CloudFlare is building a faster, safer web from the ground up

Cloudflare

cloudflare

Corient Private Wealth LLC trimmed its position in Cloudflare, Inc. (NYSE:NET – Free Report) by 14.6% during the 4th quarter, according to the company in its most recent filing with the Securities and Exchange Commission (SEC). The firm owned 25,134 shares of the company’s stock after selling ...

Cloudflare, Inc. (NYSE:NET) Shares Sold by Corient Private Wealth LLC - Defense World

Kremlin-aligned gang used Cloudflare and GitHub resources, and they didn&#x27;t like that one bit

FlyingYeti phishing crew grounded after failed Ukraine ops • The Register

Shares of Cloudflare, Inc. (NYSE:NET – Get Free Report) were down 3.5% during trading on Thursday after an insider sold shares in the company. The stock traded as low as $72.30 and last traded at $72.55. Approximately 443,900 shares were traded during mid-day trading, a decline of 88% from ...

Cloudflare (NYSE:NET) Stock Price Down 3.5% After Insider Selling - Defense World

CrowdStrike and Cloudflare deepen strategic partnership, integrating AI-powered security tools to simplify operations and fortify defences against advanced cyber threats.

Cloudflare & CrowdStrike expand partnership for enhanced security

Shares of Cloudflare are down 18.3% YTD versus a gain of 11.5% for the Nasdaq Composite even though the cloud platform provider continues to deliver solid results.

Why Cloudflare Shares Have Tumbled 41% From The Recent 52-Week High

The war against unauthorized AI bots, scrapers, and content thieves just got a powerful new weapon. On March 19, 2025, Cloudflare launched AI Labyrinth, a free tool designed to protect websites from unauthorized data collection while simultaneously reducing server load and mitigating SEO issues caused by AI-generated duplicate content. Cloudflare launched AI Labyrinth protect websites from unauthorized data collection For years, content creators, businesses, and website owners have faced an uphill battle against bots that scrape their data without permission. These AI-powered crawlers feast on the hard work of real people, siphoning off valuable information to train AI models—often without consent, credit, or compensation. But Cloudflare is flipping the script. A Smarter Approach to Blocking AI Bots Instead of simply blocking unauthorized AI crawlers—a method that often results in an endless game of cat and mouse—AI Labyrinth plays the long game. It redirects these rogue bots into a maze of AI-generated content, tricking them into believing they’ve struck gold. AI Labyrinth plays the long game. redirects these rogue bots into a maze of AI-generated content The crawlers, thinking they’re gathering high-quality data, continue to navigate the Labyrinth, unaware they are being fed an endless loop of machine-generated pages that lead nowhere. Meanwhile, the real website remains untouched, its valuable data shielded from prying AI eyes. This clever deception not only protects original content but also forces AI bots to waste computational resources—driving up their costs while leaving them empty-handed. real website remains untouched Why This Matters in the Fight Against Content Theft Unauthorized AI scraping isn’t just a nuisance—it’s an existential threat to original content creators. When AI bots ingest stolen content, they: Train models that regurgitate stolen ideas without credit


Flood search engines with duplicate AI-generated content that buries the originals


Strain website servers with excessive, unauthorized crawling


Deprive creators of the traffic, engagement, and revenue they rightfully deserve Train models that regurgitate stolen ideas without credit Train models that regurgitate stolen ideas without credit regurgitate stolen ideas without credit Flood search engines with duplicate AI-generated content that buries the originals Flood search engines with duplicate AI-generated content that buries the originals duplicate AI-generated content buries the originals Strain website servers with excessive, unauthorized crawling Strain website servers with excessive, unauthorized crawling Strain website servers Deprive creators of the traffic, engagement, and revenue they rightfully deserve Deprive creators of the traffic, engagement, and revenue they rightfully deserve the traffic, engagement, and revenue they rightfully deserve Cloudflare’s AI Labyrinth is more than just a security measure—it’s a declaration of war against unethical AI scraping. By redirecting and deceiving these bots, Cloudflare is sending a clear message: if you want high-quality data, you have to play fair. declaration of war against unethical AI scraping. redirecting and deceiving these bots if you want high-quality data, you have to play fair. As a journalist covering the tech industry, I’ve seen my fair share of innovation aimed at tackling online piracy, but Cloudflare’s new AI-powered Labyrinth tool has really caught my attention. online piracy In my opinion, this could be a game-changer in stopping sites like Tech-Gate.org, which racked up over 1,000 DMCA takedown notices for shamelessly lifting content from legitimate publishers. Tech-Gate.org 1,000 Let’s be real—content theft has become a relentless plague in digital media. As soon as we publish an article, it’s often scraped, republished, and monetized by bad actors with zero regard for intellectual property rights. It’s frustrating, to say the least. So when I hear that Cloudflare is leveraging AI to build a more sophisticated defense system, I can’t help but be intrigued. Of course, the proof will be in the results. Will Labyrinth be the much-needed fortress against digital theft, or just another well-intentioned but ultimately ineffective solution? I, for one, look forward to seeing how it performs in the wild—especially against repeat offenders like Tech-Gate.org, who have been stealing content left, right, and center for far too long. far too long The Future of Web Protection Internet security tools like AI Labyrinth prove that innovation isn’t one-sided. Website owners now have a powerful and free way to fight back, ensuring that AI companies can’t simply vacuum up the internet’s collective knowledge without permission. powerful and free With AI-generated content increasingly dominating the online space—reportedly making up four of the top 20 Facebook posts and an estimated 47% of content on Medium—tools like AI Labyrinth are essential for preserving the integrity and security of authentic web content. The fight against unauthorized AI scraping is far from over—but Cloudflare just turned the tide. AI bots beware: the Labyrinth awaits. AI bots beware: the Labyrinth awaits.

Cloudflare’s AI Labyrinth Puts an End to Content Thieves by Trapping Bots in a Maze

Mitigating Attacks on WordPress When Running Under a Full CDN Like Cloudflare: A Guide

Welcome to HackerNoon’s Writing Prompts! Would you like to take a stab at answering some of these questions? Click the log in or create account button on the top right to be redirected back to this template! Cloudflare is a great company that has been innovating and challenging the cloud providers that be. Here is an overview of their products and why you should care. Cloud Security Tools: Cloudflare Turnstile For Bot Prevention Instead of Google Captcha, where to prove you are not a bot, a user would need to select all the buses or cars, Cloudflare has made waves with its free alternative to bot prevention: Cloudflare Turnstile. Released in 2022, but already seeing major adoption, you probably have seen it before. One of the great things about this product is that it does not cause as many users to get frustrated and leave. R2: A Cloud Storage Alternative to S3 AWS, GCP, and Azure all charge egress fees per gigabyte - usually in the range of .05$ to .09 $ per GB. This has led to a lot of caching of the objects, and generally, has slowed innovation surrounding video and VPN products on these clouds. When coupled with Cloudflare CDN, this product can be very powerful. It is ideal for web content delivery, video streaming, static site hosting, and applications that benefit from low egress costs and edge delivery. Cloud Security Solutions: Adaptive DDoS protection One of Cloudflare’s earliest successful products was its DDOS protection. One can imagine that a company that routes 20% of all internet traffic, can have very advanced security tools. The CEO tells a story of how they protected Eurovision from DDOS attacks, and Cloudflare's DDoS protection is often considered superior to other solutions for several reasons. Here are the key factors that contribute to its effectiveness and popularity: Cloudflare’s Web Application Firewall (WAF) Web Application Firewalls that Cloudflare provides have proved to be highly useful for securing public APIs. They can set IP-based throttle limits, and protect against malicious payloads. One of the great things about this cloud security tool is that it is easy to set up and has so much power. From my experience, the first time I noticed it was when I was using a public API for stock history. I was being throttled after 25 calls on my IP address. Cloudflare managed rules


Core OWASP rules (Top 10 exploit patterns)


Advanced rate limiting In the modern world, data is key. There are so many companies (ChatGPT) that scrape the web consistently by using bots. While in the competitive cyberspace, bots will often adapt and find a way, for the majority of naive scraping tools, Cloudflare is able to block them. Conclusion At the end of the day, we all assume that the major clouds have ‘figured it out’. From my software development experience in Big Tech, there is a lot of misguided designs and principles. All of us have grown up on the internet, have seen it change human dialogue, the way we think about the world, and has entertained us for years. The most shocking thing about Cloudflare is that they don’t discriminate or judge their customers and they don’t overcharge their customers. If you would like to follow me for more, here is my Cloud security substack, cheers! Welcome to HackerNoon’s Writing Prompts! Would you like to take a stab at answering some of these questions? Click the log in or create account button on the top right to be redirected back to this template! Welcome to HackerNoon ’s Writing Prompts! Would you like to take a stab at answering some of these questions? Click the log in or create account button on the top right to be redirected back to this template! HackerNoon Cloudflare is a great company that has been innovating and challenging the cloud providers that be. Here is an overview of their products and why you should care. Cloud Security Tools: Cloudflare Turnstile For Bot Prevention Instead of Google Captcha, where to prove you are not a bot, a user would need to select all the buses or cars, Cloudflare has made waves with its free alternative to bot prevention: Cloudflare Turnstile . Released in 2022, but already seeing major adoption, you probably have seen it before. One of the great things about this product is that it does not cause as many users to get frustrated and leave. Cloudflare Turnstile Cloudflare Turnstile R2: A Cloud Storage Alternative to S3 AWS, GCP, and Azure all charge egress fees per gigabyte - usually in the range of .05$ to .09 $ per GB. This has led to a lot of caching of the objects, and generally, has slowed innovation surrounding video and VPN products on these clouds. When coupled with Cloudflare CDN, this product can be very powerful. It is ideal for web content delivery, video streaming, static site hosting, and applications that benefit from low egress costs and edge delivery. Cloud Security Solutions: Adaptive DDoS protection One of Cloudflare’s earliest successful products was its DDOS protection. One can imagine that a company that routes 20% of all internet traffic, can have very advanced security tools. The CEO tells a story of how they protected Eurovision from DDOS attacks, and Cloudflare's DDoS protection is often considered superior to other solutions for several reasons. Here are the key factors that contribute to its effectiveness and popularity: Cloudflare’s Web Application Firewall (WAF) Web Application Firewalls that Cloudflare provides have proved to be highly useful for securing public APIs. They can set IP-based throttle limits, and protect against malicious payloads. One of the great things about this cloud security tool is that it is easy to set up and has so much power. From my experience, the first time I noticed it was when I was using a public API for stock history. I was being throttled after 25 calls on my IP address. Web Application Firewalls Web Application Firewalls cloud security tool cloud security tool Cloudflare managed rules Core OWASP rules (Top 10 exploit patterns) Advanced rate limiting Cloudflare managed rules Cloudflare managed rules Core OWASP rules (Top 10 exploit patterns) Core OWASP rules (Top 10 exploit patterns) Advanced rate limiting Advanced rate limiting In the modern world, data is key. There are so many companies (ChatGPT) that scrape the web consistently by using bots. While in the competitive cyberspace, bots will often adapt and find a way, for the majority of naive scraping tools, Cloudflare is able to block them. Conclusion At the end of the day, we all assume that the major clouds have ‘figured it out’. From my software development experience in Big Tech, there is a lot of misguided designs and principles. All of us have grown up on the internet, have seen it change human dialogue, the way we think about the world, and has entertained us for years. The most shocking thing about Cloudflare is that they don’t discriminate or judge their customers and they don’t overcharge their customers. If you would like to follow me for more, here is my Cloud security substack, cheers! Cloud security

A Deeper Look at the Curious Case of Cloudflare

Your web scraper has just been blocked by Cloudflare again! You've looked online for solutions, but nothing seems to do the trick. Is there a secret sauce for Cloudflare bypass? You bet there is. You just haven't stumbled upon the right recipe! Embark on a guided tour through the maze of Cloudflare anti-bot technology. We're here to demystify Cloudflare's inner workings, unveil some game-changing tricks, and ensure you never hit a dead end again! What Is Cloudflare? , the cyber fortress of the Internet, offers top-notch services to boost the security and performance of websites and web applications. But you probably knew that already! Cloudflare Now, what's the secret sauce that makes Cloudflare so popular? Drumroll, please—it’s anti-bot solutions! Ever come across that page? Well, you’re the only one! That’s a result of the , a high-tech shield against the mischief of malicious bots. Cloudflare Bot Management service Here are the technologies that power that cybersecurity platform: Behavioral analysis: Picture it as Sherlock Holmes scrutinizing user behavior on the Web to decipher whether users are humans or not. Machine learning magic: Think of it as the Dumbledore's Army of bot detection, learning from a colossal dataset with billions of records to give every request a reliable bot score. Fingerprinting: Use millions of internet properties to precisely identify bot requests without compromising user privacy. The result? Bots, begone! From stopping credential stuffing to thwarting content scrapers, spammers, inventory hoarders, credit card stuffers, and even putting up a solid defense against application DDoS attacks—Cloudflare's got it covered. Cloudflare’s Bot Management stands tall as the Batman of bot management, vigilantly safeguarding websites to thwart any hint of malicious bot behavior. 🦸‍♂️ But every superhero has its kryptonite! Dive into the next section to unravel more. Tips to Bypass Cloudflare Before digging into this chapter, let's tackle a burning question. Is it even possible to bypass Cloudflare? Absolutely! No online solutions can or should restrict users from accessing public sites as they wish, even through automated software. Thus, the million-dollar question is: how to outsmart Cloudflare? 🤔 Dive into the tips below to master the art of eluding Cloudflare! Use a Reliable IP Cloudflare gives IPs a score based on their reputation. If your IP has been caught red-handed performing actions considered malicious by Cloudflare, the chances of getting blocked become high. The solution? Having a large pool of IPs to distribute your requests on. In that scenario, even if one IP gets blocked, you have many others available in your arsenal. That’s where IP rotation via proxy comes into play. Find out more on . how to rotate an IP address Leverage Browser Automation Only browsers can overcome the challenges set by Cloudflare’s Bot Management. When you navigate a page shielded by that solution, your browser uses JavaScript to tackle those challenges without you even realizing it. Requests made by an automated script through a traditional HTTP client will then hit an immediate roadblock. Why? Because HTTP clients can’t execute JavaScript! That’s something only a browser can do. Thus, your Cloudflare-ready automated script should rely on an approach for programmatically executing actions on a web page in a browser, such as . controlling a headless browser via Playwright Reverse Engineering the Cloudflare Challenges This is where the real challenge kicks in—how to outsmart the anti-bot defenses intricately embedded into a web page by Cloudflare? The answer: dive deep into their workings, unravel the puzzle through reverse engineering, and conquer. That's easier said than done 😅 Step one involves finding the request linked to the retrieval of the JavaScript challenge. This will involve encrypted code that you must somehow decipher: Suppose you managed to do that. Now, all that remains is to figure out the original code's intentions and create a strategy to fool it into thinking that the Cloudflare challenge has been overcome. On paper, this sounds like a valid approach, but in the real world, it may be a one-way ticket to a colossal headache. Don’t worry, though. Overcoming Cloudflare is possible—you just need the right tools! My Solution Works… Until It Doesn't Imagine you're an IT genius who manages to crack the code to get past Cloudflare's bot protection technologies. Bear in mind that Cloudflare is an ever-evolving beast, so no guarantees your solution will work tomorrow. To reach the goal, you've got to embrace a different approach! A real-world, ideal, intelligent solution to overcome Cloudflare would be a controllable browser that: Can overcome advanced anti-bot measures for you. Is always up-to-date because it runs in the cloud. Automatically rotates IPs and handles request attempts. Avoids fingerprinting to improve your privacy. Can resolve CAPTCHAs on its own. Allows cookie management and header customization. Ensures amazing scalability and performance by executing in the cloud. It can be controlled by any browser automation library, such as Selenium, Playwright, or Puppeteer. This isn't a mere dream, but a palpable reality. The browser you're imagining is real, developed by Bright Data, and goes by the name of ! Scraping Browser Final Thoughts Cloudflare reigns supreme in safeguarding websites against malicious bots. Here, we've delved into the technologies it uses for detecting and stopping automated software, offering you some handy tips to avoid them. But let's be real, that’s a grueling, trial-and-error, demoralizing task. Need to bypass Cloudflare once and for all? Embrace Scraping Browser from and join our mission to transform the Internet into a public space accessible to everyone, everywhere, even via automated scripts! Bright Data Until next time, keep exploring the Internet with freedom!

Cracking the Code of Cloudflare Bypass

ক্লাউডফ্লেয়ার বাইপাসের কোড ক্র্যাক করা

Den Code von Cloudflare Bypass knacken

Descifrando el código de omisión de Cloudflare

Déchiffrer le code du contournement de Cloudflare

क्लाउडफ्लेयर बाईपास के कोड को क्रैक करना

Cloudflareバイパスのコードを解読する

Cloudflare Bypass 코드 크래킹

Quebrando o código do Cloudflare Bypass

Взлом кода обхода Cloudflare

Cloudflare Bypass Kodunu Kırmak

Bẻ khóa mã bỏ qua Cloudflare

破解 Cloudflare 绕过代码

I can't go a day (or two) without building a bot of some sort, and last week was no exception. I've been a fan of the for nearly a decade now, and one of my favorite examples of it is my bot. I thought I'd use the Marvel API as a way to build bot, but this time on the platform. Here's how I did it. Marvel API random comic book cover another Cloudflare Workers The Architecture So, obviously, I'm using Cloudflare Workers, but I decided to make this project a two-step process. In my , I shared how you can "connect" one Worker to another via service bindings. Obviously, this is practical for reuse, and while it may be a bit overkill for this, I wanted to put what I learned in the last post into practice. My project's two workers are: last post A scheduled Worker that makes use of their support. This worker will be run on a schedule, call the second Worker for its data, and then post to Mastodon. Cron triggers A Worker that wraps calls to the Marvel API, specifically one to get a random character. The latter of the two is the simpler one, so let's start with that. Using the Marvel API for Character Information So, much like my 'random comic book' logic, I get a random Marvel character by: Figure out the total number of characters. I did this using their interactive tester and making note of the total number of results. As of August 2023, that number is 1562. Here's an example of that output: Then, simply call the character endpoint with a limit of one and an offset of a random number in that range. For the most part, it's all relatively simple, except Marvel requires you to sign your API requests. This was a minor sticking point for me as my previous Node.js code didn't work on Cloudflare, and I had to switch to Web Crypto, which was nicely documented here: . https://developers.cloudflare.com/workers/runtime-apis/web-crypto/#web-crypto Here's the entirety of the Worker: // Based on checking the API (in Aug 2023) to see how the max number of characters
const CHAR_TOTAL = 1562;

const getRandomInt = (min, max) => {
    return Math.floor(Math.random() * (max - min + 1)) + min;
}

async function getSuperHero(privateKey, publicKey) {

    let selected = getRandomInt(0, CHAR_TOTAL);
    let url = `https://gateway.marvel.com:443/v1/public/characters?limit=1&apikey=${publicKey}&offset=${selected}`;

    // add hash
    let ts = new Date().getTime();
    let myText = new TextEncoder().encode(ts + privateKey + publicKey);

    let hash = await crypto.subtle.digest({
        name:'MD5'
    }, myText);

    // Credit: https://developers.cloudflare.com/workers/runtime-apis/web-crypto/
    const hexString = [...new Uint8Array(hash)]
      .map(b => b.toString(16).padStart(2, '0'))
      .join('');

    url += '&hash='+encodeURIComponent(hexString)+'&ts='+ts;

    let resp = await fetch(url);
    let data = await resp.json();
    return data.data.results[0];
}

export default {
    async fetch(request, env, ctx) {
        const PRIVATE_KEY = env.MARVEL_PRIVATE_KEY;
        const PUBLIC_KEY = env.MARVEL_PUBLIC_KEY;
        let hero = await getSuperHero(PRIVATE_KEY, PUBLIC_KEY);
        console.log(`I got the hero ${hero.name}`);

        return new Response(JSON.stringify(hero), {
            headers: {
                'Content-Type':'application/json;charset=UTF-8'
            }
        });
    },
}; By the way, notice the ? Later this week, I'll show how that works in production. In case you're curious, here's what that JSON response looks like. It randomly selected one of my favorite characters, Galactus. In order to save on space, I removed items from the arrays of data. console.log many {
    "id": 1009312,
    "name": "Galactus",
    "description": "",
    "modified": "2014-09-30T16:47:03-0400",
    "thumbnail": {
        "path": "http://i.annihil.us/u/prod/marvel/i/mg/5/03/528d31a791308",
        "extension": "jpg"
    },
    "resourceURI": "http://gateway.marvel.com/v1/public/characters/1009312",
    "comics": {
        "available": 236,
        "collectionURI": "http://gateway.marvel.com/v1/public/characters/1009312/comics",
        "items": [
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/comics/12638",
                "name": "Alpha Flight (1983) #10"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/comics/12639",
                "name": "Alpha Flight (1983) #100"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/comics/4788",
                "name": "Annihilation (2006) #1"
            },

        ],
        "returned": 20
    },
    "series": {
        "available": 112,
        "collectionURI": "http://gateway.marvel.com/v1/public/characters/1009312/series",
        "items": [
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/series/2116",
                "name": "Alpha Flight (1983 - 1994)"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/series/3613",
                "name": "Annihilation (2006 - 2007)"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/series/1864",
                "name": "Annihilation: Heralds of Galactus (2007)"
            }
        ],
        "returned": 20
    },
    "stories": {
        "available": 259,
        "collectionURI": "http://gateway.marvel.com/v1/public/characters/1009312/stories",
        "items": [
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/stories/694",
                "name": "Cover #694",
                "type": "cover"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/stories/898",
                "name": "Fantastic Four (1998) #520",
                "type": "cover"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/stories/899",
                "name": "1 of 5 - Galactus",
                "type": "interiorStory"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/stories/922",
                "name": "Fantastic Four (1998) #518",
                "type": "cover"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/stories/923",
                "name": "AVENGERS DISASSEMBLED TIE-IN! \"FOURTITUDE\" PART 2 (OF 3) With public opinion of the FF at an all-time low and with all of Manhat",
                "type": "interiorStory"
            }
        ],
        "returned": 20
    },
    "events": {
        "available": 9,
        "collectionURI": "http://gateway.marvel.com/v1/public/characters/1009312/events",
        "items": [
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/events/229",
                "name": "Annihilation"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/events/234",
                "name": "Avengers Disassembled"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/events/318",
                "name": "Dark Reign"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/events/302",
                "name": "Fear Itself"
            }
        ],
        "returned": 9
    },
    "urls": [
        {
            "type": "detail",
            "url": "http://marvel.com/comics/characters/1009312/galactus?utm_campaign=apiRef&utm_source=fe877c0bf61f995fc8540d9eac4704f1"
        },
        {
            "type": "wiki",
            "url": "http://marvel.com/universe/Galactus?utm_campaign=apiRef&utm_source=fe877c0bf61f995fc8540d9eac4704f1"
        },
        {
            "type": "comiclink",
            "url": "http://marvel.com/comics/characters/1009312/galactus?utm_campaign=apiRef&utm_source=fe877c0bf61f995fc8540d9eac4704f1"
        }
    ]
} Notice the property is both a path and an extension. Here's our lovely world devourer. thumbnail Now, on to the next Worker. The Scheduled Tooter I love that subhead. So the next Worker is responsible for running on a schedule and actually doing the Mastodon posting. The cover how these are set up and how you can test them. For the most part, this is just a plain worker. The biggest difference in the code is that you have a handler, not . Here's a barebones Worker for scheduled execution. Cloudflare docs scheduled fetch export default {
    async scheduled(event, env, ctx) {


    },
}; And your schedule is defined in the file. In this case, every two hours: wrangler.toml [triggers]
crons = ["0 */2 * * *"] The first issue I ran into was how to connect this worker to the first one. In the , you'll see it's rather simple: last blog post const backResponse = await env.backlogic.fetch(request.clone()); But, this expects an incoming request, an HTTP-driven Worker. There isn't a request in a Cron-triggered worker. I on the Cloudflare forums and got help from the most Internet nickname ever, . The fix is to simply make an empty (mostly) Request object like so: fetch asked Cyb3r-Jak3 let heroRequest = await env.randomsuperhero.fetch(new Request('http://127.0.0.1'));
let hero = await heroRequest.json(); Next, I prepare my data for my toot: /*
Generate the text for the toot.
I'm using the 'detail' link which is not always the best, better than the wiki though :( 
*/
let toot = `
Your random Marvel superhero of the moment is: ${hero.name}.
More information here: ${hero.urls[0].url}
`;

let image = `${hero.thumbnail.path}.${hero.thumbnail.extension}`; This is where I should point out that while my random comic book cover bot is , the character information is - unfortunately - a bit slim, especially for obscure characters. To be clear, I don't mean a lack of information, but a lot of 404s and no images. Honestly, I almost punted on this as a source of data, but I figured I'd let it go and see how it looks after a while "in the wild." great Now, at this point, I've got the text for my toot, as well as the image URL. I had intended to make use of the npm module I'd used in the past, , however, when I included this in my Worker, I got compatibility errors with the Workers environment. It didn't seem like an easy workaround, and I almost gave up when I thought, why not actually look at their and try using it without a wrapper? mastodon-api API documentation Creating a "toot" was incredibly simple: let data = new FormData();
data.append('status', toot);

let resp = await fetch('https://botsin.space/api/v1/statuses', {
    body:data,
    method:'post',
    headers:{
        'Authorization':`Bearer ${KEY}`
    }
}); Literally ten lines or so of code. I was pleasantly surprised. I then looked into the image aspect. Like Twitter, if you want to associate an image with a toot, you first upload the image, get the ID, and then associate it with the new message. To do this, I needed to first get the bits from the URL on Marvel's side and then send that to Mastodon. In the past, I would have simply saved the image to , but I don't think Cloudflare supports that. Instead, I did everything in memory. This took me the longest time, so hopefully, this code can help others. /tmp async function uploadMedia(url, key) {
    // first, grab the bits of the url
    let imgreq = await fetch(url);
    let blob = new Blob([await imgreq.blob()]);

    let data = new FormData();
    data.append('file', blob);

    let mediaupload = await fetch('https://botsin.space/api/v2/media', {
        body:data,
        method:'post',
        headers:{
            'Authorization':`Bearer ${key}`
        }
    });

    return await mediaupload.json();

} Woot. Okay, with that in play, here's the updated toot code (I love saying toot): let toot = `
Your random Marvel superhero of the moment is: ${hero.name}.
More information here: ${hero.urls[0].url}
`;

let image = `${hero.thumbnail.path}.${hero.thumbnail.extension}`;
let mediaOb = await uploadMedia(image, KEY);

let data = new FormData();
data.append('status', toot);
data.append('media_ids[]', mediaOb.id);

let resp = await fetch('https://botsin.space/api/v1/statuses', {
    body:data,
    method:'post',
    headers:{
        'Authorization':`Bearer ${KEY}`
    }
}); And voila - magic: Random Super Hero @myrandomsuperhero@botsin.space Your random Marvel superhero of the moment is: Alex Power.More information here: http://marvel.com/characters/1387/alex_power?utm_campaign=apiRef&utm_source=fe877c0bf61f995fc8540d9eac4704f1 (UTC) 12:01 PM • August 13, 2023 This is Alex Power, part of Power Pack, a favorite of mine when I was a teenager, and unfortunately, the detail links to a 404. :( All in all, this is kind of par for the course for Cloudflare Workers. I hit a few snags, learn how Cloudflare does things, and then it all just works. I know I said this before, but I'm absolutely digging into the Cloudflare developer experience. If you want to follow the bot, you can find the account at . The source code for generating the random hero may be found here: . The source for the cron/Mastodon Worker may be found here: https://botsin.space/@myrandomsuperhero https://github.com/cfjedimaster/cloudflareworkers-demos/tree/main/randomsuperhero https://github.com/cfjedimaster/cloudflareworkers-demos/tree/main/randomsuperherobot Also published . here

Mastodon: Building a Marvel Character Info Bot with Cloudflare Workers

This article describes how to deploy a static site. We would use (IaC) approach to describe site changes and deploy them automatically with Cloudflare Pages. Infrastructure as Code If you are not familiar with one of the tools, don't worry, all steps are described in the details. You can use this article as a guide for your own static site for 10 minutes and 100% free. Table of Content Description Pre-requirements Tips Quick Start Useful links Conclusion For a quick start, you can jump to the “Quick Start” section. Description The site will be hosted on , and the domain name will be managed by you. The site will be built using the Hugo framework. The whole process will be automated using Cloudflare pages. We will use GitHub as a source of truth for our site. Cloudflare Pages So we describe an infrastructure as code and deploy it automatically; it's a little different from the site constructor approach, there is no user interface (UI) for site management, but it's more flexible and powerful. Just follow these , and you will have your own personal site for free. 5 steps Initialize your site Choose a theme for your site Commit the changes to your repository on GitHub Assign your site to your project on Cloudflare Pages Enjoy your site To update your site, you just need to commit changes to your repository on GitHub. Then Cloudflare Pages will automatically rebuild your site. Pre-requirements GitHub account. If you don't have one, you can create it , it's free. here Cloudflare account. If you don't have one, you can create it , it's free. here Domain name. This step is . optional If you don't have a domain name, you can use a free subdomain from Cloudflare. It's up to you. Your free subdomain will look like this: . yourname.pages.dev TIPS TIP 1 If you would like to make a similar page to my , just grab my repo in 3 steps: kazakov.xyz Fork my repo kksudo/kazakov.xyz Update content on your new site. Please, see the repo structure. You need to update 2 files: - Edit the links to your social networks or remove unused ones. For the main pic, I use , but you can use this setting . To read more about Hugo config.toml, please read . ./config.toml Gravatar avatarurl = "images/avatar.jpg" here - Add a few words about yourself. about.md Follow to step “#4 Cloudflare Pages” TIP 2 To verify your changes, run and open docker run --rm -it -v $(pwd):/src -w /src -p 1313:1313 klakegg/hugo:alpine server -D http://localhost:1313/ TIP 3 You can add a new page by . docker run --rm -it -v $(pwd):/src -w /src klakegg/hugo:alpine new <SECTIONNAME>/<FILENAME>.<FORMAT> TIP 4 To exclude unnecessary dynamic files, update your .gitignore Quick Start Step 1. Initialize Your Site For our site, we will use the Hugo framework. We will use some steps from the to initialize our site. So, to start it, we need to a Hugo binary on your machine, but I prefer a docker to keep my OS clean. Hugo quick start guide install Let's create a root structure for your site : hugo new site business-card docker run --rm -it -v $(pwd):/src -w /src klakegg/hugo:alpine new site business-card Docker options: - remove the container after exit --rm - interactive mode -it - mount the current directory to directory in the container -v $(pwd):/src /src - set the working directory to in the container -w /src /src - latest docker image with Hugo, based on the Alpine Linux klakegg/hugo:alpine Tip for a lazy person like me: smart Check this step in my repository kksudo/kazakov.xyz Step 2. Chose a Theme for Your Site To find a theme for our site, we will use an . official Hugo hub There are a lot of themes; you can choose any of them. But my favorite option is to discover it via GitHub. Because you can filter repositories by stars, forks, maintaining, etc. For example, is a simple filter. here So, let's start with the theme with more than starts on GitHub. Coder 2k+ Move to the root directory of your site. cd business-card Initialize an empty Git repository in the current directory. Skip it if your site already has a Git repository. git init -b main # Initialize the local directory as a Git repository. Clone the Coder theme into the directory, and add it to your project as a Git submodule. themes git submodule add https://github.com/luizdepra/hugo-coder themes/hugo-coder Update the site configuration file , indicating to use theme. Also, we should add some settings for this theme. Grab these settings from my file. ./config.toml hugo-coder config.toml You can find more information about the theme settings . here Note: If you would like to use a simple theme, without any settings, you can use the theme. It's a default theme for Hugo, URL. Ananke quick start Tip for a lazy person like me: smart You could grab this step from my repository kksudo/kazakov.xyz To verify changes, run and open docker run --rm -it -v $(pwd):/src -w /src -p 1313:1313 klakegg/hugo:alpine server -D http://localhost:1313/ Step 3. Commit the Changes to Your GitHub Repository Let's add some content, and change the default site settings before the first commit. Add an About page docker run --rm -it -v $(pwd):/src -w /src klakegg/hugo:alpine new about.md Add some markdown to the body of the post, but do not change the draft value, code content/about.md ---
title: "About"
date: 2023-04-11T21:14:37Z
draft: true
---

I'm a member of "I want to know everything" and "I'm happy to share what I've already learned" groups.
DevOps with love. Update as described in the theme page. Let's add some menu settings for the About page. Or grab these settings from my file, and then edit. ./config.toml hugo-coder config.toml Push our changes to GitHub origin. git add -A . # Adds the files in the local repository and stages them for commit
git commit -m "First commit" # commit all the files in your project

git remote add origin git@github.com:<owner>/<repo-name>.git
git push -u origin main Note: Read more about GitHub quick start here Tip for a lazy person like me: grab this step from my repository smart kksudo/kazakov.xyz Step 4. Assign Your Site to Your Project on Cloudflare Pages Create an account on , and log in to your account. Cloudflare Create a new project on , and connect to your GitHub repository. Cloudflare Pages Connect Cloudflare to your GitHub repository. Install Cloudflare Pages. Grand permissions Cloudflare integration in GitHub. Setting up Cloudflare settings for the Hugo site Chose a Hugo framework Add a variable to use on the latest Hugo version, HUGO_VERSION=0.111.3 : If you would like to revert Cloudflare permissions for your repo, you should go to and change settings for Cloudflare Pages. TIP https://github.com/settings/installations So, after Cloudflare will be assigned, deployment will be triggered. And after it, Cloudflare started deployment of your site. By default, Cloudflare will build and deploy all your branches and generate a unique URL for each branch; the main URL will be also available for your default branch. So, Cloudflare shows you a lot of details about your build and deployment process. Step 5. Enjoy Your Site Open and enjoy :) https://yourname.pages.dev Tip for a lazy person like me: smart Sorry, no tips :) Useful Links Install docker Install Hugo Join to GitHub Deploy a Hugo site CloudFlare Hugo quickstart Coder theme configuration documentation based on this article My page with code for this article GitHub repository Disclaimer, Why Did I Choose This Set of Tools? Why Hugo? Hugo it's the world’s fastest framework for building websites with more than 66 000 stars on GitHub. It's very easy to use, pretty fast, open-source, and has a lot of features. You can find more information about Hugo on the . official website Why GitHub? GitHub is a code hosting platform for version control and collaboration. It lets you and others work together on projects from anywhere. You can find more information about GitHub on the . official website Why Cloudflare? Firstly, I will explain why, I didn't choose . The last time I tried to use GitHub Pages, I faced some incipient problems. Also, I chose Cloudflare Pages because this provider is network focused. GitHub Pages I know Cloudflare as a network infrastructure and security company. I really like their products, like DNS, CDN, Firewall, etc. But I didn't know that they have a product for hosting static sites. I was surprised when I found out about it. So, I decided to use network focused provider for hosting static sites. Conclusion I hope this article was useful for you. If you have any questions, feel free to ask me in the comments. I will be happy to help you. Also, feel free to connect with me . kazakov.xyz P.S.: For this flow, we could use a lot of other approaches, services, and tools; in the section above, I described why I sue these techs. For example, you can use GitHub pages instead of Cloudflare Pages, and use another static site generator instead of Hugo. #contest-tags

Deploy Your Personal Web-Page With Hugo, Cloudflare and GitHub 100% For Free

Introduction You want to shorten the long links generated by your application because the URLs are too long to fit in an SMS. You want to keep scalability in mind and want to handle millions of shortened URLs and millions of users flowing through the links. Most importantly, you want this new service to be as cheap as possible - it can't break the bank. In this article, we will use Cloudflare Workers and KV to create a scalable URL shortener service that you can use to shorten your application links. The Stack is a Serverless platform that allows you to deploy code and
have it available globally with extremely low latency. Cloudflare Workers... Cloudflare Workers is much cheaper than competitors such as Azure Functions and AWS Lambda. ($5/10 million requests) can run code instantly without suffering from cold starts like the competitors. can automatically scale to millions of users - you are only charged per request and not the number of concurrent users. The reason why Cloudflare can do this is that it doesn't rely on the traditional serverless model of using the virtual machine architecture. Instead, it utilizes the Chrome V8 engine with the Service Workers API to run code extremely fast. If you are a nerd like me - will amaze you. reading this Alternatively, you would probably write a simple Express or Flask app, dockerize it, and deploy it through or Kubernetes. docker-compose is a key-value store that is globally distributed. We're going to use it to map from a short link to the original long URL. A URL shortener service is one of the best cases to utilize serverless functions. It does 1 simple job, so you shouldn't waste your time thinking about infrastructure, and most importantly, an expensive bill to pay at the end of the month. Workers KV Alternatively, this might be a mix of a PostgreSQL server for permanent storage and Redis to cache results. The Preparation Sign up for a . Cloudflare Account Install Cloudflare's CLI tool, . Wrangler npm install wrangler --save-dev Authenticate yourself with Wrangler by following instructions on the screen to complete OAuth with Cloudflare. npx wrangler login The Plan We want to expose 2 endpoints for the user - one to create a shortened URL, and the other to serve all shortened URLs, perform 302 redirects if the shortened link is found, and 404 otherwise. The code Step 1: Generate a boilerplate for Cloudflare Workers In a local directory, run . It's going to create the necessary files and folder structure for you to create our URL shortener. You will be prompted to select a few options, to which you will select yes to. npx wrangler init shortener Would you like to use git to manage this Worker? y No package.json found. Would you like to create one? y Would you like to use TypeScript? y Would you like to create a Worker at shortener/src/index.ts? Fetch Handler To further simplify development, since we are going to make more than 1 endpoint, we'll
install which is an Express-like router to automatically parse out the HTTP method, query params, and path params, etc. I also prefer to use to format the code so that it looks consistent. itty-router prettier cd shortener
rm package-lock.json
yarn
yarn add itty-router
yarn add --dev prettier Step 2: Create a KV namespace KV is basically the replacement for the database and cache you were going to use to make your Express URL shortener. The difference is, KV is as easy to use as the browser's localStorage. To create a KV namespace on your Cloudflare account, run the following commands. We'll need to run both because we'll need to reserve one for development as well. npx wrangler kv:namespace create SHORTENER_KV
npx wrangler kv:namespace create SHORTENER_KV --preview Note down the KV ids and edit to look like the following. When you deploy with Wrangler, it will set the correct variables and link the KV namespace to your worker. wrangler.toml name = "shortener"
main = "src/index.ts"
compatibility_date = "2022-08-11"

kv_namespaces = [
 {
 binding = "SHORTENER_KV",
 id = "<Your KV ID>",
 preview_id = "<Your KV preview ID for development>"
 }
]

[vars]
HOST_URL = "http://localhost:8787" Step 3: Make the API to shorten a URL I like to write out the skeleton of what I'm going to implement before actually implementing the logic behind it, as I feel like it will lead to better code structure. In the below snippet, I am creating an that will triage API requests to the POST and GET functions respectively. itty-router Also, I am linking the KV namespace in the environment. Finally, from my experience, it is ideal to add statements here and there to help with debugging in case my Worker goes wrong. console.log import { Router } from "itty-router";
const router = Router();

export interface Env {
 SHORTENER_NAMESPACE: KVNamespace; // the KV namespace we setup in the previous step
 HOST_URL: string;
}

router.post("/", async (request: Request, env: Env) => {
 // TODO: Post request to shorten a URL
});

router.get("/:slug", async (request: Request, env: Env) => {
 // TODO: redirect shortened URL to application URL
});

export default {
 async fetch(
 request: Request,
 env: Env,
 ctx: ExecutionContext
 ): Promise<Response> {
 return router.handle(request, env, ctx).then((res: Response) => {
 console.log("HTTP Response", request.url, res.status);
 return res;
 });
 },
}; Now, we can fill in the logic to create a shortened URL. In the below code snippet, I am generating a random slug that has not been used before and using that as the key to store the long URL that the user has passed in. Finally, I am returning the user a shortened URL that they may use to redirect to their URL. router.post("/", async (request: Request, env: Env) => {
 const json = await request.json();
 const longLink = json?.url;
 // generate a random slug - this is not cryptographically random but works in our case
 let slug = btoa(Math.random() + "").slice(0, 9);
 // generate a new slug if the previous one is already existing
 let existing = await env.SHORTENER_KV.get(slug);
 while (existing) {
 slug = btoa(Math.random() + "").slice(0, 9);
 existing = await env.SHORTENER_KV.get(slug);
 }
 // generate shortened URL and return to user
 await env.SHORTENER_KV.put(slug, longLink);
 return new Response(JSON.stringify({ url: `${env.HOST_URL}/${slug}` }), {
 headers: { "content-type": "application/json" },
 });
}); At this point, we should test our code. To run a dev server, run . Then, you can use Postman to test out the server for now. npm run start Step 4: Handle the URL redirect When the user visits the shortened URL, we would like to fetch the long URL from KV, and provide the user with a 302 redirect response, so that the browser can handle the redirection for us. If the shortened slug cannot be found, we can return a 404 telling them nothing is here. router.get("/:slug", async (request: Request, env: Env) => {
 const slug = request.params.slug;
 const redirectTo = await env.SHORTENER_KV.get(slug);
 if (redirectTo) {
 return Response.redirect(redirectTo, 302);
 }
 return new Response("URL not found", {
 status: 404,
 });
}); Hopefully by now, 8 minutes in, you have a fully working URL shortener that is deployed locally. We'll leave the last 2 minutes for deployment, which is more than enough. The deployment Simply run and Wrangler will handle the rest. By default, Cloudflare Workers will provide you a subdomain but you can also configure your own by going into the Cloudflare Dashboard under Workers. npm run deploy workers.dev - KV Namespaces:
 - SHORTENER_KV: <your KV ID>
- Vars:
 - HOST_URL: "http://localhost:8787"

Published shortener (3.83 sec)
 https://shortener.cho.workers.dev As you can see, the used is still the one we defined at the beginning, using localhost:8787. Remember to change the URL to the published URL, in my case, . (Feel free to check this out!) HOST_URL https://shortener.cho.workers.dev After you modify , run again. wrangler.toml npm run deploy Debugging with Workers One of the hardest things about programming something in a new language or system is debugging. Although can bring you very far into the debugging journey, sometimes, you will encounter issues only visible after deploying to Cloudflare. wrangler dev Luckily, Cloudflare provides a way to tail production logs - as long as you write in your worker code for debugging purposes. After that, you can tail the logs using wrangler. console.log npx wrangler tail --format=pretty Go further This example is meant to showcase the power of a serverless URL shortener. Here are a couple of enhancements I would make to the code, and I will leave it as an exercise to the reader: Input validation for POST request Authenticate via Bearer/Basic token Return a fancy UI if the link is not found. ... (let's not limit creativity here) Source Code With ~50 lines we were able to completely recreate a URL shortener. You can check out all the code used in this tutorial . That's 10 minutes of your life writing (maybe) your first Worker and also (maybe) your first URL shortener. Hope you enjoyed this! here Also published . here

How to Build a Scalable URL Shortener With Cloudflare Workers and KV Under 10 Minutes

We've recently announced that WunderGraph is now fully open source. Today, we'd like to explain how you can leverage our API Developer Framework to add Edge Caching to your GraphQL APIs without locking yourself into a specific vendor. Caching GraphQL on the Edge should be vendor-agnostic. Services like Akamai and GraphCDN / Stellate offer proprietary solutions to solving this problem.
We'll compare the different approaches and their tradeoffs. Why did we create proprietary GraphQL CDN solutions A good question to start with is why we've created proprietary GraphQL CDN solutions in the first place. Most GraphQL implementations ignore how the web works The problem with most GraphQL implementations is that they don't really use the "platform" they are operating on. By platform, I mean the web, or more specifically HTTP and the REST constraints. The web has a lot to offer if you're using it in the right way. If read requests (Queries) were using the GET verb, combined with Cache-Control Headers and ETag, Browsers, CDNs, Cache Servers like Varnish, Nginx, and many other tools could handle Caching out of the box. You wouldn't really need a service that understands GraphQL. However, the reality is that most GraphQL APIs make you send Queries over HTTP POST. Cache-Control Headers and ETags don't make sense in this case, as all participants on the web think we're trying to "manipulate" something. So, we've created Edge Caching solutions that rewrite HTTP POST to GET and are capable of invalidating in very smart ways. By analyzing Queries and Mutations they're able to build up a cache and invalidate Objects as mutations flow through the system. Limitations and Problems of GraphQL CDNs / Edge Caches A GraphQL CDN creates a secondary source of truth As discussed in the constraint by Fielding, intermediaries can be used to implement a shared cache to improve the latency of requests. The problem I see with "smart" GraphQL Edge Caches is that they look into the GraphQL Operations to figure out what to cache and what to invalidate. Some GraphQL CDN implementations even allow you to use their API to invalidate objects. Layered System What sounds very smart creates a huge problem, you're building a second source of truth. Before using the GraphQL CDN, all you've had to do is implement your resolvers. Now, you've got to think about how to invalidate the CDN. Even worse, you're now programming against a single vendor and their implementation, coupling your application to a single service provider. You can't just switch easily from one GraphQL CDN provider to another. Contrary to REST/HTTP APIs, there's no standard on how to Cache GraphQL APIs, hence every implementation will be different. Another issue is that we're creating a secondary source of truth. Imagine we're putting a GraphQL CDN in front of the GraphQL API of GitHub to cache the Issues for a repository. If we're using "smart" Cache invalidation using Mutations, we're not able to update the cache if a user is bypassing our CDN and uses the GitHub API directly. A GraphQL CDN really only works if 100% of the traffic flows through the system. Your GraphQL Edge Cache won't work on localhost or in your CI/CD Testing takes up a huge part of software development, and we definitely want to have a great Developer Experience when building our APIs. Using a proprietary, cloud-only, Caching solution means that we're not able to test it locally or from within our Continuous Integration systems. You'll be developing on your local machine without the CDN, only to find out that something behaves weirdly when you're using the CDN in production. If we were using standardized caching directives, we'd be able to use any Cache Server, like Nginx or Varnish within our CI/CD pipeline to test our APIs. Proprietary GraphQL CDNs impose a vendor lock-in problem As mentioned earlier, there's no specification for how GraphQL Caching should work. Every implementation is different, so you're always tied to a specific vendor. They might get broke, they might get acquired, they might cancel their service, and they might change their pricing model. In any case, it's a risk that needs to be managed. A GraphQL CDN can introduce cross-origin requests Depending on the setup, adding a GraphQL CDN to your architecture could mean that your browser-based applications have to make cross-origin requests. That is, when your application is running on and your GraphQL CDN runs on , the browser will always make an extra Preflight request. It's possible to cache the Preflight request, but this still means that we'd have to do at least one extra request on the initial page load. example.com example.cdn.com A GraphQL CDN might not work well for authenticated Requests with Server Side Rendering (SSR) Let's say your application requires your users to be logged in, but you still want to be able to apply to cache. Aside from that, you'd also like to implement Server-Side Rendering (SSR). In the ideal scenario, you'd have your users log into your authentication server, which sets a cookie on your apex domain so that you're logged in on all subdomains. If your CDN is running on a different domain, it's going to be impossible to server-side render a page as the browser will not send the user's cookie to the CDN domain. Luckily, some providers offer custom domains, so cookie-based authentication might still work for you. Invalidating Deeply Nested GraphQL Operations might not work Here's an example of a GraphQL Operation that invalidates easily. mutation UpdateProfile {
    updateProfile(update: {id: 1, name: "Jannik"}) {
        id
        name
        friendsCount
    }
} If you've previously Queried the user with id 1, you can invalidate all records with that id and invalidate the following Query: query {
    profile(id: 1){
        id
        name
        friendsCount
    }
} Let's make it a bit more complex and query for all your friends: query {
    me {
       friends {
            id
            name
        }
    }
} Now, you made some good friends at the last conference. Let's add them: mutation AddFriend {
    addFriends(where: {user:{hasConnection: {conferences: {id: {eq: 7}}}}}){
        id
        name
    }
} We've now added all users as friends that had visited a conference whose id equals 7. At this point, we don't get back all the data when we query again for all your friends because the cache can't know that the mutation has invalidated the cache for the query. addFriends friends At this point, you've got to start adding response tagging, surrogate keys, or analyzing the GraphQL return types to make your invalidation strategy "smarter". We'll pick up this topic again after the benefits. Benefits of using a GraphQL CDN / Edge Cache When there's a cost, there are also benefits! Using a GraphQL CDN means, you don't have to change much of your application. In an ideal scenario, you simply change the URL of the GraphQL Server and everything should work. You also don't have to deploy and configure any tooling. You're buying a ready-to-use service that just works. Despite the problems discussed earlier, a GraphQL CDN can probably improve the performance of many applications. When not to use a GraphQL CDN As discussed in the section on Cache Invalidation, building a smart CDN service that doesn't know anything about your GraphQL Backend is actually extremely hard. The problem lies in the fact that the Backend is the source of truth, but doesn't share all this information with the Cache. In fact, if you're running into such issues where invalidation becomes hard, you might want to apply caching at a different level, inside the resolvers, or even one level below, at the entity level, using the DataLoader pattern. Additionally, if your data is expected to change frequently, it might make not much sense for you to cache at the Edge. Another big underrated flaw is that the invalidation of a distributed cache is eventually consistent. This means you serve stale content after a mutation for a short period of time. If you don't respect that in your architecture it has the potential to break the business logic of clients. Client (US) -> Query all posts to fill the cache
Client (US) -> Run mutation to update the post with ID:1
System fires a web-hook to a third-party service in FRA
Client (FRA) -> Query all posts -> stale content This is not GraphQL specific but with HTTP Caching the semantics are better understood. In summary, don't use a GraphQL CDN if you need write-after-read consistency. If your requirement is to support write-after-read consistency, there are multiple solutions to the problem. One is to not cache at the Edge, but rather at the application layer. In this case, you'd trade latency for consistency, which can be a good trade. Another way of solving the problem is by distributing the state across the edge and sharding the data based on location. This model of sharding is not always possible, but if the shared state is only used across groups of users in the same location, this solution could work very well. One example of this is Cloudflare Workers + Durable Objects, which gives you a simple Key-Value store that is persisted in a specific location, meaning that all users close to that one location can have a consistent state at low latency. When a GraphQL Edge Cache makes the most sense If you've got a single GraphQL API, and this API is the only API your frontend is talking to, you fully own this API and no traffic is bypassing your Cache, then such a CDN might actually make sense. Otherwise, I doubt you get the results you're expecting, especially not with the extra costs discussed earlier. WunderGraph - An Alternative Approach to GraphQL Edge Caching without vendor lock-in We've discussed the pros and cons of GraphQL CDNs, now I'd like to propose another approach that makes you stay vendor independent. When it comes to solving problems, sometimes it's smart to be dumb. I think a Cache can be a lot smarter when it's dumb and playing by the rules of the web, and that's exactly what we're doing with WunderGraph. and can be deployed everywhere. Our solution is Open Source How does it work? I've been working with GraphQL for many years now, and I've realized that when we deploy an application that uses GraphQL APIs, I've never seen the application change the GraphQL Operations at runtime. What applications do at runtime is change the Variables, but the Operations usually stay static. So, what we've done is, we've created a GraphQL to JSON-RPC compiler, which treats GraphQL Operations like "Prepared Statements", you've probably heard of the term from using a database. When first using a SQL statement, you send it to the Database and get back a handle to "execute" it later. Subsequent requests can now execute the Statement just by sending the handle and the variables. This makes the execution a lot faster. Because we're not changing GraphQL Operations at runtime, we can actually do this "compilation" step during development. At the same time, we're replacing HTTP POST for Queries with HTTP GET, while sending the Variables as a Query Parameter. By sending GraphQL Queries via JSON-RPC with HTTP GET,
we're able to automatically enable us to use Cache-Control Headers and ETags. And that's the whole magic of WunderGraph's Caching Story. We don't build a "smart" Cache. We don't cache Objects, and we don't have to build complex invalidation logic. We simply cache the response of unique URLs. For each Operation, you can define if it should be Cached and for how long. If a response might change frequently, you can also set the Cache time to 0, but configure "stale-while-revalidate" to a non-negative number. The client will automatically send the request to the origin with the ETag, the server will either send a refreshed response or 304, not modified. It's a very dumb approach. If the cached value is expired or stale, we're asking the origin if they have an update. Depending on the configuration, that might create a few more requests to the origin, but we're also not creating a second source of truth, or have to deal with the complexity of managing cache invalidation tags for nested objects. It's similar to using Change Data Capture (CDC) as a source for Subscriptions vs. simple Polling. CDC can get extremely complex to get right, while simple server-side Polling might work just fine most of the time. It's simple and robust. There's really not much we've invented here, all this is standard Caching behavior and any service, like Cloudflare, fastly, Varnish or Nginx supports it out of the box. There's a standard for how all the participants of the web treat Cache-Control and ETag headers, we've simply implemented this standard. By removing GraphQL from the equation, we've made it compatible with the web. If you build tools for the web, you should respect how the web is built, otherwise, you're creating more problems than you're solving. Additional Benefits of the WunderGraph GraphQL Caching Approach It's not just CDNs and Servers who understand the Cache-Control and ETag Headers. Browsers also automatically cache and invalidate your responses, without adding a single line of code. Additionally, because we've removed GraphQL from the runtime, we've automatically reduced the attack surface of our application. If our frontend doesn't change GraphQL Queries at runtime, why expose an API that allows you to do so? Limitations of the JSON-RPC Caching Approach One of the limitations of this approach is that we're no longer able to use regular GraphQL Clients, as they'd expect us to send GraphQL Operations over HTTP POST, and Subscriptions over WebSockets. That's not an issue for new projects, but might be a blocker for existing applications. Btw. we've got an internal RFC to add a compatibility mode, allowing e.g. Apollo client / urql, etc. to work with WunderGraph through an Adapter. If you're interested, please let us know. That said, using plain JSON-RPC would be very inconvenient. That's why we're not just compiling GraphQL to JSON RPC, but also generating fully type-safe clients. One such client-side integration is the , making it super easy to use WunderGraph with NextJS, including Server-Side Rendering, Authentication, File Uploads, etc... NextJS package Another limitation, as of now, is that you have to self-host WunderGraph. We're working on a hosted Serverless solution, but as of now, you'd have to deploy and run it yourself. While it might not be super convenient, this also comes with an advantage: WunderGraph is Apache 2.0 licensed, you can run it anywhere. How can you deploy WunderGraph Now that we've discussed the pros and cons of the WunderGraph approach to caching, let's see how we can deploy WunderGraph to achieve good Caching results. First, you don't have to deploy WunderGraph globally. It's possible to run it close to your origin, e.g. the (micro-) services that you'd like to use. In this scenario, it's possible to run WunderGraph as an Ingress to your other services. The architecture of this scenario looks like this: Client -> WunderGraph Server -> Origin Deploy WunderGraph with Nginx or Varnish as an additional Caching Layer If you're deploying WunderGraph close to the origin, it will automatically add the required Cache-Control & ETag Headers. This setup might already be enough for your scenario. However, in some scenarios, you'd like to add another layer of Caching Servers. This could be e.g. a cluster of Nginx or Varnish servers, placed in front of your WunderGraph server. Architecture updated with this scenario: Client -> Cache Server -> WunderGraph Server -> Origin Deploy WunderGraph with Cloudflare or fastly as an Edge Cache Depending on where your users are, a centralized Caching Layer might not be sufficient for your use case. In this scenario, you can use services like Cloudflare (Workers) or fastly, to add a globally distributed GraphQL CDN / Edge Cache. What's important to note here is that you're not tied to a specific solution. As mentioned before, we're using standardized Cache-Control directives, supported by all Cache Server solutions, so you're not tying yourself to a specific vendor. Updated Architecture: Client -> CDN -> WunderGraph Server -> Origin Deploy WunderGraph directly on the Edge, using fly.io Another option is to deploy WunderGraph directly on the Edge, removing the need for an additional Caching Layer. Services like allow you to deploy Containers as close to your users as possible. The architecture of this scenario looks like the following: fly.io Client -> WunderGraph Server (on the Edge) -> Origin Deploying a service on the Edge is not always beneficial There's one important point I'd like to make that applies to all solutions mentioned in this post. If we're deploying a workload on the Edge, this service will be very close to the users, which is generally a good thing. At the same time, depending on where on the "Edge" we're processing a request, there might be a random latency between ~0-300ms to your origin server(s) per roundtrip. If we have to do multiple roundtrips to fetch the data for a single client request, this latency can add up. As you can see, it's not always beneficial to the overall performance to have logic running on the edge. There's one more thing! This solution works for GraphQL, REST, gRPC, and other Protocols as well! You've heard that right. WunderGraph is not just about GraphQL origins. We support a wide array of upstream protocols, like REST, gRPC (coming soon!), and Databases like PostgreSQL, MySQL, Planetscale, etc... WunderGraph ingests all your services and creates a "Virtual Graph", a GraphQL Schema representing all your services. The Caching layer we've described above does not just work for GraphQL origins, but all services that you've added to your Virtual Graph. This means, you're able to apply one unified layer of Authentication, Authorization, and of course Caching to all your APIs. From our experience, it's rarely the case that you're connecting a frontend to a single GraphQL server. Instead, you're probably going to connect it to many services, which is what we're trying to simplify, and caching is one part of the story. Conclusion We've discussed various options to improve application performance with different kinds of Caching systems. There are ready-to-use solutions that come with simplicity but also lock you into specific vendors. With WunderGraph, we're trying to offer an Open Source alternative that's based on standards and implemented by many tools and vendors, so you're able to choose the best-of-breed solution for your situation. If you're looking into adding Caching to your stack, we'd love to talk! or . Join us on Discord connect on Twitter Also published . here

How to Leverage Open Source GraphQL CDN/Edge Cache With Cloudflare, Fastly, and fly.io

Writers, your just got a backend lift! If you’ve published a story on April 20th, or since, you will probably see a larger number of reads on your stories. The reason for this is that we’ve started collecting data from . Why did we do this? For more accurate readership reports! story stats Cloudflare Analytics and so using Google Analytics API (as we previously were) to measure readership per story is problematic.  Cloudflare has the advantage of being so integral to how our site’s content is delivered, that their servers’ get a chance to log the traffic before a client could ever block it. counts a successful request to a URL delivering content of type HTML. In this way, we gain a straight forward view into story traffic. 58% of tech-savvy audiences block Google Analytics Cloudflare’s approach “This doesn't mean you can go to your advertisers and start demanding more ad revenue. It should, however, mean that you now have a more accurate picture into the actual resource demands required to run your site.” - Cloudflare’s Matthew Prince Why not?  Well, one reason is that Cloudflare includes more bot and crawler traffic in their analytics. Thankfully, we can filter page views by what Cloudflare calls a botManagementDecision such that the traffic has been deemed to be “likely human” in origin.  Are you a ‘likely human’ reading this story? Then you will count towards the reads at the top of this story page and writers will see that in . Here is the filter used in the GraphQL query: their story stats "pageviewsFilter": {
        "AND": [
          {
            "botManagementDecision": "likely_human"
          },
          {
            "datetime_geq": startDate,
            "datetime_leq": endDate
          },
          {
            "requestSource": "eyeball"
          },
          {
            "AND": [
              {
                "edgeResponseStatus": 200,
                "edgeResponseContentTypeName": "html"
              }
            ]
          }
        ] This was my first time using GraphQL and while it looks like there is sort of living documentation supported, I had a hard time figuring out how to get the data I wanted.  Most of their documentation was focused on firewalls and threat detection. I ended up opening up to see what requests were being sent as I changed filters while viewing their graphs on their own site. Thankfully, I was able to paste those into , an open source client to help you write queries, and verify I was getting the same page reads seen in the charts. my browser’s network tab GraphiQL I really like the idea of using these server logs over downloading and executing javascript on the client.  It is more efficient and less intrusive.  It would be nice to remove the Google Analytics code from our site, but we still rely on it for tracking the time spent on pages and some click events. And keeping it on site just may help with search traffic ;-) Perhaps we’ll roll our own system there in the future. Currently we still only fetch stats once a day, but I’ll be looking into fetching these stats more often so stay tuned for that. Okay, so that’s the update, enjoy! You should be seeing a lot less of these guys on your story now 🙂 Head on over to today and see for yourself! https://app.hackernoon.com/stats

HackerNoon Story Stats Update: Reads Powered By Cloudflare

Actualización de estadísticas de HackerNoon Story: lecturas impulsadas por Cloudflare

Mise à jour des statistiques de l'histoire de HackerNoon : lectures optimisées par Cloudflare

HackerNoon Story Stats Update: Reads Power by Cloudflare

HackerNoon ストーリー統計の更新: Cloudflare による読み取り

Atualização das estatísticas da história do HackerNoon: leituras alimentadas pela Cloudflare

Cập nhật số liệu thống kê về câu chuyện của HackerNoon: Số lượt đọc được cung cấp bởi Cloudflare

HackerNoon 故事统计更新：由 Cloudflare 提供支持的读取

Last week, I had a problem. I wanted to start blogging using . But for SEO reasons, I wanted to put my new blog at , instead of either a completely different domain, or a subdomain like blog.cloak.ist. Ghost cloak.ist/blog That's because I want to build up SEO value for my principal domain, . Google sees subdomains as being a whole other thing. So it's muuuuch better to use if you can. cloak.ist yoursite.com/blog I spent 3 days figuring out how to set up Ghost at cloak.ist/blog. It was pretty painful. So I've decided to celebrate my blog finally existing by writing a guide to how I set it up. I hope it helps you out! Before we get started - You need to own a domain and be using Cloudflare to manage its DNS - You need to be able to buy a $5/month droplet on DigitalOcean Step 1: create a DigitalOcean 1-Click Ghost Droplet Ghost has a hosted option which you can pay for. But I'm going cheap – and also I'm pretty sure, from reading a few things online, that their hosted option won't work with this guide. So head on over to DigitalOcean and . Choose the $5/month droplet. set up a new droplet with their 1-click option Once the droplet is set up, don't go and do anything in it just yet. Just make a note of the droplet's ipv4 address . Step 2: point a subdomain at your droplet's IP You need to set Ghost up at a subdomain at your domain, like . blog.yoursite.com This is so that we can later tell to get all content from that subdomain. But don't worry, we'll also have the option of telling Cloudflare to restrict access to that subdomain to you and Cloudflare, so that you don't have a competing location for your blog living out there. yoursite.com/blog So go to Cloudflare and create a new A record, with Name set to whatever you want the subdomain to be (e.g. Name will be blog if you want to use blog.yoursite.com), and IPv4 address set to the droplet's IP that you have noted. Click the orange cloud to change it from Proxied to DNS Only! This caused me a of pain. VERY IMPORTANT: lot Step 3: drink some tea The next step will be to go back to your droplet and set up Ghost on it. But if your subdomain hasn't propagated yet (in other words, if the internet hasn't yet realised that you added the A record from the previous step), the set up will fail. DNS should propagate pretty fast, but please, please: Have a little break to give DNS time to propagate, hence the tea. I recommend chamomile.On returning from your delicious tea session, check DNS with a before going onto the next step to make sure that your subdomain is pointed at your droplet IP. service like this If you left the orange cloud enabled in the previous step, it's very likely that your Ghost set up will fail and you'll see Cloudflare IP addresses when you do the DNS check. (Not doing this step properly meant that I went into a frenzy of destroying and then remaking droplets. I went through around 5 of them before I got it right.) Step 4: set up Ghost Use . this guide to set up Ghost on your droplet When you're asked for the URL of your Ghost droplet, enter: https: //yoursubdomain.yoursite.com All of the rest of the set up should be straightforward – use the standard options throughout. If you get an error, it could be that your DNS didn't propagate yet so Ghost was unable to issue an SSL certificate for your subdomain. I'd recommend going back to Step 1 and starting again if that happens – alternatively, if your error looks different, have a google around to see if you can fix it. Quite a lot of people seem to have had issues with DigitalOcean's '1-click' Ghost installation... At this point, you should also check that your blog is live at https://yoursubdomain.yoursite.com. Keep your SSH session open! Step 5: put your blog at /blog We're going to put your blog at a subdirectory on the subdomain now, we move it to living at your root domain instead of the subdomain. before Run the following commands while you're still in your SSH session: sudo -i -u ghost-mgr cd / /www/ghost ghost config url https: ghost restart // This will let you configure Ghost var // This will move you into the Ghost installation folder to make changes //yoursubdirectory.yoursite.com/blog // This will change the URL of your Ghost blog. You can obviously change /blog to whatever you want // IMPORTANT! This will restart your Ghost instance so that the URL change comes into effect. At this point, check that https://yoursubdirectory.yoursite.com/blog is working. If you see a 404, you probably need to run ghost restart. Also, you could now set yourself up as the admin of the blog by going to https://yoursubdirectory.yoursite.com/blog/ghost. This is not a necessary step though – you can do it later. Step 6: set up a Cloudflare worker We're going to set up a Cloudflare worker at your subdirectory https://yoursite.com/blog which is going to act as a reverse proxy – so it'll basically serve your Ghost blog at that URL. Go to Cloudflare -> Workers'Add Route' -> yoursite.com/blog*'Add Route' -> www.yoursite.com/blog* if you want to make sure you'Manage Workers''Create a Worker' At this stage, you're going to want to replace everything in the code editor with the following script, after : changing the config object to reflect your set up config = { : , : , : ,
} { url = URL(request.url) targetPath = url.pathname response = fetch( ) (
  	targetPath.includes( ) || 
    targetPath.includes( ) ||
    targetPath.includes( ) ||
    targetPath.includes( )
  ) { response
  } body = response.text() body = body.split(config.subdomain).join(config.root)

  response = Response(body, response) response
}

addEventListener( , (event) => {
  event.respondWith(handleRequest(event.request))
}) // Config for our subdomain (where the Ghost blog currently lives) // and our root domain (where we want it to live) // and our blogPath (the subdirectory we want it at) // Change these to suit your case! const subdomain "yoursubdirectory.yourdomain.com" root "yourdomain.com" blogPath "blog" // Function that processes requests to the URL the worker is at async ( ) function handleRequest request // Grab the request URL's pathname, we'll use it later const new const // Change request URLs to go through to the subdomain let await `https:// ` ${config.subdomain} ${targetPath} // Ghost loads assets like JS and CSS from 3 subdirectories // We don't need to change these requests at all // So if we're getting stuff from those subdirectories, // we return the response of the fetch request from above // immediately. if '/blog/favicon.png' `/ /assets/` ${config.blogPath} `/ /public/` ${config.blogPath} `/ /content/` ${config.blogPath} return // In other cases - which will usually be pages of the // Ghost blog - we want to find any reference to our subdomain // and replace it with our root domain. // This is so that things like our canonical URLs and links are // set up correctly, so we NEVER see our subdirectory in the code. // First we get the body of the response from above let await // Then we search in the body to replace the subdomain everywhere // with the root domain. new return "fetch" I've written comments explaining what this code is doing. Paste this code in, give the Worker a slightly clearer name – e.g. yoursite-blog – hit Save and Deploy, and then go back to the main Workers page: Here, you'll want to hit Edit on the Route you created earlier, and set the Worker on it to the one you just made. Step 7: test it out At this point, if all has gone well, you should have a fully working blog at yoursite.com/blog (or whichever subdirectory you chose). If you don't, check through the instructions to see if there's anything you missed. Step 8 (optional): Restrict access to the subdomain blog So to make things cleaner, you might want to do one last thing: use a Cloudflare Firewall rule to make access to yoursubdirectory.yoursite.com impossible for everyone apart from: 1) you (because you're going to need to go there still to use the CMS) 2) Cloudflare Workers. I feel like this may be a good thing to do for SEO, since otherwise you have a blog with exactly the same content and with competing canonical URLs. (I could be wrong here though – please let me know via the live chat if so!) First, we'll create an IP list for Cloudflare IP addressesGo to Configurations at your main Cloudflare menu: Select Lists, 'Create new list': Paste everything from into your new list and hit SaveGo to your domain from the main Cloudflare menu, then Firewall -> Firewall Rules -> Create a Firewall RuleSet your rule up like this: this list - My Ghost subdomain is cloak.ist – you'll want to replace this in the URI Full filter. - You'll want to change the list filter to use your list of Cloudflare IPs. - Finally, you'll want to use filters like the last 3 to add your own IP addresses. (I have 3 here because I move between different locations a lot.) - ⚠️ You'll need to do this to ensure you have access to editing the Ghost blog! That's it! At this point you should have: - A working blog at yoursite.com/blog......which has correct links to itself, and correct canonical URLs at your root domain (yoursite.com) - A subdomain where you can go to manage your blog, which is only accessible to you and Cloudflare Hopefully this will be much better for your SEO than if you'd had to put your Ghost blog at the subdomain! If you have any questions at all, feel free to tweet @cloakist . Also published at: https://cloak.ist/blog/ how-to-put-a-ghost-blog-at-a-subdirectory-using-cloudflare-workers/

How to set up a Ghost blog on a subdirectory via Cloudflare Workers

When you buy a Domain Name from a registrar (e.g. GoDaddy) to host a website, there is high probability that you would also use their DNS zone hosting services too. However, a secure website requires an SSL certificate, which is not free. Therefore, some people utilize Cloudflare services to get a free SSL bundled with their DNS services. You might have many reasons to choose Cloudflare as your preferred DNS service provider, but you realize very soon that it has started posing new challenges due to some inaccurate configurations. In fact, there is nothing wrong at the Cloudflare-side. It's merely due to lack of our understanding and insufficient documentation for novice users. Imagine a scenario where you subscribed to a web-based application that is hosted by a third-party (e.g. Okta, Zoho). It simply means that you do not have direct access to their web-server instance. You would assume that a simple CNAME record configuration in Cloudflare's DNS setting should be enough to start serving your website. It's not that straight forward. Firstly, you must understand that the Cloudflare is running an HTTP proxy server (i.e. kind of a web-broker) that has two sides to be managed. Being a web-proxy, the Cloudflare allows two different options to you. First option is a Proxy-mode (i.e. traffic intercepted by the Cloudflare) and another one is a DNS-only mode (i.e. traffic sent directly to  web origin servers without changing anything). On one side, it is accepting web request from your user's web browser (e.g. Chrome, Safari) that expects a secure traffic (i.e. HTTPS) therefore the SSL certificates used by Cloudflare must be compatible and acceptable to your user's web browser. This SSL certificate is a major hurdle many times in the configuration. On the other side, the Cloudflare must contact the web-application's origin servers that might be expecting a secure traffic (i.e. HTTPS) therefore SSL certificates acceptable to your origin servers must be used by Cloudflare in a proxy-mode else choose DNS-only mode. Many times people choose to use DNS-only, but the website is not served properly yet. Perhaps, additional configuration is required in Page Rules to route the traffic in right manner to origin servers. Sometimes the problem is due to type of SSL certificate you are using. Also, have some patience while you migrate from some other DNS provider because DNS settings propagation time could also frustrate sometimes. Typically, it takes approx. 24 ~ 48 hours to get DNS settings working properly after you make changes. Acquire some knowledge related to SSL certificates verification method as well, if you are a novice. Some companies (e.g. Okta) have simplified configuration of DNS settings using Cloudflare. I enjoy using Cloudflare due to its analytics capabilities and simple to use user interface.

How To Link a Third-Party Hosted Website using Cloudflare Services

Two companies that design and manufacture wedding dresses are suing Cloudflare for copyright infringement after it failed to terminate service to 99 'repeat infringer' websites. Allure Bridals and Justin Alexander claim that they sent 7,000 DMCA complaints to the CDN company but aside from passing the notices on, Cloudflare failed to take more meaningful action. When copyright holders feel they have exhausted all options to have websites stop their allegedly-infringing activities, there is a growing trend to move further up the chain. Sites now regularly have copyright complaints filed against their hosting companies and domain registries, for example, demanding that they take action to prevent contentious behavior. Since millions of websites now use Cloudflare’s services, that makes the CDN provider a prime candidate for pressure. A new case filed yesterday in a Tennessee district court provides yet another example. American Clothing Express Inc., which does business as Allure Bridals and Justin Alexander, designs and manufactures wedding dresses. As part of the companies’ sales and marketing efforts, they claim to spend hundreds of thousands of dollars per year on photoshoots featuring models wearing their creations. According to the companies, however, the resulting photographic images are also being deployed by unauthorized overseas websites (sample below) in an effort to drive customers to unaffiliated bridal stores in local markets selling “cheap imitation” dresses. The plaintiffs state that they lack a meaningful remedy against such sites, noting that the majority are hosted on servers in China, other locations in South East Asia, or on offshore servers that advertise their non-compliance with United States’ copyright laws. “Complaints sent by Plaintiffs, or their agents, to the Infringing Website Defendants, or to the entities hosting them in these far-away jurisdictions, largely fall on deaf ears. Domestic judgments obtained against the Infringing Website Defendants are often unenforceable against them in their home jurisdictions,” the complaint reads. The filing lists 99 websites (represented by Does 1-200) falling into these categories that all have something in common – they are or have been customers of US-based Cloudflare. As a result, the plaintiffs have resorted to filing infringement notices with the CDN company, hoping it will take action to restrict the availability of the infringing images. Indeed, over the past three years the companies claim that they sent several thousand infringement notifications to Cloudflare which included the URLs of pages on the allegedly infringing sites where unlicensed images were being used. The complaint acknowledges that Cloudflare forwarded the complaints to its customers and their hosts but due to the nature of the clients, the hosting providers mostly ignored the takedown demands. The complaint targets the operators of the 99 sample sites with claims of direct copyright infringement but additionally, due to Cloudflare’s involvement, the CDN company itself is accused of contributory copyright infringement. “CloudFlare had actual knowledge of the specific infringing activity at issue here because anti-counterfeiting vendors retained by Plaintiffs delivered more than seven thousand notifications to CloudFlare of the ongoing infringement being prosecuted herein over the course of three years,” the complaint reads. In common with a in California involving another bridal company, the plaintiffs in this matter also state that Cloudflare should have taken more permanent action when they realized that complaints were being made against the same customers time and again, as illustrated by the sample in the image below. similar on-going case “CloudFlare could have stopped this infringement being perpetrated through its CDN by simply terminating the accounts of repeat infringers,” the complaint continues. “CloudFlare has never terminated a repeat infringer in response to notifications sent by Plaintiffs or other bridal manufacturers. Consequently, an exceedingly disproportionate amount of websites infringing Plaintiffs’ copyrights are optimized by CloudFlare, as opposed to other providers of CDNs, due to CloudFlare’s well-known policy of refusing to terminate repeat infringers.” While the plaintiffs don’t mention Cloudflare’s competitors by name, the complaint alleges that in response to similar copyright infringement notices, other CDN providers told their clients that if the images weren’t removed, their entire website accounts would be terminated. The term ‘repeat infringer’ is becoming increasingly common in United States copyright infringement cases. In December 2019, Cox Communications was hit with a after a Virginia federal jury determined that the ISP didn’t do enough to stop repeat infringers. Cox was found to be contributorily and vicariously liable for the alleged pirating activities of its subscribers on more than 10,000 copyrighted works. $1 billion copyright infringement verdict For comparison, Allure Bridals and Justin Alexander state that Cloudflare is liable for contributory copyright infringement relating to more than 5,000 infringing images published on 99 different websites. Overall, Cloudflare serves many thousands of pirate sites, making the outcome of this and similar cases of particular interest. In respect of the “willful and intentional” direct infringement claims against the 99 websites themselves, Allure Bridals and Justin Alexander request actual or statutory damages, injunctive relief to prevent the ongoing infringements, and the destruction of all copies of copyright works made in violation of the bridal companies’ rights. The contributory copyright infringement claim against Cloudflare asserts that the CDN company assisted the direct infringers by storing copies of the infringing images on servers in the United States, improving the performance of the infringing websites, while concealing their true locations. As a result, Cloudflare’s behavior is also described as “willful and intentional”, with the plaintiffs demanding a similar injunction in addition to actual or statutory damages. The complaint can be obtained here ( pdf ) Originally published as “ Cloudflare Sued For Failing to Terminate 99 ‘Repeat Copyright Infringing’ Sites " with the Attribution-NonCommercial 3.0 Unported (CC BY-NC 3.0) license

Cloudflare Sued For Failing to Terminate 99 ‘Repeat Copyright Infringing’ Sites

On July 18th, 2020, caused an outage for about 27 minutes affecting businesses worldwide. a configuration error in Cloudflare’s backbone network The error configuration sent all traffic across the backbone to Cloudflare’s Atlanta node and unfortunately “overwhelmed” the router. Cloudflare later mentioned they saw traffic drop by about 50% across the network. Although not entirely catastrophic thanks to the architecture design, but many locations with heavy internet usage were affected including San Jose, Seattle, Los Angeles, Chicago, Washington, DC, London, Amsterdam, Frankfurt, Paris, Stockholm, Moscow, São Paulo, and the list goes on. How our system handled it Cloudflare is one of the many CDNs mlytics offer via Power-Ups (CDN marketplace) on the platform. When the outage occurred, the AI Load Balancing swapped out Cloudflare with the next best-performing CDN (varies in different regions) for requests. The chart below is the AI Load Balancing optimization chart, it shows which CDNs were used (swapped over) during a certain timeframe. In this case, we’re using a demo site with Akamai and Cloudflare installed. As shown, Akamai had several query spikes in the states due to the Cloudflare performance drop – the system is actively swapping over to Akamai. Frequent swaps were made from Cloudflare to Akamai in the US This had successfully helped many of our customers mitigated the outage, and we received no complaints over the course. On the chart, we’re also seeing a latency spike and availability drop for Cloudflare in the same time frame. This helps illustrate a clear picture of what happened when aligning this with the chart above. Pulse It shows a latency spike and availability drop for Cloudflare on the Pulse chart We believe all the CDN providers are doing their best to deliver the best performance possible, but unfortunately, things do happen. And our job at mlytics is to help our customers minimize the exposure of such risk and maximize the uptime at all times. Again, outages DO happen This is a wake-up call reminding all businesses will eventually have to deal with outages if offering any form of web services. Thus, a solid cloud redundancy and disaster recovery plan in place to prevent any event from taking down your service is imperative. Imagine losing money and customers hate you… horrible right? But stopping users from doing what they were doing (especially if they enjoy it so much) is your express train ticket to PR hell. Small or medium businesses might have room for mistakes, but some of the bigger companies affected by the outage such as Discord, Feedly, Politico, Shopify, and League of Legends might not be so lucky. These are some of the most used online services today, and imagine millions of users got disconnected without being informed. We've talked about in the past, and there are just so many factors that can bring a website off the grid. A website without a backup plan is exactly like sailing a cruise without liferafts – you’re betting you won’t need it. website downtime causes and solutions Previously published at https://mlytics.com/blog/how-we-helped-our-users-mitigated-the-cloudflare-outage/

Helping Our Users Mitigate The 2020 Cloudflare Outage - How We Did It

In 2018, four of Japan's largest manga publishers filed a motion at a Tokyo court demanding that Cloudflare stop providing services to several 'pirate' sites, including Mangamura replacement Hoshinoromi. The companies now reveal that a settlement has been reached with Cloudflare to "stop the replication" of the sites on its Japan-based servers, if a court declares them illegal. As the largest CDN and DDoS mitigation service on the planet, Cloudflare provides services to millions of websites. A tiny proportion of these sites are on the radars of entertainment and publishing companies since they either directly offer or link to unlicensed copies of copyrighted works. As a result, Cloudflare is under almost continual pressure to cease doing business with these entities. As first reported in September 2019, Japan-based publishers Shueisha, Kadowaka, Kodansha, and Shogakukan sued the ‘pirate’ site Hoshinoromi in a New York federal court. The platform, which positioned itself as a replacement for self-shuttered pirate site Mangamura, was accused of “willful and massive infringement” of the publishers’ copyrights. here on TF That case is still ongoing and according to a filing late last week , the publishers are having considerable difficulty identifying and serving the defendants, so need an extension. Cloudflare was mentioned in that lawsuit too and it now transpires that the same publishers previously targeted the CDN company in a Japan court back in 2018. ( pdf ) In common with other lawsuits in Japan, details are hazy. However, according to a joint statement issued late last week, Shueisha, Kadowaka, Kodansha, and Shogakukan filed a motion at the Tokyo District Court in August 2018 with a demand that Cloudflare should stop providing services to several ‘pirate’ platforms, Hoshinoromi included. Due to caching, that amounted to Cloudflare delivering infringing content to the public, they argued. For reasons that appear related to the ongoing case in the United States, they have waited until now to reveal a settlement of sorts with Cloudflare. It was reportedly reached in June 2019 and seems to hinge on whether a court determines that the ‘pirate’ sites in question are copyright-infringing and therefore illegal. The publishers’ statement indicates that when the ‘pirate’ sites using Cloudflare are viewed by users in Japan, most of those users will be accessing them via Cloudflare’s Japan-located servers. So, if the Tokyo District Court rules that the sites are illegal, Cloudflare has reportedly agreed to “stop the replication of the sites to Cloudflare’s servers in Japan.” At least in part, the announcement is designed to be a warning to other ‘pirate’ sites that may be considering using Cloudflare’s services to improve uptime and general accessibility. Whether it will make much of a difference on the ground remains to be seen, however. While this particular matter appears to be settled, last December by Takeshobo, another major publisher based in Japan that distributes dozens of manga publications, many under the Bamboo Comics label. Cloudflare was sued The publisher said it was forced to sue Cloudflare because takedown notices sent to the CDN company in respect of a nameless ‘pirate’ site were effectively ignored, allowing infringing material to stay online via Cloudflare’s services. Progress in that particular case is unknown but the settlement with Shueisha, Kadowaka, Kodansha, and Shogakukan may provide a possible solution for Takeshobo. Cloudflare is obviously extremely cautious when faced with similar lawsuits, always insisting that as a service provider it is not responsible for the activities of its users. Last week, however, the effects of a ruling handed down in December by a German court saw Cloudflare under the threat of serious fines. disconnect pirate music platform DDL-Music Originally published as “ Cloudflare Agrees to Stop Caching Pirate Content in Japan, If Court Declares Sites Illegal ” with the Attribution-NonCommercial 3.0 Unported (CC BY-NC 3.0) license

Cloudflare Agrees to Stop Caching Pirate Content in Japan, If Court Declares Sites Illegal

With the advent of Cloudflare’s new , it is now easy to download and inspect a zone’s access logs by a single RayID (Cloudflare’s fancy term for a request ID), a defined time period, or from the last downloaded. In our case, we ship all of our other logs to so everyone can analyze and parse them in any way imaginable! Enterprise Log Share (ELS) REST API RayID Sumo Logic Unfortunately, there seems to be no cut-and-dry way to integrate the two. Thankfully, Lambda functions are now able to be run on a scheduled interval using — sort of like you would a cron job on a server. AWS CloudWatch Scheduled Events There are a couple of clear benefits of using a Lambda function instead of a job on a physical/virtual server you run yourself: Execution of a Lambda function , and it may actually cost nothing if you remain within the free tier AWS hands out every month Low cost: costs next to nothing In my case, I used the to allow me to quickly get up and running, invoke a function locally (which is key to developing and iterating quickly), and it makes deployments simple. I also recommend checking out , too. Ease of development: Serverless framework apex Lambda functions are This is important for things which require speed, such as parsing and ingesting logs for a high-traffic site like ours. Speed: fast. Write the code, test and deploy — that’s it! No Chef, no Docker, no nothing… just write your function, ship it and it is live. No servers to build/maintain: How It Works First, you’re going to want to to ingest the Cloudflare logs. Set the source category, host, etc. at SumoLogic to make the configuration of the Lambda function easier. create a new HTTP source Next, create a new Serverless function: $ mkdir -p ~/cloudflareToSumoLogic $ cd ~/cloudflareToSumoLogic $ serverless create \--name cloudflareToSumoLogic \--template aws-nodejs \--path ~/cloudflareToSumoLogic Let’s walk through the configuration file: serverless.yml functions: cloudflareToSumoLogic: # This is the name of the Lambda function **_\# This is the name of the handler; this is the "main"_    \# _function called by Lambda_**  
handler: index.handler

**_\# This function was written to be run every one minute  
# Any adjustments will require code changes..._**  
events:  
  - schedule: rate(1 minute)

**_\# Required environment variables:_**  
environment:  
  **\# Grab this URL from the Sumo Logic source you just created**  
  SUMO\_ENDPOINT: ...  
  **\# Get these values from the Cloudflare admin UI**  
  CLOUDFLARE\_ZONE\_ID: ...  
  CLOUDFLARE\_AUTH\_EMAIL: ...  
  CLOUDFLARE\_AUTH\_KEY: ... Finally, you’re going to need the function: Now, try it out! $ serverless invoke local -f cloudflareToSumoLogic startTime: 2017-02-11T15:07:00.000ZendTime: 2017-02-11T15:08:00.000Z cloudflareOpts: { method: 'GET',hostname: 'api.cloudflare.com',path: '/client/v4/zones/.../logs/requests?start=1486825620&end=1486825680',headers:{ 'X-Auth-Email': ' ..','X-Auth-Key': '...' } } . res.statusCode:  200 res.headers:  { date: 'Sat, 11 Feb 2017 15:38:40 GMT','content-type': 'application/json','transfer-encoding': 'chunked',connection: 'close','set-cookie': [ '__cfduid=...; expires=Sun, 11-Feb-18 15:38:39 GMT; path=/; domain=.cloudflare.com; HttpOnly' ],vary: 'Accept-Encoding','x-cf-request-id': '00000000-0000-0000-0000-000000000000','strict-transport-security': 'max-age=31536000','served-in-seconds': '0.355',server: 'cloudflare-nginx','cf-ray': '...-ORD' } Log events: 659 Everything look good? It’s time to deploy: $ serverless deploy Serverless: Packaging service...Serverless: Uploading CloudFormation file to S3...Serverless: Uploading service .zip file to S3 (9.45 KB)...Serverless: Updating Stack...Serverless: Checking Stack update progress........................Serverless: Stack update finished...Serverless: Removing old service versions... Service Informationservice: cloudflareToSumoLogicregion: us-east-1api keys:Noneendpoints:Nonefunctions:cloudflareToSumoLogic Analyzing Ingested Logs in Sumo Logic Now that your logs are being ingested every minute, you can run the following query and you should be able to start seeing results like the screenshot above: _sourceCategory=...| json auto| toLong(num(timestamp)) as timestamp| timestamp as _messageTime| fields - timestamp| fields - _raw Replace with the HTTP source category that you created earlier for your Cloudflare logs. This snippet parses the JSON lines automatically, translates the log timestamp to a time object, replaces the with the log timestamp, then remove the raw JSON object and its ingest timestamp. _sourceCategory timestamp _messageTime

How to Ship Cloudflare logs to Sumo Logic with Lambda

Why? For me, self-hosting is like having my own personal playground where I can experiment, tinker, and learn. It’s a great way to explore new technologies, try out different setups, and have fun with my projects. As part of my job, I need to have a deep understanding of developer experience. The best way to build this understanding is to be the developer, both initial experience with any development tool, as well as day-to-day work with these systems. Self-hosting is in a way, building empathy with the developer community. Understanding differences, and good/bad versions. My main reason is “learning.” There are a bunch of other reasons one may choose to do this: Privacy: your data
Full control: You own it (well, both good and bad)
Cost-effective: (may not be always true, but mostly true) You Shouldn’t Do This… But if You Really Want to… I don’t suggest this route for the majority of the people. It’s hard, you’ll hit walls way more frequently than you want. You have to be a warrior. If your reason is similar to mine, go for it. There is a strong determining factor though, your connectivity. Let’s get started. I’ll start the connectivity, then the hardware and software. Sounds Common but Not So Common: It’s a Privilege to Be on Fios While high-speed internet has become more commonplace, it’s still a privilege, especially in the United States. We’re (I’m) definitely taking it for granted. I use Verizon Fios is a fiber internet service. If it wasn’t for this, I wouldn’t self-host my stuff. The big practical separation for Fios is how stable it is, regardless of the “mpbs/gbps” package you have. I used Fios in residential and office setups in New York City for years. I got off as I moved to different neighborhoods, and I really really missed it when I didn’t have it, even though I got 1gbps service packages from other providers. Back in the day when we used dial-ups and tried to play Half-Life (or Counter-Strike) online, your connection speed mattered but “lag” mattered even more. I lived in Turkey back then, and we had a cable internet provider vs ADSL services the difference was that you were getting super low lag/ping in cable network even though you were 1/4, 1/3th of the connection speed you had compared to other services that bragged how fast they were. Didn’t matter when playing online. I have 300/300mbps which is what’s called a “symmetrical” connection. 300mbps is already way higher than the average internet connectivity worldwide (although certain countries/cities regions have way faster networks, overall world citizen gets access to the internet at lower speeds). But it would be ok even if it was slower because it’s a Fiber network, and it’s symmetrical which means download and upload speeds are the same. Often, you see traditional ISPs advertise something ridiculous with high speeds like 500mbps, but it’s often only referred to as download speeds. And in the majority of consumer scenarios, this is fine. But you need the upload speed to be high and consistent/stable when you want to serve upstream. Hardware Since it’s hobby purposes, I initially searched some “old” servers (like servers sitting on racks) on eBay. Then I realized it had a million combinations of hardware components, like CPU architectures, and network interfaces. I quickly went down the rabbit hole of Reddit threads both fun and scary stories. These “serious” server hardware were electric eating, heating sources that are also giant, require space kinda machines turned me off, and I backed out quickly. Then I explored mini PCs, which are more common computers that could handle my applications really easily. Think like you’re looking for a computer that you could use, but instead, you just host stuff and it sits somewhere in your home in a closet, without being a fire risk or a thing that you need to worry about how to keep it cool. I bought an “HP Elite Desk Mini” which is a decent computer if I were to use it as my desktop. It has 16GB memory, an i7 quad-core CPU, and 510 GB SSD. I think I bought it under $150 on Amazon. You can go fancier with a much beefier machine with a few hundred dollars if you’re being much more serious about this. I’m thinking of buying another (same machine) and stacking them. The footprint of this machine is super small. It snuck under my Verizon router in a closet, has almost zero noise, and barely heats. I’m sure if I find an ARM version of this thing (or a Raspberry Pi) I can go smaller and have almost no heat but I’ve never seen overheating on these. Whether this machine is a good or bad hardware decision, it’s debatable, but I’m really happy a few years in. Software: Ubuntu & Docker The first thing I did was to clean it up and install Ubuntu (LTS). Almost bare-bones Ubuntu then right away docker installed. I have Nginx and PHP on it for some early play of a WordPress blog (not this one) but then abandoned it. I run almost everything exclusively in Docker (more on this below). I try to update & upgrade Ubuntu once a year. Nothing else. Access: Cloudflare Zero Trust The machine itself is completely closed to direct internet access. Its IP Tables don’t allow connections even from the local network (except SSH port accepts local network IP range). Traditionally, the machine needs to open ports outside, then have a router port forwarding and set up all public IP sort things. More than 2 decades ago, I did that with static IP from my ISP. Man, all the hustle… None of that is necessary anymore. I use Cloudflare Access and Tunnels which has an agent always running in the server, and from remote configuration, I can listen to any internal port (without opening it up) and forward the port directly to a subdomain of my domains. This shortcut the DNS works for me too. On top of that, most of my private apps run on subdomains that are protected by Cloudflare Zero Trust access (only me). I love Cloudflare’s feature that solves 2 to 3 problems at once. One might wonder, what happens if Cloudflare has an outage, and their Zero Trust tools stop working, does it open my apps to the public all of a sudden? No, because my apps are not open to the public in the first place. Zero Trust tunnel has to work in order to open it up to the public, and if Zero Trust authentication is down, the subdomain will also not be accessible (because it’s proxied through Zero Trust “application” record. Worst case scenario, I lose access to my private apps from outside. Even with that, I can SSH to my server and create a tunnel, port forwarding to the specific port the app is running. On a normal day, I simply join to Zero Trust network using Cloudflare’s desktop app WARP, which replaces VPN for me. All things considered, I’m sure there are still holes and paranoia in this plan. You can go through a more traditional route that is not any different than hosting this instance in Digital Ocean or AWS and replicate what you think is “more secure”, but I’m pretty happy with the baseline Cloudflare brings, and solves a few unnecessary things I have to take care (like no need to do reverse proxy for all apps I’m running). Deploy Apps: Portainer + Gitops I use Portainer to both set up deployments and manage my containers. Portainer is essentially a nice UI version of your docker command line tools. But where it shines is the gitops integration that integrates with GitHub via webhooks, so when I push any change to any of my app repos (which all have docker-compose.yml that includes their infra and application configurations), my apps get re-deployed by portainer. This makes spinning up a new app, or an open-source tool in my server, a breeze. I covered Portainer and its gitops integration in this article: Portainer + gitops ❤️: A simple way to deploy and manage your self-hosted applications Why? For me, self-hosting is like having my own personal playground where I can experiment, tinker, and learn. It’s a great way to explore new technologies, try out different setups, and have fun with my projects. As part of my job, I need to have a deep understanding of developer experience. The best way to build this understanding is to be the developer, both initial experience with any development tool, as well as day-to-day work with these systems. Self-hosting is in a way, building empathy with the developer community. Understanding differences, and good/bad versions. My main reason is “learning.” There are a bunch of other reasons one may choose to do this: Privacy: your data Full control: You own it (well, both good and bad) Cost-effective: (may not be always true, but mostly true) Privacy : your data Privacy Full control : You own it (well, both good and bad) Full control Cost-effective : (may not be always true, but mostly true) Cost-effective You Shouldn’t Do This… But if You Really Want to… I don’t suggest this route for the majority of the people. It’s hard, you’ll hit walls way more frequently than you want. You have to be a warrior. If your reason is similar to mine, go for it. There is a strong determining factor though, your connectivity. Let’s get started. I’ll start the connectivity, then the hardware and software. Sounds Common but Not So Common: It’s a Privilege to Be on Fios While high-speed internet has become more commonplace, it’s still a privilege, especially in the United States. We’re (I’m) definitely taking it for granted. I use Verizon Fios is a fiber internet service. If it wasn’t for this, I wouldn’t self-host my stuff. The big practical separation for Fios is how stable it is, regardless of the “mpbs/gbps” package you have. I used Fios in residential and office setups in New York City for years. I got off as I moved to different neighborhoods, and I really really missed it when I didn’t have it, even though I got 1gbps service packages from other providers. Back in the day when we used dial-ups and tried to play Half-Life (or Counter-Strike) online, your connection speed mattered but “lag” mattered even more. I lived in Turkey back then, and we had a cable internet provider vs ADSL services the difference was that you were getting super low lag/ping in cable network even though you were 1/4, 1/3th of the connection speed you had compared to other services that bragged how fast they were. Didn’t matter when playing online. I have 300/300mbps which is what’s called a “symmetrical” connection. 300mbps is already way higher than the average internet connectivity worldwide (although certain countries/cities regions have way faster networks, overall world citizen gets access to the internet at lower speeds). But it would be ok even if it was slower because it’s a Fiber network, and it’s symmetrical which means download and upload speeds are the same. Often, you see traditional ISPs advertise something ridiculous with high speeds like 500mbps, but it’s often only referred to as download speeds. And in the majority of consumer scenarios, this is fine. But you need the upload speed to be high and consistent/stable when you want to serve upstream. Hardware Since it’s hobby purposes, I initially searched some “old” servers (like servers sitting on racks) on eBay. Then I realized it had a million combinations of hardware components, like CPU architectures, and network interfaces. I quickly went down the rabbit hole of Reddit threads both fun and scary stories. These “serious” server hardware were electric eating, heating sources that are also giant, require space kinda machines turned me off, and I backed out quickly. Then I explored mini PCs, which are more common computers that could handle my applications really easily. Think like you’re looking for a computer that you could use, but instead, you just host stuff and it sits somewhere in your home in a closet, without being a fire risk or a thing that you need to worry about how to keep it cool. I bought an “HP Elite Desk Mini” which is a decent computer if I were to use it as my desktop. It has 16GB memory, an i7 quad-core CPU, and 510 GB SSD. I think I bought it under $150 on Amazon. You can go fancier with a much beefier machine with a few hundred dollars if you’re being much more serious about this. I’m thinking of buying another (same machine) and stacking them. The footprint of this machine is super small. It snuck under my Verizon router in a closet, has almost zero noise, and barely heats. I’m sure if I find an ARM version of this thing (or a Raspberry Pi) I can go smaller and have almost no heat but I’ve never seen overheating on these. Whether this machine is a good or bad hardware decision, it’s debatable, but I’m really happy a few years in. Software: Ubuntu & Docker The first thing I did was to clean it up and install Ubuntu (LTS). Almost bare-bones Ubuntu then right away docker installed. I have Nginx and PHP on it for some early play of a WordPress blog (not this one) but then abandoned it. I run almost everything exclusively in Docker (more on this below). I try to update & upgrade Ubuntu once a year. Nothing else. Access: Cloudflare Zero Trust The machine itself is completely closed to direct internet access. Its IP Tables don’t allow connections even from the local network (except SSH port accepts local network IP range). Traditionally, the machine needs to open ports outside, then have a router port forwarding and set up all public IP sort things. More than 2 decades ago, I did that with static IP from my ISP. Man, all the hustle… None of that is necessary anymore. I use Cloudflare Access and Tunnels which has an agent always running in the server, and from remote configuration, I can listen to any internal port (without opening it up) and forward the port directly to a subdomain of my domains. This shortcut the DNS works for me too. On top of that, most of my private apps run on subdomains that are protected by Cloudflare Zero Trust access (only me). I love Cloudflare’s feature that solves 2 to 3 problems at once. One might wonder, what happens if Cloudflare has an outage, and their Zero Trust tools stop working, does it open my apps to the public all of a sudden? No, because my apps are not open to the public in the first place. Zero Trust tunnel has to work in order to open it up to the public, and if Zero Trust authentication is down, the subdomain will also not be accessible (because it’s proxied through Zero Trust “application” record. Worst case scenario, I lose access to my private apps from outside. Even with that, I can SSH to my server and create a tunnel, port forwarding to the specific port the app is running. On a normal day, I simply join to Zero Trust network using Cloudflare’s desktop app WARP, which replaces VPN for me. All things considered, I’m sure there are still holes and paranoia in this plan. You can go through a more traditional route that is not any different than hosting this instance in Digital Ocean or AWS and replicate what you think is “more secure”, but I’m pretty happy with the baseline Cloudflare brings, and solves a few unnecessary things I have to take care (like no need to do reverse proxy for all apps I’m running). Deploy Apps: Portainer + Gitops I use Portainer to both set up deployments and manage my containers. Portainer is essentially a nice UI version of your docker command line tools. But where it shines is the gitops integration that integrates with GitHub via webhooks, so when I push any change to any of my app repos (which all have docker-compose.yml that includes their infra and application configurations), my apps get re-deployed by portainer. This makes spinning up a new app, or an open-source tool in my server, a breeze. I covered Portainer and its gitops integration in this article: Portainer + gitops ❤️: A simple way to deploy and manage your self-hosted applications Portainer + gitops ❤️: A simple way to deploy and manage your self-hosted applications

How I Hosted My Hobby Projects From a Cheap HP Mini Desktop From My Closet (Verizon Fios)

I want to talk about a quick tweak you can do in your CDN TTL settings to radically improve your site’s performance. Direct impact on Time-To-First-Byte (TTFB) metric, but as a halo effect, pretty much every other Web Vital. You can do this in any CDN since TTL customization is a pretty standard need, and most CDN providers have easy ways to create rules for various rule configurations. I use Cloudflare for my blog’s CDN layer. Cloudflare already comes with nice defaults for optimizing the delivery of static assets like images, javascript, css files. But for HTML documents, CDNs use cache-control headers to determine how to cache, and how long to cache. Applications return this header, and it’s a way for the application (origin) to tell CDNs how to behave on certain pages. But in this optimization method, we’ll simply override all (or most) of our pages to be highly cached and served from the cache while revalidating in the background. The way this works is CDN always serves the “last” cached HTML to the reader (or crawler) from the edge network, really really really fast (in some cases double-digit milliseconds), and triggers a request to the origin server to get the “latest” version. Most applications also return proper response codes if the content hasn’t changed from the timestamp that CDN will ask if there is a new update to the content. How to Configure Custom TTL in Cloudflare To set up custom edge TTL in Cloudflare, navigate to your site page, Caching > Cache Rules page. Create a new rule, give it a name, and then set up the request path configuration. You can set multiple expressions, and exclude patterns that you know are Admin, or Rest API, or other URLs that should NOT be cached long. I use WordPress for my blog, and I exclude paths containing things like wp-admin, wp-json, cron… Then, select “Ignore cache-control header,” and use this TTL in the Edge TTL section. And finally, select how long you want to cache. Longer is better because longer means most of your site content, including long-tail content that doesn’t get consistent traffic, will also be cached at the edge. I started with 1 day, then 1 week, then I tried 1 month, but then had some pages stuck in the cache too long and dialed it back to 1 week as my sweet spot. Even if you’re not using Cloudflare, I’m sure there is an equivalent of this in your CDN provider. What Is the Impact on Page Speed? After the change, I saw a big drop (like 90% reduced load) in my server’s load. It meant CDN was doing what it was supposed to do. It’s one of the positive side effects of doing higher cache offload to CDN, to be able to scale higher traffic without needing powerful hosting resources. My Time-To-First-Byte decreased (improved) 70%, coming from shy of 500ms down to 100–160ms range 🤯 More importantly, the real user experience on the page became even more mind-blowing because things became super snappy. Click click click, bam bam bam, nothing was in a visible loading state anymore. Even if metrics didn’t move, I am super happy with this aspect of the change. 🤯🤯🤩 I got my Cloudflare Web Analytics email and noticed almost all Web Vitals moved positively at least 30% improvement. I wasn’t expecting other Web vitals like CLS and LCP to be directly impacted (or impacted as much as they did). But it makes sense. When the assets load much faster like this, the “wait time” (or blocking time) goes down, therefore, the layout shift or the largest paint goes down. SEO Impact It’s a well-known fact that Google takes your “core web vitals” into account when determining your ranking in the search results. This change has more impact on crawlers than you think. Because most of the time, crawlers’ requests are the ones that hit “cache cold” pages. It means Google (or other search engines) is reading your site holistically way more than your real users. Imagine every single article you wrote. There is no user who reads every single one of them — Google does 🙂 (and does it regularly). When a crawler tries to visit a page that nobody read in a long time, its request will have cache-miss more likely than cache-hit, so it will “wait” longer for your web server to render the page. When you put yourself in the crawler’s shoes, imagine you try to read 10,000 articles/pages on a site over a day or two period (maybe it takes longer, who knows…). Now, consider the percentage of those pages that will have to be rendered, or served from the CDN cache. The more pages Google sees “slow”, it will think your whole site is slow. This is where the real value of super-long TTLs comes in. Especially if you combine that with serve-stale-while-revalidate (SSWR) which most CDNs automatically do (if not, I’m sure there is a setting you can enable these together). SSWR, with a super-long TTL (like 7 days, or more), basically creates an infinite loop of “always cached” scenarios. And with that, your crawler traffic gets served from the cache (at cost/risk of “stale content” which is OK in the vast majority of use cases), and directly increases your site’s overall speed score and, therefore your SEO scores. Content Freshness There is one caveat though, which is content freshness. When you bump the Edge TTL up to multi-day TTLs like I did, you need to make sure your CMS/site is nicely integrated with your CDN’s cache clear systems, in the case you make updates. Two scenarios: You update existing content (like fixing a typo or changing the cover image of a post); the change should be reflected on the content’s detail page right away. You publish new content, so the new content is supposed to show up in common places like your homepage. You can use your CDN’s cache clear UI or APIs to trigger “purge” on URLs you think it’s impacted (homepage, section pages, etc.), or highly visible pages like the homepage can be configured with a lower TTL in a separate cache rule set. I use WordPress for my content management system and Cloudflare has a WordPress plugin to listen to publish/update hooks to trigger these cache clear nicely. Another way to think about this is to find the balance. What is the “staleness” you can tolerate on a page? Let’s say another article detail page showing “recent articles” or “related articles” sections to NOT show your most recent article there. As long as that time length is not something you can’t afford, cache longer to achieve better site/page performance. I want to talk about a quick tweak you can do in your CDN TTL settings to radically improve your site’s performance. Direct impact on Time-To-First-Byte (TTFB) metric, but as a halo effect, pretty much every other Web Vital. You can do this in any CDN since TTL customization is a pretty standard need, and most CDN providers have easy ways to create rules for various rule configurations. I use Cloudflare for my blog’s CDN layer. Cloudflare already comes with nice defaults for optimizing the delivery of static assets like images, javascript, css files. But for HTML documents, CDNs use cache-control headers to determine how to cache, and how long to cache. Applications return this header, and it’s a way for the application (origin) to tell CDNs how to behave on certain pages. But in this optimization method, we’ll simply override all (or most) of our pages to be highly cached and served from the cache while revalidating in the background. The way this works is CDN always serves the “last” cached HTML to the reader (or crawler) from the edge network, really really really fast (in some cases double-digit milliseconds), and triggers a request to the origin server to get the “latest” version. Most applications also return proper response codes if the content hasn’t changed from the timestamp that CDN will ask if there is a new update to the content. How to Configure Custom TTL in Cloudflare To set up custom edge TTL in Cloudflare, navigate to your site page, Caching > Cache Rules page. Create a new rule, give it a name, and then set up the request path configuration. You can set multiple expressions, and exclude patterns that you know are Admin, or Rest API, or other URLs that should NOT be cached long. I use WordPress for my blog, and I exclude paths containing things like wp-admin, wp-json, cron… Then, select “Ignore cache-control header,” and use this TTL in the Edge TTL section. And finally, select how long you want to cache. Longer is better because longer means most of your site content, including long-tail content that doesn’t get consistent traffic, will also be cached at the edge. I started with 1 day, then 1 week, then I tried 1 month, but then had some pages stuck in the cache too long and dialed it back to 1 week as my sweet spot. Even if you’re not using Cloudflare, I’m sure there is an equivalent of this in your CDN provider. What Is the Impact on Page Speed? After the change, I saw a big drop (like 90% reduced load) in my server’s load. It meant CDN was doing what it was supposed to do. It’s one of the positive side effects of doing higher cache offload to CDN, to be able to scale higher traffic without needing powerful hosting resources. My Time-To-First-Byte decreased (improved) 70%, coming from shy of 500ms down to 100–160ms range 🤯 More importantly, the real user experience on the page became even more mind-blowing because things became super snappy. Click click click, bam bam bam, nothing was in a visible loading state anymore. Even if metrics didn’t move, I am super happy with this aspect of the change. 🤯🤯🤩 I got my Cloudflare Web Analytics email and noticed almost all Web Vitals moved positively at least 30% improvement. I wasn’t expecting other Web vitals like CLS and LCP to be directly impacted (or impacted as much as they did). But it makes sense. When the assets load much faster like this, the “wait time” (or blocking time) goes down, therefore, the layout shift or the largest paint goes down. SEO Impact It’s a well-known fact that Google takes your “core web vitals” into account when determining your ranking in the search results. This change has more impact on crawlers than you think. Because most of the time, crawlers’ requests are the ones that hit “cache cold” pages. It means Google (or other search engines) is reading your site holistically way more than your real users. Imagine every single article you wrote. There is no user who reads every single one of them — Google does 🙂 (and does it regularly). When a crawler tries to visit a page that nobody read in a long time, its request will have cache-miss more likely than cache-hit, so it will “wait” longer for your web server to render the page. When you put yourself in the crawler’s shoes, imagine you try to read 10,000 articles/pages on a site over a day or two period (maybe it takes longer, who knows…). Now, consider the percentage of those pages that will have to be rendered, or served from the CDN cache. The more pages Google sees “slow”, it will think your whole site is slow. This is where the real value of super-long TTLs comes in. Especially if you combine that with serve-stale-while-revalidate (SSWR) which most CDNs automatically do (if not, I’m sure there is a setting you can enable these together). SSWR, with a super-long TTL (like 7 days, or more), basically creates an infinite loop of “always cached” scenarios. And with that, your crawler traffic gets served from the cache (at cost/risk of “stale content” which is OK in the vast majority of use cases), and directly increases your site’s overall speed score and, therefore your SEO scores. Content Freshness There is one caveat though, which is content freshness. When you bump the Edge TTL up to multi-day TTLs like I did, you need to make sure your CMS/site is nicely integrated with your CDN’s cache clear systems, in the case you make updates. Two scenarios: You update existing content (like fixing a typo or changing the cover image of a post); the change should be reflected on the content’s detail page right away. You update existing content (like fixing a typo or changing the cover image of a post); the change should be reflected on the content’s detail page right away. You publish new content, so the new content is supposed to show up in common places like your homepage. You publish new content, so the new content is supposed to show up in common places like your homepage. You can use your CDN’s cache clear UI or APIs to trigger “purge” on URLs you think it’s impacted (homepage, section pages, etc.), or highly visible pages like the homepage can be configured with a lower TTL in a separate cache rule set. I use WordPress for my content management system and Cloudflare has a WordPress plugin to listen to publish/update hooks to trigger these cache clear nicely. Another way to think about this is to find the balance. What is the “staleness” you can tolerate on a page? Let’s say another article detail page showing “recent articles” or “related articles” sections to NOT show your most recent article there. As long as that time length is not something you can’t afford, cache longer to achieve better site/page performance.

How to Tweak CDN/Edge TTL to Radically Improve Site Performance (and SEO): A Quick Guide

In recent weeks, webmasters have raised urgent concerns regarding Facebook's aggressive crawling practices, particularly highlighting the behavior of its user agent, “facebookexternalhit.” Many site owners report that these bots are creating significant strain on their web hosting servers, leading to alarming spikes in traffic that threaten site reliability. Several site owners have come forward with their experiences, describing the overwhelming impact of Facebook’s crawling activities on their websites. One webmaster recounted their situation, said, “Our website gets hammered every 45 to 60 minutes with spikes of approximately 400 requests per second from 20 to 30 different IP addresses within Facebook’s netblocks. Between these spikes, the traffic is manageable, but the sudden load is risky.” This sentiment resonates with many, as webmasters are advocating for a more balanced distribution of requests from Facebook’s bots, akin to the behavior exhibited by Googlebot and other reputable search engine crawlers. The consequences of these excessive requests extend beyond mere inconvenience; they disrupt the user experience and lead to costly resource consumption for site owners. Smaller websites, in particular, have found themselves severely impacted. In response to the relentless onslaught, some webmasters have taken proactive measures by implementing stricter rules in their robots.txt files to shield their servers from the overwhelming traffic. However, because Facebook’s bot functions as a scraper rather than a traditional crawler, it disregards these instructions, further complicating the situation. This growing issue has sparked widespread discussions within the web development community, with experts urging Facebook to reconsider its crawling strategies. The collective voice of these webmasters underscores a critical need for a more sustainable approach to web scraping and crawling practices. In a bid to manage the excessive requests, many webmasters are turning to tools like Cloudflare, which provides robust features for managing traffic and implementing rate limiting. By configuring a rate-limiting WAF rule, webmasters can effectively throttle the number of requests originating from Facebook’s bots, alleviating server strain during peak traffic periods. One webmaster expressed their perspective on the necessity of a balanced approach, stating, “I don’t want to block the bot entirely, but the current pattern is unsustainable. Using Cloudflare’s rate limiting has allowed us to protect our site while still enabling Facebook to access our content for link previews.” Concerns regarding Facebook’s crawling practices have been echoed on various platforms. In a post on Cloudflare, one user articulated their frustrations: “I am writing to express my concern about the excessive crawling activity of Facebook’s crawler. “This excessive crawling is causing significant performance issues and potential downtime for our website.” They went on to detail, “Our web server logs indicate that Facebook’s crawler (facebookexternalhit/1.1 – 2a03:2880:22ff:7::face is making multiple requests to our WordPress website every second, even during off-peak hours,” “During peak hours, the crawler’s activity spikes to tens of thousands of requests per minute. This excessive crawling is overwhelming our servers and causing them to slow down or even crash.” While webmasters recognize the necessity of Facebook’s crawler for indexing purposes and making their content accessible to users, they firmly believe that the current level of crawling is excessive and unreasonable. As a result, many remain vigilant, closely monitoring their server performance and adjusting settings to mitigate the challenges posed by Facebook’s bots. The unfolding situation highlights a critical juncture for the web development community, with potential implications for how major tech companies manage web scraping and crawling practices in the future. As webmasters advocate for a more equitable solution, the outcome of this discussion could set important precedents in the industry, influencing the relationship between webmasters and tech giants moving forward. In recent weeks, webmasters have raised urgent concerns regarding Facebook's aggressive crawling practices , particularly highlighting the behavior of its user agent, “facebookexternalhit.” Many site owners report that these bots are creating significant strain on their web hosting servers, leading to alarming spikes in traffic that threaten site reliability. Facebook's aggressive crawling practices Several site owners have come forward with their experiences, describing the overwhelming impact of Facebook’s crawling activities on their websites. Several site owners have come forward with their experiences, describing the overwhelming impact of Facebook’s crawling activities on their websites. One webmaster recounted their situation, said, “Our website gets hammered every 45 to 60 minutes with spikes of approximately 400 requests per second from 20 to 30 different IP addresses within Facebook’s netblocks. Between these spikes, the traffic is manageable, but the sudden load is risky.” This sentiment resonates with many, as webmasters are advocating for a more balanced distribution of requests from Facebook’s bots , akin to the behavior exhibited by Googlebot and other reputable search engine crawlers. Facebook’s bots The consequences of these excessive requests extend beyond mere inconvenience; they disrupt the user experience and lead to costly resource consumption for site owners. Smaller websites, in particular, have found themselves severely impacted. In response to the relentless onslaught, some webmasters have taken proactive measures by implementing stricter rules in their robots.txt files to shield their servers from the overwhelming traffic. However, because Facebook’s bot functions as a scraper rather than a traditional crawler, it disregards these instructions, further complicating the situation. This growing issue has sparked widespread discussions within the web development community, with experts urging Facebook to reconsider its crawling strategies. The collective voice of these webmasters underscores a critical need for a more sustainable approach to web scraping and crawling practices. In a bid to manage the excessive requests, many webmasters are turning to tools like Cloudflare, which provides robust features for managing traffic and implementing rate limiting. By configuring a rate-limiting WAF rule, webmasters can effectively throttle the number of requests originating from Facebook’s bots, alleviating server strain during peak traffic periods. One webmaster expressed their perspective on the necessity of a balanced approach, stating, “I don’t want to block the bot entirely, but the current pattern is unsustainable. Using Cloudflare’s rate limiting has allowed us to protect our site while still enabling Facebook to access our content for link previews.” Concerns regarding Facebook’s crawling practices have been echoed on various platforms. In a post on Cloudflare, one user articulated their frustrations: “I am writing to express my concern about the excessive crawling activity of Facebook’s crawler. “This excessive crawling is causing significant performance issues and potential downtime for our website.” They went on to detail, “Our web server logs indicate that Facebook’s crawler (facebookexternalhit/1.1 – 2a03:2880:22ff:7::face is making multiple requests to our WordPress website every second, even during off-peak hours,” “During peak hours, the crawler’s activity spikes to tens of thousands of requests per minute. This excessive crawling is overwhelming our servers and causing them to slow down or even crash.” While webmasters recognize the necessity of Facebook’s crawler for indexing purposes and making their content accessible to users, they firmly believe that the current level of crawling is excessive and unreasonable. As a result, many remain vigilant, closely monitoring their server performance and adjusting settings to mitigate the challenges posed by Facebook’s bots. The unfolding situation highlights a critical juncture for the web development community, with potential implications for how major tech companies manage web scraping and crawling practices in the future. As webmasters advocate for a more equitable solution, the outcome of this discussion could set important precedents in the industry, influencing the relationship between webmasters and tech giants moving forward.

Stories, News & Mentions about CLOUDFLARE

Cloudflare’s Stories on HackerNoon

Cloudflare's News & Mentions