Cloudflare will block AI web crawlers by default, and introduces new Pay Per Crawl scheme that means AI companies will have to fork out for the privilege to scrape

2025-07-02 05:16:00
www.pcgamer.com

As time goes on, just saying ‘no’ to AI feels more and more futile. However, Cloudflare is announcing a few more tools for your anti-AI arsenal that put some of the power back in your hands. Cloudflare offers a range of network services from domain registration and content delivery to DDOS mitigation. Adding to that quiver, Cloudflare is launching the sharp and pointy Pay Per Crawl scheme, which aims to hit AI companies scraping online content where it hurts—namely, their deep pockets.

Last year, Cloudflare began allowing customers to simply block AI bots from their websites with one click. This AI Audit tool also allows users to pick and choose which bots they want to block, as well as offering insights into which LLM or gen-AI is regurgitating their content. Now, Cloudflare has announced that AI crawlers will be blocked by default, describing it as “the first step toward a more sustainable future for both content creators and AI innovators.”

Users who want to have their content scraped for free can still turn this default setting off, but Cloudflare’s second announcement seemingly asks the question, ‘Why would you want to?’

Pay Per Crawl, a scheme announced last year that allows users to charge AI companies wanting to scrape their content, is now in beta. To date, Cloudflare has not revealed the names of any Pay Per Crawl participants aside from ProRata, the startup behind AI search engine Gist.AI.

Founder and CEO Bill Gross offered comment as part of Cloudflare’s announcement regarding AI crawlers, saying, “Our whole mission is to protect and elevate human creativity in the AI era. That’s why we’re proud to be one of the first AI companies to participate in Cloudflare’s initiative to create a new permission-based model for the Internet. We believe that creators and publishers deserve to be fairly compensated for the value they bring, and we’re thrilled to support Cloudflare in helping make that vision a reality.”

It doesn’t seem unlikely that bigger AI companies would go for a scheme like this. After all, OpenAI has already struck content partnerships with a number of media companies, including PC Gamer’s own parent company, Future. Pay Per Crawl feels like an attempt to level the playing field, giving smaller content creators a leg to stand on, who would otherwise simply not have an entire team to dedicate to working out the terms of a content partnership.

But let’s reverse for a second, especially because, as Wired notes, not all web crawlers are bad. For instance, the Internet Archive’s preservation work in particular relies heavily on them. Last year, the Pew Research Center found that a quarter of all webpages that existed between 2013 and 2023 are now no longer accessible. Web crawlers are one way to preserve pages like that before they disappear in a puff of broken links and long-forgotten hosting services.

AI crawlers, however, are a bit of a menace as they scrape without permission from the owner of the content, and can end up overly taxing servers by behaving not unlike a DDOS attack. As previously mentioned, Cloudflare is no stranger to DDOS attacks, telling our Nick last month that their network blocks about ‘225,000,000,000 attacks per day.’ The company also shared that its network recently mitigated the largest attack ever recorded, battering a single IP address with 37.5 TB of data over the course of 45 seconds.

As such, Cloudflare’s AI blocking tools leverage a mix of machine learning and behavioural analysis to differentiate between these two types of web crawlers. Cloudflare has also found some success fingerprinting naughty bots by punting AI web crawlers down a rabbit warren of AI-generated pages. Now that’s what I call fighting fire with fire…or a flinging slop fight. At any rate, would it be naive to feel just a little optimistic about Cloudflare’s push for a ‘Permission-Based Model for the Internet’? Time will tell.

Best gaming PC 2025

Source Link

Take your gaming to the next level! The Redragon S101 RGB Backlit Gaming Keyboard is an Amazon’s Choice product that delivers incredible value. This all-in-one PC Gamer Value Kit includes a Programmable Backlit Gaming Mouse, perfect for competitive gaming or casual use.

With 46,015 ratings, an average of 4.6 out of 5 stars, and over 4K+ bought in the past month, this kit is trusted by gamers everywhere! Available now for just $39.99 on Amazon. Plus, act fast and snag an exclusive 15% off coupon – but hurry, this offer won’t last long!

Score this incredible deal now!

Start your free Amazon Prime trial
today and unlock unlimited streaming and more!

Help Power Techcratic’s Future – Scan To Support

If Techcratic’s content and insights have helped you, consider giving back by supporting the platform with crypto. Every contribution makes a difference, whether it’s for high-quality content, server maintenance, or future updates. Techcratic is constantly evolving, and your support helps drive that progress.

As a solo operator who wears all the hats, creating content, managing the tech, and running the site, your support allows me to stay focused on delivering valuable resources. Your support keeps everything running smoothly and enables me to continue creating the content you love. I’m deeply grateful for your support, it truly means the world to me! Thank you!

BITCOIN

Bitcoin Logo