Cloudflare experiment will block AI bot scrapers unless they pay a fee

Cloudflare has rolled out a couple of new measures meant to keep AI bot crawlers at bay. To start with, every new domain customer that signs up with the company to manage their website traffic will now be asked if they want to allow AI crawlers or to block them altogether. The company released a free tool in 2024 to block AI bots, but with this change, users can block them by default without having to tinker with their settings. Several big publishers, including Condé Nast, TIME and The Associated Press have already signed up to block crawlers. In addition, Cloudflare has launched a private beta experiment called “pay per crawl,” which would only allow crawlers to access a website’s content if they pay for it.

Matthew Prince, Cloudflare’s CEO, recently went on record to say that publishers are facing an existential threat, because people aren’t clicking on chatbots’ source links. If users don’t visit those sources, the websites don’t get the ad revenue they need to be able to keep running. “Original content is what makes the Internet one of the greatest inventions in the last century, and it’s essential that creators continue making it,” Prince said in a statement released with the company’s latest updates. “AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone.”

Cloudflare believes publishers should be able to charge AI bots for access if they want to, and pay per crawl is its first experiment for that particular purpose. “Each time an AI crawler requests content, they either present payment intent via request headers for successful access (HTTP response code 200), or receive a 402 Payment Required response with pricing,” Cloudflare explained. The company records those transactions and provides the underlying technical infrastructure. Publishers will be able to allow certain crawlers to access their content for free if they want to, and they can define a flat, per-request price across its websites for other crawlers.

The company says pay per crawl is still in its very early stages, and it expects the tool to evolve in the future. It also says that it supports the development of other marketplaces and ways to charge AI crawlers for content. A marketplace could, for instance, allow dynamic pricing that enable publishers to charge different rates for different types of content.

Explore new destinations with ease using the Garmin Drive 52 GPS Navigator! With over 17,988 ratings and a solid 4.4/5-star rating, this GPS system has been a top choice for travelers. Over 500+ units were bought in the past month, all for only $144.99.

This 5″ GPS navigator comes with essential driver alerts, real-time travel data, and external memory storage. The simple on-screen menus and bright, easy-to-see maps make it easy to navigate wherever you are. Plus, it’s road trip-ready with The HISTORY Channel database, featuring notable historic sites and much more to enhance your journey.

Don’t miss out—get your hands on the Garmin Drive 52 today for a smoother ride ahead! Buy Now for $144.99 on Amazon!

Help Power Techcratic’s Future – Scan To Support

If Techcratic’s content and insights have helped you, consider giving back by supporting the platform with crypto. Every contribution makes a difference, whether it’s for high-quality content, server maintenance, or future updates. Techcratic is constantly evolving, and your support helps drive that progress.

As a solo operator who wears all the hats, creating content, managing the tech, and running the site, your support allows me to stay focused on delivering valuable resources. Your support keeps everything running smoothly and enables me to continue creating the content you love. I’m deeply grateful for your support, it truly means the world to me! Thank you!