Internet firm, Cloudflare will begin automatically blocking AI crawlers from accessing website content without the site owner’s consent or compensation, a shift that could disrupt how AI companies gather training data.

Beginning Tuesday, all new domains registering with Cloudflare will be prompted to choose whether to allow AI bots, giving website owners more control over data scraping.

Cloudflare, a content delivery network, helps businesses speed up the delivery of online content and applications by caching data closer to users. It plays a key role in ensuring smooth and reliable access to web content worldwide.

According to a 2023 report by the company, about 16 per cent of global internet traffic flows directly through Cloudflare’s CDN.

“AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate,” said Matthew Prince, co-founder and CEO of Cloudflare, in a statement Tuesday.

“This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone,” he added.

AI crawlers are automated bots that scan and extract vast amounts of data from websites, databases, and other sources. This information is used to train large language models developed by companies like OpenAI and Google.

Cloudflare argues that while the internet once rewarded content creators by directing users to original websites, AI crawlers are now disrupting that model. By collecting text, articles, and images to generate responses, these bots often eliminate the need for users to visit the source directly.

As a result, the company says, publishers are losing valuable web traffic—and with it, critical revenue from online advertising.

This move builds on a tool Cloudflare introduced in September 2023, which allowed publishers to block AI crawlers with a single click. Now, the company is going further by making that block the default setting for all websites using its services.

OpenAI, backed by Microsoft, declined to participate in Cloudflare’s preview of the plan.

It argued that Cloudflare was inserting itself as an unnecessary middleman in the process.

The company emphasized its early adoption of robots.txt — a standard web protocol used to block automated scraping — and said its crawlers already respect publishers’ preferences.