AI web crawlers are overwhelming websites and disrupting the economic foundation of online publishing, according to Cloudflare CEO Matthew Prince. The cybersecurity company executive has launched a “pay-per-crawl” initiative to force AI companies to compensate content creators whose sites they harvest for training data.
Prince argues that AI chatbots have fundamentally broken the traditional web economy. Search engines like Google previously sent traffic back to websites after indexing their content, supporting billions of dollars in advertising revenue. AI chatbots instead provide direct answers without referring users to original sources, eliminating the traffic that funds most online content.
The scale of AI crawling has become destructive for many websites. Cloudflare reports that 30 percent of global web traffic now comes from bots, with AI bots representing the fastest-growing segment. According to web hosting company InMotion, AI crawlers are more aggressive than traditional search crawlers and often ignore guidelines designed to prevent server overload.
Fastly, another web infrastructure company, warns that AI crawlers cause “performance degradation, service disruption, and increased operational costs” by generating traffic spikes up to twenty times normal levels within minutes. Steven J. Vaughan-Nichols from The Register reports that his small website gets knocked offline by AI bot attacks despite having protection against traditional cyber attacks.
Major AI companies are driving this traffic surge. Meta accounts for 52 percent of AI searchbot traffic, followed by Google at 23 percent and OpenAI at 20 percent, according to industry data. Even Wikipedia reported in April that AI crawlers had increased their bandwidth costs by 50 percent.
Cloudflare’s blocking strategy
Prince believes Cloudflare’s position gives it unique leverage in this dispute. The company provides security and traffic management services for 20 percent of websites globally. By blocking AI crawlers on behalf of customers, Cloudflare can force AI companies to negotiate payment agreements.
The company has already identified problematic actors. Prince specifically called out Perplexity for ignoring blocking measures and allegedly copying content summaries while falsely attributing them to major publications like People magazine and The New York Times.
Prince sees Google as the key to solving the broader problem. He argues that other AI companies won’t pay for content access until Google, with its dominant search position, starts paying first. Google’s transition from search to AI-powered answers makes this change inevitable, he contends.
Traditional blocking methods have proven ineffective against AI crawlers. Many ignore robots.txt files, the standard way websites communicate crawling restrictions. Publishers report that “do not crawl” requests are increasingly ignored, with blocking failure rates rising from 3.3 percent at the end of 2024 to 13 percent by March 2025.
Three possible outcomes
Prince outlined three scenarios for the web’s future. The “nihilistic outcome” would see journalists and content creators lose their livelihoods as AI companies harvest their work without compensation. The “Black Mirror version” would result in five major AI companies vertically employing all content creators, creating knowledge silos that reverse the internet’s democratizing effect.
His preferred third option resembles the entertainment industry model, where AI companies would compete for exclusive content access similar to Netflix bidding for shows. Content creators would receive payments based on their material’s value to AI training, potentially expanding beyond traditional advertising and subscription revenue.
Some AI companies have begun voluntary licensing deals. OpenAI has signed agreements with several publishers, and Prince reports these negotiations have become easier since Cloudflare implemented its blocking measures in July.
The broader implications extend beyond individual websites. Prince argues that the current trajectory toward fragmented, siloed information systems could increase societal polarization. He sees the pay-per-crawl system as necessary to preserve the open web that has facilitated knowledge sharing for decades.
Sources: Crazy Stupid Tech, The Register