Cloudflare expands AI bot controls and launches pay-per-use monetization

Cloudflare has unveiled a major expansion of its tools for managing AI bots and monetizing content, one year after the company first let website owners block AI crawlers with a single click. The update introduces a new bot classification system, tighter default settings, and a push toward paying content owners for how their work is actually used.

The company’s updated taxonomy sorts automated traffic into three categories: Search, Agent, and Training. Search covers crawlers that index a site to answer future queries. Agent describes bots acting in real time on a person’s behalf, such as a chatbot fetching a page during a conversation. Training refers to crawlers pulling content to build or fine-tune AI models. All Cloudflare customers, including those on the free tier, can now allow or block traffic by these categories.

Starting September 15, 2026, new domains joining Cloudflare will block Training and Agent bots by default on pages that carry ads, while Search crawlers stay allowed. Crawlers that combine multiple purposes, including Google’s, Apple’s, and Bing’s bots, will be blocked wherever a site owner blocks any one of their functions. Existing customers can opt out of the new defaults before the deadline.

The data behind the shift

Cloudflare backs the changes with figures from its own network, through which more than 20 percent of the web routes traffic. AI training now accounts for 52 percent of crawler requests, up from 22 percent in spring 2025, and non-human traffic makes up more than half of all internet traffic. According to a Pew Research Center study Cloudflare cites, users click a traditional search result only 8 percent of the time when Google shows an AI summary, roughly half the rate without one. Time spent on the open web has also dropped, with Cloudflare reporting just 15 minutes of open-web browsing for every hour people spend online.

Cloudflare also flags Google’s crawler as a persistent transparency problem, since it fuses search and AI functions into one bot and gives Google roughly twice the access of competing AI companies. More than 50 publisher licensing deals have been signed since 2023, the company notes, though it admits the resulting revenue still falls short of what publishers lost from declining referral traffic.

Getting paid for use, not just crawls

Beyond access controls, Cloudflare is retooling its Pay Per Crawl service into a “Pay Per Use” model, working with Ceramic.ai and You.com to test payments tied to how often content actually appears in AI answers rather than how often it gets crawled. “Cloudflare allows us to easily and programmatically scale our operations,” says Ceramic.ai founder and CEO Anna Patterson.

The company is also launching a Monetization Gateway, letting customers charge for any resource behind Cloudflare, including web pages, datasets, APIs, and AI tool calls, using the x402 payment protocol and stablecoin settlement. Payments as small as fractions of a cent can clear without a signup or existing account, positioning agents as autonomous buyers on the web. The gateway is open for a waitlist now, with broader capabilities planned later this year.

Sources

• Your site, your rules: new AI traffic options for all customers – The Cloudflare Blog
• Making AI search smarter – The Cloudflare Blog
• Content Independence Day, one year on: building the business model for the agentic internet – The Cloudflare Blog
• Announcing the Monetization Gateway: charge for any resource behind Cloudflare via x402 – The Cloudflare Blog