Cloudflare introduces a marketplace for paid AI scraping

Cloudflare has launched a new marketplace called “Pay per Crawl”, allowing website owners to charge AI companies for scraping their content. According to a report by Maxwell Zeff in TechCrunch, the system, currently in a private beta, aims to give publishers more control over their data. Website owners can set a price for each “crawl”, allow free access, or block AI bots individually. Cloudflare also announced it will now block all AI crawlers by default for new websites, requiring owners to grant permission explicitly. This initiative addresses the widespread, uncompensated use of online content for training AI models. The company highlighted a significant imbalance, noting that OpenAI’s crawler scraped sites 1,700 times for every single user referral it provided.