AI web crawlers are overwhelming websites and disrupting the economic foundation of online publishing, according to Cloudflare CEO Matthew Prince. The cybersecurity company executive has launched a “pay-per-crawl” initiative to force AI companies to compensate content creators whose sites they harvest for training data. Prince argues that AI chatbots have fundamentally broken the traditional web …
The IT security company Cloudflare claims that the AI answer engine Perplexity is using undeclared web crawlers to access content on websites that explicitly forbid it. According to a blog post authored by several Cloudflare engineers, Perplexity ignores standard protocols designed to respect the preferences of website owners. Cloudflare conducted an experiment to verify the …
Cloudflare has launched a new marketplace called “Pay per Crawl”, allowing website owners to charge AI companies for scraping their content. According to a report by Maxwell Zeff in TechCrunch, the system, currently in a private beta, aims to give publishers more control over their data. Website owners can set a price for each “crawl”, …
Publishers face an existential crisis as artificial intelligence companies extract their content without providing fair compensation, Cloudflare CEO Matthew Prince warned at an event in Cannes. Christine Wang reported for Axios. Prince revealed striking statistics about the imbalance between content scraping and traffic referrals. Ten years ago, Google crawled two pages for every visitor it …
A Ukrainian-American company, Triplegangers, experienced severe website disruption when OpenAI’s web crawler attempted to scrape its entire database of 65,000 3D human model products. According to Julie Bort’s report in TechCrunch, the incident occurred when OpenAI’s bot sent tens of thousands of server requests using 600 different IP addresses. CEO Oleksandr Tomchuk described the bot’s …
Cloudflare is introducing new tools to protect websites from AI bots and control their scraping. The company is offering monitoring and selective blocking of AI data scraping bots to all customers, including its 33 million free users. CEO Matthew Prince says these measures are designed to give website owners more control over how and when …