Artificial intelligence companies are deploying a new generation of bots that scrape web content in real time to provide users with instant summaries instead of traditional search results. Traffic from these retrieval bots surged 49 percent in the first quarter of 2025 compared to the previous quarter, according to data from TollBit, a startup that helps publishers monitor AI content usage.
The findings, reported by Nitasha Tiku in The Washington Post, reveal how companies like OpenAI and Anthropic are fundamentally changing how information flows across the internet. These bots retrieve and process far more content than humans would typically read to generate AI-powered answers for users switching from Google search to tools like ChatGPT.
TollBit analyzed data from 266 websites, including many news organizations. CEO Toshit Panigrahi warned that “this is coming for everyone,” not just publishers. The growth of retrieval bots outpaced traditional training bots by 2.5 times between late 2024 and early 2025.
The shift creates significant challenges for content creators. More than 26 million AI scrapes bypassed website blockers in March alone, according to TollBit’s findings. While some news organizations have secured licensing deals with AI companies, many publishers struggle to receive fair compensation for their content.
An OpenAI spokesperson acknowledged that referral traffic to publishers from ChatGPT searches may be lower in volume but reflects stronger user intent. The development signals a major restructuring of the internet as AI-powered answers gain popularity over traditional search results.