An open-source program named Anubis is helping website operators protect their sites from being overwhelmed by AI data scrapers. According to a report by Emanuel Maiberg for 404 Media, the developer Xe Iaso created the tool in her free time after her own server was repeatedly crashed by a bot harvesting data for AI models.
Since its launch in January, Anubis has been downloaded nearly 200,000 times. It is used by notable organizations like the GNOME Foundation, FFmpeg, and UNESCO.
The software works as an invisible test for website visitors. Iaso describes it as an “uncaptcha”. It uses JavaScript to make the visitor’s browser perform a quick cryptographic calculation. This proves the visitor is a human using a standard browser, not an automated bot. While a single calculation is trivial for a user’s computer, performing this check across millions of websites becomes prohibitively expensive for large-scale scraping operations.
Iaso explained that traditional methods like the robots.txt file are now often ignored by AI companies. She considers Anubis a more practical solution for smaller organizations than resource-intensive methods like trying to poison datasets. The tool is designed to be free, lightweight, and easy to implement.
Iaso continues to develop Anubis, acknowledging the constant cat-and-mouse game with bot creators. She is currently seeking funding to work on the project full-time.
