AI crawlers overwhelm open source infrastructure
Free and open source software (FOSS) projects are facing severe infrastructure challenges due to aggressive crawling by AI companies. According to a report by Niccolò Venerandi, multiple FOSS projects have experienced outages and service disruptions from AI crawlers that ignore standard protocols like robots.txt. SourceHut founder Drew DeVault reported that LLM crawlers are accessing expensive …