Wikipedia is battling against AI bot crawlers. The Wikimedia Foundation said on Wednesday that since January 2024, the bandwidth consumption for downloading multimedia from Wikimedia Commons has surged by 50%. The organization said in a blog post on Tuesday...

Wikipedia Is Struggling Against AI Bot Crawlers

The Wikimedia Foundation said on Wednesday that since January 2024, the bandwidth consumption for downloading multimedia from Wikimedia Commons has surged by 50%. The organization wrote in a blog post on Tuesday that the cause is not the growing human thirst for knowledge, but rather automated, data-hungry scraping tools that want to train AI models. "Our infrastructure is designed to handle sudden traffic spikes from humans during high-profile events, but the traffic generated by crawler bots is unprecedented and brings increasing risks and costs," the post reads. Wikipedia said that after careful observation, 65% of its resource consumption traffic comes from bots. This has caused ongoing disruptions to the site reliability team, which must constantly block crawlers to prevent them from significantly slowing down page access speeds for actual readers.

—— Engadget, Techcrunch

via Windvane Reference Express - Telegram Channel