Academic libraries around the world are fighting off relentless swarms of AI data-scraping bots that are slowing or crashing their websites, according to a report by CBC News. The bots, used by AI companies and third-party services to harvest content for training large language models, are knocking entire digital collections offline and forcing libraries to choose between open access and staying online at all.
The targets are not random. Libraries, museums, and other cultural heritage institutions hold large volumes of quality, vetted information that is generally not hidden behind paywalls or logins — exactly the kind of clean, authoritative text that AI developers prize for training models like Claude, Gemini, and ChatGPT. For more context on this story, see our ongoing latest AI developments.
Collections Knocked Offline
The scale of the problem has grown to the point where whole archives become inaccessible. "Sometimes the bot traffic gets so bad that it causes the sites to go down completely for a little while," said Dean Seeman, director of collections and digital infrastructure at University of Victoria Libraries, in the CBC report. He described the outages as creating a "real problem" for public access — when a library's site collapses under bot traffic, students, researchers, and ordinary citizens are locked out together with the machines.
At San Jose State University, the experience is one of constant pressure. "They're just out there on the internet, just bombarding us with trying to grab up as much material as possible," said Jill Strykowski, the university's cataloguing and government documents lead. Her institution digitizes government documents and other public materials, making it a prime feeding ground for scrapers.
The report describes one school that was overwhelmed by more than 11,000 searches on Finnish music in a single day — traffic no human audience could plausibly generate, and the kind of narrowly focused, high-volume crawling typical of automated harvesting operations.
A Problem That Keeps Growing
Researchers tracking the phenomenon found that reports of this kind of aggressive scraping started appearing in late 2024 across North America, Europe, and Oceania, and experts told CBC News the problem has continued to get worse since. As AI companies race to secure high-quality training data — and as the open web's publishers increasingly block crawlers — the pools of unguarded, high-quality text keep shrinking, and the pressure on institutions that refuse to lock down their collections keeps rising.
That pressure has a cost. Strykowski said the bots are "causing an incredible amount of work and headache for every institution that does digitization work and every institution that is trying to make information openly available." Libraries must now run bot-filtering infrastructure, chase abusive crawlers, and absorb outages — spending scarce budget on defense rather than digitization and access.
An Extractive Relationship, Critics Say
Beyond the technical burden, the episode has revived a familiar criticism of how AI companies treat public-interest institutions. Peter Lewis, Canada Research Chair in Trustworthy Artificial Intelligence at Ontario Tech University, told CBC the pattern amounts to large private companies making use of common-good institutions "in a way that's very extractive" — assuming public material is free for the taking, then using it to build highly commercial models that are sold back to the same public.
The dynamic echoes earlier conflicts between AI developers and news publishers, authors, and artists, but with a twist: libraries rarely have legal departments or paywalls to fall back on. Their mandate is to make information openly available, which makes blocking traffic — even hostile traffic — a direct contradiction of their mission.
Good Bots, Bad Bots, and What Comes Next
The CBC report notes that not all automated traffic is hostile, and libraries do distinguish between well-behaved crawlers that respect rate limits and robots.txt exclusions and the far less polite harvesters hammering their archives. The difficulty is scale: even a share of abusive traffic small enough to slip past individual rate rules becomes crushing when thousands of clients coordinate or rotate identities.
Institutions are responding with a mix of tactics — rate limiting, bot fingerprinting, blocking known scraper networks, and in some cases taking collections offline entirely during attacks. None of these are free, and all of them degrade the open access that libraries exist to provide.
The irony is hard to miss: the same institutions that spent decades digitizing humanity's recorded knowledge so anyone could read it for free are now being pushed to close their digital doors because AI systems want to read everything at once. How libraries, funders, and AI companies resolve that tension — whether through licensing deals, compensated access, or stricter technical enforcement — will shape what the open web of training-data-quality information looks like for years to come.
For now, the people caught in the middle are the readers. As Seeman's experience at University of Victoria shows, every outage caused by a bot swarm takes public access down with it.
---
Stay Ahead of AIGet the latest AI news, analysis, and breakthroughs — all in one place.
Read more AI news →
