DPLA Coffee Chat: Keeping Online Collections Online

Libraries and other heritage web sites, especially our digital collections and catalogs full of high-value data, are contending with a surge of automated traffic. Aggressive botnets of AI scrapers and crawlers overwhelm servers, causing site performance issues and downtime affecting the legitimate users we serve. DPLA itself experienced an explosion of such traffic starting at the end of 2025, and has dealt with occasional instability. We have also heard from partners facing the same difficulty, and directly experienced mitigation effects—such as CAPTCHAs, rate limiting, and IP blocking—that have necessitated changes to our traditional harvesting methods. Library missions are all about making information widely accessible, but how do we detect and keep out abusive automated traffic without shutting out our community?

In this informal virtual coffee chat, DPLA will share insights from our own recent efforts keeping our catalog online under sustained bot pressure. There will also be an open floor for peers to trade questions, experiences, and approaches so we can learn from one another as a field facing this shared challenge.