Auror Office
Google Warns CDN Migrations Can Swamp Origin Servers, Decrees Caution on Bot Blocking
Google warns that moving a site to a CDN can briefly flood its origin server with crawler traffic, and that blanket bot protection may push pages out of Search.
Wednesday, 11 November 2026 · 2 min read
Behold a fresh decree from the Google Search Central Blog, issued on Wednesday, November 11, 2026: site owners who move behind a Content Delivery Network, or CDN, may find their own servers besieged by crawlers at the very moment they sought relief. A CDN spreads a site's files across fast edge servers placed closer to visitors, which shields the origin server, the backend machine that holds the original content. Google explains that the passage to such a network can be a strange one, and that careless defences may knock a site out of Search altogether. The cause lies in what Google calls the crawl capacity limit, also known as hostload, meaning the pace at which its crawlers judge a server can bear their visits. Slow responses make the crawlers ease off, while fast and stable ones, given enough crawl demand, gradually persuade them to quicken. When a site's DNS, the system that directs visitors to a server, points to a swift CDN edge network, Google says, it looks as though the server just received a massive hardware upgrade, and the crawlers ramp up their requests to match. Here the plot thickens. A brand-new CDN begins with an empty, or cold, cache, so the first request for any URL at an edge location is a cache miss, a request the edge cannot answer from its own stored copy and must fetch from the origin. Human visitors cluster around a small set of popular pages that warm up quickly, but crawlers set out to explore the full breadth of a site's addresses. Until the caches are warmed, Google notes, the origin server still does the heavy lifting. Nervous administrators may be tempted to switch on blanket bot protection in their CDN dashboards, yet Google cautions that such rules treat all automated traffic alike. When a CDN or a Web Application Firewall, a filter that screens incoming requests, challenges automated requests with a CAPTCHA, a test meant to tell humans from bots, Google's crawlers fail the test and are left stuck on an error page. If that persists, the crawlers back off and pages and shopping listings begin to drop out of Search. Google's remedy reads like an arcane manual for the cautious. It advises allowing verified bots in CDN and firewall settings, or targeting specific unwanted bots through robots.txt, a file that tells crawlers which pages to avoid, and pre-warming caches by fetching the busiest and most frequently crawled URLs through the CDN at a controlled pace before routing all traffic there. Should the origin still strain, servers may return a 429 (Too Many Requests) or 503 (Service Unavailable) status code for a short window of a few hours, or 1-2 days at most. The guidance accompanies Episode 5 of The Non-Canonical Cabinet, the Season 1 finale of Google's video series, in which a rubber chicken stands in for an edge server. Networking diagrams, Google concedes, are subjective.
Source: developers.google.com