What Is Crawl Rate Limit?

How a Crawl Rate Limit Works

A crawler cannot fetch a site’s pages as fast as it likes without risking overwhelming the server, so it works within a ceiling. Google names this the crawl capacity limit and defines it as “the maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches.” Those two dials — how many connections at once, and how long between requests — together set how hard Googlebot leans on your infrastructure.

The defining feature of the modern limit is that it is automatic and adaptive. Google’s documentation explains: “If the site responds quickly for a while, the limit goes up, meaning more connections can be used to crawl. If the site slows down or responds with server errors, the limit goes down and Google crawls less.” In other words, your server’s health is the throttle. A fast, reliable server earns a higher crawl rate; a struggling one is crawled more gently, automatically.

Crawl capacity is only one half of a site’s overall crawl budget. The other half is crawl demand — how much Google actually wants to crawl a site, driven by its popularity and how frequently its content changes. Effective crawling is capped by whichever is lower: a site can have plenty of capacity but little demand, or high demand throttled by a slow server.

The End of the Manual Limiter

For over a decade, Search Console offered a manual Crawl Rate Limiter Tool that let owners cap Googlebot directly. Google retired it. As covered below, the tool was slow and largely misused, and Googlebot’s automatic tuning made it redundant. Today there is no dial in Search Console for raising or lowering the crawl rate — the only supported controls are emergency error responses and a report form for unusual crawl activity.

Example of a Crawl Rate Limit in Action

The clearest documented example is the retirement of that manual tool. In a November 2023 Search Central blog post, Google announced the deprecation of the Crawl Rate Limiter Tool, effective January 8, 2024. The tool had existed since 2008. Google’s stated reasoning is a textbook illustration of how crawl rate limits now work: Googlebot “reacts to how the site’s server responds to Googlebot’s HTTP requests — for example, if the server persistently returns HTTP 500 status codes, Googlebot will automatically slow down crawling,” and it slows down similarly when response times grow. By contrast, the old manual tool was sluggish: requests to limit crawling “typically took about a day to go into effect and remained in effect for 90 days.”

Two documented details make the example concrete. First, after removing the tool Google “set the minimum crawling speed to a lower rate, comparable to the old crawl rate limits,” so it effectively keeps honoring the low rates cautious owners had chosen — without a manual control. Second, Google preserved the real-time levers that actually matter in a crisis. Its guidance instructs owners who need an urgent slowdown to “return 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests,” because “Google’s crawling infrastructure reduces your site’s crawling rate when it encounters a significant number of URLs with 500, 503, or 429” responses. That reaction is fast and automatic — the modern replacement for the day-late manual dial.

The takeaway from the whole episode: crawl rate is something Google manages by reading your server, not something you set by hand. Give it a fast, healthy server and it crawls generously; serve it errors and it backs off within days.

Frequently Asked Questions

The Bottom Line

A crawl rate limit is the speed governor on a search crawler — how many connections it opens and how long it waits between them. Google renamed the concept crawl capacity limit and made it automatic, tuning it up when a server is fast and healthy and down when it is slow or erroring. The old manual control is gone; the modern levers are a fast server, fewer wasteful URLs, and, for true emergencies, error status codes that tell Google to back off.