What Is Crawl Budget?

How Crawl Budget Works

Crawl budget answers a simple question: of all the URLs on your site, how many will Googlebot actually fetch in a given window, and which ones? Google’s large-site crawl budget guide defines it as “the set of URLs that Google can and wants to crawl,” and breaks it into two independent factors that both have to be satisfied.

The first is the crawl capacity limit — the can. Google describes it as “the maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches.” Crucially, this limit is dynamic. It rises when your site responds quickly and stays healthy, and it falls when responses slow down or the server starts returning errors. Google also caps it by its own available resources. In practice, a fast, stable server earns a higher ceiling; a struggling one gets throttled, because Google does not want its crawling to be what takes your site down.

The second factor is crawl demand — the wants. Even if Google could crawl more, it only will if it has reason to. Google names three drivers: perceived inventory (Google tries to crawl the URLs it knows about, and duplicates or unwanted pages “waste Google’s crawling time”), popularity (more popular URLs get crawled more often), and staleness (Google recrawls to detect changes). A site full of near-duplicate parameter URLs signals low-quality inventory and can suppress demand across the whole property.

Your effective crawl budget is the intersection of the two. Raising capacity (faster server) without demand (fresh, valuable, unique URLs) does little, and vice versa.

Google also notes a related trap: making URLs faster to serve can actually increase how much Google crawls, because a healthier server raises the capacity limit. That is usually good, but it means server performance and crawl volume move together — a slow site both gets crawled less and, when it recovers, may see a crawl surge. Crawl budget is therefore not a fixed allowance you’re granted once; it’s a moving equilibrium Google renegotiates continuously based on how your site behaves.

Who Actually Needs to Manage Crawl Budget

Google is unusually direct that most sites should not spend time on this. Its guidance targets three profiles:

• Very large sites — more than 1 million unique pages with content that changes moderately often (about weekly).
• Medium-to-large sites — more than 10,000 unique pages with very rapidly changing content (about daily).
• Any site with a large share of URLs classified as “Discovered — currently not indexed” in Search Console.

Google explicitly labels these as rough estimates, not exact cutoffs. The takeaway is that below this scale, crawl budget is rarely the constraint — the real bottleneck is usually indexing or content quality.

The third profile is worth dwelling on, because it’s the one that catches sites that aren’t obviously huge. A large share of URLs sitting in “Discovered — currently not indexed” is a symptom that Google has found more URLs than it is willing to crawl and process, which is a crawl-and-index prioritization problem even on a site well under a million pages. When you see that status at scale, the fix usually isn’t publishing more — it’s reducing the count of thin, duplicate, or auto-generated URLs competing for the same finite attention.

Example of Crawl Budget

Google’s large-site crawl budget guide is the authoritative source, and its own numbers make the concept concrete. Google states that crawl-budget management is worth the effort primarily for sites with “more than 1 million unique pages” changing roughly weekly, or “more than 10,000 unique pages” changing daily. Those figures are the closest thing to an official threshold, and they reframe the whole topic: a 2,000-page blog worrying about crawl budget is optimizing a problem Google says it doesn’t have.

The capacity mechanic is documented just as precisely. Google explains that the crawl capacity limit reflects “the maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches,” and that “if the site responds quickly for a while, the limit goes up… If the site slows down or responds with server errors, the limit goes down.” A worked implication: imagine a large retailer whose faceted navigation generates millions of filter combinations — color, size, price, sort order — each a crawlable URL. Googlebot spends its finite parallel connections fetching these near-identical pages instead of new product listings. Google’s guidance addresses exactly this: because duplicates and low-value URLs “waste Google’s crawling time,” blocking or consolidating them frees demand for the URLs that matter. Nothing about the site’s server changed; the budget was recovered by removing waste. The example holds because every claim in it is Google stating its own crawling economics.

How to Protect Crawl Budget

For sites that genuinely operate at scale, the levers follow directly from Google’s two factors:

• Raise capacity — Improve server response times and reliability so Google’s crawl capacity limit trends up rather than down.
• Cut wasted demand — Block or consolidate faceted-navigation URLs, strip tracking parameters, eliminate soft 404s, and fix redirect chains so each hop isn’t a separate fetch.
• Guide discovery — Keep XML sitemaps accurate and use internal linking to signal which URLs deserve frequent crawling.
• Measure it — Use the Crawl Stats report and server log analysis to see where Googlebot actually spends its requests, rather than guessing.

The through-line is that crawl budget is managed by subtraction more than addition. You rarely convince Google to crawl more; you stop it from crawling the wrong things.

Frequently Asked Questions

The Bottom Line

Crawl budget is Googlebot’s finite attention on your site: how much it can crawl, times how much it wants to. For most sites it’s a non-issue. For very large or fast-changing ones, it becomes a resource you protect — mainly by not squandering it on URLs that never deserved a crawl.