What Is Crawl Error?

How Crawl Errors Work

A crawl error is any response — or non-response — that stops Googlebot from successfully fetching a URL during crawling. Google’s Crawl Stats report organizes these into two very different tiers, and the distinction is the single most important thing to understand about them.

The first tier is host status: problems that affect your site’s overall availability. Google’s Crawl Stats documentation tracks three host-level categories over the past 90 days and rates them green (no significant issues), yellow (an issue more than a week ago), or red (an issue within the last week that needs attention):

• robots.txt fetching — Google needs a successful response to your robots.txt request. It accepts an HTTP 200 with a file, or a 403/404/410 meaning the file simply doesn’t exist. But a 429 or 5xx response is treated as a failure, and if it persists too long, Google halts crawling of the site.
• DNS resolution — Failures here mean your DNS server “didn’t recognize your hostname or didn’t respond during crawling.” Google flags the host when the failure rate crosses roughly 5%.
• Server connectivity — Triggered when “your server was unresponsive or did not provide a full response for a URL during a crawl.”

The second tier is per-URL crawl responses: the HTTP outcome of fetching an individual page. Google’s Crawl Stats groups these into good responses (200 OK, 301/308 permanent redirects, 302/307 temporary redirects, 304 Not Modified) and problematic ones (404 Not Found, 401/407 unauthorized, 5xx server error, other 4xx, DNS errors, fetch errors, page timeouts, and redirect errors).

How Google Reacts to Each Error

Google’s documentation on HTTP status codes spells out the consequences, and they differ sharply by type:

• 4xx (client errors) — Google doesn’t index 4xx URLs, and it removes already-indexed URLs that start returning 4xx. A 404 on a genuinely deleted page is expected and handled gracefully; crawl frequency for that URL simply tapers off.
• 429 (Too Many Requests) — Read as an overload signal. Google slows its crawling rather than treating the page as broken.
• 5xx (server errors) — Prompt Googlebot to temporarily slow down. Already-indexed content survives at first but is eventually dropped if the errors persist.
• Soft 404 — A page that returns 200 but looks empty or shows an error message. Search Console flags it as a soft 404 because the success status is lying about the real state of the page.

The reason these distinctions matter operationally is that Google’s response to an error is about persistence, not a single occurrence. A transient 5xx during a deploy is shrugged off; the same 5xx sustained for days causes Google to slow crawling and then start dropping indexed URLs. The same logic governs robots.txt: a brief blip is tolerated, but prolonged unavailability halts crawling. So the metric to watch in Crawl Stats is not whether an error ever appeared, but whether it is recent and recurring — which is exactly what the green/yellow/red host status is designed to signal.

Example of Crawl Errors

Google’s Crawl Stats report documentation and its HTTP status codes guide together give the authoritative, sourced picture of how crawl errors behave. The robots.txt rule is the most consequential and the most specific: Google states it will accept an HTTP 200 with a robots.txt file, or a 403/404/410 indicating the file is absent, but that a 429 or 5xx response is a failure — and if robots.txt stays unavailable “too long,” Google stops crawling the site entirely. That is the mechanism behind a whole class of outages: a server misconfiguration that returns 503 for /robots.txt doesn’t break one page, it can freeze the crawl for the whole domain.

The DNS threshold is equally concrete. Google’s Crawl Stats documentation notes that DNS resolution problems raise a host-status warning once failures exceed roughly 5%. A worked scenario: a team migrates DNS providers and a fraction of Googlebot’s lookups start timing out. Cross that ~5% line and the host status flips to red, signaling that Google is now struggling to reach the site at all — a far bigger problem than any individual broken link, and one that no amount of on-page fixing will address.

On the per-URL side, Google’s HTTP status guide states the rule plainly: “Google doesn’t index URLs that return a 4xx status code, and URLs that are already indexed and return a 4xx status code are removed from the index,” while 5xx errors “prompt Google’s crawlers to temporarily slow down.” The practical reading is a triage order. A spike in 5xx across many URLs means Google is throttling your whole site and may start dropping indexed pages — urgent. A single expected 404 on a retired page is routine. The examples are trustworthy because each behavior is quoted from Google’s own crawling documentation rather than inferred from ranking observations.

Triaging Crawl Errors by Blast Radius

Because Google reacts so differently to each error, the right response is to sort by how much of the site is affected, not by how alarming the error name sounds:

1. Host-level first — Fix DNS failures, server unavailability, and robots.txt fetch errors immediately. These can suppress crawling across the entire domain.
2. Widespread 5xx next — A server-error spike throttles crawling and threatens indexed pages. Restore healthy responses before Google drops them.
3. Redirect and soft-404 issues — Redirect chains and loops waste crawl budget and confuse consolidation; soft 404s hide real content failures behind a 200.
4. Individual 404s last — Expected on any evolving site. Only worth fixing when an important, linked URL is 404ing by mistake, in which case restore it or redirect it to the right page.

Diagnose the pattern before you touch anything. A wall of errors in Crawl Stats that all resolve to one unreachable host is one fix, not a thousand.

Frequently Asked Questions

The Bottom Line

A crawl error is Googlebot hitting a wall. Some walls are harmless — an expected 404 on a deleted page — and some are catastrophic, like an unreachable robots.txt that can freeze crawling site-wide. Fix the ones with the widest blast radius first: DNS, server availability, and robots.txt before any individual broken URL.