What Is Crawling?

How Crawling Works

Crawling is the entry point to Google Search. In Google’s own words, the first stage is where “Google downloads text, images, and videos from pages it found on the internet with automated programs called crawlers.” That crawler is Googlebot, also called a robot, bot, or spider, and it works through a constantly updated list of URLs discovered from previous crawls, submitted sitemaps, and links found on pages it has already fetched.

Crawling is not exhaustive. Google states that “Googlebot uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.” That algorithmic budgeting is why discovery and prioritization matter: Googlebot will not necessarily fetch every URL on a large site, and it revisits pages at a cadence it decides based on signals like popularity and how often a page changes. On big or fast-moving sites this becomes an explicit constraint called crawl budget.

A step people routinely forget is that crawling includes rendering. Google renders the page and “runs any JavaScript it finds using a recent version of Chrome,” much like a browser. This matters because a page’s raw HTML and its rendered output can differ significantly. If your primary content, links, or canonical tags are injected by client-side JavaScript, Googlebot only sees them after rendering succeeds — and rendering can be queued or resource-limited. Server-rendered HTML removes that uncertainty.

Crawling also isn’t a single event. Google recrawls pages on a cadence it sets from signals like popularity and how often a page changes, so a URL that was fetched last week may be refetched today or not for weeks. This is why a change you publish can take time to be reflected in results: Google has to recrawl the page before it can reindex the update. Frequently updated, well-linked pages get revisited far more often than static, deeply buried ones.

Once a page is crawled and rendered, it becomes eligible for the next stage. Crawling alone does nothing for visibility; it simply puts the page in front of Google’s indexing systems.

What Controls Whether a Page Gets Crawled

Several technical levers decide whether, and how often, Googlebot fetches a URL:

• Discoverability — Internal links and sitemaps are how Google finds URLs. An orphan page with no inbound links may never be crawled.
• robots.txt — A Disallow rule stops Googlebot from fetching a path entirely, including resources the renderer might need.
• Server health — Fast, stable responses raise Google’s crawl capacity limit; timeouts and 5xx errors lower it.
• Crawl demand — Popular and frequently updated URLs get crawled more often; stale, low-value URLs get crawled rarely.

You can influence discovery directly. Submitting an accurate XML sitemap tells Google which URLs exist and roughly when they changed, and the URL Inspection tool in Search Console lets you request a crawl of a specific page. Neither forces Google to crawl — the algorithmic budget still applies — but both raise a URL’s chances of being fetched sooner, especially for new pages that have few internal links pointing at them yet.

Example of Crawling

Google’s How Search Works documentation gives the canonical worked description of crawling, and the specifics are more constraining than most people assume. Google names the mechanism precisely: pages are downloaded “with automated programs called crawlers,” the crawler is Googlebot, and the decision of what to fetch is explicitly algorithmic — Googlebot “uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.”

The rendering detail is the part that trips up modern sites. Google documents that during crawling it “renders the page and runs any JavaScript it finds using a recent version of Chrome, similar to how your browser renders pages you visit.” A concrete consequence: if a product page ships an almost-empty HTML shell and fills in its title, description, and body through client-side JavaScript, then everything Google indexes for that page depends on the render step completing correctly. Block the script in robots.txt, or have it fail, and Googlebot crawls a blank page — technically fetched, effectively empty.

Google also documents the ceiling on crawling in its large-site crawl budget guide, where it explains that crawl capacity is “the maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches,” and that this limit rises when your server responds quickly and falls when it slows down or returns server errors. The lesson from Google’s own framing is direct: crawling is a negotiated, rate-limited process shaped by how discoverable your URLs are and how healthy your server is, not a guarantee that every page will be seen.

Crawling vs Indexing

The two are often conflated because they happen in sequence and both are handled by Google behind the scenes. But they fail for different reasons and are diagnosed differently: a crawling problem means Google never fetched the page, while an indexing problem means Google fetched it and chose not to store it. Confusing the two sends you fixing sitemaps when the real issue is a duplicate canonical, or vice versa.

Frequently Asked Questions

The Bottom Line

Crawling is Google finding and fetching your pages — the moment your URL enters the machine at all. It is necessary for everything that follows and sufficient for none of it. Make your pages easy to discover and cheap to fetch, then worry about whether they get indexed and ranked.