What Is Robots.txt?

How Robots.txt Works

Before most well-behaved crawlers request pages from a site, they fetch one file: robots.txt. It is a plain-text file, written to the Robots Exclusion Protocol, that lists rules telling each crawler which URL paths it may or may not request. Google describes its primary job plainly: “A robots.txt file is used primarily to manage crawler traffic to your site.” It is a gate on fetching, applied before a single page is downloaded.

Location is strict. Google requires the file in “the top-level directory of a site,” so https://example.com/robots.txt is valid but https://example.com/folder/robots.txt is not. The rules “apply only to the host, protocol, and port number where the robots.txt file is hosted,” which means http:// and https:// versions, and every subdomain, each need their own file. The document must be UTF-8 encoded plain text, and Google enforces “a robots.txt file size limit of 500 kibibytes (KiB)” — anything after that byte count is silently ignored.

A minimal file looks like this:

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help
Sitemap: https://example.com/sitemap.xml

Google supports exactly four fields: user-agent (which crawler the block applies to), disallow (paths not to request), allow (paths that may be requested, used to carve exceptions out of a broader disallow), and sitemap (the full URL of a sitemap). The most important thing to internalise is what these rules do not do, which is the subject of the next section.

What Robots.txt Does Not Do

Robots.txt controls crawling, not indexing, and conflating the two is the source of nearly every robots.txt mistake. Google’s warning is unambiguous: robots.txt “is not a mechanism for keeping a web page out of Google.” It continues: “A page that’s disallowed in robots.txt can still be indexed if linked to from other sites,” and in that case “the URL address and, potentially, other publicly available information such as anchor text” can appear in search results.

So a Disallow says only “don’t fetch this.” It does not say “don’t list this,” it does not hide the content from users, and it does not offer any confidentiality — the robots.txt file itself is public, so disallowing /secret-launch/ announces exactly what you are hiding. For keeping a page out of results Google points to other tools: a noindex tag on a still-crawlable page, password protection, or removing the content. Critically, noindex and a robots.txt disallow work against each other — block the page and the crawler can never see the noindex.

Example of Robots.txt

A precise, verifiable example comes straight from Google’s specification of how it parses the file. Google states the enforced ceiling: “Google enforces a robots.txt file size limit of 500 kibibytes (KiB). Content which is after the maximum file size is ignored.” That single sentence has a practical consequence large sites hit in the wild.

Picture an e-commerce platform that, over years, appends a new Disallow line for every parameter, campaign, and legacy path it wants to keep crawlers away from. The file grows past 500 KiB. Every rule after that boundary — say, the disallow that was supposed to keep Googlebot out of an “infinite space” of filter URLs — is simply not read. Google crawls those URLs anyway, crawl budget drains into worthless variants, and the site owner, looking at a file that clearly contains the rule, cannot understand why it is being ignored. The fix is not a bigger file; it is a leaner one, with broad path patterns instead of thousands of individual lines.

The same spec fixes another common failure: scope. Because rules “apply only to the host, protocol, and port,” a robots.txt served only on https://www.example.com does nothing for https://blog.example.com or the bare http:// host. Teams that migrate to a new subdomain and forget to ship its robots.txt discover their staging blocks evaporated. The lesson is that robots.txt is exact and literal: right location, right host, under the size limit, and understood as a crawl gate rather than an index or privacy control.

Robots.txt vs llms.txt

The two share a naming pattern and a root-level home, which is why they get muddled, but they answer opposite questions. Robots.txt is a restriction mechanism that established crawlers are built to respect. The llms.txt proposal is a promotion mechanism — a voluntary hint for AI systems — with no guarantee any given model reads it. Do not treat llms.txt as a way to block AI crawlers; blocking is still robots.txt’s job.

Frequently Asked Questions

The Bottom Line

Robots.txt is a root-level instruction file that shapes crawler traffic through allow and disallow rules under the Robots Exclusion Protocol. It is a crawl-management tool, not a confidentiality or deindexing tool: a blocked URL can still be listed, and blocking a page you want removed prevents the crawler from ever seeing the noindex that would remove it. Keep it small, keep it at the root, and use it for the one job it does.