Robots.txt Generator & Validator

What robots.txt does, and what it does not

Robots.txt controls crawling. It tells well-behaved crawlers which paths they should not request. That is all it does, and the gap between that and what people expect it to do causes most of the damage.

It does not keep a page out of search results. A URL that is blocked in robots.txt can still be indexed if other pages link to it, and it shows up as a bare URL with no description, which is worse than either indexing it properly or removing it. To keep a page out of the index you need a noindex tag on the page, which means the crawler has to be allowed to fetch the page to see it.

It is also not a security control. The file is public, at a predictable address, and listing your admin paths in it is the same as publishing a map of them.

What the validator checks

• Syntax on every line, with the line number, so you can find the problem in the file rather than guessing.
• Rules that appear before any User-agent line, which no crawler will apply.
• A blanket Disallow: / under User-agent: *, which is the line that removes a live site from search.
• Blocked CSS and JavaScript, which stops Google rendering the page the way a visitor sees it.
• Duplicate groups for the same crawler, where most crawlers obey only one of them.
• Directives that no longer work, like Noindex:, which Google stopped supporting in 2019.
• Whether the file is actually served as plain text, or whether the server is quietly returning an HTML 404 page with a 200 status.

Testing a specific path

The path tester applies the same longest-match rule real crawlers use: among the rules that match a path, the one with the longest pattern wins, and Allow beats Disallow on a tie. That is why Disallow: /wp-admin/ followed by Allow: /wp-admin/admin-ajax.php works, and it is worth checking rather than assuming, because the interaction between wildcards and anchors is where people get surprised.

Blocking AI crawlers

Robots.txt is currently the main lever for AI training crawlers. The preset in the generator blocks GPTBot, CCBot, Google-Extended and Bytespider while leaving search crawlers alone, which is the split most sites want.

Worth understanding before you use it: Google-Extended controls training data for Gemini, not whether your pages appear in AI Overviews, which is governed by ordinary Googlebot access. And blocking the assistant crawlers means those assistants cannot cite you either. For most sites that is a real cost, since being quoted by an assistant is becoming a traffic source in its own right. Compliance is also voluntary, so this keeps out the crawlers that respect the file and nothing else.

Where the file has to live

At the root of the host, always: https://example.com/robots.txt. Not in a subfolder, not on a different subdomain. Each subdomain needs its own, so shop.example.com and blog.example.com are separate files that do not inherit anything from the main site. Serve it as text/plain, and keep it under 500KB, because Google stops reading there.

Frequently asked questions