What Is llms.txt?

How llms.txt Works

llms.txt is deliberately simple. You place a Markdown file at your domain root that opens with an H1 site name, a short blockquote summary, and then a set of link lists grouped under H2 headings — “Docs,” “API reference,” “Guides” — each link followed by a one-line description of what the page covers. A model or an AI crawler that supports the convention can read this file first and use it as a table of contents, jumping straight to the canonical pages instead of guessing from a sitemap or crawling the whole site.

An optional companion file, llms-full.txt, inlines the full text of those key pages into one document, so a model can ingest the entire relevant corpus in a single fetch. The format is designed to fit inside a context window cleanly — no navigation chrome, no ads, no boilerplate, just the content and its structure.

llms.txt vs robots.txt — the distinction that trips people up

The two files share a location and a naming pattern, which is exactly why they get confused. robots.txt is a directive: it tells a crawler which paths it is and is not allowed to fetch, and well-behaved crawlers obey it. llms.txt is a recommendation: it tells a model which content is worth reading, but grants no permissions and blocks nothing. You could publish an llms.txt pointing at pages your robots.txt disallows, and both would be technically valid and mutually contradictory. Access control stays with robots.txt; llms.txt only ever suggests.

Example of llms.txt

The best way to see what a good llms.txt looks like is to read the ones shipped by companies whose entire business depends on developers understanding their docs. Anthropic, Stripe, and Cloudflare all publish them, and their choices are instructive. Anthropic pairs a compact llms.txt (about 8,400 tokens — a curated index) with an llms-full.txt of roughly 481,000 tokens that inlines its entire API documentation for a model to ingest in one fetch. Cloudflare organizes its file by product so an assistant can pull only the context for the service in question; its full version runs into the millions of tokens.

Stripe’s is the most telling. Beyond listing its docs, Stripe added a section no one else had: explicit instructions for LLM agents on how to integrate Stripe correctly. That is the real intent of the format on display — not “rank me,” but “here is how to represent my product accurately.” The company treats the file as documentation aimed at a machine reader, and shapes it to prevent the model from getting the integration wrong.

Adoption at the publishing end has been fast: by BuiltWith’s tracking, over 844,000 websites had an llms.txt file as of October 2025. The consumption end is where the honesty is required — no major AI provider has confirmed that it reads the file as a retrieval or ranking signal, and reporting through 2025 noted that adoption among the AI platforms themselves has stalled. Publishing one is a bet on a convention the big models have not yet endorsed.

The lesson these examples teach is consistent. The three companies did not ship llms.txt to rank; they shipped it so that an AI assistant answering a developer’s question about their API pulls from the current, canonical source instead of a stale blog post. The payoff is answer accuracy, not visibility — a documentation-hygiene move, worth doing for that reason and not oversold as a growth lever.

llms.txt vs Robots.txt

Same address, opposite jobs — the full side-by-side is on the llms.txt vs robots.txt page. If you remember one thing: robots.txt decides access, llms.txt decides attention.

Frequently Asked Questions

The Bottom Line

llms.txt is a voluntary Markdown index that hands language models a shortlist of your best content and a sentence on why each page matters. It is not an access-control file and, on current evidence, not a ranking signal. Treat it as a way to make your documentation machine-legible and accurately quotable — a hygiene improvement, not a visibility hack.