llms.txt Explained: Complete Guide for AI Discovery and GEO

What is llms.txt?

An llms.txt file is a curated, Markdown-formatted index living at your site root (https://example.com/llms.txt). Its job is not to list every URL you own — that is the sitemap’s role — and not to block crawlers — that is robots.txt. It exists so large language models and AI agents can load a short, human-edited overview of what your product is and which pages matter first.

The idea was published as an open community proposal at llmstxt.org. It is deliberately simple: one H1, a short blockquote summary, optional prose, then H2 sections of annotated links. Because the format is Markdown, both people and models can read it without a special binary schema.

Use the free llms.txt Validator to check structure before you deploy — H1 placement, link list shape, relative URLs, and file size all matter in practice.

Why teams add one now

Documentation sites, API products, SaaS marketing sites, and open-source projects are increasingly consumed not only by humans but by coding agents, chatbot browsers, and RAG pipelines. Those consumers often fail when they:

• Land on marketing chrome without a single canonical “start here” path
• Follow every blog post and exhaust context on low-value pages
• Cannot distinguish primary documentation from changelog noise

llms.txt is a cheap signal: edit a short file once, keep absolute links to your best pages, and give agents an ordered reading list. That practice is sometimes lumped under GEO (generative engine optimization) — making your brand easy to cite and summarize correctly — even though it is not a formal SEO ranking algorithm.

The format, in order

A conforming file contains sections in this sequence:

1. Optional byte-order mark (BOM) — rarely needed; validators should tolerate UTF-8 BOM.
2. H1 — # Project or Site Name. This is the only section the core spec calls required. It should be the first non-empty line.
3. Blockquote summary — one or more > lines that summarize the product in a sentence or two. Not strictly required by the bare minimum reading of the spec, but every serious implementation treats it as strongly recommended. Our validator warns if it is missing.
4. Optional free Markdown body — paragraphs or lists without additional headings. Use this for how agents should interpret the list (e.g. “prefer docs over blog”).
5. Zero or more H2 sections — each a file list: list items of the form - [name](url) with an optional : notes suffix.

A special heading title matters: ## Optional. Links under that heading may be skipped when the agent needs a shorter context. Put secondary resources there (blog, changelog, community) and keep must-read docs in earlier named sections.

List item grammar

Inside file-list sections, prefer:

- [Authentication](https://docs.example.com/auth): API keys and OAuth

Not bare bullets without links, not nested lists, and not HTML anchor tags. Absolute https:// URLs are strongly recommended so the file still makes sense when cached or copied outside your origin.

How it fits with other root files

These files work together. You might Disallow admin areas in robots.txt, list product URLs in the sitemap, and list only five documentation entry points in llms.txt. Confusing them is the most common mistake — see llms.txt vs robots.txt.

Writing guidelines that work

• Curate ruthlessly. Ten excellent links beat fifty mediocre ones.
• Lead with the truth. The blockquote should say what the product is, not only slogan copy.
• Use section names agents understand: Docs, API, Examples, Policies, Optional.
• Describe every link. A few words after the colon reduce wrong-page retrieval.
• Keep it short. Aim under ~20 KB for the index; put the long form in llms-full.txt.
• Stay honest. Do not list pages that 404 or require login without saying so in the notes.

For a checklist from blank file to deploy, follow How to create an llms.txt. For quality patterns (sections, notes, pitfalls), see llms.txt best practices.

Validation and publishing

1. Draft Markdown in your editor.
2. Paste into the llms.txt Validator or use Fetch URL against a staging domain.
3. Fix errors (H1, bad list items, extra headings). Fix warnings (summary, relative URLs, size).
4. Publish to the site root as llms.txt with Content-Type: text/plain; charset=utf-8 when your host allows.
5. Confirm curl -I https://yoursite.com/llms.txt returns 200 and is not blocked by robots.txt.

Revisit the file when you reorganize docs. Unlike sitemaps, you should edit this by hand — automation that dumps every URL reintroduces the noise problem the format tries to solve.

Realistic expectations

llms.txt is not:

• A legal force field against training scrapers
• A guarantee that any particular vendor will read your file
• A replacement for clear HTML headings, sitemap, or structured data

It is a low-cost, portable convention that many documentation and product teams now ship because agents benefit from a short, stable reading list. Validate the structure, keep the links alive, and treat the file as product docs — not as a set-and-forget SEO plug-in.