llms.txt Best Practices: Sections, Links, Size, and Common Mistakes

Quality beats length

A good llms.txt feels like a product manager wrote a one-page reading list for a new engineer — not like a crawler dump. These practices keep the file useful for agents and maintainable for humans. After edits, re-check with the llms.txt Validator.

H1 and summary

• H1 is the product or site name, not a SEO keyword paragraph.
• Blockquote summary should pass the “stranger skim” test in under ten seconds.
• Mention the problem category, not only brand adjectives.
• Prefer factual tone over campaign slogans.

Section architecture

Suggested default stack for SaaS / docs sites:

1. Docs — start here, concepts, tutorials
2. API or Reference — machines-oriented specs
3. Examples — sample apps, recipes
4. Policies — privacy, terms, AI use (when relevant)
5. Optional — blog, changelog, community, status

Guidelines:

• Reuse plain English names agents already know.
• Avoid duplicate sections that cross-link the same ten URLs.
• Put primary paths above Optional so partial-context agents see them first.
• Leave empty sections out entirely — an H2 with zero links is noise (and a validator warning).

Link items

Every file-list line should look like:

- [Human title](https://absolute.example/path): one-line purpose

Avoid nested bullets, HTML <a> tags, or mixing tables into list sections.

Size budget

Long canonical prose belongs in pages linked from the index or in a llms-full.txt pack.

Optional section etiquette

Use ## Optional only when you understand the semantic: “safe to skip under tight context.” Do not hide your pricing or authentication docs there. Do put:

• Historical changelogs
• Marketing blog hubs
• Community forums
• Conference talk dumps

Spelling matters: title the heading Optional, not “Optional resources” if you want tools and agents that special-case the exact name.

Anti-patterns

1. Sitemap dump — every product URL with no narrative.
2. H1 missing / buried — YAML frontmatter or paragraphs before # Title.
3. robots directives in Markdown — Disallow: belongs in robots.txt; generate rules with the robots.txt Generator.
4. Relative-only site — (/docs) looks fine on origin, fails for many consumers.
5. Stale launch copy — still links to /docs/v1 when you live on /docs/v3.
6. Internal-only URLs — staging hosts, localhost, VPN paths.
7. HTML blobs — scraped chrome, cookie banners, nav scripts as text.
8. Claiming legal force — llms.txt is not a contract against scrapers.

Process recommendations

• Treat the file as docs, store it in the docs repo, review it in PRs.
• Add a monthly or release-checklist item: “llms.txt links still 200?”
• After major IA changes, update the index before marketing announces.
• Keep crawl controls separate (robots) and search inventory separate (sitemap).
• For on-page SEO of human pages, still use meta tags and schema — llms.txt does not replace them.

Minimal quality scorecard

Before you merge:

• H1 first, meaningful product name
• Non-empty > summary
• Absolute links only (or intentional, documented relatives)
• Every H2 has ≥1 valid list link
• Optional used only for skippable content
• Size under ~20 KB
• Validator shows zero errors
• Live /llms.txt returns 200 and is not Disallowed

Run the llms.txt Validator until the scorecard is green, then deploy with confidence. Deeper overview: llms.txt complete guide.