llms-full.txt Explained: When and How to Ship the Companion File

Index vs full dump

The llms.txt convention intentionally leaves long documentation out of the root index. That file should be scannable: H1, summary, and annotated links. When an agent needs the actual prose of many pages at once — for example offline analysis, bulk embedding, or “read everything about our API” — the community pattern is a second file:

https://example.com/llms-full.txt

llms-full.txt is typically Markdown that concatenates (or carefully excerpts) the content of your priority pages. It is optional. Many marketing sites never ship it. Documentation-heavy products benefit the most.

The llms.txt Validator focuses on the index format. Use it for /llms.txt. Treat /llms-full.txt as a content packaging exercise — still Markdown, but not bound to the same H2 file-list grammar.

When you need it

Ship llms-full.txt when:

• Agents keep fetching dozens of linked pages and still miss details
• You want a single URL for “all official product knowledge we are willing to expose”
• Your team already exports docs as Markdown (Docusaurus, Mintlify, mdbook, etc.)
• Context windows are large enough that a 100–500 KB download is acceptable

Skip it when:

• You only have a handful of short pages
• Docs change hourly and you cannot keep a dump fresh
• Sensitive or authenticated content should never be bulk-exported

Recommended relationship between the files

In llms.txt, link the full dump explicitly so agents that understand both conventions can jump there:

## Docs

- [Getting Started](https://example.com/docs/start): First setup
- [Full documentation (llms-full.txt)](https://example.com/llms-full.txt): Concatenated Markdown of priority docs

Keep the dump outside the Optional section if you want it treated as primary material, or inside Optional if it is an advanced path.

Structure patterns for llms-full.txt

There is no single rigid schema as strict as llms.txt’s file lists, but useful patterns include:

1. Repeat identity — start with # Product and a short summary so the full file stands alone.
2. Page separators — insert clear H2 headings for each original page title.
3. Canonical URLs — under each section, state the source URL in a line like Source: https://….
4. Stable order — same order as the llms.txt link list so humans can compare.
5. No secrets — scrub API keys, customer names, internal runbooks.

Example skeleton:

# Northwind API — full documentation pack

> Generated markdown pack of public docs for agent consumption.

## Quickstart

Source: https://docs.northwind.example/quickstart

…full page body…

## Authentication

Source: https://docs.northwind.example/auth

…full page body…

Size and freshness

Validators commonly warn when llms.txt exceeds roughly 20–50 KB because that file should stay an index. For llms-full.txt, larger sizes are expected, but:

• Prefer under a few hundred KB when possible
• Split into topic packs (API full, guides full) if you exceed ~1 MB
• Rebuild from CI when docs merge to main so the dump does not rot

If your validator (or agent) times out, shrink or shard.

Generation approaches

Whatever you choose, do not automatically dump every blog post and category page. The value of both files collapses when they become another noisy sitemap.

Security and crawl notes

• Full dumps increase leakage risk — review for confidential material
• Ensure neither file is accidentally Disallowed if you want public agents to fetch them
• Still keep private admin UIs out of both robots allowlists and Markdown packs

How to verify

1. Validate curated index: llms.txt Validator
2. Fetch https://yoursite.com/llms-full.txt and confirm status 200
3. Spot-check a few sections for accuracy against live pages
4. Mention both files in internal launch checklists next to sitemap and robots

For authoring the index itself, see How to create llms.txt and the broader llms.txt guide.