Two files, two jobs
When people hear “AI file at the domain root,” they often mix robots.txt and llms.txt. Both can sit at https://example.com/…, both relate to automated agents, and both show up in the same SEO conversations — but they solve opposite classes of problem.
| robots.txt | llms.txt | |
|---|---|---|
| Primary job | Restrict crawling of paths | Curate high-value pages for LLMs |
| Syntax | User-agent + Allow/Disallow | Markdown H1, >, H2 + link lists |
| Can block bots? | Yes (voluntary protocol) | No |
| Audience | Googlebot, Bingbot, GPTBot, … | LLMs / agents looking for context |
| Exhaustive? | Rules for paths, not a full site map | Intentionally incomplete |
One-line summary:
- robots.txt — what crawlers should not (or may) fetch
- llms.txt — what models should prefer to read if they follow the convention
Build robots rules with the robots.txt Generator. Check Markdown structure with the llms.txt Validator.
What robots.txt is for
robots.txt implements the Robots Exclusion Protocol. It is a set of voluntary rules that well-behaved crawlers apply before deciding whether to request a URL. Typical uses:
- Keep
/admin/, staging, or infinite filter URLs out of crawl budget - Point crawlers at a sitemap with
Sitemap: - Opt out of specific AI user-agents (e.g. GPTBot) when the vendor says they respect robots.txt
It is not a security control, and Disallow alone does not equal noindex for Google. For the full story, read the robots.txt guide and how sitemaps and robots interact.
What llms.txt is for
llms.txt is a community Markdown format. It opens with the site name (H1), a short project summary (blockquote), optional prose, and then sections of annotated links. The goal is a compact, trustworthy reading list for agents that would otherwise scrape noisy marketing pages or miss your real docs.
It does not:
- Forbid crawlers
- Replace authentication
- Automatically improve Google rankings
- Guarantee OpenAI, Anthropic, Google, or anyone else will load the file
It does give a portable, reviewable answer to: “If you only have time for N pages about this product, which ones?”
The dangerous mental model
| Myth | Reality |
|---|---|
| “llms.txt blocks ChatGPT from training” | Training scrapers are controlled via crawler policies / robots — not llms.txt |
| “robots.txt is enough for AI readiness” | robots can block; it does not explain your product for agents |
| “Put Disallow rules inside llms.txt” | Invalid — wrong vocabulary for that file |
| “One file can replace both” | You still need both for most serious sites |
If your goal is privacy / opt-out, prioritize robots.txt (and vendor documentation for each bot). If your goal is accurate AI answers about your product, ship a clean llms.txt and good docs.
How they work with sitemap.xml
Think of a three-layer stack:
- robots.txt — guardrails on crawl
- sitemap.xml — breadth for search indexing
- llms.txt — depth/priority for language models
A page can appear in the sitemap, be crawl-allowed, and still be omitted from llms.txt because it is secondary (old blog posts, tag archives). Conversely, you should not promote in llms.txt a URL you Disallow in robots.txt if you expect classic crawlers and agents to behave consistently.
Generate a sitemap with the sitemap.xml Generator when you need exhaustive coverage.
Side-by-side examples
robots.txt (crawl control)
User-agent: *
Disallow: /admin/
Disallow: /cart/
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlllms.txt (AI reading list)
# Example App
> Example App is a project management tool for product teams.
## Docs
- [Quickstart](https://example.com/docs/start): First project in 5 minutes
- [API](https://example.com/docs/api): REST reference
## Optional
- [Blog](https://example.com/blog): Product updatesNotice the first file uses User-agent / Disallow. The second uses Markdown headings and links. Never mix directive names across files.
Practical decision flow
- Do you need to stop crawlers from some paths? → robots.txt
- Do you need search engines to discover many pages? → sitemap.xml
- Do you want agents to open the right docs first? → llms.txt
- Is the file Markdown index format? → validate with llms.txt Validator
Most product sites eventually want all three. None of them replaces clear product documentation, OpenGraph titles, or proper canonical tags — they are complementary root signals.
Bottom line
robots.txt is a gate for crawlers. llms.txt is a guidebook for language models. Confusing them wastes time and can produce false confidence (“we blocked AI with llms.txt”) or false fear (“we cannot allow search bots if we ship llms.txt”). Ship both with intention, keep absolute links living, and validate structure before you deploy.