robots.txt Explained: Complete Guide for Crawl Control and SEO

What is robots.txt?

The robots.txt file is a plain-text document placed at the root of your domain — for example, https://example.com/robots.txt. When a search engine crawler arrives at your site for the first time, it fetches this file before visiting any other page. The file tells it which parts of the site to crawl and which to leave alone.

The protocol is called the Robots Exclusion Standard, and it has been around since 1994. It is simple by design: a series of User-agent blocks, each containing Allow and Disallow directives for specific URL paths. You can use the robots.txt Generator to build the file without memorising the syntax, then copy the output to your server root.

One critical caveat before anything else: robots.txt is a voluntary protocol. Well-behaved crawlers — Google, Bing, DuckDuckGo, and reputable AI companies — follow it. Malicious scrapers and spambots typically do not. If you need security, use your server’s authentication, firewalls, or access controls. robots.txt is a signal, not a lock.

Where to place robots.txt

The file must be at the exact root of the domain or subdomain you want to control. These are valid locations:

• https://example.com/robots.txt
• https://www.example.com/robots.txt (separate from the apex)
• https://blog.example.com/robots.txt (controls the subdomain only)

A robots.txt at /blog/robots.txt will be ignored — the file only works at the domain root. If you are on a shared hosting platform, place it in the public_html or www folder. On a CDN or static host, upload it to the root of your origin.

The directives: User-agent, Allow, Disallow, Crawl-delay, Sitemap, Host

User-agent

Every rule block starts with a User-agent line. The value is the name of the bot you want to address. Use * to target all bots. Named values — Googlebot, Bingbot, GPTBot — apply only to that crawler. More specific rules take precedence over the wildcard:

User-agent: *
Disallow: /private/

User-agent: Googlebot
Disallow: /nogoogle/

In this example, Googlebot follows the second block, not the first.

Disallow

Disallow tells a crawler not to fetch URLs matching the given path prefix. A path of /admin/ blocks everything under /admin/. A path of / blocks the entire site. An empty Disallow: line means nothing is blocked — it is equivalent to allowing everything for that agent.

User-agent: *
Disallow: /admin/
Disallow: /checkout/
Disallow: /account/

Allow

Allow is a Google-specific directive (and supported by most modern crawlers) that creates an exception to a broader Disallow. If you block /images/ but want Googlebot to access /images/product/, you can write:

User-agent: Googlebot
Disallow: /images/
Allow: /images/product/

The more specific rule wins.

Crawl-delay

Crawl-delay: 5 asks a crawler to wait five seconds between fetching pages. This can reduce server load from aggressive bots. One major caveat: Google ignores Crawl-delay entirely. If you need to throttle Googlebot, use the crawl rate limiter in Google Search Console instead. Bingbot and Yandex do respect this directive.

Sitemap

The Sitemap directive at the bottom of the file tells crawlers where to find your XML sitemap. This is one of the most valuable things you can put in robots.txt because it helps search engines discover all your pages quickly:

Sitemap: https://example.com/sitemap.xml

You can list multiple sitemaps, one per line. This is separate from the user-agent blocks and applies to all crawlers.

Host

The Host directive is used by Yandex to indicate the preferred domain. Google ignores it. It looks like Host: example.com. The robots.txt Generator includes this as an optional field.

Disallow vs noindex: a critical distinction

This is one of the most common misunderstandings in SEO. Disallow prevents a crawler from visiting a URL. It says nothing about whether the URL can appear in search results.

If other websites link to a URL you have disallowed, Google may still index a stub entry showing the URL (without any content, since it has never crawled it). The stub can rank for the URL itself.

noindex is a meta tag or HTTP header on the page itself that tells search engines not to include the page in their index. But here is the catch: if you disallow a URL, bots cannot read the noindex tag on it. The two directives are in conflict. The correct approach for pages you want off the internet:

1. Allow crawling (remove from robots.txt).
2. Add <meta name="robots" content="noindex"> to the page’s <head>.
3. Let Google crawl and process the noindex tag.
4. The page will drop from the index within days to weeks.

Never disallow a page you also want noindexed — you will trap the content in limbo.

Crawl budget and why it matters for large sites

Crawl budget is the number of pages Googlebot will crawl on your site in a given period. For small sites of a few hundred pages, budget is almost never a concern. For large sites — e-commerce stores with hundreds of thousands of product and faceted-filter URLs, news publishers with years of archive content, forums with user-generated content — crawl budget determines how quickly new or updated content gets indexed.

There are two components: crawl rate limit (how fast Googlebot can crawl without overloading your server) and crawl demand (how often Google wants to come back based on freshness signals and PageRank). You improve budget by reducing wasted crawl on thin or duplicate pages.

robots.txt is one tool for protecting budget. Disallowing entire categories of low-value URLs — pagination beyond page 2, search result pages, filtered product listings, printer-friendly duplicates — prevents Googlebot from spending time on pages you do not need indexed. For a deeper dive, see our guide to crawl budget optimization.

Blocking AI training crawlers

Recent years have brought a wave of AI training crawlers that ingest web content to train large language models. The major ones respect robots.txt when you opt out:

• GPTBot (OpenAI) — documented at https://openai.com/gptbot-uis/
• anthropic-ai (Anthropic) — documented in Anthropic’s crawling policy
• CCBot (Common Crawl) — widely used as a data source by many AI labs

To block all three while keeping search engine crawlers:

User-agent: GPTBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/

The robots.txt Generator includes all three in its dropdown for quick selection. Add each as a separate rule block and set Disallow: / to opt out of training data entirely, or use a specific path if you want to allow crawling of some public sections.

Testing your robots.txt in Google Search Console

After deploying your robots.txt, test it in Google Search Console:

1. Open Search Console for your property.
2. Go to Settings → robots.txt.
3. The tester shows your current live file and lets you enter any URL to see which rule applies.
4. Check that important pages like your homepage, key product pages, and blog posts are not accidentally blocked.
5. Check that private sections like /admin/ and /api/ are correctly blocked.

Recheck after any major update to your site structure. A refactor that moves URLs can accidentally expose previously blocked paths or block newly created ones.

Top 5 robots.txt mistakes

1. Disallow: / for User-agent: *

This blocks every bot from every page. It is appropriate for staging servers. On a production site, it is catastrophic — Google will deindex your entire site within weeks. The robots.txt Generator warns you explicitly if you generate this rule.

2. Disallowing pages you also want noindexed

As explained above, bots cannot read noindex tags on pages they cannot crawl. Allow the crawl; use noindex to suppress indexing.

3. Paths without a leading slash

The robots.txt standard requires paths to start with /. A rule like Disallow: admin/ may be interpreted differently across crawlers. Always write Disallow: /admin/.

4. Blocking CSS and JavaScript files

If Googlebot cannot load your site’s CSS and JS, it cannot render the page and may misunderstand your content. Avoid disallowing /assets/, /static/, or /js/ unless you have a specific reason. Google’s Mobile-Friendly Test will warn you if rendering is blocked.

5. A single robots.txt trying to control subdomains

Each subdomain needs its own robots.txt. Your file at example.com/robots.txt does not apply to blog.example.com. If you have a staging subdomain at staging.example.com, add a separate robots.txt there to block crawlers.

A quick checklist

Before you push robots.txt to production:

• Check that Disallow: / does not appear under User-agent: * unless intentional.
• Confirm your sitemap URL is listed and accessible.
• Test each blocked path in Google Search Console’s robots.txt tester.
• Verify that pages you want indexed are not accidentally blocked.
• Make sure paths all start with /.

A correct robots.txt takes five minutes to write and can save hours of troubleshooting. Use the robots.txt Generator to build yours with warnings included, then verify with Google Search Console before deploying.