Skip to content
T
Tools.Town
Free Online Tools for Everyone
robots.txt

robots.txt Generator

100% Free

Generate a valid robots.txt file with per-user-agent allow/disallow rules, crawl delay, sitemap URL, and host directive. Copy-paste ready, with warnings for common mistakes.

Multi User-Agent
Instant Copy
Warns Mistakes
SEO Essential

User-Agent Rules

Rule 1
Allow Paths
Disallow Paths

Sitemap URLs

Yandex uses the Host directive; Google ignores it.

Generated robots.txt

4
Lines
1
Rule
User-agent: *
Disallow: /admin/
Disallow: /private/

Place this file at the root of your domain as https://yourdomain.com/robots.txt.

Embed This Tool

Easy to Embed

Add robots.txt Generator to your website or blog in seconds.

  • Responsive design
  • Lightweight & fast
  • No backend required
  • Always up-to-date 
<iframe
  src="https://tools.town/embed/robots-txt-generator/"
  width="100%"
  height="600"
  style="border:none; border-radius:12px;"
  loading="lazy"
  title="robots.txt Generator">
</iframe>

Share This Tool

Share Instantly

Share robots.txt Generator with anyone — no login required.

  • Shareable link
  • No login required
  • Works on any device
  • No account needed

Share via

Advertisement

How to Use

  1. 1 Add your user-agent rules — start with the wildcard (*) rule for all bots
  2. 2 Set allow paths (directories bots may access)
  3. 3 Set disallow paths (directories to block from crawling)
  4. 4 Optionally add sitemap URL(s) and crawl delay per bot
  5. 5 Click Generate and copy the result to your server root as robots.txt

Features

  • Generate rules for multiple user agents in one file
  • Pre-populated popular bot names including GPTBot and anthropic-ai
  • Add sitemap URL(s) and canonical host directive
  • Crawl-delay support per user-agent
  • Warnings: paths not starting with /, blocking all crawlers
  • Correct RFC 7230 formatting — no hidden characters or encoding issues

Why it Matters

A missing robots.txt is fine — bots crawl freely. But an incorrect one can accidentally block Google from your entire site. This generator builds the correct syntax, warns about dangerous rules like Disallow: / for all agents, and includes your sitemap location to speed up Google's discovery.

★★★★★

Use Cases

SEO Professionals

Precise crawl rules directing bots to important pages and away from duplicates

Web Developers

Generate correct robots.txt syntax without memorising the format

Block AI Crawlers

Opt out of GPTBot, anthropic-ai, CCBot while allowing search engines

Pre-Launch Setup

Configure crawl rules before going live to prevent private pages being indexed

What this tool does

Configure allow and disallow rules per user-agent, add sitemap URLs, and generate a correctly formatted robots.txt. Copy the output to your domain root (/robots.txt).

Common patterns

Block private directories from all bots:

User-agent: *
Disallow: /admin/
Disallow: /private/

Block AI training crawlers:

User-agent: GPTBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

Verify your robots.txt after uploading using Google Search Console’s robots.txt tester.

Frequently Asked Questions

What is a robots.txt file?
A robots.txt file lives at your domain root (e.g. https://example.com/robots.txt) and tells web crawlers which pages to visit or skip. It is voluntary — well-behaved bots follow it, malicious scrapers may not. It is not a security measure.
Does Disallow stop pages from being indexed?
Not directly. Disallow prevents crawling. If other sites link to a blocked URL, Google may still index a stub entry for it. To prevent indexing completely, add a noindex meta tag or X-Robots-Tag header on the page itself.
How do I block AI training crawlers?
Add separate User-agent blocks for GPTBot (OpenAI), anthropic-ai (Anthropic), and CCBot (Common Crawl) with Disallow: /. This is honoured by compliant AI companies per their published policies.
Does Google support Crawl-delay?
No. Google ignores Crawl-delay in robots.txt. Use Google Search Console's crawl rate settings instead. Bingbot and Yandex do respect Crawl-delay.
Do I need robots.txt if I want everything crawled?
No. Absence means bots crawl freely. But it is good practice to have one declaring your sitemap: that alone helps Google find and prioritise your pages faster.
📖

Want to understand the math?

Browse all guides →
robots.txt Explained
robots.txt

robots.txt Explained: Complete Guide for Crawl Control and SEO

Everything you need to know about robots.txt — what it controls, Disallow vs noindex, crawl budget, blocking AI crawlers, testing in Google Search Console, and common mistakes.

25 Jun 2026

5 min read

sitemap.xml vs robots.txt
sitemap

sitemap.xml vs robots.txt: What Each File Does for SEO

A clear explanation of the difference between sitemap.xml and robots.txt — what each file controls, how they interact, the common confusion about sitemaps and disallowed pages, and when you need each one.

25 Jun 2026

9 min read

Related Tools You Might Like

sitemap.xml Generator

Generate a valid sitemap.xml from a list of URLs — with lastmod, priority, and changefreq support. Paste URLs, set defaults, download or copy the XML.

Canonical Tag Generator

Generate the correct rel=canonical link tag for any URL — force HTTPS, strip tracking parameters, and normalise the trailing slash.

Meta Description Tester

Test a meta description's length, pixel width, and how it truncates in Google results on desktop and mobile — with a live SERP snippet preview, instantly and in-browser.

Meta Tag Generator

Generate a complete, copy-paste block of SEO and social meta tags — title, description, canonical, Open Graph, and Twitter cards — with a live Google preview and length checks, all in your browser.

Base64 Encoder

Encode and decode text and files to Base64.

Browse more free tools
More Developer Tools All categories