Robots.txt

Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.