robots.txt Generator

robots.txt is a text file placed at the root of a website that tells search engine crawlers which pages to crawl or ignore.

No. robots.txt only controls crawling — pages may still appear in search results if linked from other sites. Use a noindex meta tag for complete removal.

In robots.txt, name each company's published crawler — GPTBot, ClaudeBot, PerplexityBot and so on — and give it a Disallow. A newer approach, Content-Signal, lets you state permission by purpose on a single line, such as search=yes, ai-train=no. Remember that robots.txt is a request honored only by crawlers that choose to obey it, not a technical block, so anything that truly must not be taken should sit behind a login.

Yes — a single line reading Disallow: / removes the entire site from crawling. Shipping a staging server's robots.txt to production without clearing that line is a genuinely common accident. Open robots.txt in a browser right after every deployment and run your key pages through Google Search Console's robots.txt tester. Recovering from a de-indexing takes days to weeks, long after the file is fixed.