Robots.txt Generator
Use this generator to build a robots.txt file without remembering the syntax. Choose a default rule, add paths to block, set rules for individual crawlers, block or allow common AI crawlers and list your sitemaps. The file updates as you go, ready to copy or download and upload to the root of your site.
Robots.txt Generator
Full URLs, one per line.
robots.txt controls crawling, not indexing. A blocked URL can still appear in search results if other pages link to it. To keep a page out of Google, allow crawling and add a noindex meta tag or header instead.
Upload the file to the root of your host, e.g. https://www.example.com/robots.txt. Each subdomain needs its own file.
Runs in your browser. Nothing you enter is uploaded or saved.
How to use this tool
- Choose the default. Allow all crawling for a normal site, or disallow all for a staging site that is also password-protected.
- Add paths and rules. List paths to block for every crawler, then add rules for specific user-agents if needed.
- Set AI crawlers and sitemaps. Block or allow AI crawlers individually and add the full URL of each XML sitemap.
- Download and upload. Download robots.txt, upload it to the root of your domain and test it in Google Search Console.
What robots.txt does (and does not do)
robots.txt tells well-behaved crawlers which URLs they may fetch. It does not stop a page being indexed: a blocked URL can still appear in results if other pages link to it. To keep a page out of Google, let it be crawled and use a noindex meta tag or X-Robots-Tag header.
How crawlers pick a group
A crawler follows the most specific User-agent group that matches it and ignores the rest. If you add a group for Googlebot, it no longer reads the * rules, so repeat any paths it should still avoid. When Allow and Disallow rules conflict, Google uses the most specific (longest) matching rule.
Blocking AI crawlers
GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and CCBot (Common Crawl) can be blocked by name. Google-Extended is a control token: blocking it limits use of your content for Gemini, without affecting Google Search. Crawlers that ignore robots.txt need server-level blocking instead.
Crawl-delay and sitemaps
Googlebot ignores Crawl-delay; some other crawlers respect it. Sitemap lines must use full URLs and can sit anywhere in the file. Listing sitemaps here helps search engines find them even without Search Console.
Frequently asked questions
Where does robots.txt go?
At the root of each host, e.g. https://www.example.com/robots.txt. A file in a subfolder is ignored, and each subdomain needs its own file.
Should I block CSS and JavaScript files?
No. Google needs them to render pages properly. Blocking them can make pages look broken to Google.
Will blocking a page remove it from Google?
Not reliably. Use noindex on a crawlable page, or the Removals tool in Search Console for urgent cases, then block crawling later if you need to.
Is robots.txt a security measure?
No. The file is public and only polite crawlers obey it. Protect private areas with authentication, and avoid listing secret paths in it.