Robots.txt Generator
Build a valid robots.txt file that tells Google, Bing and AI crawlers what they may crawl — with your sitemap linked.
ROBOTS.TXT
Private by design — everything is processed on your device. Nothing you type or upload is sent to our servers.
What it does and why it matters
Robots.txt is a small text file at the root of your website that tells search engines and other crawlers which parts of the site they may visit. A good robots.txt keeps bots out of admin, cart and search-result pages while making sure your important pages and sitemap are easy to find.
Today you can also decide whether AI crawlers — such as OpenAI's GPTBot, Anthropic's ClaudeBot, PerplexityBot and Google-Extended — may read your content. Allowing them can help your business appear in AI answers. This generator writes the rules for you, so you only need to upload one file.
How to use the Robots.txt Generator
- Choose default accessAllow or block all crawlers by default.
- Pick folders and botsBlock private paths and choose which AI crawlers are allowed.
- Add your sitemapPaste your sitemap URL so search engines find every page.
- Upload the fileDownload robots.txt and upload it to your website's root folder.
Where does robots.txt go?
At the root of your domain, so it opens at https://yourdomain.com/robots.txt.
Does robots.txt hide pages from Google?
No. It controls crawling, not indexing. To keep a page out of search results, use a noindex meta tag or password protection.
Should I allow AI crawlers?
If you want your business to be cited in ChatGPT, Claude, Perplexity and Google AI answers, allowing their crawlers helps. You can block them if you prefer not to share content.
Can a wrong robots.txt hurt SEO?
Yes. A single "Disallow: /" for all user agents blocks your whole site. Always test the file in Google Search Console after uploading.
Do I need a sitemap line?
It is recommended. A Sitemap line helps every search engine discover your sitemap without submitting it manually.