Robots.txt Generator
Create a robots.txt file for your website.
Runs in your browser Your input never leaves your device.
Report a problem with this tool
About this tool
Build a robots.txt file with one rule group. Enter one or more crawler names separated by commas (use * for all), paths to disallow and allow separated by commas, an optional sitemap URL and an optional crawl delay. Every crawler you list gets its own User-agent line in the same group, so they all share the same rules. The text updates as you type and is generated in your browser. For different rules per crawler, generate each group separately and join them with a blank line. Upload the finished file to the root of your site, for example example.com/robots.txt.
How to use it
- Set the User-agent (the default * means all crawlers).
- List the paths to disallow and allow, separated by commas.
- Add your sitemap URL and, if wanted, a crawl delay.
- Copy the output and save it as robots.txt in your site root.
Example
User-agent *, Disallow /admin, /private, Sitemap https://example.com/sitemap.xml
User-agent: * Disallow: /admin Disallow: /private Sitemap: https://example.com/sitemap.xml
The directives explained
- User-agent
- Which crawler the group applies to. * means any crawler that has no group of its own. A crawler follows only the most specific group that names it, not the * group as well.
- Disallow
- A path prefix the crawler should not fetch. Disallow: /admin also blocks /admin-login and /admin/users. An empty Disallow allows everything.
- Allow
- Re-opens a path inside a blocked folder, for example Disallow: /private with Allow: /private/press. Google and Bing use the most specific (longest) matching rule.
- Sitemap
- The full URL of your XML sitemap. It applies to the whole file, not one group, and you can list more than one.
- Crawl-delay
- Seconds between requests. Bing and Yandex read it; Google ignores it.
Rules Google applies
The file must sit at the root of each host and protocol: https://example.com/robots.txt does not cover https://shop.example.com or http://example.com. Paths are case-sensitive, so /Admin and /admin are different. Google supports * as a wildcard and $ to mark the end of a URL, for example Disallow: /*.pdf$. Do not block the CSS and JavaScript files your pages need, because Google renders pages to understand them.
Blocking a page does not remove it from search results. A blocked URL can still be indexed from links, shown without a description. To keep a page out, allow crawling and add a noindex tag.
Common recipes
- Allow everything
- User-agent * with no paths. The output is User-agent: * and Disallow: with no value.
- Block a staging site
- User-agent * and Disallow /. Password protection is safer, because robots.txt is only a request.
- Block AI training crawlers
- User-agent GPTBot, CCBot, Google-Extended and Disallow /. Google-Extended controls use of your content for Google's AI models and does not affect Google Search.
Common problems
- The file blocks nothing
- With no disallow or allow paths the output is "Disallow:" with no value, which allows everything. Add the paths you want blocked.
- Pages still appear in search results
- robots.txt asks crawlers not to fetch pages, but it does not remove already indexed URLs or guarantee compliance. Use a noindex directive or authentication for that.
- Crawl-delay has no effect
- Not every crawler supports Crawl-delay; Google ignores it.
Frequently asked questions
Where do I put the robots.txt file?
In the top folder of your site, so it loads at https://yourdomain/robots.txt. A robots.txt in a subfolder is ignored. On WordPress, which serves a generated robots.txt, a real file at the root replaces it.
How do I check the file works?
Use the Robots.txt Tester on this site to see whether a URL is allowed for a given crawler, or the robots.txt report in Google Search Console to see the version Google fetched.