robots.txt Tester
Test whether a crawler such as Googlebot may fetch a given path under a site's robots.txt.
Looks up live data What you enter is sent to our server to fetch live results from public sources. Results are cached briefly. A Cloudflare Turnstile bot check runs when you press the button (it loads a script from challenges.cloudflare.com and may show a short challenge).
A Cloudflare Turnstile bot check runs when you press the button (it loads a script from challenges.cloudflare.com and may show a short challenge).
Report a problem with this tool
About this tool
Enter a website, a path and a crawler name. The tool fetches that site's robots.txt, picks the group that applies to the crawler (the most specific matching user-agent, or *), and applies Google's matching rules: wildcards (*) and end anchors ($) work, the longest matching rule wins, and Allow beats Disallow when lengths tie. You see whether the path is allowed, the exact rule that decided it, any crawl delay, the sitemaps listed and the groups found. A missing robots.txt (404) means everything is allowed, while a server error is treated by Google as everything blocked for now. robots.txt only asks well-behaved crawlers to stay away; it does not stop access or remove pages already indexed.
How to use it
- Enter the website, for example example.com.
- Enter the path to test, such as /private/page.html, or leave it blank to use the URL you entered.
- Choose the crawler, for example Googlebot or Bingbot.
- Press Test robots.txt and read the deciding rule.
Common problems
- A page is blocked but still appears in search
- Blocking crawling does not remove a URL from the index. Use a noindex tag on a crawlable page, or remove the URL in Search Console.
- The result differs from another tester
- Crawlers differ on edge cases. This tool follows Google's documented rules.
- The fetch fails
- The site may block our server, time out, or sit on a private address, which is refused.