robots.txt Analyzer
Paste or load a robots.txt to lint it, see which paths it asks crawlers to avoid (and why they matter), and test URLs against it. Nothing is fetched.
Runs in your browser Your input never leaves your device.
Checks
What it asks crawlers to avoid
A Disallow line is a request, not protection. Every path listed here is visible to anyone who reads the file.
| Line | Rule | Applies to | Hint |
|---|
Test URLs
The file is read in your browser; nothing is fetched. Matching follows Google's rules: the most specific user-agent group wins, the longest matching rule wins, and Allow beats Disallow on a tie. Other crawlers can differ.
Report a problem with this tool
About this tool
robots.txt is a public file that asks crawlers which paths to skip, so it is both a configuration to get right and a map of what a site would rather not have crawled. This tool reads a robots.txt you paste or load, and nothing is fetched, so it works on drafts and saved copies. It lints the file: lines without a colon, misspelled directives such as Dissallow, rules that appear before any User-agent line, paths that do not start with a slash, a misplaced $ anchor, spaces in paths, duplicate or conflicting rules, empty or repeated User-agent groups, relative Sitemap URLs, Disallow: / for all crawlers (a classic leftover from a staging site), Noindex lines that Google stopped supporting in 2019, Crawl-delay (ignored by Google), Host and other non-standard directives, and files over Google's 500 KiB limit. It lists every rule with its line, the crawlers it applies to and a hint when the path looks like an admin area, login, API, debug page, backup or configuration file, because a Disallow line does not protect anything: it publishes the path to everyone, so sensitive areas need authentication. You can copy the disallowed paths as full URLs for a site you enter, and test URLs or paths for any crawler name using Google's rules (the most specific User-agent group wins, the longest matching rule wins, Allow beats Disallow on a tie), with the deciding rule shown. This is the same matching used by the live robots.txt tester. Group parsing was compared with Python's urllib.robotparser on random files. Other crawlers can interpret edge cases differently.
How to use it
- Paste your robots.txt or load the file.
- Read the checks and fix the high and medium items first.
- Review the Disallow paths with hints and decide which need real protection.
- Test URLs for a crawler such as Googlebot or Bingbot and copy the disallowed URLs if you need them.
Common problems
- It says the file looks like an HTML page
- The site redirects robots.txt to a page or error page. Fix the redirect so /robots.txt returns plain text.
- A crawler is blocked and I do not know why
- Test its URL in the test box with the crawler name. The deciding rule is shown, and remember that a more specific User-agent group replaces the * group.
- Noindex in robots.txt does not remove my pages
- Google no longer supports it. Use a noindex meta tag or an X-Robots-Tag header, and leave the page crawlable so the tag can be seen.