Free robots.txt checker
Paste a robots.txt or enter a domain. Pick a crawler and a path, and see which rule allows or blocks it.
Googlebot can crawl /
No matching rule. Crawling is allowed by default.
| User agents | Rules | Crawl delay |
|---|---|---|
| No crawler groups found. | ||
Unknown directives, missing colons and rules without a crawler group appear below.
- No syntax issues found.
Test the crawler and path that matter
Rules match from the start of the path. A * covers a variable section and a $ anchors the end.
Load the file
Paste your robots.txt, or enter a website and we fetch the live file for you.
Pick a crawler and path
Try Googlebot, Bingbot or any other user agent against the exact URL path you care about.
Fix what is flagged
Resolve syntax issues, then re-test until the important paths show as allowed.
One broad rule can block your best pages
Recheck the file when a staging site goes public, when routes change and when you add a crawler.
Is robots.txt a security tool?
No. Anyone can read the file and open a public URL. Protect sensitive areas with authentication and server access controls.
Can a blocked page still be indexed?
Yes. Search engines can discover the URL through links without crawling its content. Use the correct indexing controls for removal.
Should CSS and JavaScript be blocked?
Usually not. Search crawlers need page resources to render and understand the content properly.
Why add a sitemap?
The Sitemap directive gives compliant crawlers a direct path to your XML sitemap at discovery time.
Fix the file and check the site
See every technical issue in one audit
Rankauto crawls your site, ranks the problems and shows which pages each one affects.
Start 3-day trial →