Robots.txt checker
Find syntax problems and test whether a crawler is allowed to access a specific path.
Googlebot can crawl /
No matching rule. Crawling is allowed by default.
| User agents | Rules | Crawl delay |
|---|---|---|
| No crawler groups found. | ||
Issues to review
Unknown directives, missing colons and rules without a crawler group appear below.
- No syntax issues found.
Test the exact crawler and path that matter.
Robots rules match from the start of a URL path. Wildcards can cover variable sections and a dollar sign can anchor the end.
Load the file
Paste your robots.txt, or enter a website and we fetch the live file for you.
Pick a crawler and path
Try Googlebot, Bingbot or any other user agent against the exact URL path you care about.
Fix what is flagged
Resolve syntax issues, then re-test until the important paths show as allowed.
One broad rule can block your best pages.
Review the file when a staging site becomes public, when routes change and when a new crawler needs separate rules.
Is robots.txt a security tool?
No. Anyone can read the file and open a public URL. Protect sensitive areas with authentication and server access controls.
Can a blocked page still be indexed?
Yes. Search engines can discover the URL through links without crawling its content. Use the correct indexing controls for removal.
Should CSS and JavaScript be blocked?
Usually not. Search crawlers need page resources to render and understand the content properly.
Why add a sitemap?
The Sitemap directive gives compliant crawlers a direct path to your XML sitemap at discovery time.
Fix the file and check the site
See every technical issue in one audit.
Rankauto crawls your site, prioritizes the problems and shows which pages each issue affects.
Get free trial