Free robots.txt checker

Paste a robots.txt or enter a domain. Pick a crawler and a path, and see which rule allows or blocks it.

Pasted text is analyzed locally in your browser.

Googlebot can crawl /

No matching rule. Crawling is allowed by default.

Allowed
Crawler groups0
Allow and disallow rules0
Sitemaps0
Syntax issues0
Crawler groups
User agentsRulesCrawl delay
No crawler groups found.
Issues to review

Unknown directives, missing colons and rules without a crawler group appear below.

  • No syntax issues found.
We fetch /robots.txt from the site root, then load it into the checker so you can test paths.

Limited use per visitor per day.

Test the crawler and path that matter

Rules match from the start of the path. A * covers a variable section and a $ anchors the end.

01

Load the file

Paste your robots.txt, or enter a website and we fetch the live file for you.

02

Pick a crawler and path

Try Googlebot, Bingbot or any other user agent against the exact URL path you care about.

03

Fix what is flagged

Resolve syntax issues, then re-test until the important paths show as allowed.

One broad rule can block your best pages

Recheck the file when a staging site goes public, when routes change and when you add a crawler.

Is robots.txt a security tool?

No. Anyone can read the file and open a public URL. Protect sensitive areas with authentication and server access controls.

Can a blocked page still be indexed?

Yes. Search engines can discover the URL through links without crawling its content. Use the correct indexing controls for removal.

Should CSS and JavaScript be blocked?

Usually not. Search crawlers need page resources to render and understand the content properly.

Why add a sitemap?

The Sitemap directive gives compliant crawlers a direct path to your XML sitemap at discovery time.

Fix the file and check the site

See every technical issue in one audit

Rankauto crawls your site, ranks the problems and shows which pages each one affects.

Start 3-day trial →