Public site
Leave both path boxes empty to allow everything and just add your sitemap.
Robots.txt generator
Create crawler rules, add your sitemap and download a robots.txt file ready for your website root.
Set crawler accessOne path per line
Generated robots.txt
2 lines
Start open, then block only what needs blocking.
Public websites usually allow crawling by default and exclude admin areas, duplicate pages or internal search results.
Block private paths
Disallow admin, cart or internal search paths to protect crawl budget.
Block the whole site
Disallow / for staging sites you do not want compliant crawlers to visit.
A small file can affect your whole site.
Robots.txt controls crawling, not guaranteed indexing. One broad disallow rule can hide important pages from search crawlers.
What does Disallow do?
It asks a crawler not to request URLs that begin with the specified path. An empty Disallow value blocks nothing.
What does Allow do?
It creates a more specific exception inside a broader blocked path when the crawler supports the directive.
Should I block AI crawlers?
That is a business choice. Blocking training crawlers may limit reuse for model training, but different bots have different purposes and compliance policies.
Does robots.txt remove a page from Google?
No. A blocked URL can still be indexed if other pages link to it. Use a noindex directive on a crawlable page when removal from search is the goal.
Test and improve the technical setup
Audit more than one technical file.
Rankauto crawls your site, groups issues by severity and shows the pages that need attention.
Get free trial