Public site
Leave both path boxes empty to allow everything, then add your sitemap.
Choose what crawlers may visit, add your sitemap URL and download a robots.txt file for your domain root.
2 lines
Most public sites allow everything and exclude admin, cart and internal search paths.
Leave both path boxes empty to allow everything, then add your sitemap.
Disallow admin, cart or internal search paths to protect crawl budget.
Disallow / for staging sites you do not want compliant crawlers to visit.
One line, Disallow: /, tells every compliant crawler to stay out. Check for it first.
It asks a crawler not to request URLs that begin with the specified path. An empty Disallow value blocks nothing.
It creates a more specific exception inside a broader blocked path when the crawler supports the directive.
That is a business choice. Blocking training crawlers may limit reuse for model training, but different bots have different purposes and compliance policies.
No. A blocked URL can still be indexed if other pages link to it. Use a noindex directive on a crawlable page when removal from search is the goal.
Rankauto crawls up to 10,000 pages a month, groups issues by severity and shows the pages that need attention.
Start 3-day trial →