Free robots.txt generator

Choose what crawlers may visit, add your sitemap URL and download a robots.txt file for your domain root.

Set crawler accessOne path per line
Blocks the path for User-agent: *.
Use a more specific allow rule inside a blocked section.
Adds GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Bytespider and CCBot rules.
Googlebot ignores Crawl-delay. Other crawlers may honor it.

Do not use robots.txt to protect private information.

Generated robots.txt

2 lines

Start open. Block only what needs blocking

Most public sites allow everything and exclude admin, cart and internal search paths.

Open

Public site

Leave both path boxes empty to allow everything, then add your sitemap.

Private

Block private paths

Disallow admin, cart or internal search paths to protect crawl budget.

Closed

Block the whole site

Disallow / for staging sites you do not want compliant crawlers to visit.

A small file can hide your whole site

One line, Disallow: /, tells every compliant crawler to stay out. Check for it first.

What does Disallow do?

It asks a crawler not to request URLs that begin with the specified path. An empty Disallow value blocks nothing.

What does Allow do?

It creates a more specific exception inside a broader blocked path when the crawler supports the directive.

Should I block AI crawlers?

That is a business choice. Blocking training crawlers may limit reuse for model training, but different bots have different purposes and compliance policies.

Does robots.txt remove a page from Google?

No. A blocked URL can still be indexed if other pages link to it. Use a noindex directive on a crawlable page when removal from search is the goal.

Test and improve the technical setup

Want the whole technical setup audited?

Rankauto crawls up to 10,000 pages a month, groups issues by severity and shows the pages that need attention.

Start 3-day trial →