R1Robots line

robots.txt generator

Start from a preset or build groups by hand: which crawlers, which paths they may or may not fetch, and where your sitemaps are. The file updates as you type and is checked by the same parser as the tester.

Rules

Presets

Group 1

User agents

AI training crawlers to block

Each ticked token gets Disallow: / in one shared group. Search crawlers such as Googlebot and Bingbot are not affected.

GPTBot · OpenAICrawler that collects content OpenAI may use to train its models.
ClaudeBot · AnthropicCrawler that collects content that may be used to train Anthropic's models.
Google-Extended · GoogleA control token, not a crawler: Googlebot does the fetching, and this token says whether that content may be used for Gemini model training and grounding. Blocking it does not affect Google Search.
Applebot-Extended · AppleA control token, not a crawler: Applebot fetches pages, and this token says whether they may train Apple's generative models. Search features that use Applebot are unaffected.
CCBot · Common CrawlBuilds the open Common Crawl archive, which many AI models are trained on.
Meta-ExternalAgent · MetaMeta's crawler for uses such as AI model training.
Bytespider · ByteDanceByteDance's web crawler.
PerplexityBot · PerplexityIndexes pages for Perplexity's AI answer engine; Perplexity says it is not used to train foundation models. Untick it to stay in Perplexity's answers.

Sitemap URLs, one per line

robots.txt

—

User-agent: *
Disallow:

Upload the file to the root of your site, e.g. https://example.com/robots.txt.

How the file is put together

Each group starts with one or more User-agent lines and lists rules for those crawlers. A crawler obeys only the group naming its product token, falling back to *; groups are not added up. Within a group the longest matching path wins, and Allow wins a tie, so Disallow: /admin/ plus Allow: /admin/help/ opens one folder. * matches any characters and a trailing $ anchors the end, as in /*.pdf$. An empty Disallow: allows everything.

Google ignores Crawl-delay; some other crawlers honour it. Google-Extended and Applebot-Extended are control tokens rather than separate crawlers: they govern how content fetched by Googlebot or Applebot may be used for AI, so blocking them leaves search untouched. robots.txt is a request that well-behaved crawlers follow, not access control.

FAQ

Does Disallow keep a page out of Google?

No. It stops crawling, but a blocked URL can still be indexed from links, without its content. To keep a page out of results, allow crawling and add a noindex robots meta tag or header.

Where does robots.txt go?

At the root of each host, as /robots.txt. Subdomains and each protocol need their own file.

Will blocking GPTBot remove my site from ChatGPT search?

OpenAI documents separate tokens for search and for user-requested visits, so blocking GPTBot alone is about training. Check each company's documentation for its current tokens.