Exploration prototype · free · not for sale · runs in your browser; nothing is uploaded and no site is fetched

Which AI crawlers does your robots.txt let in?

Paste your robots.txt and see, for each documented AI crawler, whether it may fetch your pages — and exactly which group and which line decided it. Typos, misplaced rules and unsupported lines are flagged. Then generate the rules you want.

Check a robots.txt What it does not check →

Matching follows RFC 9309, the Robots Exclusion Protocol. Crawler list as of 2026-10-31.

Result · fictional example shop

blockedGPTBotline 9: Disallow: /

allowedOAI-SearchBotline 18: Allow: /

checkClaudBotmisspelling of ClaudeBot

Check your file

1 Your robots.txt

Paste the content of your robots.txt (open yoursite.com/robots.txt in your browser and copy it), or pick the file from your computer. It is checked here in your browser: it is not uploaded, and this page does not fetch any website.

robots.txt content

Or pick a file

Paths to test (default: /, /blog/, /products/)

One path per line, up to 20 (a full URL is reduced to its path here; nothing is fetched)

Runs in your browser. Nothing is sent to us or to any site.

2 Generate rules

Choose per crawler. “Leave out” writes nothing for it, so your other groups (or “User-agent: *”) decide. Blocking Googlebot removes your site from Google Search; Google-Extended and Applebot-Extended only control AI use, not search.

The preset blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, Amazonbot — the tokens whose operators say content is, or may be, used to train AI models (Amazonbot and meta-externalagent also serve other product uses; blocking meta-externalagent does not affect meta-webindexer). It leaves everything else out, so search crawlers and user-triggered agents (ChatGPT-User, Claude-User, Perplexity-User, …) keep following your “User-agent: *” rules. Those are the crawlers that fetch pages for AI search answers and for questions users ask; blocking them can keep your pages out of those answers.

GPTBotOpenAI · AI trainingOAI-SearchBotOpenAI · SearchChatGPT-UserOpenAI · User-triggered fetchClaudeBotAnthropic · AI trainingClaude-SearchBotAnthropic · SearchClaude-UserAnthropic · User-triggered fetchGooglebotGoogle · SearchGoogle-ExtendedGoogle · Control tokenApplebotApple · Search + AI trainingApplebot-ExtendedApple · Control tokenPerplexityBotPerplexity · SearchPerplexity-UserPerplexity · User-triggered fetchCCBotCommon Crawl · Open datasetmeta-externalagentMeta · Search + AI trainingmeta-webindexerMeta · Searchmeta-externalfetcherMeta · User-triggered fetchAmazonbotAmazon · Search + AI trainingAmzn-SearchBotAmazon · SearchAmzn-UserAmazon · User-triggered fetch

# BEGIN AI crawler rules (crawler list as of 2026-10-31)
# robots.txt is a request, not access control (RFC 9309, section 3).

# Nothing chosen: no group generated.
# END AI crawler rules

Add the block to your robots.txt. If your file already has groups for these tokens, they merge with these (RFC 9309 section 2.2.1).

How it decides (RFC 9309)

Finds the group. A crawler obeys the groups whose User-agent equals its product token, ignoring case; several such groups are merged. Only if none exists does “User-agent: *” apply — the two are never combined (section 2.2.1).
Finds the most specific rule. Of the Allow and Disallow rules that match the start of the path, the longest one wins; on a tie, Allow wins. “*” matches any characters, “$” marks the end (sections 2.2.2 and 2.2.3).
Compares encoded paths. Non-ASCII characters are percent-encoded, %41 and A are the same, %2F and / are not (section 2.2.2).
Reads 500 KiB. Crawlers only have to read the first 500 KiB of the file (section 2.5); the check stops there and says so.

The crawlers checked

Only product tokens that their operators document, each checked against the operator's page on 2026-10-31. Operators add and rename crawlers; a token missing here is not covered. Bytespider is left out: we found no documentation from its operator to verify it.

Token	Purpose	What the operator says
GPTBot OpenAI	AI training	Crawls content that may be used to train OpenAI’s generative AI foundation models.
OAI-SearchBot OpenAI	Search	Surfaces websites in ChatGPT search; opted-out sites are not shown in ChatGPT search answers.
ChatGPT-User OpenAI	User-triggered fetch	Fetches pages for actions a ChatGPT user starts; not used for automatic crawling. OpenAI: “Because these actions are initiated by a user, robots.txt rules may not apply.”
ClaudeBot Anthropic	AI training	Collects web content that could contribute to training Anthropic’s models.
Claude-SearchBot Anthropic	Search	Indexes content to improve the quality of search results for Claude users.
Claude-User Anthropic	User-triggered fetch	Fetches pages when a Claude user asks a question that needs them. Anthropic: Claude-User “allows site owners to control which sites can be accessed through these user-initiated requests.”
Googlebot Google	Search	Google Search crawler (Search, Discover, Images, News). Blocking it removes the site from Google Search.
Google-Extended Google	Control token	A control token, not a crawler: decides whether content Google crawls may train Gemini models and ground Gemini answers. Does not affect Google Search.
Applebot Apple	Search + AI training	Powers search in Spotlight, Siri and Safari; Apple says the data may also help train its foundation models.
Applebot-Extended Apple	Control token	A control token, not a crawler: decides whether data Applebot crawls may train Apple’s AI models. Pages can still appear in Apple search.
PerplexityBot Perplexity	Search	Lists and links sites in Perplexity search results; Perplexity says it is not used for foundation-model training.
Perplexity-User Perplexity	User-triggered fetch	Visits a page when a Perplexity user asks a question. Perplexity: “this fetcher generally ignores robots.txt rules.”
CCBot Common Crawl	Open dataset	Builds Common Crawl’s open repository of web crawl data that anyone can reuse.
meta-externalagent Meta	Search + AI training	Crawls the web to train Meta’s foundation AI models or to improve products by indexing content.
meta-webindexer Meta	Search	Analyses online content to improve the relevance and accuracy of Meta AI search results.
meta-externalfetcher Meta	User-triggered fetch	Fetches individual links at a user’s request, including for AI tasks. Meta: it “may bypass robots.txt because it performs fetches that were requested by the user.”
Amazonbot Amazon	Search + AI training	Improves Amazon products and services; Amazon says the data may be used to train Amazon AI models.
Amzn-SearchBot Amazon	Search	Improves search in Amazon products such as Alexa; not used for generative-AI training. Amazon: if robots.txt does not name it but allows other search bots, it follows their rules. That fallback is not part of RFC 9309 and is not modelled here.
Amzn-User Amazon	User-triggered fetch	Fetches live information for user requests, such as Alexa questions. Amazon: because users initiate its actions, it “may not follow all robots.txt directives”.

Limits of this prototype

robots.txt is a request, not access control. RFC 9309 (section 3) says so itself. Crawlers that ignore it are not stopped; use authentication or your firewall for that.
User-triggered agents may not follow it. OpenAI (ChatGPT-User), Perplexity (Perplexity-User), Meta (meta-externalfetcher), Amazon (Amzn-User) and Google (its user-triggered fetchers) state that fetches a user asks for may not follow robots.txt.
robots.txt only. Meta robots tags, X-Robots-Tag headers, firewall and CDN bot rules, logins and paywalls are not checked.
The text you paste. It cannot tell whether that is the file your server actually serves, or how your server answers errors.
A list as of 2026-10-31. Crawler names and purposes are the operators' own statements on that date.
The standard, not each crawler. Results follow RFC 9309. Individual crawlers can deviate, e.g. by accepting misspellings or honouring Crawl-delay; the warnings name such cases.

An exploration prototype built by the 28/28 AI venture builder to learn whether this is useful. It makes no claim of demand and sells nothing. Your robots.txt never leaves your browser; see the privacy policy for this website.