AI crawler robots.txt checker
If an assistant's crawler is blocked, it cannot read your pages and is less likely to cite you. Enter your site, or paste your robots.txt, to check.
| Crawler | What it does | Verdict |
|---|---|---|
| GooglebotGoogle Search and AI Overviews | Search index, and the AI answers built on it | Allowedno rule matches |
| Google-ExtendedGemini training and groundingControl token: never crawls itself | Collects data to train AI models | Allowedno rule matches |
| Google-AgentGoogle agents acting for peopleMay not follow robots.txt | Fetches a page when someone asks about it | Allowedno rule matches |
| BingbotBing and Microsoft Copilot | Search index, and the AI answers built on it | Allowedno rule matches |
| OAI-SearchBotChatGPT search | Finds pages to cite in AI answers | Allowedno rule matches |
| ChatGPT-UserChatGPTMay not follow robots.txt | Fetches a page when someone asks about it | Allowedno rule matches |
| GPTBotOpenAI | Collects data to train AI models | Blockedby its own rule “Disallow: /” |
| Claude-SearchBotClaude search | Finds pages to cite in AI answers | Allowedno rule matches |
| Claude-UserClaude | Fetches a page when someone asks about it | Allowedno rule matches |
| ClaudeBotAnthropic | Collects data to train AI models | Allowedno rule matches |
| PerplexityBotPerplexity | Finds pages to cite in AI answers | Allowedno rule matches |
| Perplexity-UserPerplexityMay not follow robots.txt | Fetches a page when someone asks about it | Allowedno rule matches |
| ApplebotApple Siri, Spotlight and Safari | Search index, and the AI answers built on it | Allowedno rule matches |
| Applebot-ExtendedApple IntelligenceControl token: never crawls itself | Collects data to train AI models | Allowedno rule matches |
| meta-webindexerMeta AI search | Finds pages to cite in AI answers | Allowedno rule matches |
| meta-externalfetcherMeta AIMay not follow robots.txt | Fetches a page when someone asks about it | Allowedno rule matches |
| meta-externalagentMeta AI | Collects data to train AI models | Allowedno rule matches |
| DuckAssistBotDuckDuckGo AI answers | Finds pages to cite in AI answers | Allowedno rule matches |
| Amzn-SearchBotAmazon Alexa search | Finds pages to cite in AI answers | Allowedno rule matches |
| Amzn-UserAmazon AlexaMay not follow robots.txt | Fetches a page when someone asks about it | Allowedno rule matches |
| AmazonbotAmazon | Collects data to train AI models | Allowedno rule matches |
| MistralAI-IndexMistral search | Finds pages to cite in AI answers | Allowedno rule matches |
| MistralAI-UserMistral | Fetches a page when someone asks about it | Allowedno rule matches |
| MistralAI-TrainingMistral | Collects data to train AI models | Allowedno rule matches |
| CCBotCommon Crawl | Collects data to train AI models | Allowedno rule matches |
Runs in your browser. Nothing you type is sent to Notable.
How it works
- The checker reads robots.txt the way RFC 9309 says crawlers must: a crawler follows the section naming it, or the * section if none does, and the longest matching rule wins.
- Blocking training crawlers keeps your content out of future models but does not stop live AI answers. Blocking search and answers crawlers can.
- Some crawlers that fetch a page because someone asked about it say they may not follow robots.txt; they are marked. Firewalls and CDNs can block crawlers too, and this checker cannot see those. The list was last checked against each vendor's documentation in October 2026.
The background
Search, answers and training crawlers
Search crawlers build the index that classic results and AI Overviews draw from. Answer crawlers fetch pages that assistants cite in live answers. Training crawlers collect text for future models. Each can be allowed or blocked on its own.
What to block, if anything
If you want to be recommended by AI assistants, keep search and answer crawlers allowed. Blocking training crawlers is a reasonable choice about how your content is used, and it does not remove you from live answers.
Control tokens
Google-Extended and Applebot-Extended never visit your site. They are names you can use in robots.txt to say whether pages Googlebot or Applebot already fetched may be used for AI. You will not see them in your server logs.
Want these numbers for your brand, every day?
Notable asks AI assistants your customers' questions daily and shows how often you are recommended, with the margin of error beside every number.