Which AI crawlers should you allow in robots.txt?
By Ben Frampton · · 5 min read
Short answer: if you want to appear in AI answers, allow the search crawlers that assistants use to find pages to cite: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude), PerplexityBot (Perplexity) and, as always, Googlebot and Bingbot. Blocking training crawlers such as GPTBot, ClaudeBot or Google-Extended is a separate choice. It affects what future models learn from your site, not whether today's assistants can cite you.
Why AI crawlers come in different types
Most AI companies now run more than one crawler, each with a different job. Broadly:
- Training crawlers collect pages to train future models.
- Search crawlers build an index that the assistant searches when it answers.
- User-triggered fetchers load a specific page because a person asked the assistant to look at it, or because the assistant chose to open a link while answering.
Blocking one type doesn't block the others. That lets you decide, for example, to stay out of training data but still be citable in answers.
The main AI crawlers, as of October 2026
| User agent | Company | Purpose | Affects appearing in answers? |
|---|---|---|---|
GPTBot | OpenAI | Training data for models | Indirectly, over time |
OAI-SearchBot | OpenAI | Index for ChatGPT search | Yes |
ChatGPT-User | OpenAI | Fetches pages for a user's request | Yes, for that request |
ClaudeBot | Anthropic | Training data for models | Indirectly, over time |
Claude-SearchBot | Anthropic | Index for Claude's search | Yes |
Claude-User | Anthropic | Fetches pages for a user's request | Yes, for that request |
PerplexityBot | Perplexity | Index for Perplexity answers | Yes |
Perplexity-User | Perplexity | Fetches pages for a user's request | Yes, for that request |
Googlebot | Google Search, including AI Overviews and AI Mode | Yes | |
Google-Extended | A robots.txt token controlling use for Gemini model training and grounding, not a separate crawler | Not for Search | |
Bingbot | Microsoft | Bing Search, which Copilot answers draw on | Yes |
Applebot-Extended | Apple | A token controlling use for Apple's AI training | Indirectly |
CCBot | Common Crawl | Open web archive used by many model builders | Indirectly, over time |
Companies update these lists, so check each provider's current documentation before you change anything: OpenAI's crawlers, Google's crawlers and Perplexity's crawlers.
Note that some providers treat user-triggered fetchers differently from automated crawlers, and may not apply robots.txt rules to a fetch that a person explicitly asked for.
Does blocking Google-Extended remove me from AI Overviews?
No. Google states that Google-Extended doesn't affect a site's inclusion in Google Search. AI Overviews and AI Mode are part of Search and use pages crawled by Googlebot. The only way to keep a page out of them is the same as for any search feature: nosnippet, data-nosnippet, max-snippet or noindex, which also affect how the page appears in normal results. See Google's guide to AI features and your website.
A robots.txt you can start from
This version allows assistants to find and cite your pages, while opting out of model training:
# Search and answer crawlers: allowed, so assistants can cite us
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Bingbot
Allow: /
# Model training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
# Everyone else
User-agent: *
Disallow: /app/
If you are happy for your public pages to be used in training, which can help future models know your brand, simply leave the training crawlers allowed. There's no single right answer: it's a trade-off between control over your content and how well future models know you.
How do I check whether AI crawlers can actually read my pages?
robots.txt is only the first gate. Also check:
- Your CDN or firewall. Bot protection from your hosting provider or CDN can block AI crawlers even when robots.txt allows them. Look at your bot settings and at your logs for blocked requests from the user agents above.
- JavaScript rendering. Several AI crawlers fetch the raw HTML and don't run JavaScript. If your main content only appears after scripts run, they may see an empty page. View the page source (not the inspector) and make sure the text is there.
- Server logs. Search your access logs for
OAI-SearchBot,PerplexityBotorClaude-SearchBot. If they never appear, they may not be reaching you. - Login walls and interstitials. Cookie banners are fine. Content behind a sign-in or a hard paywall can't be cited.
Frequently asked questions
Will blocking GPTBot remove my brand from ChatGPT?
Not immediately. GPTBot only collects data for training future models. Whether ChatGPT can find and cite your pages today depends on OAI-SearchBot and on search results. Over time, blocking training crawlers may mean newer models know less about you.
Should I block AI crawlers to protect my content?
It depends on what the content is for. For marketing pages you want people to find, blocking search crawlers mostly removes you from answers. For paid or proprietary content, blocking training crawlers and keeping it behind a login is reasonable.
Do AI crawlers respect robots.txt?
The major providers listed above say their automated crawlers do. Some user-triggered fetchers may not, and less reputable scrapers may ignore it entirely. robots.txt is a request, not access control.
Is llms.txt a replacement for robots.txt?
No. robots.txt controls crawling. llms.txt is a proposed file that summarises your site for language models. We look at whether it helps in do llms.txt and schema markup help with AI search?