Which AI crawlers to allow in robots.txt, and what each one does
Last reviewed
Short answer
Separate training crawlers from retrieval bots. Blocking GPTBot or ClaudeBot limits model training use. Blocking OAI-SearchBot, PerplexityBot or Claude-SearchBot removes you from the answers those products generate. Google-Extended is not a crawler at all, it is a robots.txt control token with no user agent of its own.
The bots, and what each is for
The single most useful distinction here is training versus retrieval. Training crawlers gather content that may be used to train models. Retrieval bots fetch pages at the moment a user asks something, to ground an answer. Blocking the first is a licensing decision. Blocking the second removes you from answers.
| User agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train foundation models |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search results |
| ChatGPT-User | OpenAI | User-initiated fetches, not automated crawling |
| PerplexityBot | Perplexity | Surfaces and links sites in search results; not for model training |
| ClaudeBot | Anthropic | Collects web content for model training |
| Claude-SearchBot | Anthropic | Retrieval for search-grounded answers |
The one everyone gets wrong
Google-Extended is listed in a lot of blog posts as an AI crawler. It is not. Google's documentation states plainly that Google-Extended does not have a separate HTTP request user agent string, that crawling is done with existing Google user agents, and that the robots.txt token is used in a control capacity only.
So it is a real thing to configure, and a meaningful opt-out for Gemini training and grounding. It is just not something you will ever see in your server logs, and any audit tool reporting Google-Extended visits is reporting fiction.
What to actually do
Decide once whether you are comfortable with your content being used for training. That is a business decision and there is a reasonable case either way. Then, separately, make sure retrieval bots are allowed, because blocking those is nearly always an accident.
Shopify stores rarely block anything by default, but themes, security apps and CDN rules can. Worth checking rather than assuming. Entitled audits crawler access per bot as part of the free channel checklist, alongside the channel connections themselves.
Frequently asked questions
Should I block GPTBot?
That is a licensing preference, not an SEO decision. Blocking GPTBot limits use of your content for model training. It is distinct from OAI-SearchBot, which is what surfaces you inside ChatGPT search results, and blocking that one costs you visibility.
Is Google-Extended a crawler?
No. Google documents that Google-Extended has no separate user agent string; crawling uses existing Google user agents and the robots.txt token acts only as a control for Gemini training and grounding.
Is Bingbot an AI crawler?
Bingbot is Microsoft's general search crawler rather than an AI-specific one. It matters here indirectly, because Bing's index feeds some AI search retrieval, which is also why IndexNow is worth using.
Sources
Keep reading
Score your own catalog
Entitled grades every product against the rules on this page, explains each failure, and fixes them with AI you approve. The score is free forever.