Every AI crawler user agent, in one table
Last reviewed
Short answer
Eleven AI crawlers publish an official robots.txt token. OpenAI runs GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot. Perplexity runs PerplexityBot and Perplexity-User. Anthropic runs ClaudeBot, Claude-User and Claude-SearchBot. Google runs Google-Extended and Google-CloudVertexBot. Each token is case-insensitive, and each does one of three jobs: training, search indexing, or fetching a page for a user.
The table
Every token below was read off the operator's own documentation on 23 August 2026. Tokens change, and vendors add new agents without announcement, so the source links at the foot of this page are the thing to trust if this table and the docs ever disagree.
| robots.txt token | Operator | What it does | IP ranges |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train foundation models | openai.com/gptbot.json |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT's search features | openai.com/searchbot.json |
| ChatGPT-User | OpenAI | Fetches a page because a user asked ChatGPT to | openai.com/chatgpt-user.json |
| OAI-AdsBot | OpenAI | Checks the safety of pages submitted as ChatGPT ads | openai.com/adsbot.json |
| PerplexityBot | Perplexity | Indexes sites to link them in Perplexity results | perplexity.com/perplexitybot.json |
| Perplexity-User | Perplexity | Visits a page to answer a user's live question | perplexity.com/perplexity-user.json |
| ClaudeBot | Anthropic | Collects content that may contribute to model training | claude.com/crawling/bots.json |
| Claude-User | Anthropic | Visits a site on behalf of a Claude user | claude.com/crawling/bots.json |
| Claude-SearchBot | Anthropic | Indexes content to improve Claude search results | claude.com/crawling/bots.json |
| Google-Extended | Controls Gemini training and grounding only | Standard Googlebot ranges | |
| Google-CloudVertexBot | Crawls sites at their own owner's request for Vertex AI agents | Standard Googlebot ranges |
The three jobs these bots do
Grouping by operator is how the docs are laid out, but grouping by job is how you should decide. There are only three, and they carry very different consequences for a store.
Training crawlers collect text that may go into a future model. Blocking one costs you nothing today and is a values call rather than a traffic call. Search crawlers build the index an assistant picks from; blocking one removes you from that assistant's results. User-triggered fetchers load a page because someone asked a question right now, and blocking one means the assistant answers about your product without having read your page.
- Training: GPTBot, ClaudeBot, Google-Extended
- Search indexing: OAI-SearchBot, PerplexityBot, Claude-SearchBot
- Live user fetch: ChatGPT-User, Perplexity-User, Claude-User
Why the distinction costs merchants money
Most blocklists circulating in Shopify forums are copied from publisher advice, and a publisher's incentives are the opposite of a store's. A newspaper loses when a model trains on its archive. A store loses when an assistant cannot read its product page.
The common mistake is pasting a block for every agent with an AI-sounding name, which takes out the search and fetch bots along with the training ones. That removes the store from AI shopping results in exchange for nothing. We wrote out the decision separately.
How you actually change this on Shopify
Shopify generates robots.txt for you and there is no field in the admin for it. To change a rule you add a `robots.txt.liquid` template to your theme, which Shopify then renders instead of the default while still keeping its own future updates.
Shopify's help page is blunt about the risk: editing this file is an unsupported customization, Shopify Support will not help with it, and it says in as many words that incorrect use of the feature can result in loss of all traffic. Copy the default rules first, add your block, and read the rendered file at /robots.txt afterwards.
- Online Store, then Themes, then Edit code
- Add a new template called robots.txt.liquid
- Keep the default rules, then add your own user-agent group
- Load yourstore.com/robots.txt and read what actually rendered
Verifying a bot is really the bot
A user-agent header is a claim, not proof, and anything can send one. Every operator here except Google publishes a JSON file of the IP ranges its crawlers use, listed in the table above.
If you are checking a specific request in your logs, resolve it against the published ranges rather than trusting the string. Rate-limiting or blocking by user agent alone stops honest crawlers and does nothing about dishonest ones.
Frequently asked questions
Are robots.txt user agent tokens case sensitive?
No. The robots exclusion protocol matches product tokens case-insensitively, so GPTBot and gptbot behave the same. Write them as the operator documents them anyway, because a human reading your file later will be comparing against the docs.
Does blocking an AI crawler affect my Google ranking?
Blocking Google-Extended does not. Google states plainly that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal. Blocking Googlebot itself is a different matter and will remove you from Search.
Is this list complete?
It is complete for operators that publish official documentation we could read at source. Other crawlers exist without published tokens or with documentation we could not verify, and we do not list a token we have not read from the operator.
Do I need to allow these bots for my products to reach ChatGPT?
Not for the product data itself. Shopify syndicates products to agentic surfaces through Catalog regardless of robots.txt. Crawler access governs whether your web pages can be read and cited, which is a separate path to the same shopper.
Sources
Keep reading
Score your own catalog
Entitled grades every product against the rules on this page, explains each failure, and fixes them with AI you approve. The score is free forever.