AI Crawler List: Every AI Bot and What It Does
Checked against each operator's own documentation on 2026-11-19. Also available as JSON (CC BY 4.0). To see which of these can reach your site, use the AI crawler checker.
Search and Answer Crawlers
These decide whether your pages can appear and be cited in AI search answers. Blocking one removes you from that platform.
| Token | Operator and product | Obeys robots.txt | What it does |
|---|---|---|---|
| OAI-SearchBot | OpenAIChatGPT search | Yes | Controls whether pages appear in ChatGPT search answers. Independent of GPTBot.Official docsIP rangesMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot |
| Claude-SearchBot | AnthropicClaude search | Yes | Anthropic says blocking it may reduce your visibility in Claude search results.Official docsMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com) |
| PerplexityBot | PerplexityPerplexity search | Yes | Surfaces and links sites in Perplexity answers. Perplexity says it is not used to train foundation models.Official docsIP rangesMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) |
| Googlebot | GoogleGoogle Search, AI Overviews and AI Mode | Yes | Google Search, including AI Overviews and AI Mode, uses Googlebot. There is no separate AI Overviews crawler.Official docsMozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
| Bingbot | MicrosoftBing, Copilot and partner AI | Yes | Powers Bing search and Copilot answers. Bing Webmaster Tools has an AI Performance report.Official docsMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/131.0.0.0 Safari/537.36 |
| Applebot | AppleSiri, Spotlight, Safari and Apple Intelligence answers | Yes | Follows your Googlebot rules when robots.txt does not name Applebot.Follows the Googlebot rules when not named.Official docsMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) |
| Meta-WebIndexer | MetaMeta AI search | Yes | Indexes pages for citations in Meta AI answers.Official docs |
| Amzn-SearchBot | AmazonAlexa and Amazon search | Yes | Search experiences such as Alexa. Not used for training. Follows other search crawlers’ rules when not named.Follows the Googlebot rules when not named.Official docs |
Fetchers for User Questions
These visit a page when someone asks an assistant about it. Some operators say robots.txt may not apply because a person asked.
| Token | Operator and product | Obeys robots.txt | What it does |
|---|---|---|---|
| ChatGPT-User | OpenAIChatGPT | May not | Visits a page when a ChatGPT user asks it to. OpenAI says robots.txt rules may not apply. Not used for search inclusion.Official docsIP rangesMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot |
| Claude-User | AnthropicClaude | Yes | Fetches pages when a Claude user asks. Anthropic says blocking it stops Claude retrieving your content for users.Official docsMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com) |
| Perplexity-User | PerplexityPerplexity | May not | Fetches a page when a user asks. Perplexity says it generally ignores robots.txt.Official docsIP rangesMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) |
| Meta-ExternalFetcher | MetaMeta AI | May not | Fetches a page for a user; Meta says it may bypass robots.txt.Official docs |
| Amzn-User | AmazonAlexa | May not | User-requested fetches; may not follow every robots.txt rule.Official docs |
Training Crawlers
These collect pages to train AI models. Blocking them is a business decision and does not remove you from AI search.
| Token | Operator and product | Obeys robots.txt | What it does |
|---|---|---|---|
| GPTBot | OpenAIOpenAI model training | Yes | Training opt-out only. Blocking it does not remove you from ChatGPT search.Official docsIP rangesMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot |
| ClaudeBot | AnthropicAnthropic model training | Yes | Training opt-out. Anthropic advises against IP blocking, which also stops it reading robots.txt.Official docsMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) |
| meta-externalagent | MetaMeta AI training and indexing | Yes | Crawls for training and indexing.Official docs |
| Amazonbot | AmazonAmazon model training | Yes | May be used to train Amazon AI models. Honours noarchive as a training opt-out.Official docs |
| CCBot | Common CrawlCommon Crawl (open datasets) | Yes | Feeds the open Common Crawl corpus used to train many models.Official docs |
| Bytespider | ByteDanceByteDance models | Not confirmed | ByteDance publishes no crawler policy, so whether it honours robots.txt is unverified.Official docs |
Control Tokens
Not crawlers: robots.txt names that switch an AI use on or off for pages another crawler collects.
| Token | Operator and product | Obeys robots.txt | What it does |
|---|---|---|---|
| Google-Extended | GoogleGemini training and grounding | Yes | A robots.txt token only (no crawler of its own). Controls Gemini training and grounding; does not affect Google Search or AI Overviews.Official docs |
| Applebot-Extended | AppleApple model training | Yes | A robots.txt token only: opts out of Apple using your content to train its models. Does not crawl.Official docs |
Retired Names
Old tokens that no longer do anything. Rules for them can be removed.
| Token | Operator and product | Obeys robots.txt | What it does |
|---|---|---|---|
| anthropic-ai | Anthropicnone (retired) | Not a crawler | Old token that Anthropic no longer documents. Rules for it do nothing: use ClaudeBot, Claude-User and Claude-SearchBot.Official docs |
| Claude-Web | Anthropicnone (retired) | Not a crawler | Old token that Anthropic no longer documents. Rules for it do nothing.Official docs |
More free tools
GEO Ready Checker
56 checks of whether ChatGPT, Claude, Perplexity, Google and Copilot can reach, read and cite a page, with copy-paste fixes.
OpenAI Crawler Checker
Can each AI crawler get to your site? robots.txt rules and live fetches as eight AI bots, in about ten seconds.
Openllms.txt Validator
Check an llms.txt against the format, with line numbers, the Lighthouse audit, broken links and prompt injection.
Openllms.txt Generator
Draft an llms.txt for your site from its sitemap, ready to edit and publish.
OpenChrome Extension
Our free GEO and AI visibility checker for Chrome: check any page you are looking at in one click.
Open




