AI Crawler List: Every AI Bot and What It Does

Checked against each operator's own documentation on 2026-11-19. Also available as JSON (CC BY 4.0). To see which of these can reach your site, use the AI crawler checker.

These decide whether your pages can appear and be cited in AI search answers. Blocking one removes you from that platform.

TokenOperator and productObeys robots.txtWhat it does
OAI-SearchBotOpenAIChatGPT searchYesControls whether pages appear in ChatGPT search answers. Independent of GPTBot.Official docsIP rangesMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Claude-SearchBotAnthropicClaude searchYesAnthropic says blocking it may reduce your visibility in Claude search results.Official docsMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com)
PerplexityBotPerplexityPerplexity searchYesSurfaces and links sites in Perplexity answers. Perplexity says it is not used to train foundation models.Official docsIP rangesMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
GooglebotGoogleGoogle Search, AI Overviews and AI ModeYesGoogle Search, including AI Overviews and AI Mode, uses Googlebot. There is no separate AI Overviews crawler.Official docsMozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
BingbotMicrosoftBing, Copilot and partner AIYesPowers Bing search and Copilot answers. Bing Webmaster Tools has an AI Performance report.Official docsMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/131.0.0.0 Safari/537.36
ApplebotAppleSiri, Spotlight, Safari and Apple Intelligence answersYesFollows your Googlebot rules when robots.txt does not name Applebot.Follows the Googlebot rules when not named.Official docsMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
Meta-WebIndexerMetaMeta AI searchYesIndexes pages for citations in Meta AI answers.Official docs
Amzn-SearchBotAmazonAlexa and Amazon searchYesSearch experiences such as Alexa. Not used for training. Follows other search crawlers’ rules when not named.Follows the Googlebot rules when not named.Official docs

Fetchers for User Questions

These visit a page when someone asks an assistant about it. Some operators say robots.txt may not apply because a person asked.

TokenOperator and productObeys robots.txtWhat it does
ChatGPT-UserOpenAIChatGPTMay notVisits a page when a ChatGPT user asks it to. OpenAI says robots.txt rules may not apply. Not used for search inclusion.Official docsIP rangesMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Claude-UserAnthropicClaudeYesFetches pages when a Claude user asks. Anthropic says blocking it stops Claude retrieving your content for users.Official docsMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com)
Perplexity-UserPerplexityPerplexityMay notFetches a page when a user asks. Perplexity says it generally ignores robots.txt.Official docsIP rangesMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
Meta-ExternalFetcherMetaMeta AIMay notFetches a page for a user; Meta says it may bypass robots.txt.Official docs
Amzn-UserAmazonAlexaMay notUser-requested fetches; may not follow every robots.txt rule.Official docs

Training Crawlers

These collect pages to train AI models. Blocking them is a business decision and does not remove you from AI search.

TokenOperator and productObeys robots.txtWhat it does
GPTBotOpenAIOpenAI model trainingYesTraining opt-out only. Blocking it does not remove you from ChatGPT search.Official docsIP rangesMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
ClaudeBotAnthropicAnthropic model trainingYesTraining opt-out. Anthropic advises against IP blocking, which also stops it reading robots.txt.Official docsMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
meta-externalagentMetaMeta AI training and indexingYesCrawls for training and indexing.Official docs
AmazonbotAmazonAmazon model trainingYesMay be used to train Amazon AI models. Honours noarchive as a training opt-out.Official docs
CCBotCommon CrawlCommon Crawl (open datasets)YesFeeds the open Common Crawl corpus used to train many models.Official docs
BytespiderByteDanceByteDance modelsNot confirmedByteDance publishes no crawler policy, so whether it honours robots.txt is unverified.Official docs

Control Tokens

Not crawlers: robots.txt names that switch an AI use on or off for pages another crawler collects.

TokenOperator and productObeys robots.txtWhat it does
Google-ExtendedGoogleGemini training and groundingYesA robots.txt token only (no crawler of its own). Controls Gemini training and grounding; does not affect Google Search or AI Overviews.Official docs
Applebot-ExtendedAppleApple model trainingYesA robots.txt token only: opts out of Apple using your content to train its models. Does not crawl.Official docs

Retired Names

Old tokens that no longer do anything. Rules for them can be removed.

TokenOperator and productObeys robots.txtWhat it does
anthropic-aiAnthropicnone (retired)Not a crawlerOld token that Anthropic no longer documents. Rules for it do nothing: use ClaudeBot, Claude-User and Claude-SearchBot.Official docs
Claude-WebAnthropicnone (retired)Not a crawlerOld token that Anthropic no longer documents. Rules for it do nothing.Official docs

More free tools

  • GEO Ready Checker

    56 checks of whether ChatGPT, Claude, Perplexity, Google and Copilot can reach, read and cite a page, with copy-paste fixes.

    Open
  • AI Crawler Checker

    Can each AI crawler get to your site? robots.txt rules and live fetches as eight AI bots, in about ten seconds.

    Open
  • llms.txt Validator

    Check an llms.txt against the format, with line numbers, the Lighthouse audit, broken links and prompt injection.

    Open
  • llms.txt Generator

    Draft an llms.txt for your site from its sitemap, ready to edit and publish.

    Open
  • Chrome Extension

    Our free GEO and AI visibility checker for Chrome: check any page you are looking at in one click.

    Open