AI Crawler Checker: Who Can Reach Your Site?

We read your robots.txt as each AI crawler would, then fetch the page as eight of them to catch firewall and CDN blocks. For content, indexing and structured data too, run the full GEO ready checker. Every crawler is listed in our AI crawler list.

Questions and Answers

Which AI crawlers matter most?

The search crawlers: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Googlebot for Google AI Overviews and AI Mode, and Bingbot for Copilot. Blocking any of them removes you from that platform’s answers.

If I block GPTBot, will I disappear from ChatGPT?

No. GPTBot collects pages for training. ChatGPT search uses OAI-SearchBot, which you control separately. The same split applies to Anthropic (ClaudeBot and Claude-SearchBot) and Google (Google-Extended and Googlebot).

Why does it say possibly blocked?

We fetched the page with that crawler’s user agent and the server refused it. Many firewalls block requests that claim to be a bot but come from the wrong IP addresses, so the real crawler may still get through. Check your firewall or CDN logs for the real bot before changing anything.

More free tools

  • GEO Ready Checker

    56 checks of whether ChatGPT, Claude, Perplexity, Google and Copilot can reach, read and cite a page, with copy-paste fixes.

    Open
  • llms.txt Validator

    Check an llms.txt against the format, with line numbers, the Lighthouse audit, broken links and prompt injection.

    Open
  • llms.txt Generator

    Draft an llms.txt for your site from its sitemap, ready to edit and publish.

    Open
  • AI Crawler List

    Every AI crawler user agent, who runs it, what it is for and whether it obeys robots.txt, from the official docs.

    Open
  • Chrome Extension

    Our free GEO and AI visibility checker for Chrome: check any page you are looking at in one click.

    Open