SEO glossary

robots.txt

robots.txt is a plain text file at the root of a website that tells crawlers which URLs they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309 in 2022. It controls crawling, not indexing: a URL blocked in robots.txt can still appear in Google if other pages link to it.

Reviewed by Gareth Hoyle, Managing Director
Last reviewed 31 October 2026

How does robots.txt work?

Rules are grouped by user agent, the name a crawler identifies itself with, and each group lists paths the crawler may not (disallow) or may (allow) fetch. The file must sit in the top-level directory, such as example.com/robots.txt, and applies only to that host, protocol and port. Google reads the first 500 KiB, follows the most specific group that matches it and ignores crawl-delay.

The protocol dates from 1994. RFC 9309 says its rules are not a form of access authorisation, so use noindex or a password to keep a page out of results.

Controlling AI crawlers in robots.txt

AI companies publish their own user agents, and most separate model training from search, so blocking one need not block the other.

OpenAI: GPTBot crawls content that may train its models and OAI-SearchBot indexes pages for ChatGPT search; each is set independently. ChatGPT-User fetches pages when a user asks, and OpenAI says robots.txt may not apply to it
Anthropic: ClaudeBot collects training data, Claude-SearchBot indexes pages for search results and Claude-User fetches pages for user questions. Anthropic follows robots.txt and supports crawl-delay
Google: Google-Extended controls whether content Google crawls is used to train Gemini models and for grounding in Gemini Apps and Vertex AI. It has no user agent of its own and doesn't affect inclusion or ranking in Google Search
AI Overviews and AI Mode are part of Google Search, so they follow Googlebot rules and snippet controls, not Google-Extended

Example

A publisher wants to appear in ChatGPT search and Claude's search results but keep its articles out of model training. It adds groups that disallow GPTBot, ClaudeBot and Google-Extended from the whole site and leaves OAI-SearchBot, Claude-SearchBot and Googlebot free to crawl. Its Google rankings and AI Overview eligibility are unaffected.