AI companies now run separate crawlers for training models and for answering questions in search-style products. That split matters: you can keep your content out of model training and still appear, with a link, when someone asks ChatGPT or Claude about your category. This guide lists the main user agents and shows robots.txt rules for the common choices.
Training crawlers and search crawlers are different
OpenAI documents GPTBot for collecting content that may train its models, and OAI-SearchBot for surfacing websites in ChatGPT search. Blocking GPTBot signals you do not want your content used for training; blocking OAI-SearchBot keeps you out of ChatGPT search answers.
Anthropic made the same split in February 2026: ClaudeBot gathers content that may be used for training, Claude-SearchBot indexes content to improve search answers, and Claude-User fetches pages when a person asks Claude a question.
The main AI user agents
Use these names exactly in robots.txt. Rules are matched per user agent, and a named group overrides the generic * group for that bot.
| User agent | Company | Purpose | robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Content that may train models | Respected |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | Respected |
| ChatGPT-User | OpenAI | Visits pages when a user asks | May not apply, user initiated |
| ClaudeBot | Anthropic | Content that may train models | Respected |
| Claude-SearchBot | Anthropic | Indexes for search answers | Respected |
| Claude-User | Anthropic | Fetches pages for a user's question | Blocking may reduce visibility |
| PerplexityBot | Perplexity | Indexes for Perplexity answers | Respected per Perplexity docs |
| Google-Extended | Controls use in Gemini models, not Search ranking | Respected |
Common robots.txt choices
To stay visible in AI search but opt out of training, add a group for each training crawler with Disallow: / and leave the search crawlers allowed. To block everything AI, add groups for every agent in the table. Do not block Googlebot to avoid AI Overviews; that removes you from Google Search entirely.
Remember that robots.txt is a public request, not access control. Content that must stay private needs authentication.
- Opt out of training: Disallow GPTBot, ClaudeBot, Google-Extended, CCBot
- Stay in AI answers: allow OAI-SearchBot, Claude-SearchBot, PerplexityBot
- Keep Googlebot allowed: AI Overviews use normal Search crawling
- List your sitemap at the bottom of the file
Check what you actually block
Many sites block AI crawlers by accident through a CDN bot rule or a security plugin, not robots.txt. Check server or CDN logs for these user agents and confirm they receive 200 responses on pages you want cited.
XDQ xAgents' AI Visibility agent reads your robots.txt and reports, for each of eight AI crawlers, whether it is allowed, blocked by name or blocked by a catch-all rule.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
Not from ChatGPT search. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, so you can block GPTBot and still allow OAI-SearchBot.
What does Google-Extended do?
Google-Extended lets you control whether content crawled by Google is used for Gemini models. It does not affect Google Search ranking, and blocking it does not remove you from Search.
Can I block AI Overviews without leaving Google Search?
Google has not introduced a separate control for AI Overviews. Snippet controls such as nosnippet limit what can be shown, but blocking Googlebot removes the page from Search entirely.
Which Anthropic crawler should I allow?
Allow Claude-SearchBot and Claude-User if you want to appear when people use Claude to search. Block ClaudeBot if you do not want content used for training.
Sources
- Overview of OpenAI crawlers, OpenAI, 2026
- Anthropic clarifies what its three web crawlers do and how to block them, PPC Land, 2026
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic, 2026
- Google's common crawlers, Google Search Central, 2026
- Perplexity crawlers, Perplexity, 2026
