What is an AI crawler?
AI companies run several bots, each with a different job. OpenAI, Anthropic and Perplexity now split them into three kinds: a training crawler, a search crawler and a fetcher that visits a page because a user asked. Google uses one control token for AI use instead of a separate bot.
What are the main AI crawlers?
| Company | Bot | Used for | robots.txt |
|---|---|---|---|
| OpenAI | GPTBot | Training models | Follows it |
| OpenAI | OAI-SearchBot | ChatGPT search results | Follows it |
| OpenAI | ChatGPT-User | Actions a user starts | "robots.txt rules may not apply" |
| Anthropic | ClaudeBot | Training models | Follows it |
| Anthropic | Claude-SearchBot | Search result quality | Follows it |
| Anthropic | Claude-User | Pages a user asks about | Follows it |
| Perplexity | PerplexityBot | Perplexity search results, not training | Follows it |
| Perplexity | Perplexity-User | Pages a user asks about | "generally ignores robots.txt rules" |
| Google-Extended (a token, not a bot) | Gemini training and grounding | A robots.txt token only |
Sources: OpenAI, Anthropic, Perplexity, Google, all read 1 November 2026.
Why do AI crawlers matter for B2B SaaS?
Blocking the wrong bot can take you out of AI answers. Anthropic says turning off Claude-SearchBot "may reduce your site's visibility and accuracy in user search results". OpenAI says "Each setting is independent of the others". You can allow OAI-SearchBot for ChatGPT search and block GPTBot to opt out of training.
Google is different. It says "Google-Extended does not impact a site's inclusion in Google Search", and it has no separate user agent. Its normal crawlers fetch the pages; the token only controls AI use.
How do you control AI crawlers?
- Decide per purpose: training, AI search, user requests.
- Write a robots.txt rule for each bot by name. See robots.txt.
- Check your server or CDN logs to see which bots really visit, and whether a firewall blocks the ones you want.
- Remember that robots.txt is a request. RFC 9309 says its rules "are not a form of access authorization" (RFC 9309).
- Watch the new shared signal. On 30 October 2026 OpenAI said it will honor a common robots.txt signal for AI training, drawing on the work of the IETF AI Preferences working group, and aims to implement it "in the coming months" (OpenAI, IETF).
What do the rules look like? (hypothetical example)
This example is hypothetical. A SaaS company wants to appear in AI search but stay out of training:
- User-agent: GPTBot, then Disallow: /
- User-agent: ClaudeBot, then Disallow: /
- User-agent: Google-Extended, then Disallow: /
- User-agent: OAI-SearchBot, Claude-SearchBot and PerplexityBot, then Allow: /
Note the trade-off: Google says Google-Extended also covers grounding in Gemini Apps, so blocking it can affect more than training.
AI crawler vs search crawler vs user fetcher
- GPTBot and ClaudeBot collect pages for training.
- PerplexityBot builds a search index for answers with links.
- Google-Extended is a control, not a crawler.
- User fetchers visit one page because a person asked, so some do not follow robots.txt.
For the full set of steps that get a site into AI answers, see LLM SEO: how B2B SaaS brands get named in AI answers.