What is robots.txt?

robots.txt is a file at yourdomain.com/robots.txt. Google describes it simply: it "tells search engine crawlers which URLs the crawler can access on your site" (Google Search Central: Introduction to robots.txt, read 1 November 2026).

The rules are a standard. RFC 9309, published by the IETF in 2022, defines the Robots Exclusion Protocol, first written by Martijn Koster in 1994. It also says plainly: "These rules are not a form of access authorization."

How does robots.txt work?

The file is made of groups. Each group starts with one or more user-agent lines, followed by allow and disallow rules for paths.

  1. A crawler fetches /robots.txt before it crawls.
  2. It finds the group whose user-agent matches its own product token, such as GPTBot. Matching is case-insensitive. If no group matches, it uses the "User-agent: *" group.
  3. It follows the most specific matching allow or disallow rule for each URL.
  4. It may cache the file. RFC 9309 says crawlers should not use a cached copy for more than 24 hours, unless the file cannot be reached.

Two details catch people out. If robots.txt returns a 4xx error, crawlers may crawl everything. If it returns a 5xx server error, the standard says crawlers must assume everything is disallowed.

What does robots.txt not do?

  • It does not keep a page out of Google. Google says robots.txt "is not a mechanism for keeping a web page out of Google"; use noindex or a password for that.
  • It does not force anyone. Google notes that its rules "cannot enforce crawler behavior". Respectable crawlers obey; others might not.
  • It does not cover every AI fetch. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a user started the fetch.

How does robots.txt work for AI crawlers?

OpenAI, Anthropic and Perplexity split their crawlers by job, and Google uses a separate token for Gemini, so one file can allow search and block training. The table lists the names their own pages give today.

AI crawler names to use in robots.txt
CompanySearch or user fetchesModel training
OpenAIOAI-SearchBot (search), ChatGPT-User (user fetches)GPTBot
AnthropicClaude-SearchBot (search), Claude-User (user fetches)ClaudeBot
PerplexityPerplexityBot (search), Perplexity-User (user fetches)None listed: Perplexity says neither of its agents is used to collect training content for AI foundation models
GoogleGooglebot (Search, including AI features)Google-Extended (a token for Gemini training and grounding)

Sources, read 1 November 2026: OpenAI, Anthropic, Perplexity, Google common crawlers, Google AI features.

Why does robots.txt matter for B2B SaaS?

A single old line can hide your site from AI search. A rule that blocks every AI bot by name can also block the search crawlers that let ChatGPT, Claude and Perplexity link your pages. Check what your file says, and check your CDN too: Google lists "ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" as a basic for AI features.

Example robots.txt (hypothetical)

A hypothetical SaaS site that allows search crawlers, blocks training crawlers and asks all crawlers to skip its app area:

  • User-agent: *
  • Disallow: /app/
  • User-agent: GPTBot
  • Disallow: /
  • User-agent: ClaudeBot
  • Disallow: /
  • User-agent: Google-Extended
  • Disallow: /
  • Sitemap: https://www.example.com/sitemap.xml

Note: a crawler that matches its own group ignores the "*" group. Here GPTBot obeys only "Disallow: /". Repeat shared rules in each group if you need them.

robots.txt vs llms.txt

robots.txt says what crawlers may fetch. llms.txt is a proposed file that points language models to a site's key pages. It sets no access rules. See the AI crawler entry for the bots themselves and the LLM SEO guide for how pages get cited.