AI Crawlers and robots.txt: A Configuration Guide That Keeps Your GEO Door Open

Two Categories of AI Crawler

Every crawler aimed at a language model falls into one of two categories, and conflating them is the main source of configuration errors.

The first is training crawlers: OpenAI's GPTBot, Anthropic's ClaudeBot, Apple's Applebot-Extended. They collect content for model training or improvement. Blocking them has no direct effect on whether your current content appears in answers.

The second is retrieval and live-fetch crawlers: OpenAI's OAI-SearchBot, Anthropic's Claude-SearchBot, Perplexity's PerplexityBot, plus user-triggered agents ChatGPT-User, Claude-User, and Perplexity-User. This category determines whether you appear at all: OAI-SearchBot builds the ChatGPT search index, PerplexityBot builds Perplexity's index, and the User agents fetch your page live when a reader clicks a source link.

The implication is direct: if you want to be cited, the second category must be fully allowed. Many sites copy a block-all-AI list from the internet and block OAI-SearchBot along with everything else, then wonder why they vanished from ChatGPT search.

Reference Table of Major AI Crawlers

OpenAI operates three: GPTBot for training collection, safe to block at your discretion; OAI-SearchBot for the ChatGPT search index, must be allowed; ChatGPT-User for live access during a user session, recommended to allow.

Anthropic operates ClaudeBot (training, may block), Claude-User (live user access, recommended to allow), and Claude-SearchBot (search index, must be allowed).

Perplexity operates PerplexityBot (index, must be allowed) and Perplexity-User (live user access, recommended to allow).

Google-Extended is a special case: it controls only whether content is used for Gemini and Vertex AI model training. It does not affect normal Google crawling or ranking, and blocking it will not remove you from Google, nor will it prevent AI Overviews from using your content.

Others include meta-externalagent (Meta), Applebot-Extended (Apple), CCBot (Common Crawl, an indirect source for many training corpora), Bytespider, and Amazonbot. Weigh each by its purpose.

Writing robots.txt Correctly

The most robust strategy is not exhaustive enumeration but allow-by-default with explicit exceptions: under User-agent: * use Disallow for directories that genuinely need protection, such as admin, account, search-result pages, and internal endpoints; then add separate User-agent blocks only for the training bots you intend to block.

Three details cause frequent errors. First, robots.txt matching is evaluated per User-agent block with no inheritance, so every block must list its own complete Disallow set. Second, Crawl-delay is honored by only some crawlers and should not be your primary throttle; real rate limiting belongs at the CDN or WAF layer. Third, robots.txt must sit at the site root and is case-sensitive; placing it in a subdirectory or behind a redirect renders it inert.

One more point often overlooked: robots.txt only constrains well-behaved crawlers. Malicious scraping ignores it entirely and must be handled with a WAF, rate limits, and behavioral analysis.

Diagnosing an Accidental Block

Step one is a static review: fetch robots.txt and check each crawler name above against every rule, confirming OAI-SearchBot, PerplexityBot, and Claude-SearchBot are not matched by anything. Remember that matching uses longest-prefix-wins, so a loose Disallow: / overrides everything after it.

Step two is log analysis: aggregate Nginx or CDN access logs by User-agent and count requests and status-code distribution per AI crawler over the last 30 days. Zero requests means it never arrived; a large share of 403 or 429 means your WAF or rate limiter is rejecting it.

Step three is cross-validation: many WAF default rule sets classify unfamiliar AI crawlers as malicious and block them, leaving robots.txt correct while access still fails. You must explicitly allowlist the retrieval crawlers in your WAF.

Finally, test on the platforms themselves: ask Perplexity and ChatGPT search a question tightly related to your site and see whether your domain appears. Three consecutive rounds with no appearance strongly indicates an access problem.

Block or Allow: A Decision Framework

Whether to allow training crawlers is a business decision, not a technical one, and it depends on how your content generates value.

If your model depends on content being read and cited (media, SaaS, professional services, ecommerce), the brand exposure from allowing usually outweighs the downside; allow retrieval crawlers and decide on training crawlers case by case.

If your core asset is proprietary data or paid content that a model could absorb and substitute for your product, blocking training crawlers is reasonable, but you should still allow retrieval crawlers, because they generate cited referral traffic that is a net gain.

A middle path is directory-level tiering: allow all crawlers on public blog posts, guides, and documentation; block all crawlers on paywalled content and user data areas. That granularity fits most sites better than a site-wide binary.

Making Access Part of Routine GEO Monitoring

Access configuration is not a one-time task. It breaks when CDN policies change, WAF rules update, or the site is redesigned. Put it on a quarterly checklist.

Track three indicators: whether 30-day request volume from retrieval crawlers is stable, whether the share of non-200 responses is climbing abnormally, and whether the number of times your own domain appears as a cited source declines in your Broccoli AI GEO sampling results.

The third is the most sensitive. The technical accessibility dimension in Broccoli AI GEO automatically checks crawlability and AI crawler reachability, so using it as a routine monitor surfaces problems before citation rates drop, rather than after traffic has already declined.