robots.txt for AI Agents

Summary

robots.txt now controls three distinct bot behaviors: training crawlers (index for model training), search-indexing bots (crawl on a schedule to power a search feature), and user-initiated fetch bots (fetch one page because a person asked the assistant about it right now). Blocking GPTBot does not prevent OAI-SearchBot or ChatGPT-User from reading your docs. Recommended: block training crawlers, allow the other two. One caveat: OpenAI's and Perplexity's user-initiated bots state outright that they may not check robots.txt at all, since a human explicitly asked for that one page.

Training Crawlers       Search-Indexing Bots         User-Initiated Bots
─────────────────       ─────────────────────        ─────────────────────────────
GPTBot (OpenAI)         OAI-SearchBot (OpenAI)        ChatGPT-User (OpenAI) - robots.txt caveat
ClaudeBot (Anthropic)   Claude-SearchBot (Anthropic)  Claude-User (Anthropic)
Google-Extended         Googlebot                     Perplexity-User (Perplexity) - robots.txt caveat
Applebot-Extended       PerplexityBot (Perplexity)
cohere-ai               Applebot, Spotlight, Siri

robots.txt has a new dimension in the AI era: the distinction between bots that train models and bots that retrieve content at inference time. Blocking a training crawler prevents your content from influencing model weights. Blocking a retrieval crawler prevents agents from finding your content when answering questions. These are different decisions with different business implications.

The Training vs Retrieval Split

Most AI companies run at least two retrieval bot types alongside their training crawler: one that indexes on a schedule to power a search feature, and one that fetches a specific page because a user asked the assistant about it in the moment. Verified against each vendor's own crawler documentation:

Blocking GPTBot does not prevent your content from appearing in ChatGPT - that is OAI-SearchBot (scheduled indexing) and ChatGPT-User (user-initiated). Blocking ClaudeBot does not prevent Claude from reading your docs - that is Claude-SearchBot and Claude-User. These are separate decisions that require separate entries.

The robots.txt caveat matters operationally: disallowing ChatGPT-User or Perplexity-User is a stated preference, not a guaranteed block, since both vendors say the directive may not apply once a person has directly asked their assistant to fetch that page. Anthropic makes no such carve-out for Claude-User - its documentation states all three of its bots honor robots.txt.

Recommended Configuration for Agent Discoverability

A SaaS product that wants to appear in AI-generated answers, but does not want its content used to train competitors' models or general-purpose LLMs, uses this configuration:

User-agent: *
Allow: /

# Block training crawlers
# These crawlers index content for model training, not retrieval
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Omgilibot
Disallow: /

# Allow retrieval bots
# These bots fetch content during AI inference - blocking them
# means your content won't appear in AI-generated answers.
# They are allowed by the wildcard rule above, but listed
# explicitly for clarity and auditability.

# User-agent: OAI-SearchBot
# Allow: /

# User-agent: ChatGPT-User
# Allow: /

# User-agent: Claude-SearchBot
# Allow: /

# User-agent: Claude-User
# Allow: /

# User-agent: PerplexityBot
# Allow: /

# User-agent: Perplexity-User
# Allow: /

# User-agent: Googlebot
# Allow: /

# Reference LLM index
Sitemap: https://example.com/sitemap.xml

The explicit Allow entries for retrieval bots are commented out because they are already covered by User-agent: * Allow: /. They are included as documentation of intent - future engineers can see the decision was deliberate.

Selective Content Blocking

Some content warrants training exclusion even if your default is to allow training crawlers:

# Allow training crawlers generally
User-agent: GPTBot
Allow: /docs
Allow: /blog
Disallow: /customers    # Customer case studies - contractually restricted
Disallow: /pricing      # Pricing changes frequently, stale training data is harmful
Disallow: /private      # Internal content

User-agent: ClaudeBot
Allow: /docs
Allow: /blog
Disallow: /customers
Disallow: /pricing
Disallow: /private

Content Signals

Crawler rules describe who may fetch content. Content Signals describe permitted uses after retrieval: search, input to an AI answer, and model training. Decide these separately; allowing a fetch does not answer all three usage questions. The declarations communicate policy to systems that honor them, rather than technically enforcing it.

They can be emitted through managed platform settings or written directly into robots.txt. This example permits search and AI input while declining training; choose values that match your policy:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Some platforms can also emit the same signal as a response header:

Content-Signal: search=yes, ai-input=yes, ai-train=no

The three core signal types, described in Cloudflare’s policy documentation:

search - Whether content may be used to build search indexes and results, excluding AI-generated summaries. This is a use preference, not a rule allowing a particular crawler to fetch the page.

ai-input - Whether content may be used as input to an AI system, such as retrieval context for an answer.

ai-train - Whether content may be used to train or fine-tune models. Choose this independently of search and AI input.

The header approach has an advantage over robots.txt: it applies per-response rather than per-path, which means you can vary the policy based on the request (authenticated users versus public content, for example).

Content Signals does not replace or override crawl access rules. A client must consider both whether it may fetch the content and whether the intended use is permitted. Bots that do not support Content Signals still see the crawler rules in robots.txt.

RSL and the IETF aipref drafts

Content Signals is Cloudflare's own vocabulary, already shipping. Two other efforts describe the same problem - what a crawler may do with content after it fetches it - at different levels of formality, and it's worth being precise about which is which.

RSL (Really Simple Licensing) 1.0 is a ratified standard, not a draft (rslstandard.org). It lets a publisher attach machine-readable licence terms - pay-per-crawl, subscription, pay-per-inference, attribution (including Creative Commons), or free use - to content via robots.txt, an HTTP header, an HTML tag, an RSS feed, or a media file. Cloudflare, Reddit, Akamai, O'Reilly Media, and Vox Media back it; the nonprofit RSL Collective runs collective licensing on top of it. Reach for RSL when you want to state a price or a specific licence condition rather than a plain yes/no preference.

IETF aipref is still two Internet-Drafts, not an RFC. draft-ietf-aipref-vocab (revision -08, 13 September 2026) defines the preference vocabulary - train-ai, ai-use, and search, each allowed/disallowed/unknown - and states outright that its contents "do not reflect consensus of the working group." draft-ietf-aipref-attach (revision -05, 18 August 2026) defines how a preference attaches to content: a Content-Usage HTTP response header, and a Content-Usage rule added directly to robots.txt alongside Allow/Disallow, so access and use live in the same file with different verbs. Treat both as unstable until they reach RFC status - don't build a compliance requirement around field names that are still moving.

Content Signals is what you can ship today. RSL is the standard to reach for if you want to state a price. aipref is the IETF's parallel attempt to standardize the same robots.txt-attached preference model Content Signals already implements.

Referencing llms.txt from robots.txt

There is no official standard for linking llms.txt from robots.txt, but the informal convention is a comment line:

# AI agent index
# llms-txt: https://example.com/llms.txt
# llms-full: https://example.com/llms-full.txt

Some crawlers parse these comment conventions. It is low cost to include them.

The Enforcement Gap

robots.txt is unenforceable. It is a convention that any crawler can ignore, and many commercial scrapers do. A Disallow directive in robots.txt states your preference.

For content you genuinely need to protect from AI training or unauthorized use:

• Authentication gates - content behind login walls is not indexable by crawlers without credentials
• Legal agreements - Terms of Service that explicitly prohibit scraping create legal liability for violators
• noindex meta tags - <meta name="robots" content="noindex"> instructs search engines not to index a page; some AI crawlers respect this
• Rate limiting - throttling requests from known bot user-agents limits scraping throughput

For most documentation and public content, robots.txt is the appropriate tool. It works for well-behaved crawlers - which includes all the major AI companies - and documents your policy for any future disputes.

Monitoring Bot Traffic

Understanding which AI bots are actually crawling your site helps you verify robots.txt is being respected and identify undeclared crawlers:

• Filter your access logs for known bot user-agent strings
• Watch for high-volume requests from cloud provider IP ranges with no matching user-agent
• Use Cloudflare's Bot Management dashboard if your site is on Cloudflare
• Check the Referer header on server-side rendered pages - retrieval bots often include a Referer pointing to the AI product that triggered the fetch
• Track agent traffic as its own segment in analytics rather than folding it into human sessions - count requests carrying a verified Web Bot Auth signature separately from those merely claiming a recognized user-agent string, since only the signed ones are hard to spoof

If you see traffic from a bot not covered by your robots.txt, look up the bot's documentation to identify its user-agent string and add an appropriate directive.

User-agent strings are trivially spoofable, so log-based verification only goes so far. Web Bot Auth is the emerging answer: signature-based verification that identifies bot traffic cryptographically rather than by user-agent, building on RFC 9421 HTTP Message Signatures so an accountable agent signs its requests and you verify them against a published key directory. Cloudflare has run it in production since March 2026 (folded into Verified Bots). It lets you admit known "good bots" while blocking automation that merely claims a friendly user-agent. The IETF's webbotauth working group adopted its own protocol draft, draft-ietf-webbotauth-httpsig-protocol, on 1 September 2026, replacing the earlier individual submission it was based on; it defines the Signature-Agent header, a JWKS-based key directory format, and the well-known URI for serving it - implement against this draft going forward. See Agent Identity for how request signing fits the broader agent-authorization model.

Cloudflare's three-way bot classification

Cloudflare replaced its old binary "block AI bots or don't" toggle with three independent categories, live for all customers including free tier since 1 July 2026:

• Search - bots that index content to answer questions later, typically in exchange for referral traffic.
• Agent - automated action taken in real time on a person's behalf right now (ChatGPT-User, and browser-driving agents such as Claude or Gemini operating a real browser).
• Training - crawlers that fold content into a model's weights permanently.

Bot Preference Sync, rolled out from 21 August 2026, writes your dashboard settings for these three categories into your robots.txt automatically, prepending Cloudflare-managed directives ahead of your existing rules rather than replacing them. Enabled by default for new zones; existing zones are prompted to confirm before it activates.

From 15 September 2026, new domains get a default that depends on monetization: ad-supported sites default Training to Disallow (blocking training-only crawlers from operators including Amazon, Anthropic, Meta, and OpenAI, on the reasoning that an AI answer built from training data replaces the visit that would have earned ad revenue); non-ad-monetized sites default Training to Allow. Search and Agent stay allowed by default in both cases.

Disallowing training does not cost you search visibility: Apple and Google both state plainly that a Disallow rule for Applebot-Extended or Google-Extended in robots.txt has no effect on Search ranking or inclusion. Microsoft currently supports only the NOARCHIVE meta tag for this purpose - it has said it is building robots.txt-based support for a "no training" preference, targeted for early 2027, so a Microsoft-specific Disallow line has no effect yet.

What to configure: decide Search, Agent, and Training separately - they are different business decisions, not one setting. If you're on Cloudflare, use the three-way dashboard controls and turn on Bot Preference Sync so robots.txt reflects them automatically; otherwise hand-maintain the equivalent per-bot Allow/Disallow rules from the table above. Block Training without hesitation using the Google-Extended/Applebot-Extended pattern - it doesn't touch search ranking. Leave Search and Agent open unless you have a specific reason to exclude them: blocking Agent bots turns away a real person's in-flight task, not model training.

Cloudflare's Pay Per Crawl turns crawl access itself into a priced transaction: a request from a designated crawler gets HTTP 402 Payment Required with a crawler-price header, the crawler replies with crawler-max-price (or crawler-exact-price) to indicate what it will pay, and a successful 200 response carries crawler-charged once payment clears. Crawlers authenticate with Web Bot Auth - Ed25519-signed signature-agent/signature-input/signature headers per RFC 9421 - so the price can't be evaded by spoofing a user-agent string. Cloudflare acts as merchant of record, billing crawlers and paying out publishers. Per crawler, choose Allow (free), Charge (a flat site-wide price), or Block. See Payments for how the underlying 402 payment flow works.