--- title: "robots.txt for AI Agents" description: "Configuring robots.txt to control training crawler access while keeping retrieval bots enabled" url: "https://agentsurface.dev/docs/discovery/robots-txt" lastVerified: 2026-09-25 lastModified: 2026-09-25T05:46:01.000Z --- ## Summary [#summary] robots.txt now controls three distinct bot behaviors: training crawlers (index for model training), search-indexing bots (crawl on a schedule to power a search feature), and user-initiated fetch bots (fetch one page because a person asked the assistant about it right now). Blocking GPTBot does not prevent OAI-SearchBot or ChatGPT-User from reading your docs. Recommended: block training crawlers, allow the other two. One caveat: OpenAI's and Perplexity's user-initiated bots state outright that they may not check robots.txt at all, since a human explicitly asked for that one page. ``` Training Crawlers Search-Indexing Bots User-Initiated Bots ───────────────── ───────────────────── ───────────────────────────── GPTBot (OpenAI) OAI-SearchBot (OpenAI) ChatGPT-User (OpenAI) - robots.txt caveat ClaudeBot (Anthropic) Claude-SearchBot (Anthropic) Claude-User (Anthropic) Google-Extended Googlebot Perplexity-User (Perplexity) - robots.txt caveat Applebot-Extended PerplexityBot (Perplexity) cohere-ai Applebot, Spotlight, Siri ``` `robots.txt` has a new dimension in the AI era: the distinction between bots that train models and bots that retrieve content at inference time. Blocking a training crawler prevents your content from influencing model weights. Blocking a retrieval crawler prevents agents from finding your content when answering questions. These are different decisions with different business implications. ## The Training vs Retrieval Split [#the-training-vs-retrieval-split] Most AI companies run at least two retrieval bot types alongside their training crawler: one that indexes on a schedule to power a search feature, and one that fetches a specific page because a user asked the assistant about it in the moment. Verified against each vendor's own crawler documentation: | Bot Name | Operator | Type | Respects robots.txt? | | ------------------- | ---------- | -------------------- | ------------------------------------------------------------------------------------ | | `GPTBot` | OpenAI | Training | Yes | | `OAI-SearchBot` | OpenAI | Search indexing | Yes | | `ChatGPT-User` | OpenAI | User-initiated fetch | No - OpenAI states robots.txt rules "may not apply" since a user requested the fetch | | `ClaudeBot` | Anthropic | Training | Yes | | `Claude-SearchBot` | Anthropic | Search indexing | Yes | | `Claude-User` | Anthropic | User-initiated fetch | Yes - Anthropic states all three of its bots honor robots.txt | | `PerplexityBot` | Perplexity | Search indexing | Yes | | `Perplexity-User` | Perplexity | User-initiated fetch | No - Perplexity states this fetcher "generally ignores robots.txt rules" | | `Google-Extended` | Google | Training | Yes | | `Googlebot` | Google | Search + retrieval | Yes | | `Applebot-Extended` | Apple | Training | Yes | | `Applebot` | Apple | Search + retrieval | Yes | | `cohere-ai` | Cohere | Training | Yes | Blocking `GPTBot` does not prevent your content from appearing in ChatGPT - that is `OAI-SearchBot` (scheduled indexing) and `ChatGPT-User` (user-initiated). Blocking `ClaudeBot` does not prevent Claude from reading your docs - that is `Claude-SearchBot` and `Claude-User`. These are separate decisions that require separate entries. The robots.txt caveat matters operationally: disallowing `ChatGPT-User` or `Perplexity-User` is a stated preference, not a guaranteed block, since both vendors say the directive may not apply once a person has directly asked their assistant to fetch that page. Anthropic makes no such carve-out for `Claude-User` - its documentation states all three of its bots honor robots.txt. ## Recommended Configuration for Agent Discoverability [#recommended-configuration-for-agent-discoverability] A SaaS product that wants to appear in AI-generated answers, but does not want its content used to train competitors' models or general-purpose LLMs, uses this configuration: ``` User-agent: * Allow: / # Block training crawlers # These crawlers index content for model training, not retrieval User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: cohere-ai Disallow: / User-agent: CCBot Disallow: / User-agent: Omgilibot Disallow: / # Allow retrieval bots # These bots fetch content during AI inference - blocking them # means your content won't appear in AI-generated answers. # They are allowed by the wildcard rule above, but listed # explicitly for clarity and auditability. # User-agent: OAI-SearchBot # Allow: / # User-agent: ChatGPT-User # Allow: / # User-agent: Claude-SearchBot # Allow: / # User-agent: Claude-User # Allow: / # User-agent: PerplexityBot # Allow: / # User-agent: Perplexity-User # Allow: / # User-agent: Googlebot # Allow: / # Reference LLM index Sitemap: https://example.com/sitemap.xml ``` The explicit `Allow` entries for retrieval bots are commented out because they are already covered by `User-agent: * Allow: /`. They are included as documentation of intent - future engineers can see the decision was deliberate. ## Selective Content Blocking [#selective-content-blocking] Some content warrants training exclusion even if your default is to allow training crawlers: ``` # Allow training crawlers generally User-agent: GPTBot Allow: /docs Allow: /blog Disallow: /customers # Customer case studies - contractually restricted Disallow: /pricing # Pricing changes frequently, stale training data is harmful Disallow: /private # Internal content User-agent: ClaudeBot Allow: /docs Allow: /blog Disallow: /customers Disallow: /pricing Disallow: /private ``` ## Content Signals [#content-signals] Crawler rules describe who may fetch content. [Content Signals](https://contentsignals.org/) describe permitted uses after retrieval: search, input to an AI answer, and model training. Decide these separately; allowing a fetch does not answer all three usage questions. The declarations communicate policy to systems that honor them, rather than technically enforcing it. They can be emitted through managed platform settings or written directly into `robots.txt`. This example permits search and AI input while declining training; choose values that match your policy: ``` User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / ``` Some platforms can also emit the same signal as a response header: ``` Content-Signal: search=yes, ai-input=yes, ai-train=no ``` The three core signal types, described in [Cloudflare’s policy documentation](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/#content-signals-policy): **`search`** - Whether content may be used to build search indexes and results, excluding AI-generated summaries. This is a use preference, not a rule allowing a particular crawler to fetch the page. **`ai-input`** - Whether content may be used as input to an AI system, such as retrieval context for an answer. **`ai-train`** - Whether content may be used to train or fine-tune models. Choose this independently of search and AI input. The header approach has an advantage over `robots.txt`: it applies per-response rather than per-path, which means you can vary the policy based on the request (authenticated users versus public content, for example). Content Signals does not replace or override crawl access rules. A client must consider both whether it may fetch the content and whether the intended use is permitted. Bots that do not support Content Signals still see the crawler rules in `robots.txt`. ## RSL and the IETF aipref drafts [#rsl-and-the-ietf-aipref-drafts] Content Signals is Cloudflare's own vocabulary, already shipping. Two other efforts describe the same problem - what a crawler may do with content after it fetches it - at different levels of formality, and it's worth being precise about which is which. **RSL (Really Simple Licensing) 1.0** is a ratified standard, not a draft ([rslstandard.org](https://rslstandard.org)). It lets a publisher attach machine-readable licence terms - pay-per-crawl, subscription, pay-per-inference, attribution (including Creative Commons), or free use - to content via `robots.txt`, an HTTP header, an HTML tag, an RSS feed, or a media file. Cloudflare, Reddit, Akamai, O'Reilly Media, and Vox Media back it; the nonprofit RSL Collective runs collective licensing on top of it. Reach for RSL when you want to state a price or a specific licence condition rather than a plain yes/no preference. **IETF aipref** is still two Internet-Drafts, not an RFC. `draft-ietf-aipref-vocab` (revision -08, 13 September 2026) defines the preference vocabulary - `train-ai`, `ai-use`, and `search`, each `allowed`/`disallowed`/`unknown` - and states outright that its contents "do not reflect consensus of the working group." `draft-ietf-aipref-attach` (revision -05, 18 August 2026) defines how a preference attaches to content: a `Content-Usage` HTTP response header, and a `Content-Usage` rule added directly to `robots.txt` alongside `Allow`/`Disallow`, so access and use live in the same file with different verbs. Treat both as unstable until they reach RFC status - don't build a compliance requirement around field names that are still moving. Content Signals is what you can ship today. RSL is the standard to reach for if you want to state a price. aipref is the IETF's parallel attempt to standardize the same `robots.txt`-attached preference model Content Signals already implements. ## Referencing llms.txt from robots.txt [#referencing-llmstxt-from-robotstxt] There is no official standard for linking `llms.txt` from `robots.txt`, but the informal convention is a comment line: ``` # AI agent index # llms-txt: https://example.com/llms.txt # llms-full: https://example.com/llms-full.txt ``` Some crawlers parse these comment conventions. It is low cost to include them. ## The Enforcement Gap [#the-enforcement-gap] `robots.txt` is unenforceable. It is a convention that any crawler can ignore, and many commercial scrapers do. A `Disallow` directive in `robots.txt` states your preference. For content you genuinely need to protect from AI training or unauthorized use: * **Authentication gates** - content behind login walls is not indexable by crawlers without credentials * **Legal agreements** - Terms of Service that explicitly prohibit scraping create legal liability for violators * **`noindex` meta tags** - `` instructs search engines not to index a page; some AI crawlers respect this * **Rate limiting** - throttling requests from known bot user-agents limits scraping throughput For most documentation and public content, `robots.txt` is the appropriate tool. It works for well-behaved crawlers - which includes all the major AI companies - and documents your policy for any future disputes. ## Monitoring Bot Traffic [#monitoring-bot-traffic] Understanding which AI bots are actually crawling your site helps you verify `robots.txt` is being respected and identify undeclared crawlers: * Filter your access logs for known bot user-agent strings * Watch for high-volume requests from cloud provider IP ranges with no matching user-agent * Use Cloudflare's Bot Management dashboard if your site is on Cloudflare * Check the `Referer` header on server-side rendered pages - retrieval bots often include a `Referer` pointing to the AI product that triggered the fetch * Track agent traffic as its own segment in analytics rather than folding it into human sessions - count requests carrying a verified Web Bot Auth signature separately from those merely claiming a recognized user-agent string, since only the signed ones are hard to spoof If you see traffic from a bot not covered by your `robots.txt`, look up the bot's documentation to identify its user-agent string and add an appropriate directive. User-agent strings are trivially spoofable, so log-based verification only goes so far. Web Bot Auth is the emerging answer: signature-based verification that identifies bot traffic cryptographically rather than by user-agent, building on RFC 9421 HTTP Message Signatures so an accountable agent signs its requests and you verify them against a published key directory. Cloudflare has run it in production since March 2026 (folded into Verified Bots). It lets you admit known "good bots" while blocking automation that merely claims a friendly user-agent. The IETF's webbotauth working group adopted its own protocol draft, `draft-ietf-webbotauth-httpsig-protocol`, on 1 September 2026, replacing the earlier individual submission it was based on; it defines the `Signature-Agent` header, a JWKS-based key directory format, and the well-known URI for serving it - implement against this draft going forward. See [Agent Identity](/docs/authentication/agent-identity) for how request signing fits the broader agent-authorization model. ## Cloudflare's three-way bot classification [#cloudflares-three-way-bot-classification] Cloudflare replaced its old binary "block AI bots or don't" toggle with three independent categories, live for all customers including free tier since 1 July 2026: * **Search** - bots that index content to answer questions later, typically in exchange for referral traffic. * **Agent** - automated action taken in real time on a person's behalf right now (`ChatGPT-User`, and browser-driving agents such as Claude or Gemini operating a real browser). * **Training** - crawlers that fold content into a model's weights permanently. **Bot Preference Sync**, rolled out from 21 August 2026, writes your dashboard settings for these three categories into your `robots.txt` automatically, prepending Cloudflare-managed directives ahead of your existing rules rather than replacing them. Enabled by default for new zones; existing zones are prompted to confirm before it activates. **From 15 September 2026**, new domains get a default that depends on monetization: ad-supported sites default Training to Disallow (blocking training-only crawlers from operators including Amazon, Anthropic, Meta, and OpenAI, on the reasoning that an AI answer built from training data replaces the visit that would have earned ad revenue); non-ad-monetized sites default Training to Allow. Search and Agent stay allowed by default in both cases. Disallowing training does not cost you search visibility: Apple and Google both state plainly that a `Disallow` rule for `Applebot-Extended` or `Google-Extended` in `robots.txt` has no effect on Search ranking or inclusion. Microsoft currently supports only the `NOARCHIVE` meta tag for this purpose - it has said it is building `robots.txt`-based support for a "no training" preference, targeted for early 2027, so a Microsoft-specific `Disallow` line has no effect yet. **What to configure:** decide Search, Agent, and Training separately - they are different business decisions, not one setting. If you're on Cloudflare, use the three-way dashboard controls and turn on Bot Preference Sync so `robots.txt` reflects them automatically; otherwise hand-maintain the equivalent per-bot `Allow`/`Disallow` rules from the table above. Block Training without hesitation using the `Google-Extended`/`Applebot-Extended` pattern - it doesn't touch search ranking. Leave Search and Agent open unless you have a specific reason to exclude them: blocking Agent bots turns away a real person's in-flight task, not model training. Cloudflare's Pay Per Crawl turns crawl access itself into a priced transaction: a request from a designated crawler gets HTTP `402 Payment Required` with a `crawler-price` header, the crawler replies with `crawler-max-price` (or `crawler-exact-price`) to indicate what it will pay, and a successful `200` response carries `crawler-charged` once payment clears. Crawlers authenticate with Web Bot Auth - Ed25519-signed `signature-agent`/`signature-input`/`signature` headers per RFC 9421 - so the price can't be evaded by spoofing a user-agent string. Cloudflare acts as merchant of record, billing crawlers and paying out publishers. Per crawler, choose Allow (free), Charge (a flat site-wide price), or Block. See [Payments](/docs/agentic-commerce/payments) for how the underlying 402 payment flow works.