--- title: "Agents can only recommend what they can find" description: "Discoverability decides whether an AI agent evaluating your business on a customer's behalf ever gets past the front door." url: "https://agentsurface.dev/guide/discoverability" lastModified: 2026-09-25T10:35:24.000Z --- A customer asks their AI assistant to find a supplier that can handle a specific job, compare a few options, and pull up pricing and setup requirements for each. The assistant works through this the way a diligent junior researcher would: it visits each company's website, tries to read the pages, checks whether there's a sitemap or index pointing at the useful information, and looks for a name, an address, and some way to confirm who actually runs the business. Companies that pass this test get compared side by side. Companies that fail it get dropped, usually without anyone on the losing side knowing a request ever happened. Assistants already fetch a page the moment someone asks about a company, and separately crawl sites on a schedule to keep their own answers current. A site that blocks these visits, or buries its useful facts behind logins, broken links, or pages a machine can't parse, is invisible on a channel customers are already using to shortlist vendors. Getting this right is a business decision that happens to have technical fixes. A supplier that a customer has never heard of can still win a deal once an agent puts it in front of them. A supplier an agent can't locate never enters the conversation at all. The same work that makes a site easy for agents to read also makes it easier for an agent to recommend the business by name and send the customer back to the right page. ## What good looks like [#what-good-looks-like] Start with access. `robots.txt` is a plain-text file at the root of your site that tells automated visitors which pages they may read. Separately, you can state what they're allowed to do with what they read: show it in a search result, use it to answer a customer's question right now, or feed it into training a model. Those are different decisions with different consequences. Blocking access closes the door outright; declining to allow training still lets your content appear in an agent's answer to a live question. See [robots.txt for AI Agents](/docs/discovery/robots-txt) for how the two interact and which crawlers do which job. A sitemap is a file listing every page on your site and when it last changed, so an agent can find the current version of something instead of a page that moved two years ago. Pair it with redirects when pages do move: a broken link is a dead end most agents won't try twice. Identity matters more than it might sound like it should. An agent trying to work out who runs a business checks whether the company name, domain, and contact details agree across the site, and looks for the basics: an About page, a Contact page, a Privacy page. Structured data, small blocks of machine-readable information embedded in a page, commonly in a format called JSON-LD, gives the agent something firmer to check than prose copy. This is also what keeps an agent from confusing your business with a similarly named competitor, or treating a copycat as the genuine article. For product and technical documentation specifically, `llms.txt` is worth knowing about: a short index file, again plain text, listing your most important pages with a one-line description of each. Its real use is narrower than the name suggests. Coding tools and technical agents fetch it while checking whether your product can do a particular job; general search engines and most consumer-facing assistants largely skip it. See [llms.txt](/docs/discovery/llms-txt) for where it pays off and where it doesn't. ## Where to start [#where-to-start] Ask whoever runs the website three questions. Is crawler access set on purpose, or left at the hosting platform's default? Does the sitemap cover every page and update when content changes? Does the site state, consistently and checkably, who runs the business? Those three items, covered by [robots.txt](/docs/discovery/robots-txt), a working sitemap, and a consistent identity, are the foundation everything else builds on. Right behind them: public documentation that explains what the product does and what it takes to set up, before asking anyone to create an account. An agent evaluating a purchase on a customer's behalf can't get past a login wall to check. ## How to tell it's working [#how-to-tell-its-working] Ask an assistant a few of the questions a real prospect would ask: your company name directly, then a description of the problem you solve without naming you, and see whether you show up and whether the facts it states back are accurate. Check your server logs for requests from the named retrieval bots, kept separate from ordinary human traffic, to see whether agents are visiting and which pages they land on. If a customer or colleague mentions that an AI tool gave them wrong information about your business, that's usually a discoverability gap showing up as a trust problem, and it's worth tracing back to the page the agent read rather than arguing with the assistant that produced it. Once agents can reliably find and verify the business, the next question is whether the pages they land on explain the product well enough for a customer to act on. ## Recommendations - [Let agents reach your public content](https://agentsurface.dev/docs/discovery/robots-txt): Set crawler access rules and check your firewall. Use Content Signals to separately declare whether retrieved content may be used for search, AI answers, or training. - [Give important pages stable URLs](https://agentsurface.dev/docs/discovery/aeo-checklist): Maintain working links and a sitemap. Update its dates when content changes, and redirect moved pages to their replacements. - [Identify the business behind your product](https://agentsurface.dev/docs/discovery/commercial-and-entity-discovery): Use consistent company names, domains, and contact details. Publish About, Contact, and Privacy pages, and connect official profiles through structured data. - [Make your documentation easy to find](https://agentsurface.dev/docs/api-surface/openapi-for-agents): Link to product guidance, API docs, and integrations from your website. Explain setup requirements before asking someone to sign in. - [Give agents a documentation index](https://agentsurface.dev/docs/discovery/llms-txt): Use llms.txt to introduce your product and link to your documentation. For large sites, link to separate indexes for each product area. - [Publish where agents can connect](https://agentsurface.dev/docs/discovery/well-known-endpoints): Use discovery metadata and catalogs to link to your API specification, MCP server, and other agent resources. Check that each address works and matches its description. - [Publish your product and content feeds](https://agentsurface.dev/docs/discovery/catalogs-and-feeds): Offer structured feeds, including NLWeb schema feeds, for clients that use them. Validate ARD or AI Catalog entries against the format you publish, and check every linked resource. - [Provide evidence for publisher claims](https://agentsurface.dev/docs/discovery/catalogs-and-feeds): A catalog's trust manifest can include publisher details, signatures, or attestations. Publish claims that the receiving client can verify, including who issued them and whether they remain valid. - [List your official integrations](https://agentsurface.dev/docs/discovery/commercial-and-entity-discovery): List your SDKs, skills, plugins, and apps in the directories customers use. Link each listing to your domain and keep installation instructions and publisher details current. - [Package a plugin for each supported platform](https://agentsurface.dev/docs/discovery/commercial-and-entity-discovery): Bundle related skills, tools, or MCP connections into an installable plugin. Include the platform's required manifest, setup instructions, and publisher details, then test installation in that host. - [Advertise agent services through DNS](https://agentsurface.dev/docs/discovery/dns-discovery): Publish DNS records that point to your agent services, so compatible clients can find their connection details. DNS-AID is an emerging proposal; use it when your clients support it.