llms.txt is a proposed standard file placed at a site's root or a subpath that gives AI agents a concise markdown overview of the site and links to machine-friendly versions of key pages. It is designed to be small enough to fit in an AI context window, so an agent can see the site's structure and fetch only what it needs. For AI startups, it is a cheap, high-leverage discoverability move.
What Does the File Actually Do?
It tells an agent what the site is and where the important content lives, in a format the agent can parse precisely. Instead of reconstructing meaning from HTML with navigation and scripts, the agent reads a clean map and follows the right link. That reduces wasted tokens and interpretation errors.
The llms.txt proposal describes the purpose directly: agents should view or search llms.txt to find what they need, then follow links to LLM-friendly content. The file stays small; the detail lives behind the links.
Why Does It Matter for AI Startups?
Because agents are now a real discovery channel. When a coding agent or assistant answers a question about a product, it reads pages to form the answer. A site that presents a clear map is easier to use correctly, which raises the odds of accurate mention and citation.
Adoption is no longer speculative. The proposal notes thousands of sites publish llms.txt, documentation platforms generate it automatically, Chrome's Lighthouse audits for it, and OpenAI, Anthropic, and Gemini publish their own for developer docs. It is becoming part of the agent-legible web.
What Should the File Contain?
The structure is simple: an H1 with the project name, a blockquote summary with key context, optional detail sections, and file lists of links to markdown versions of key pages, each with a short description. Our guide to structured data for AI startups covers the complementary machine-readable layer.
Keep it concise and point to genuinely useful content. The file is a map, not a content dump, and its value is in directing the agent efficiently.
How Does It Fit With Sitemaps and robots.txt?
It complements them. Sitemaps list all indexable pages for search engines; llms.txt provides a curated overview for agents on demand. It can also coexist with crawler access controls that govern what automated tools may fetch.
The distinction matters: robots.txt sets access rules, llms.txt guides use. They serve different purposes and are used at different times.
What Else Should an AI Startup Do?
Publish markdown versions of important pages, so the links in llms.txt resolve to clean, LLM-friendly content. The proposal recommends making a markdown version available at the same URL with a .md extension, which lets agents fetch exactly what they need.
That combination — a concise map plus clean markdown pages — is what makes a site easy for agents to use. Our guide to AI-agent discoverability covers the broader strategy, and docs as marketing covers the content that benefits most.
Developer adoption is broad: Stack Overflow's 2025 Developer Survey found 84% of developers use or plan to use AI tools.
What Should You Measure?
Measure adoption and citation, not impressions. For developer-facing products, the useful signals are how many developers try, adopt, and extend the tool, plus how often models and answers cite it accurately. The audience's behavior supports that focus: Stack Overflow's 2025 Developer Survey found developers value concrete recommendations and long-form articles over brand content, and that most are actively using AI tools. Vanity reach misleads; activation, retention, and accurate AI mention are the metrics that reflect real distribution. Read them per channel and per query, not as one blended number.
How Conbersa Distributes the Content
Conbersa runs a site's supporting distribution across a fleet of real physical smartphones, one identity per device, so the content that llms.txt points to also reaches human audiences on social platforms. See how it works at conbersa.ai.