What is llms.txt and why does it matter?

Best move: fetch /llms.txt directly, then judge whether it is a content map or a machine-facing endpoint before you read anything into it.

Why it works: llms.txt lives at a known path in plain markdown, so it costs one request to find. What it says about a company depends entirely on which kind you found.

Key takeaways

• llms.txt is a markdown file served at a site’s root that hands language models a curated index of the content worth reading.
• It is a community proposal from llmstxt.org, not a ratified web standard, and adoption is voluntary.
• llms.txt targets inference-time reading by models, while robots.txt gates crawlers and sitemap.xml feeds search indexes.
• The file is static and fetched at a fixed path, so detecting it needs no JavaScript rendering.
• Bassethound classifies each llms.txt as standard (a content map) or technical (a machine-facing endpoint), and only the technical form moves a domain toward a shipping verdict on its own.

What exactly does an llms.txt file contain?

llms.txt is markdown, not a new format. The llmstxt.org proposal defines a light structure: an H1 with the site or project name, an optional blockquote summary, then link lists grouped under H2 headings. Each link points to a page (often a markdown version of it) with a short description. The point is a curated map, not a dump of every URL. A companion file, /llms-full.txt, inlines the full text of those pages so a model can pull everything in a single fetch instead of crawling.

Sites publish llms.txt to steer models toward the pages that matter (docs, API references, canonical explanations) rather than leaving a model to reconstruct meaning from rendered HTML and marketing copy. Because it is plain markdown at a fixed path, it is cheap to author and cheap to consume. That low cost cuts both ways. A thoughtful llms.txt reflects real intent to serve AI readers. A thin one, auto-generated and forgotten, reflects a checkbox. The file structure alone does not tell you which you are looking at, so treat contents as a lead, not a verdict.

How is llms.txt different from robots.txt and sitemap.xml?

They sit at the same root and look alike, but they answer different questions. robots.txt is access control for crawlers. It says which user agents may fetch which paths, and it predates the AI era by decades. sitemap.xml is a discovery aid for search engines. It lists URLs, last-modified dates, and priorities so an indexer knows what exists and what changed. Neither carries content. Both describe the site to a machine that will fetch pages later.

llms.txt changes the audience and the payload. The audience is a language model reading at inference time, not a crawler building an index. The payload is curated content in markdown, not a URL list or an allow/deny rule. A model can read llms.txt and act on it in the same turn, without a separate crawl. That is the shift. robots.txt and sitemap.xml optimize for the search pipeline. llms.txt optimizes for a model that lands on your site mid-conversation and needs to be right about you fast. A site can, and often should, ship all three. They do not compete. They serve crawlers, indexers, and models in parallel.

Why does llms.txt matter for AI-readiness?

Most signals in an AI-readiness read are inferred. You detect a vector store from client libraries, a model provider from SDK calls. llms.txt is different. It is a company explicitly declaring that it builds for AI consumers. That intent is rare enough to be informative. A team that curates a markdown map for models is a team that has thought about being read by an agent, not just clicked by a person.

That is why Bassethound treats llms.txt as one of the six AI-readiness signal groups. Presence contributes to the ai_readiness score and can nudge the icp_signal warmer. But intent is not the same as a running stack. A polished llms.txt can sit in front of a site with no model, no orchestration, and no vector store behind it. So the file raises a flag worth following, not a conclusion. The value is in what it tells you about a company’s posture toward AI, paired with the harder infrastructure signals in the same dossier. Read alone, llms.txt overstates. Read alongside the other four layers, it sharpens the picture.

What is the difference between a standard and a technical llms.txt?

Bassethound splits llms.txt into two classes because they mean different things. A standard llms.txt follows the llmstxt.org shape: a curated content map pointing a model at docs and canonical pages. It is a discoverability and content move. It counts as one AI-readiness signal group and nothing more.

A technical llms.txt is machine-facing. A model or agent calls it as an endpoint, in the same family as an MCP server or a .well-known AI file. That class carries more weight. On its own, a technical llms.txt can push a domain to a shipping verdict, because a machine-facing endpoint is expensive to stand up and pointless to fake. The distinction is the whole reason Bassethound classifies rather than just reports presence. A tool that flattens both into a single “has llms.txt” boolean will misread a marketing file as production AI. The deeper mechanics of the split, and the edge cases, live in the companion node on standard versus technical llms.txt. The short version: contents and posture decide the class, and the class decides how much the signal is worth.

What can a file at /llms.txt not tell you?

Plenty, and Bassethound reports the gaps rather than papering over them. A standard llms.txt tells you a company curated content for models. It does not tell you whether a model runs in production, whether there is a vector store, or whether any agent exists. Intent and implementation are separate, and the file speaks only to intent.

It also cannot vouch for freshness or honesty. An llms.txt can be stale, auto-generated, or aspirational. The path is fixed, so absence is cheap to confirm, but absence is not proof a company ignores AI. Plenty of teams ship real AI with no llms.txt at all, because a purely server-side or fully proxied stack leaves nothing at that path to find. Detection is a static crawl plus optional JS rendering, and llms.txt happens to need neither, but the same honesty applies across the dossier: what stays server-side stays invisible, and Bassethound says so. Treat /llms.txt as one honest data point. Correlate it with the model provider, the SDKs, the vector store, and the machine-facing endpoints before you decide what a company runs.

Bassethound perspective

Publishing llms.txt has become a checkbox. Vendors race to drop one at their root, call themselves AI-ready, and most detection tools reward them with a green light. We think that is backwards. A standard llms.txt is a curated content map. It tells a model where your docs are. That is a marketing and discoverability move, and it says nothing about whether you run a model, a vector store, or an agent in production. So Bassethound splits the signal. A standard llms.txt counts as one AI-readiness group, no more. Only the technical form, a machine-facing endpoint a model or agent is meant to call, moves a domain toward a shipping verdict on its own, alongside MCP and .well-known files. A competitor who equates any llms.txt with AI adoption will disagree, and that is the point. The file at /llms.txt is a claim. Whether the claim is backed by a stack is a separate question, and it is the one worth answering. We would rather undercount a marketing file than hand out a shipping badge nobody earned.

Sources

• llms.txt proposal and specification: https://llmstxt.org
• Model Context Protocol: https://modelcontextprotocol.io
• Anthropic documentation: https://docs.anthropic.com