What One Domain Reveals
What does llms.txt tell you about a company?
llms.txt reveals a company's AI posture: they know the file exists and chose to publish a curated map for language models. The links inside name their canonical docs, products, and API surface, and the first-seen date tells you when they adopted it. Bassethound classifies it (standard vs technical) inside the five-layer dossier, in one call.
Best move: read llms.txt as a statement of intent, then verify it against the backend before you trust it.
Why it works: llms.txt is a publishing choice, not a piece of infrastructure. It tells you a team thought about how models read its pages, but a static file cannot prove what runs behind the site.
Key takeaways
- llms.txt is a proposed standard (llmstxt.org, 2024): a markdown file at the domain root that hands language models a curated map of the site.
- Its presence is a posture signal (the company knows the file exists and chose to publish it), not proof of backend AI.
- Bassethound classifies each llms.txt as standard (llmstxt.org format) or technical, so you know whether you found a real content map or something else at that path.
- The links inside reveal what a company treats as canonical: its docs, its API reference, its product surface.
- In the deep profile, a Wayback first-seen date times the adoption and separates early movers from recent followers.
What is in a well-formed llms.txt?
The llmstxt.org format is short and rigid. An H1 with the project or company name, a blockquote summary, then H2 sections holding lists of links. Each link points to a page, often a markdown version, with an optional one-line description. Common sections are Docs, API, and an Optional block a model can skip under a tight context budget. A companion llms-full.txt inlines the whole corpus for a single fetch.
Read a well-formed file and you learn what a human on that team decided a model should see first. That curation is the value. The site has thousands of URLs. The llms.txt names the twenty that matter. You get the canonical docs, the product pages the team leads with, and the API reference, stripped of navigation and HTML chrome. Bassethound parses the structure, confirms it matches the standard, and pulls the linked targets into the dossier, so the file is not just present but read.
What separates a standard llms.txt from a technical one?
Not every file at /llms.txt follows llmstxt.org. Some teams drop a plain-text note at that path, an AI-crawler directive, or an access policy borrowed from robots.txt convention. Bassethound classifies what it finds. Standard means the markdown structure parses: a title, a summary, and linked sections in the expected shape. Technical means a file exists at the path but does not match that shape.
The distinction changes what you can conclude. A standard file signals the company read the spec and followed it on purpose. A technical file signals the team wanted something at that path for its own reasons, a weaker and noisier tell. Both are data, and both beat guessing. Report the classification, not a bare present/absent boolean, and you avoid scoring an improvised text file as a deliberate content map. You know which one you are looking at, and you weight it for what it is.
Does llms.txt prove a company ships AI?
No, and that is the honest limit. llms.txt tells models how to read a company. It says nothing about that company building with models. A team can publish a pristine llms.txt and run zero AI in its product. Another can ship a serious AI backend and never touch the file. So treat it as one signal in a group, not a verdict.
Bassethound’s AI-readiness score counts distinct signal groups: model providers, AI SDKs, orchestration, vector stores, observability, and the publishing group that holds llms.txt, MCP endpoints, and .well-known AI files. llms.txt moves the score by one group. It does not carry the weight of a detected vector store or a live MCP endpoint, because those are backend infrastructure, expensive to stand up and pointless to fake. The correct read is correlation. A clean llms.txt next to an Anthropic SDK and a Pinecone client is a company that ships and publishes. A clean llms.txt next to an empty backend is a company that publishes.
What does the first-seen date of llms.txt tell you?
In the deep profile, Bassethound pulls Wayback history to date when the file first appeared. Timing separates the early movers who adopted within months of the 2024 proposal from the recent followers who added it once the convention spread.
An early date correlates with a team that tracks the AI ecosystem and moves on it. A recent date can mean a marketing-driven addition after the fact. Neither is good or bad in isolation, but paired with the other four layers it sharpens the read. Early llms.txt plus real backend infrastructure points to a company that has been building for a while. Recent llms.txt plus an empty backend points to positioning. The date turns a static file into a point on a timeline, and that timeline is where you catch a company whose posture runs ahead of its plumbing.
How does llms.txt fit with .well-known and MCP endpoints?
llms.txt is one file in a family a company can publish for agents. robots.txt sets crawler rules. .well-known/ holds standardized metadata under a reserved path (RFC 8615), including security.txt and agent manifests. A live MCP endpoint is a server an agent can call at runtime. Read together, these show how far a company has gone toward being legible to agents.
llms.txt is the entry level: a static content map anyone can write in an afternoon. An MCP endpoint is the deep end: an active interface an agent invokes, backed by real tooling. A company with llms.txt and a live MCP endpoint is publishing for AI and building for it. Bassethound checks the whole family in one pass and correlates the results, so you see the full posture rather than one file at a time. That correlation is the point. Five separate MCP lookups cost five calls and still cannot show you the story these files tell together.
Bassethound perspective
Most tools that claim to measure AI-readiness stop at “does this domain have an llms.txt” and call the checkbox a score. That is the shallow end. llms.txt is cheap to add, and grading it alone rewards companies that publish over companies that build. A pristine file next to an empty backend earns points it has not earned.
The moat is correlation, not detection. Read llms.txt next to a vector store, an AI SDK, and a live MCP endpoint, in one call, and you learn whether the posture matches the plumbing. A single-layer checker hands you a boolean. Bassethound hands you the whole trail and labels the file standard or technical, so you know what kind of signal you hold. Be clear on the limit too: a static crawl plus optional JS render can miss server-side or proxied files, and the dossier reports that gap instead of pretending the absence is proof. Read the intent in llms.txt, then verify it against the backend. That is the read a checkbox cannot give you.
Sources
- llms.txt proposal: https://llmstxt.org
- Model Context Protocol: https://modelcontextprotocol.io
- Well-Known URIs (RFC 8615): https://datatracker.ietf.org/doc/html/rfc8615
- MDN, robots.txt: https://developer.mozilla.org/en-US/docs/Glossary/Robots.txt
Frequently asked questions
Where does llms.txt live?
At the domain root, /llms.txt. Some sites also ship /llms-full.txt with the full content inlined. Bassethound checks the root path and reports what it finds.
Does every company have an llms.txt?
No. Adoption is early and uneven. Absence tells you little on its own, so Bassethound weighs it against backend signals like SDKs and vector stores.
Is llms.txt the same as robots.txt?
No. robots.txt controls crawler access. llms.txt curates content for language models. Different jobs, different files, and Bassethound reads both.
Can a company fake an llms.txt?
Yes. It is a static text file, cheap to add. That is why it is a posture signal, not proof. Correlate it with backend infrastructure before you trust it.
What is the difference between a standard and a technical llms.txt?
Standard follows the llmstxt.org markdown format, a titled summary with linked sections. Technical is a file at the same path that does not follow that shape. Bassethound labels which one you found.