BLOG / llms-txt-history-agent-readable-web.mdx
From llms.txt to an agent-readable web: the useful lesson for commerce
A dated history of llms.txt, Markdown access, AGENTS.md, and what SKU.md should learn without overstating adoption.
llms.txt did not become a universal standard. Its useful lesson is simpler: publishers need concise discovery, readable source material, and explicit scope. One resource may not handle all three jobs.
Here, SKU.md means merchant-hosted product discovery and stable product knowledge: agents can find products, understand sourced facts, and know where to recheck live commerce. It is not a new checkout protocol.
The original problem was context, not commerce
Jeremy Howard’s original proposal was published on 2024-09-03. It described a practical problem: whole sites are too large for a model context window, while navigation, scripts, and presentation make HTML expensive to reduce accurately. The proposed /llms.txt file gives a short Markdown overview plus selected links; clean Markdown variants make linked pages easier to read.
FastHTML used the proposal itself, including expanded context files, and nbdev began generating Markdown versions of documentation pages. That self-use matters: it tested discovery and retrieval as a workflow. It did not prove that every model would fetch the file or treat it as authoritative.
Implementations diverged from 2024 to 2026
| Period | Documented implementation | What it added |
|---|---|---|
| 2024-09 | llms.txt proposal and repository | Root overview, curated links, optional context expansion |
| 2024 to 2025 | Mintlify generation | Generated llms.txt, llms-full.txt, page descriptions, and Markdown page links |
| 2026 | Cloudflare AI consumability | Global, product-level, and full-text tiers plus per-page Markdown |
| 2026 | Vercel Markdown access | A .md representation and copy-as-Markdown flow for each docs page |
| 2026 | Shopify agent resources | Shopify-managed agents.md with theme overrides for its own agent and llms paths |
The dates describe the published pages and practices we reviewed, not a single coordinated rollout. Mintlify may call llms.txt an industry standard, while the originating site still presents an open proposal. Across the ecosystem, the accurate description is a proposal with real implementations, not a formal Web standard.
The agent-readable web is already plural
llms.txt curates readable content. AGENTS.md gives coding agents repository instructions such as setup, tests, and contribution rules. Sitemap lists crawlable URLs. Product JSON-LD describes page-level product entities. Cloudflare and Vercel expose Markdown representations without requiring one shared discovery design.
That branching is healthy when each file has a clear owner. It becomes risky when a filename is treated as evidence of adoption, truth, or permission. A consumer still has to know who published the data, how fresh it is, whether identifiers match, and which live system owns an action.
What SKU.md should borrow, and where it should stop
SKU.md should copy the small entry point, readable documents, stable links, and testable delivery contract. It should add commerce-specific identity, evidence, limitations, and explicit handoffs back to live systems.
It should avoid presenting a community proposal as platform adoption, keep volatile offers out of unbounded prose, and never assume that discoverability guarantees use. A product document can improve inspection without becoming a ranking signal or checkout authority.
Further reading
Read What is SKU.md? for the commerce-specific category, and compare the responsibilities in llms.txt vs agents.md vs SKU.md.