An llms.txt file is a single Markdown file at your site's root (/llms.txt) that gives AI agents a curated map of your most important pages instead of forcing them to parse full HTML navigation, ads, and scripts.
The only required part of the file is an H1 title. A blockquote summary, optional body paragraphs, and H2 sections of markdown links are all optional but recommended by the canonical spec.
llms-full.txt is a separate, companion file that compiles your entire site's text into one Markdown document — a convention Mintlify developed with Anthropic that later became part of the official proposal, and data from Mintlify-hosted docs shows agents request it more than twice as often as llms.txt.
Creating one manually takes four steps: inventory your key pages, write the H1 and summary, group links under H2 headings, and publish the file at your root. For a site with more than a few dozen pages, generating llms-full.txt by hand stops being practical.
Google has been explicit that llms.txt does not affect search rankings. Google's own generative-AI guidance says sites don't need new machine-readable files to appear in AI search results, and Google's John Mueller stated directly that llms.txt "is not done for search."
Chrome Lighthouse now checks for llms.txt anyway, just not for SEO. An experimental "Agentic Browsing" category added in Lighthouse 13.3 validates llms.txt presence and format alongside WebMCP annotations and layout stability, framing the file as an agent-compatibility signal rather than a ranking factor.
Several documentation platforms generate the file automatically, including Mintlify, GitBook, and static-site plugins like vitepress-plugin-llms and docusaurus-plugin-llms, so check your platform before writing one by hand.
An llms.txt file is a single Markdown file, ideally placed at yoursite.com/llms.txt, that gives AI agents and language models a curated map of a site's most important pages instead of forcing them to parse full HTML navigation, ads, and scripts. The canonical specification at llmstxt.org describes the format precisely: a Markdown file that gives an agent "background information, guidance, and links to detailed markdown files" instead of a raw HTML tree.
The file exists because AI agents and language models read websites far less efficiently than humans browsing with a mouse. A model that needs a pricing page or API reference has to fetch full HTML, strip out navigation bars, cookie banners, and embedded scripts, and guess which links actually matter — work that a curated Markdown index eliminates in one request. The file isn't tied to any single framework or hosting provider: anyone who can add a file to a web server's document root can publish one, whether that site runs on a static-site generator, a custom CMS, or a hand-rolled server.
Documentation sites, developer tools, and any product that expects to be looked up by an AI coding agent or chatbot need an llms.txt file most, because those are exactly the audiences trying to answer a specific technical question quickly. A support portal that gets referenced by customer-facing AI assistants, an open-source project whose README gets pasted into a coding agent, and a SaaS product with an API that developers integrate with the help of an LLM all fit this pattern. Content-heavy marketing sites with little unique technical detail get less out of the file, since there's less structured reference material worth curating in the first place.
The practical benefit shows up in two places: agents that support llms.txt spend less time and fewer tokens crawling a site to find the same handful of pages a human would bookmark, and teams get a single, versioned file they can update deliberately instead of hoping a general-purpose crawler picks the right pages out of a full sitemap.
How to Create llms.txt: Three Approaches
Three approaches cover almost every site, and the right one depends on how your content is already structured. Writing llms.txt by hand works well for small sites and single-page projects, where a developer can list every page that matters in fifteen minutes — this is the approach the FastHTML example on llmstxt.org demonstrates. Platform auto-generation is the least effort option if your documentation already runs on Mintlify, GitBook, or a static-site generator with a maintained plugin (vitepress-plugin-llms and docusaurus-plugin-llms cover the two most common static-site generators), since the platform regenerates the file whenever your docs change. Crawling your own site works best for larger sites without a supported plugin — marketing sites, custom CMSs, or documentation split across multiple subdomains — where manually copying dozens or hundreds of pages into Markdown isn't realistic.
Take a Quick Look
Building llms-full.txt for a site with hundreds of pages means converting every one of them to clean Markdown — Nstproxy Crawl does that conversion in one API call per page, with JavaScript rendering included for pages that don't work with a plain HTTP fetch.
For the crawled approach specifically, Nstproxy Crawl — launched as a dedicated AI web crawling API — is built for turning a list of URLs into the clean Markdown that both llms.txt links and llms-full.txt's full-text dump require. It's a REST API that fetches a page, renders any JavaScript the page needs, and returns cleaned Markdown or HTML in one call, with a site-level crawl mode that walks a domain under explicit maxDepth and maxPages bounds instead of wandering into pagination or login pages you never meant to include. That bounded site-level crawl is the part that matters here: it gives you the raw material for llms-full.txt without hand-copying every page, and you keep full control over which pages actually make it into the curated llms.txt links.
Site-level crawling with explicit bounds — set maxDepth, maxPages, and include/exclude URL rules so a crawl targeting your docs subdomain doesn't pull in your marketing blog or account-login pages.
Clean Markdown output per page — each crawled page returns as Markdown ready to paste into llms-full.txt, without the manual work of stripping navigation and boilerplate HTML.
JavaScript rendering included — pages built with a client-side framework render fully before conversion, so you're not left with an empty shell for any page that loads content after the initial HTML response.
What You Need Before You Start
Creating an llms.txt file requires write access to your web server's document root (or the ability to add a static file through your hosting platform, CMS, or static-site generator's public directory), since the file has to be reachable at yoursite.com/llms.txt to do anything useful. You'll also need an inventory of the pages worth including — most teams can list their top 10 to 30 pages from memory, but a full-text llms-full.txt file benefits from a complete list pulled from your sitemap or CMS. No special software is required for a hand-written file beyond a plain text editor; the crawled approach additionally needs an API key for whichever crawling service you use.
Prepare Your Tools and Workspace
Start by deciding which of the three approaches from the previous section fits your site, since that choice determines the rest of your setup. If your documentation platform already supports auto-generation, check its settings for an "llms.txt" or "AI-readable docs" toggle before doing anything manually — Mintlify, GitBook, and several CMS SEO plugins (Yoast SEO and AIOSEO among them) turn this on without any extra installation. If you're writing the file by hand, no setup is needed beyond a Markdown-capable text editor. If you're crawling an existing site to build llms-full.txt, install your crawling service's SDK (for example, npm install @nstdata-ai/crawl or the equivalent Python package) and keep your API key out of any file you plan to commit to a public repository.
Step 1: Inventory Your Most Important Pages
List every page an agent or developer would actually want when trying to understand or use your site, not every page that exists. For a documentation site, this usually means your quickstart guide, API reference, core concept pages, and a handful of worked examples; for a product site, it means your pricing page, key feature pages, and any developer-facing docs. Leave out paginated listings, login pages, and legal boilerplate — the point of the file is curation, and a list padded with low-value pages defeats that purpose.
Step 2: Write the H1 Title and Summary
Open the file with a single H1 heading naming your site or project — this is the only section the specification actually requires. Follow it immediately with a blockquote (a line starting with >) that summarizes what the project is and who it's for in one or two sentences, since this is the first and sometimes only context an agent reads before deciding whether to follow any links at all.
# Your Project Name> Your Project is a [one-sentence description of what it does and who it's for].
Step 3: Group Your Links Under H2 Categories
Break your inventoried pages into H2-headed groups such as ## Docs, ## Examples, and ## API Reference, then list each page as a markdown link with a short note describing what it contains. The specification treats an ## Optional section as a convention for secondary links an agent can skip when it needs a shorter context window, so put anything genuinely non-essential there rather than mixing it into your primary sections.
## Docs-[Quickstart](https://yoursite.com/docs/quickstart.md): Get a working example running in under five minutes.
-[API Reference](https://yoursite.com/docs/api.md): Full list of endpoints, parameters, and response fields.
## Optional-[Changelog](https://yoursite.com/changelog.md): Version history, useful for context but not required to use the product.
Step 4: Add llms-full.txt for the Full Text (Optional)
Compile the full text of your key pages into a second file at /llms-full.txt if you want agents to be able to load your entire documentation in one request instead of following links one at a time. The convention comes from a real, documented origin rather than informal practice: Mintlify's own account of the format's history states it developed the full-text file with Anthropic before it was folded into the official llms.txt proposal, and Mintlify reports that agents visiting its hosted docs request llms-full.txt more than twice as often as the curated llms.txt. For a small site, generating it can mean manually concatenating each page's Markdown source. For a larger site, run a bounded site-level crawl against your own domain, convert each returned page to Markdown, and concatenate the results in the same order your llms.txt links appear, so the two files stay logically consistent with each other.
Step 5: Validate and Publish Your File
Check your file against the spec's structural rules before publishing: an H1 present, a blockquote summary if you included one, and every list item formatted as a proper markdown link rather than a bare URL. Then upload llms.txt (and llms-full.txt, if you created one) to your site's document root so they resolve at yoursite.com/llms.txt and yoursite.com/llms-full.txt. If your site serves distinct sections from different subpaths — a /docs subdirectory with its own scope, for instance — the specification allows a more specific llms.txt at that subpath to take precedence over the root file for URLs under it.
Step 6: Test It With a Real AI Agent
Give an AI agent or chatbot only your llms.txt file as a starting point and ask it questions a real visitor would ask, then check whether it can answer correctly using just the links and descriptions you provided. If it consistently follows the wrong link or asks for information your file should have covered, that's a sign your descriptions are too vague or an important page is missing — treat this as a normal editing pass rather than a one-time check, since the file needs the same maintenance as any other piece of documentation.
The Complete llms.txt Format, With an Example
A complete, spec-compliant llms.txt file follows this structure, in this order:
# Project Name> One or two sentence summary of the project.
Optional paragraph or two of additional context that doesn't fit in the summary.
## Docs-[Page title](https://example.com/page.md): Optional one-line note about the page.
## Examples-[Example title](https://example.com/example.md): Optional one-line note.
## Optional-[Secondary resource](https://example.com/secondary.md): Optional one-line note.
Only the H1 is required; everything else — the blockquote, body paragraphs, and H2 sections — is optional but improves how usefully an agent can act on the file. Note that not every real-world implementation follows this strictly: Anthropic's own platform documentation llms.txt file skips the H1 and blockquote entirely and goes straight into H2 categories with markdown links, so treat the full spec as the target rather than a hard requirement every agent will enforce.
What llms.txt Can't Do: SEO and Other Limits
llms.txt does not influence Google search rankings, and Google has said so directly rather than leaving it ambiguous. Google's generative-AI optimization guidance states that sites don't need new machine-readable files, AI text files, or Markdown to appear in generative AI search results, and Google's John Mueller said on Bluesky in January 2026 that llms.txt "is not done for search" — a presence of the file on a Google-run site isn't an endorsement of the format either.
That doesn't make the file pointless, but it does mean the payoff is different from what many SEO-focused explanations imply. Chrome Lighthouse added an experimental "Agentic Browsing" audit category in Lighthouse 13.3 that checks for llms.txt presence and validates it against the spec's recommendations (an H1 header, sufficient length, and included links), alongside checks for WebMCP tool annotations, accessibility tree structure, and layout stability. That's a signal about how well a site works with AI agents specifically, separate from and unrelated to search visibility.
The other real limit is adoption and enforcement: there's no single authority verifying that any given AI agent actually reads or respects your llms.txt file, so publishing one is a bet that the agents visiting your site support the convention rather than a guarantee of a specific outcome. Treat it as complementary infrastructure for agent-facing use cases, not a replacement for your existing sitemap.xml or robots.txt, which continue to serve search engines and general crawlers the way they always have. If some of the terminology in this guide is unfamiliar, Nstproxy's glossary of web-data and AI-agent terms covers related concepts like crawling, rendering, and structured extraction in more depth.
Conclusion
Creating an llms.txt file comes down to writing one required H1, an optional but recommended summary and set of categorized links, and publishing the result at your site's root — a task that takes minutes by hand for a small site and benefits from a bounded crawl for a larger one. It won't move your search rankings, since Google has said plainly that it doesn't, but it does give AI agents a faster, more reliable way to understand your site, and Chrome Lighthouse now checks for it as part of judging how agent-friendly a site is. Publish the file, test it against a real agent, and revisit it on the same schedule you'd use for any other documentation.
Q: Where exactly should I upload my llms.txt file?
Upload it to your site's document root so it resolves at yoursite.com/llms.txt; you can also publish a more specific version at a subpath like /docs/llms.txt, which takes precedence over the root file for URLs under that path.
Q: Do I need both llms.txt and llms-full.txt?
No — llms.txt alone is a complete, spec-compliant file, but adding llms-full.txt lets agents load your entire documentation's text in a single request instead of following individual links, which is worth the extra effort for larger documentation sites.
Q: Will an llms.txt file improve my Google search rankings?
No, and Google has confirmed this directly: its generative-AI optimization guidance says new machine-readable files aren't needed to appear in AI search results, and Google's John Mueller has stated that llms.txt isn't used for search.
Q: Does my documentation platform already generate this file for me?
Possibly — Mintlify, GitBook, and several CMS SEO plugins (Yoast SEO, AIOSEO) generate llms.txt automatically, and static-site generators have maintained plugins like vitepress-plugin-llms and docusaurus-plugin-llms, so check your platform's settings before writing one by hand.
Q: How often should I update my llms.txt file?
Update it whenever you add, remove, or significantly restructure the pages it references, the same way you'd maintain a sitemap — a stale file that links to removed pages is worse for an agent than no file at all.
Q: Can I have different llms.txt files for different sections of my site?
Yes — the specification allows a more specific llms.txt at a subpath to take precedence over your root file for URLs under that subpath, which is useful if your docs, blog, and marketing pages need different curation.
Q: Is there a required minimum length for llms.txt?
The specification itself sets no minimum, but Chrome Lighthouse's Agentic Browsing audit checks for "sufficient length" alongside an H1 header and included links, so an extremely short file may fail that specific check even though it's technically spec-compliant.
Q: What happens if I don't create an llms.txt file at all?
Nothing breaks — the file is entirely optional infrastructure, and agents without llms.txt support simply fall back to parsing your site's regular HTML the way they already do; you're only leaving a convenience on the table for the subset of agents that do support it.
Marcus Chen
Aug. 13th 2026
110M+ real IPs with 99.9% access success
Blazing-fast average response ~0.5s for high-concurrency tasks
From only $0.1/GB
Get immediate access to premium residential, datacenter, IPv6 and ISP proxy pools.
