What is llms.txt, in plain English?
llms.txt is a Markdown discovery file you publish at yourdomain.com/llms.txt to surface your canonical pages to large language models. It is curation, not enforcement: you list the pages you want LLMs to prioritize, with one-line descriptions, grouped by section. The Answer.AI proposal by Jeremy Howard introduced the spec; the canonical reference lives at llmstxt.org. If you want the full breakdown of the spec, provider support, and decision matrix, the longer playbook is here.
Think of it as three files doing three different jobs:
- llms.txt — a curated index, like a hand-written sitemap for LLMs.
- llms-full.txt — the full content concatenation of those pages, ready for direct context-window ingestion.
- robots.txt — allow/disallow rules for crawlers. Enforcement, not curation.
The four invariants every llms.txt must satisfy:
- One
#H1 line with the site name. - One
>blockquote with a short summary. - At least one
##section header. - Every link is an absolute https:// URL inside a Markdown link.
What llms.txt does NOT do
The honest read on this file matters. Four limits worth naming up front.
1. It is not a Google ranking factor
John Mueller dismissed llms.txt as a Google ranking signal in 2024. Google Search does not parse it. Google AIO does not document any first-party fetch of it. Publishing one will not move you up the SERP - if anyone tells you otherwise, they are confusing correlation (sites that publish llms.txt also tend to do other technical SEO well) with causation.
2. It is not a guarantee of citation
Anthropic's claudebot and PerplexityBot have been observed fetching llms.txt on documentation sites in production logs through 2025-2026. That means retrieval-stage discovery, not automatic citation. AI citation is driven mostly by entity recognition (where schema markup carries weight) and brand-mention volume across the open web - off-site brand mentions remain the strongest AI-citation lever per recent correlation analyses.
3. It is not a replacement for robots.txt or sitemap.xml
If you block GPTBot, ClaudeBot, or PerplexityBot in robots.txt, none of them will ever fetch your llms.txt. robots.txt is enforcement; llms.txt is advisory. They are complementary, not competitive. Robots.txt also lets you control which AI crawlers can actually fetch your pages, and our free in-browser builder ships with the 2026 AI-bot presets already wired in.
4. It is not universally supported
As of 2026, fewer than half of the major AI engines fetch llms.txt directly. Treat it as a high-leverage signal for the engines that do (Claude, Perplexity, retrieval-aware IDE agents) and a near-zero-cost signal for the ones that do not. Also note: entity resolution for AI engines still depends on a clean JSON-LD graph (Organization plus WebSite plus WebPage nodes) in your page <head> - schema does the work llms.txt does not.
How to use the file you just generated
Five steps, total time well under five minutes.
- Copy or download the file using the buttons above the output panel.
- Place it at your site root. Next.js: public/llms.txt. WordPress: document root via FTP. Vercel and Cloudflare Pages: /public folder.
- Deploy. Ship the change. The file becomes reachable at https://yourdomain.com/llms.txt.
- Verify with curl. Run curl -I https://yourdomain.com/llms.txt and confirm a 200 response with Content-Type text/markdown or text/plain.
- Optionally serve llms-full.txt. Flip the long-descriptions toggle, paste each page body into the skeleton, and serve the result at /llms-full.txt.
Serpbays publishes its own file at /llms.txt - use it as a working reference if you want to see a real production example.
Which LLM providers actually fetch llms.txt today?
An honest read of provider support as of mid-2026. No inflation.
| Provider | Fetches llms.txt? | Notes |
|---|---|---|
| ChatGPT | Not directly | Secondary discovery via OAI-SearchBot retrieval, not a documented first-party fetch. |
| Claude (Anthropic) | Yes | claudebot has been observed fetching llms.txt on documentation sites in server logs. |
| Perplexity | Yes | PerplexityBot fetches root discovery files, including llms.txt. |
| Google AIO | Not documented | No first-party llms.txt fetch behavior published; Google uses its own index. |
| IDE agents (Cursor, Windsurf) | Yes | Active fetch via MCP servers and built-in docs-discovery flows. |
Next steps after publishing
If you publish blog content, your llms.txt only does half the job. AI engines decide which of your pages to cite based on entity recognition (schema markup) and crawler permission (robots.txt). Pair this file with our JSON-LD schema generator and the AI-aware robots.txt generator, then push the heaviest lever by earning the off-site brand mentions that actually drive AI citation. File-format optimization is necessary; brand-mention volume is decisive.