llms.txt is a new AI-native standard (proposed in 2024) that lets you give AI models a curated, markdown-formatted overview of your website — similar to what robots.txt does for crawlers. Worth 8% of your AEO score, it is the fastest single file you can create to improve AI discoverability.
Publish a plain markdown file at https://yourdomain.com/llms.txt.
Why AI cares: AI models and their tooling increasingly check for llms.txt as a first-pass orientation to a site. It is a low-effort, high-signal addition.
The file must start with an H1 (company/site name), followed by a blockquote summary, then H2 sections for different content areas.
Why AI cares: Format compliance ensures AI parsers can reliably extract the content. Non-standard formatting may cause the file to be ignored.
Every URL referenced in llms.txt should return HTTP 200. Test all links before publishing.
Why AI cares: Broken links in llms.txt signal that the file is unmaintained, which reduces its credibility as a content source.
llms-full.txt is a more detailed version of llms.txt containing full page content for AI models that want deeper context.
Why AI cares: AI models with limited crawl budgets may rely on llms-full.txt instead of crawling individual pages. It is your opportunity to control the full narrative.
Each linked section should have a descriptive label and brief explanation. The file should contain at least 10 links and 300 words.
Why AI cares: Descriptive llms.txt files give AI models more context per link, reducing the need to actually crawl each page.
Add a footer or <link> tag pointing to /llms.txt so crawlers can discover it without prior knowledge of the standard.
Why AI cares: Discovery is a prerequisite for usage. AI crawlers that do not know about the llms.txt standard can still find it via link traversal.
llms.txt is a markdown file published at the root of a website (similar to robots.txt) that is specifically designed for Large Language Models. It provides a structured, human-readable overview of a site's content — with the company description, key pages, and documentation links formatted in a way AI models can parse efficiently.
Adoption is growing. Perplexity has explicitly supported llms.txt, and several AI tooling providers check for it. Even if a model does not directly parse llms.txt today, the format discipline it enforces (clear summaries, structured links) improves the quality of your overall AI-readable content.
Sitemap.xml is a machine-readable list of URLs for crawler scheduling. llms.txt is a human-readable (and AI-readable) narrative overview of your site — it explains what each section is, not just where it is. Think of sitemap.xml as a table of contents and llms.txt as an executive summary.
ansly audits your site across all 7 AEO categories including LLMs.txt & LLM Content. Get your score in under 60 seconds.
Audit my site free →