llms.txt: the complete guide for AI crawlers

What llms.txt is

llms.txt is a proposed standard — a plain-text (Markdown) file at the root of your site (yourdomain.com/llms.txt) that points AI systems to your most important, clean content. Think of it as a curated map for language models: instead of crawling messy HTML, an AI can read a concise, structured list of your key pages and summaries.

It's the AI-era cousin of robots.txt and sitemap.xml, but with a different goal: robots.txt controls access, sitemaps list URLs, and llms.txt curates the content you most want AI to understand and quote.

What goes in an llms.txt file

A typical llms.txt is short and human-readable:

• An H1 with your site/brand name, and a short blockquote summary of what you do.
• Sections (## headings) grouping links to your most important pages — docs, key products, guides — each as a Markdown link with a one-line description.
• Optionally an ## Optional section for secondary links an AI can skip if it's short on context.

Is it worth adding?

Adoption is still early and not every AI engine reads llms.txt yet, so treat it as low-cost insurance rather than a guaranteed ranking boost. It's quick to create, it can't hurt, and it signals that your content is structured and citable — which aligns with everything else that helps AI quote you. The bigger wins still come from clean, answer-first pages and entity authority.

Practical tip: keep it accurate and current, link only to canonical pages, and don't stuff it — a focused llms.txt is more useful to a model than an exhaustive one.

Check your AI readiness

Beyond llms.txt, what matters is whether AI crawlers can actually reach and parse your pages. CiteLens includes a GEO readiness check that tests crawlability (can GPTBot, ClaudeBot and PerplexityBot reach a page) and content structure — so you can fix what's blocking AI from citing you. Free plan to start.