Does llms.txt work?

No Google ranking benefit. Useful for some agents. llms.txt is a plain Markdown file that lists useful pages for AI tools; Google Search says it does not use these files, so they do not improve visibility or rankings in Google Search, including its AI features.

Nirmal Gyanwali smiling at his desk in the WP Creative office

The short answer

Other tools can use it as a navigation aid; that is a different claim from increasing citations.

I build websites for the agentic web, and I treat llms.txt as a small part of that work. Here is what the evidence supports, where it can be useful and what I would check first.

My first check is whether the file matches the pages it describes. A generator can carry stale titles, broken URLs or unsupported claims into a tidy-looking index. Check the real output, not just whether the plugin is enabled. Fix the source pages, regenerate the file and check it again. The useful job is helping a tool find accurate material; the file cannot turn weak evidence into authority.

Google says Search does not use llms.txt. Ahrefs studied 137,210 domains with traffic in May 2026; about 28% had a valid file, and 97% of those files received no requests that month. That is a historical log finding, not a test of whether adding a file changes citations. The August 2026 proposal focuses on helping agents find useful content, particularly documentation.

Nirmal Gyanwali in his video studio with a lighting setup

What is llms.txt, and how is it meant to work?

Jeremy Howard first proposed llms.txt on 3 September 2024 and revised it to v2 on 10 August 2026. The required part is a Markdown H1 with the site or project name; summaries and linked sections can follow. A file may sit at the root or under a path such as /docs/llms.txt, covering the pages below that path.

The idea is a small guide to useful material, with detail behind the links. Version two also describes Markdown alternatives and standard link relations that help an agent discover the relevant files. It remains a proposal, with adoption varying by tool.

# Example Co
> Website design and development for Australian B2B companies.

## Services
- [AI-ready websites](https://example.com/ai-ready-websites/): what we build and for whom
- [Contact](https://example.com/contact/): book a call
Nirmal Gyanwali at his multi-screen desk with his dog on his lap

What does the evidence say?

  • What the evidence says Google: Search “itself doesn’t use them”; you don’t need “new machine readable files, AI text files, markup, or Markdown to appear in Google Search”
  • Source Google AI optimisation guide, updated 2026-07-10
  • Verdict False

AI crawlers fetch llms.txt

  • What the evidence says Some did in the Ahrefs sample; 97% of valid files had no requests in May 2026. A fetch does not show how the content was used
  • Source Ahrefs, 15 June 2026
  • Verdict Observed, uncommon in this sample

llms.txt increases AI citations

  • What the evidence says The sources reviewed here do not establish a causal citation lift
  • Source Ahrefs and Google
  • Verdict Not established

Some agents use llms.txt for documentation

  • What the evidence says Fetches were observed by Ahrefs; Cloudflare documents using these files in its own documentation work. Neither shows a general citation uplift
  • Source Ahrefs and Cloudflare
  • Verdict Supported for navigation use

It cannot cause any harm

  • What the evidence says Google’s no-harm statement concerns Search visibility. An inaccurate file or exposed private link can still cause practical problems
  • Source Google
  • Verdict Too broad

Caption: Claims checked 26 October 2026. The May crawler sample and the August proposal answer different questions; neither is a controlled test of every assistant.

A newer SE Ranking analysis, published 15 October 2026, examined about 300,000 domains and reported no useful relationship between the file’s presence and domain-level AI citation frequency in its statistical and machine-learning analysis. That is an observational finding, not a before-and-after experiment. The public methodology does not give a complete prompt set or platform and collection-date breakdown, so it cannot settle every use case.

Two caveats from Ahrefs itself: a fetch is not a read, so every figure is a ceiling on real use, and Ahrefs’ customers are more technical than the average website, so adoption in the study is higher than across the web.

Why does this matter for marketing teams?

It matters when a small convenience is sold as a ranking strategy. Ask a supplier which outcome it means: a file fetched, documentation used correctly, a citation, or a qualified visit. Require evidence for that outcome. Technical access, useful material and credible proof deserve the larger share of the effort.

Nirmal Gyanwali working at a window desk in the WP Creative office in Ultimo, Sydney

When is llms.txt still worth having?

If you add one, follow the current proposal, keep it focused, and check its URLs, media type and actual body. A 200 response can still be an HTML error page. Review the generated text after page changes; generation alone does not guarantee accuracy. This site’s llms.txt is a small example.

  • You publish developer documentation or an API, and want coding agents and AI tools to find the right pages fast.
  • You want a tidy, human-readable index of your key pages, and you’ll keep it current.
  • It takes five minutes and you understand it’s a convenience, not a strategy.
  • As a promised Google ranking or AI citation boost. The first is ruled out by Google; the wider citation benefit is not established here.
  • As a way to control AI crawling or training. Set a separate policy using each provider’s documented controls; robots.txt is not authentication and does not secure private material.
  • If nobody will maintain it. An out-of-date file sends tools to the wrong pages.

Worth it

Not worth it

Nirmal Gyanwali presenting at a whiteboard during a workshop

llms.txt vs robots.txt vs sitemap.xml

llms.txt

  • Purpose Suggest key pages to AI tools
  • Format Markdown
  • Status Voluntary proposal, v2 August 2026
  • Used by Google Search No
  • Controls AI crawlers No
  • Effect on AI citations No causal lift established here

robots.txt

  • Purpose Allow or block crawlers
  • Format Plain-text rules
  • Status Long-standing convention, formalised as RFC 9309
  • Used by Google Search Yes
  • Controls AI crawlers Rules for compliant crawlers; user-triggered access varies by provider
  • Effect on AI citations Search-bot restrictions can limit retrieval; training controls are separate

sitemap.xml

  • Purpose List URLs for search engines
  • Format XML
  • Status Long-standing protocol, supported by major search engines
  • Used by Google Search Yes
  • Controls AI crawlers No
  • Effect on AI citations Discovery aid, not a citation guarantee

Caption: What the three files are used for.

Common misconceptions about llms.txt

“It’s the robots.txt for AI.” It is a content guide. robots.txt expresses crawl preferences for bots that follow it; authentication protects private content. Neither function is supplied by llms.txt.

“Google recommends it.” Google says the opposite: you don’t need it to appear in Search, and Search doesn’t use it.

“Every big site has one, so it must work.” Adoption isn’t evidence. Many files exist because tools and plugins generate them automatically.

“No requests means it’s broken.” Not necessarily. It can mean no qualifying request was observed in the window you measured. Check what your logs capture before drawing a wider conclusion. Cached or indirect access may not appear in that record.

Nirmal Gyanwali, founder of WP Creative, working on a laptop at his desk

Why can a readiness scanner recommend it if Google ignores it?

A scanner may be checking whether an agent can navigate documentation, not whether Google will rank a page. Chrome’s Agentic Browsing category is informational and unbenchmarked. Cloudflare’s documentation work describes llms.txt alongside Markdown delivery and navigation changes. Its reported efficiency results cover that bundle; they do not isolate llms.txt or measure a Google citation lift.

For a documentation experiment, compare the same questions with and without the guide and record correct answers, time, tokens, version and failures. That is a proposed test, not a result I have measured. For a marketing site, first ask whether agents need a guided path beyond the clear pages you already publish.

Nirmal Gyanwali relaxed in an armchair in a podcast studio

What should you do instead?

  1. Choose crawler access deliberately. Search, training and user-triggered tools have different jobs. Follow current provider documentation and match the CDN policy.
  2. Test raw and rendered content. Some fetch tools do not execute JavaScript; Google can process it. Find a real failure before prescribing a rebuild.
  3. Answer buyer questions plainly with evidence, useful detail and clear internal links. Google Search remains relevant to its AI features; do not equate a ranking with a citation.
  4. Earn mentions on the sites assistants already cite: directories, industry press, podcasts, YouTube.
  5. Make your site agent-ready if you want agents to act on it, not just read it.

Want a straight answer on what moves AI visibility for your site? Book a free call. You’ll talk with me or my team, and my team at WP Creative builds the technical foundation.

Nirmal Gyanwali, founder of WP Creative

Questions about llms.txt

Does Google use llms.txt?

No. Google’s guide to AI features says Search doesn’t use llms.txt or other AI text files, and that creating one won’t help or hurt your visibility in Search. It’s fine to have one for other services. It won’t change your Google rankings or your chances of appearing in AI Overviews.

Does ChatGPT read llms.txt?

Ahrefs’ May 2026 data recorded GPTBot fetching some files. GPTBot is OpenAI’s training crawler, so that is not evidence that ChatGPT Search used the file or cited the site because of it. OpenAI documents OAI-SearchBot and ChatGPT-User separately. A tool can also read a file when a user explicitly asks; none of that establishes a citation boost.

Should I add llms.txt to my WordPress site?

Only if the maintenance cost fits a real use. A documentation site used by coding agents is a clearer case than a small brochure site. Yoast SEO and Rank Math offer generators; check the output and keep private URLs out. It will not improve Google Search visibility. The wider foundation is on AI-ready websites.

What is llms-full.txt?

llms-full.txt is a convention used by some documentation publishers to put a larger body of content in one file. The current llms.txt v2 proposal instead describes a small guide with relevant detail behind links. A full export has different size and maintenance costs. It is not a Google ranking requirement, and I would add one only for a tool or workflow that actually needs it.

Will llms.txt stop AI from training on my content?

No. llms.txt is a content guide. Set future crawl preferences separately for each provider, and protect private material with access controls. Blocking a training crawler does not necessarily block that provider’s search crawler, and a new rule does not undo earlier training. Check the provider’s documented scope and your CDN response.

Does nirmal.com.au have an llms.txt?

Yes. This site serves a small one, generated from the live pages so it stays current. I keep it as a convenience for agents, with no proven effect on search. I haven’t measured who fetches it yet, so I make no claim that anything reads it.

Got a website that should be doing more?

Tell me what the website has to earn. If my team can help, we’ll show you how. If we can’t, we’ll tell you who can.