We build llms.txt files for clients. We built one for our own site. So, you should read what follows with that in mind, and then notice that we are about to argue against most of what our industry says about this file, including things we have said ourselves.
Here is the short version, for anyone who does not want the full explanation. An llms.txt file is a plain text document at your domain root that gives AI systems a curated summary of your site. It is a reasonable idea. Google has publicly stated that it does not use these files at all and that having one will neither help nor hurt your visibility in Google Search. No major AI platform has formally committed to reading them. Independent analyses of large sets of AI-cited pages have not found any measurable citation advantage from having one. It is worth implementing if you can maintain it, as cheap insurance against where the web might go, and it is not a strategy. If an agency is quoting you for AI optimization and this file is a headline deliverable, you are paying for paperwork.
Now the long version, because the details are genuinely interesting and the honest answer is more useful than the sales answer.
What an llms.txt File Actually Is
The file lives in your root directory, accessible at yoursite.com/llms.txt, and it is written in plain text, usually formatted as Markdown for readability. It was proposed in 2024 as a way to give large language models a clean, curated view of a site rather than making them infer everything from navigation, boilerplate, cookie banners, and whatever else clutters a modern web page.
The distinction from robots.txt is worth getting right, because the two are constantly conflated, including in a lot of published explanations. Robots.txt is a permission system; it tells crawlers which parts of your site they may access. It is a door policy. An llms.txt file is a briefing; it says here is who we are, here are the pages that matter, here is what each one contains, here is how to describe us accurately. One controls entry. The other offers context to whoever comes in. It is also worth knowing that llms.txt is a community convention rather than a formal standard; there is no RFC, and the spec is maintained through the llmstxt.org project rather than by a standards body.
A well-built file typically contains a description of the organization, the services or products explained in plain language, key facts such as location, credentials, and founding details, links to the most important pages with a sentence explaining what each contains, the geographic markets served, and the areas where the business genuinely has expertise. For an agency like ours, that would cover the service lines, the areas we work in across LA, Orange County, San Diego, San Francisco, and nationally, our Premier Google Partner status, and links to the service pages, case studies, and team information.
What Google Says, and It Is Not Ambiguous
On May 15, 2026, Google Search Central published official guidance on optimizing for its generative AI features. It includes a mythbusting section, and llms.txt is the first item in it. Google states that you do not need to create AI text files, special markup, or Markdown to appear in Google Search or its generative capabilities, because Google Search does not use them. It goes on to say that creating and maintaining an llms.txt file for other services is completely fine, and that doing so will neither harm nor help your visibility or rankings in Google Search, because Google Search ignores it.
That is Google speaking about AI Overviews and AI Mode, which together reach an enormous share of your customers; Google confirmed at I/O in May that AI Mode alone has passed a billion monthly users. Google Search Advocate John Mueller has been publicly skeptical of the file for some time. If you were told this file was how you get into AI Mode, you were told something that was not true, and it was not true before the guidance was published either.
What the Independent Evidence Shows
Google only speaks for Google, so the fair question is whether the other platforms use it. The answer, as of mid-2026, is that none of the major AI platforms has formally committed to consuming llms.txt, and the measured evidence is not encouraging.
One analysis of more than ninety thousand AI-cited URLs found no measurable citation uplift associated with llms.txt adoption. Another team ran a machine learning model to test whether the presence of the file predicted citation frequency and found that removing the variable actually improved the model accuracy, meaning the file was adding noise rather than signal. Adoption itself is thin and easy to misread; the file sits at roughly five to six percent of the top ten thousand sites, and that number is heavily inflated by Shopify quietly deploying llms.txt across every store on its platform in late April and early May 2026, without merchant opt-in, acknowledged in a developer changelog weeks later. When people tell you adoption is surging, that is a substantial part of what they are looking at.
There is a legitimate use case, and it is narrower than the marketing suggests. For developer documentation, API references, and any property where coding assistants and autonomous agents actively ingest structured content, the file has real practical value. Anthropic, Cloudflare and various developer platforms adopted it early for exactly this reason. If your site is documentation, this conversation is different. If your site sells plumbing services in Torrance, it is not.
So, Should You Have One?
Our honest position: yes, probably, with the right expectations. The cost is close to zero if the file is generated properly and maintained accurately. Google has confirmed it cannot hurt you. The agent-facing web is clearly growing, and Google own guidance now includes a section on agentic experiences, so a curated machine-readable summary of your business may prove useful in ways nobody has fully mapped yet. That is a reasonable case for cheap insurance.
What it is not is a lever. It will not get you cited by ChatGPT. It will not put you in AI Mode. It is not the reason your competitor is showing up and you are not. And there is one genuine risk worth naming: an inaccurate or stale llms.txt is worse than no file at all, because you have volunteered a description of your business that is wrong. If nobody owns keeping it current, do not publish it.
The robots.txt Work That Actually Matters
This is the part of the original technical package that genuinely does something, and it deserves more attention than the file everyone talks about. Your robots.txt should reflect a deliberate decision about which AI crawlers you permit. The ones that matter for most businesses are GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, and CCBot for Common Crawl, which feeds a range of datasets. Applebot-Extended governs Apple Intelligence training.
One important correction to advice you have probably read, including in earlier versions of this article. Google-Extended does not belong in that list the way it is usually presented. Google-Extended is a control governing whether your content is used for training and grounding in Gemini and related products. It does not control AI Overviews or AI Mode, which are served from the regular Search index through Googlebot. Blocking Google-Extended does not remove you from AI Overviews, and allowing it does not get you in. rankThis distinction is wrong in a remarkable amount of published advice, and getting it right is a decent test of whether whoever is advising you has read the documentation.
A second correction. It is often claimed that older robots.txt configurations block AI crawlers by default, which is why your site is invisible to AI. This is not generally true. A standard WordPress installation does not block GPTBot or ClaudeBot. Some sites block them deliberately, some hosts and security plugins add rules, and it is absolutely worth checking your file rather than assuming. But the idea that most businesses are accidentally locked out by a default setting is a scare that sells audits.
Permitting these crawlers is a real decision with a real tradeoff, not a checkbox. Allowing them means your content can be used, including for training. Blocking them protects your content and removes you from consideration on those surfaces. For a marketing site trying to be found, permitting is almost always right. For a publisher whose content is the product, it genuinely is not. Make the call on purpose.
What to Do Instead, or Rather, As Well
If the file is not the lever, what is? The same unglamorous things that have been the answer for two years. Content carrying something a language model could not have produced on its own, meaning original data, first-hand experience, real numbers from real projects. Descriptions of your business that are accurate and consistent everywhere they appear, because inconsistency across directories, review platforms, and social profiles is the single most common reason a model describes a business wrongly. Genuine standing in the places your customers actually discuss your category. Technical health good enough that your pages are indexed and eligible to be surfaced at all, which is a hard requirement for Google AI features.
None of that fits in a text file, which is precisely why the text file became so popular.
Frequently Asked Questions
What is an llms.txt file?
An llms.txt file is a plain text document, usually formatted in Markdown, placed at the root of your website. It gives AI systems a curated summary of your organization, your services, and your most important pages, so they have clean context rather than inferring everything from your navigation and page furniture. It was proposed in 2024 and remains a community convention rather than a formal standard; there is no governing body and no RFC.
Does llms.txt actually work? Does Google use it?
Google does not use it. Google Search Central guidance, published May 15, 2026, states that site owners do not need AI text files to appear in Google Search or its generative AI features because Google Search does not use them, and that maintaining one will neither help nor hurt visibility or rankings in Google Search. No other major AI platform has formally committed to reading the file either. Independent analyses of large sets of AI-cited URLs have found no measurable citation advantage from having one.
Is llms.txt the same as robots.txt?
No. Robots.txt is a permission system that tells crawlers which parts of your site they may access; it is a door policy, and it is honored by the major AI crawlers. An llms.txt file offers context and description rather than permission, telling AI systems what your business is and which pages matter. Robots.txt does something enforceable. An llms.txt file makes an offer that platforms are free to ignore, and most currently do.
Should my business have an llms.txt file?
Probably, with realistic expectations. It costs very little, Google has confirmed it cannot hurt you, and the agent-facing web is growing in ways that may make a curated machine-readable summary useful later. Treat it as cheap insurance rather than a visibility lever. One caveat: an inaccurate or outdated file is worse than none, because you have published a wrong description of your own business. If nobody will maintain it, skip it. Developer documentation and API reference sites are the exception, where the file has genuine present-day value.
Which AI crawlers should I allow in robots.txt?
For most businesses wanting AI visibility: GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, and CCBot for Common Crawl. Note that Google-Extended is commonly misunderstood; it governs training and grounding for Gemini and related products, and it does not control AI Overviews or AI Mode, which are served from the regular Search index through Googlebot. Permitting AI crawlers is a genuine tradeoff, since it allows your content to be used including for training, so decide deliberately rather than defaulting.
If you came here looking for confirmation that an llms.txt file will fix your AI visibility, we are sorry to be the ones to tell you otherwise. The upside is that you now know something most of your competitors do not, which is where the money is actually going.
LAD Solutions builds complete GEO technical packages, and yes, that includes llms.txt and a properly considered robots.txt, because the technical hygiene is worth having and costs little. But the reason our clients get cited is not the files. It is the content nobody else could have written, the entity data cleaned up across forty places nobody had looked at since 2021, and the reputation work in the places their customers actually read. That work is slower and less impressive on a deliverables list, which is exactly why it still works.
If you want an honest assessment of where your AI visibility actually stands, and a plan that spends your money on the parts that move, contact us here or call 844.523.2556 today. If your current agency is charging you for AI-ready files as a headline service, bring the invoice.
