A few weeks ago, Twitter (sorry, X) went wild over a new miracle file: llms.txt. The pitch? Control what ChatGPT, Claude or Perplexity scrape from your website. The reality? A little less glorious.
Here is why you should not bet everything on it, and what actually works today.
llms.txt, the idea that sounded good
The concept was born in September 2024, driven by Answer.AI. The idea: create a standardized text file at the root of your site to guide LLMs in understanding your content. A bit like a “welcome, here is what matters” sign at the entrance of your digital home.
Except nobody is obligated to read the sign.
The problem nobody says out loud
robots.txt works because it is built on a web convention accepted by every player since the 90s. Google respects it. Bing too. It is baked into crawl culture.
For llms.txt? Zero official W3C standard. Zero technical obligation. Zero consensus between OpenAI, Anthropic, Google DeepMind or Mistral. Each one crawls the web however it wants, with its own rules, its own bots, its own indexing logic.
In practice: ChatGPT does not read your llms.txt. Neither does Claude. Perplexity? Same story. This file does not appear in any of their official documentation as a recognized signal.
robots.txt vs llms.txt: no contest
Here is what actually makes a difference today. The major models each have their own official crawl bot, and those bots do respect the good old robots.txt:
- GPTBot (OpenAI): documented, blockable via robots.txt
- ClaudeBot (Anthropic): same, follows classic directives
- PerplexityBot: same logic
- Google-Extended: Google’s bot for its Gemini models
Want to control what AI reads on your site? Work on your SEO strategy and update your robots.txt. Less exciting advice, but the one that actually has an impact.
So should we completely forget about llms.txt?
Not entirely. A few monitoring tools and WordPress plugins are starting to read it. And if the file becomes a standard in 2 or 3 years, early adopters will have a head start. But today, in 2025, betting on llms.txt for your AI SEO is like optimizing your site for a search engine that does not exist yet.
The real challenge of 2025 is getting cited by LLMs, not controlling them. Showing up in a Perplexity or ChatGPT answer is the new first Google result. And for that, the recipe has not changed: quality content, domain authority, clean technical structure.
No magic file. Just good work.
Sources: llmstxt.org, GPTBot OpenAI documentation, ClaudeBot Anthropic documentation