What happened
The digital marketing landscape has recently seen a surge in the adoption of `llms.txt` files, marketed as a crucial tool for Generative Engine Optimization (GEO). However, a provocative analysis by Search Engine Journal has challenged this trend. By using a satirical "Cats.txt" experiment, the author demonstrated that the arguments supporting `llms.txt` are largely based on flawed logic and lack empirical evidence. The study suggests that many of the perceived benefits are merely coincidental, leading to what the industry is now calling "GEO Astrology."
Technology context
GEO (Generative Engine Optimization) is an emerging field focused on making website content more accessible and "citable" for AI models like GPT-4 or Claude. The `llms.txt` file was proposed as a Markdown-based standard to provide AI crawlers with a simplified, text-heavy version of a website's most important pages. The core idea is to reduce token usage and noise for the AI. However, unlike traditional SEO, where `robots.txt` has a clear, documented function for search engine crawlers, the interaction between LLMs and static text files remains opaque and unverified by major AI developers.
Why it matters
This critique is vital because it highlights a growing gap between marketing hype and technical reality. As businesses rush to optimize for AI, they risk falling for "snake oil" tactics that have no measurable impact on how AI models actually retrieve and process information. Understanding that LLMs primarily rely on massive pre-trained datasets and sophisticated RAG pipelines—rather than simple text files at the root of a domain—is essential for any modern digital strategy.
Key terms explained
- GEO (Generative Engine Optimization): The practice of tailoring web content to increase its chances of being used as a source in AI-generated responses.
- LLMs.txt: A proposed community standard for a text file that summarizes a website's content specifically for AI agents.
- Tokenization: The process by which AI models break down text into smaller units (tokens) to process and understand language.
- Crawler: A bot that systematically browses the internet to index content for search engines or AI training sets.
Impact
In the short term, this revelation will likely lead to a more skeptical approach toward unverified SEO "hacks" for AI. Agencies and webmasters may pause the implementation of non-standard files until major players like OpenAI or Microsoft provide official documentation. In the medium term, the focus will shift back to high-quality, structured data and authoritative content, which remain the most reliable ways to influence AI outputs.
What's next
The industry is moving toward a more standardized approach to AI data scraping. We should expect the emergence of official protocols (perhaps through W3C or major AI labs) that will replace community-driven ideas like `llms.txt`. The focus will likely shift from "how to be read" to "how to be trusted," emphasizing E-E-A-T (Experience, Expertise, Authoritativeness, and Trustworthiness) as the primary ranking factor for generative engines.
Educational analysis generated with AI and editorially reviewed.