Google's John Mueller on llms.txt: "the hope is bigger than the reality". AI crawlers are reading plain sitemaps and RSS feeds instead.

On Google's podcast, John Mueller said AI crawlers "usually don't have any kind of a Console or any setup where you can submit a sitemap file", and gave his own observation: "I've seen that happen in my server logs where some AI crawler accesses my sitemap file." His advice was "either you stick to the generic naming, call it sitemap.xml, or you focus on RSS feeds." On llms.txt he said "I think the hope is bigger than the reality" and "Currently none of this happens", and answered "No" to whether it can stand in for a sitemap. The limits matter. He named no crawler. He speaks for Google, not for OpenAI, Anthropic or Perplexity. One person's server logs are an anecdote, and he said he does not know what those companies do with the files. He also said that a sitemap Google "couldn't fetch" is often about demand: "crawl demand is very often based on the perceived quality of a website."

Google's incident still carries only the 24 September launch post. Barry Schwartz writes: "I believe we are seeing the final phase of the Google September 2026 spam update rolling out between October 4th and October 6th, today." That is his reading of third-party tracking tools and forum posts. Google has said nothing about phases.

Google added one sentence to the help page for the setting that lets a site keep its content out of AI Overviews and AI Mode: "Owners of the top-level domain property will see a notification on the Search generative AI control page when one of their child properties overrides the parent setting." The page also now states the setting is "rolled out to all websites worldwide" as of 31 August 2026, and that a change "generally takes a few days".

The changelog now lists October entries: a Decisions API, document input, an image search tool, and "The Agent API `low`, `medium`, and `high` presets now use Fast Search" (web search down from $2.50 to $1.00 per 1,000 calls). The migration page is unchanged: Sonar requests "are being reformulated as Agent API requests, rolling out gradually by model", with no model named. The engine's `main` is still `06233ce` with `sonar` at `src/adapters.ts` line 14.

Verbatim: "As of September 29th 2026, AI Overviews appeared in 82.91% of the US desktop search results for branded keywords." The base is 232.8 million US desktop result crawls from July to September, 92.5 million of them on branded keywords. By month, the branded share went from 61.3% in July to 73.3% in September. In a separate panel of 100 major brand names checked weekly, 63 showed an overview by 30 September (five in early July), and 76.2% of those overviews sat in position one. Wikipedia was cited in 71.9% of them. Limits: US desktop only, a "branded keyword" is any search containing a brand name, the panel is 100 of the world's biggest brands, and the article says nothing about small businesses, about the spam update, or about whether the overviews are accurate.