Google's own AI agent ignores robots.txt, and our pre-audit crawler check tests neither Google's nor Bing's crawler, so two of the five engines we audit go unchecked before a client is told they are missing.

Google lists Google-Agent as the fetcher "used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request", alongside eight other user-triggered fetchers, and says these fetchers "generally ignore robots.txt rules" because a person asked for the fetch. Search Engine Journal (Matt G. Southern, 22 October, read first-hand) raised it while reporting Sundar Pichai's interview on the Sources podcast, where he said agents will reach sites both by browsing pages and through protocols such as MCP and Google's Universal Commerce Protocol, calling protocols "a far more scalable approach." Google publishes the IP ranges these fetchers use so traffic can be verified.

ChatGPT's newest entry is still locked chats (20 October). Google's newest changelog entry is still the 23 October URL hyphens note. Perplexity's changelog is unchanged and marks no current Sonar model as deprecated. Anthropic's newest posts are still 22 October, and the deprecations table is unchanged: Sonnet 4.5 retires 30 November, and Haiku 4.5 is still listed Active with no retirement notice. Bing's webmaster blog still ends at 13 October, and Cloudflare's newest posts (23 October) are about file serving and cryptography.

Method in one clause: 31 million citations across 11 AI surfaces (including ChatGPT, Gemini, Claude, Perplexity and AI Overviews) from October 2025 to July 2026, plus live robots.txt fetches for 105 US and UK news publishers against 25 AI user agents in July 2026. News was 1.3% to 1.9% of all citations in a month. The New York Times, which blocks all 13 major AI user agents, had zero ChatGPT and zero Gemini citations but 37,642 from Grok. AP blocks every OpenAI crawler, yet 80% of its citations came from ChatGPT. llms.txt adoption across the audited publishers was zero. Goodie sells AI visibility tools, and the study covers major news sites only, no small or service businesses.