Cloudflare will now rewrite a website's robots.txt for it, automatically and on by default for new customers, which turns the crawler-access file into a moving part of an audit.

Cloudflare has connected two things that used to be separate. The AI bot categories a site owner sets in the Cloudflare dashboard (Search, Agent, Training) are now mirrored automatically into that site's robots.txt. The generated block is prepended to whatever robots.txt already says, wrapped in comment markers reading "BEGIN Cloudflare Bot Preference Sync" and "END", so existing Disallow lines survive. It is available on every plan from Free to Enterprise. Cloudflare's post says that "for all new customers, Bot Preference Sync will be on by default", and existing customers on the older managed robots.txt feature get prompted to move across.

The Training category also gained a softer option. Instead of a hard block, Disallow writes a no-training preference into robots.txt in a form that lets a crawler doing both search and training keep reaching the content for indexing, provided that crawler meets four disclosure conditions: respect no-training preferences in robots.txt, offer site owners an opt-out from AI summaries, provide URL-level visibility into which pages were used for training alongside search metrics, and demonstrate publicly that disallowing training does not damage traditional search results.

On 20 August this radar recorded the September change as "Cloudflare blocks Training and Agent categories by default for new domains onboarding, Search stays allowed". The scope was wrong. Cloudflare's own wording is that "for all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads". PPC Land's 21 August write-up reads it the same way: ad-monetised domains get Disallow as their training default at onboarding, and all other new customers start with no restrictions applied. Cloudflare's stated reasoning is that an ad is the signal that a page was meant to be landed on by a person.

This closes the item flagged as worth chasing on 21 August. 5W tested 320 or more consumer-intent prompts across ChatGPT, Claude, Perplexity, Gemini and Google AI Overviews, in ten US metros (New York, Los Angeles, Chicago, Houston, Phoenix, Philadelphia, San Antonio, San Diego, Dallas, Austin) across six categories (plumbing, electrical, personal injury law, family medicine, dentistry, HVAC), giving 60 metro-category combinations. Data was collected in Q1 2026, so this is older material that has not reached this desk before.

The findings, in the report's own words: "an estimated 78% of independent local operators have effectively zero AI citation share", and "Yelp, Angi, and Zocdoc together capture an estimated 21.5% of all local-services AI citations". The most-cited individual brands are Yelp at 9.0%, Roto-Rooter 7.5%, Angi 7.0%, Zocdoc 5.5% and Morgan and Morgan 5.0%. The largest single signal effect it claims is that an entity with strong signals commands "approximately eight times the AI citation share of an independent operator with strong Yelp reviews".

Four caveats have to travel with those numbers. Both headline figures are described by the report as estimates, not counted totals. The public page does not disclose how many times each prompt was run or whether collection went through APIs or the live consumer interfaces, which are the two methodology facts that decide whether a result is reproducible. The categories are trades, medicine and law, not coaching, so the shape may transfer but the percentage does not. And 5W is a public relations and generative-engine-optimisation agency that sells the earned-media and entity-building work its own research implies you need, which is a direct interest in the finding.