On October 30, 2026, the four companies whose crawlers and models touch most of the web said the same thing at the same time: they will honor a single robots.txt instruction that lets a site refuse AI training across every participating provider at once. Google posted on its Search Central Blog. Microsoft published on the Bing Blog. OpenAI updated its policy pages. And the Anthropic engineer who wrote the specification, Kevin Kelley, submitted it to the Internet Engineering Task Force (IETF) the same afternoon. For site owners, this is the first genuinely shared control over how AI companies use web content, and the details matter more than the headline.
What was announced on October 30, 2026
Four separate announcements, one mechanism:
- Google: "Supporting a shared AI training control for websites," by John Mueller, Google Search Advocate, on the Google Search Central Blog.
- Microsoft: "Your content, your choice: clearer preferences for training and grounding," by Krishna Madhavan, Principal Product Manager, Microsoft Bing, on the Bing Blog.
- OpenAI: "Making it easier for websites to express AI training preferences," published in the company's Policies and Procedures section.
- Anthropic: Kevin Kelley submitted an Internet-Draft, draft-kelley-aipref-training-preference-signal-00, to the AI Preferences working group (AIPREF) at 15:39 UTC, and wrote that Anthropic had already started building support for it.
Until now, a site that wanted to opt out of AI training had to maintain a separate instruction for each company's crawler, because each company used its own name for that note. The draft is ten pages, is intended as Standards Track, was last updated on October 30, 2026, and expires on May 3, 2027 if not revised. One more thing worth noting: no company has said it reads the token yet today.
How the shared opt-out token works
The device is an ordinary robots.txt group whose User-Agent line names a "policy token" instead of a crawler:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That example, straight from the draft, expresses a preference against AI training for all paths except /press-kit/, for which it expresses a preference in favor. Path matching and longest-match precedence follow RFC 9309, the Robots Exclusion Protocol the IETF published in September 2022. A single group may name several policy tokens, and policy tokens may sit alongside ordinary crawler names.
A few design choices explain why this is likely to stick. The draft introduces no new directive names and no new grammar, so existing parsers behave exactly as before. Consumers must match policy tokens exactly, apart from case, and the double underscore prefix makes accidental matches by loosely written parsers unlikely. The draft also draws a hard line between access and use: the token is not a crawling directive, and each crawler's own product tokens govern what may be fetched. A consumer reading only for training purposes may refrain from retrieving pages it cannot use, but if you want a crawler to stay away entirely, you still need a group for that crawler's name.
The wildcard trap: blocking every bot says nothing
The clause that will matter most to ordinary site owners is this one: a wildcard group, User-Agent: *, MUST NOT be interpreted as matching the policy token. If no group in your robots.txt names __usage_ai-training, then no preference has been expressed. "A robots.txt file that doesn't use this mechanism, as is true of every one published today, expresses no preference at all," Kelley wrote to the AIPREF mailing list.
The reverse inference is forbidden too. A block addressed to a proprietary token such as Google-Extended keeps whatever meaning it has today, but it does not count as a shared training opt-out for OpenAI or Microsoft, and no provider may treat the shared token as covered by a proprietary one. In practice: if you think you already opted out, you probably did not.
When each company will actually honor it
| Provider | Timing | Source |
|---|---|---|
| Microsoft | From February 1, 2027, as it crawls | Bing Blog, October 30, 2026 |
| "Early 2027" | Google crawler documentation, October 30, 2026 | |
| OpenAI | "In the coming months" | OpenAI policy post, October 30, 2026 |
| Anthropic | Implementation underway, no date given | AIPREF mailing list, October 30, 2026 |
Microsoft adds a deadline of its own: reporting by PPC Land on October 31, 2026 notes that sites relying on Microsoft's older opt-out method, built on the NoArchive and NoCache directives, lose that protection on March 1, 2027 unless they switch to the new mechanism.
What it means at Google: training and grounding together
At Google, the token covers more than training. According to Google's announcements and PPC Land's October 31 reporting, a site that adopts it will also be kept out of Gemini answers that cite web pages, what the industry calls grounding. Rankings in Google and Bing search results are not affected. That distinction between access, use and citation is the whole story of this update: Google's own staff acknowledged nine days earlier, in the Non-Cabinet Canonical video series, that robots.txt governs crawling, not what a company does with the content it fetches. This token extends the notation into the "what happens next" layer.
Should your business adopt the token?
Our position is blunt. If your business earns attention through content, adopting the token is a self-inflicted wound. You would opt out of the training data that makes models know your category, and at Google you would also disappear from the Gemini answers that increasingly act as the front door to search. The businesses with a real case for using it look different: sites with genuinely private or high-value content behind the public web, publishers whose licensing deals govern their relationships (private agreements may override preferences expressed in robots.txt, per the draft), and operators who want path-level control, blocking a members area while welcoming a press kit.
There is no urgency either. Nothing is enforced until Microsoft starts in February 2027 and Google in early 2027. Sequencing is everything: decide what you want each AI system to do with your content, write the rules for that, and test the file before and after every change.
If you want a clear-eyed read on whether AI search sends you customers or quietly drains your traffic, that is exactly the work we do. Talk to us at The Vivid Company.
