On 30 October, OpenAI, Google, Microsoft and Anthropic said they will respect a single, shared signal in robots.txt that tells AI companies a site does not want its content used for training. The Verge reported the news, citing Search Engine Roundtable, and OpenAI published its own announcement the same day.
For anyone who runs a website, the appeal is simple. Today, keeping content out of AI training means tracking down each company's crawler and writing a separate rule for it. A common token would let one line in robots.txt speak to every provider that has agreed to honor it.
Where the signal comes from
According to The Verge, the shared "policy token" was proposed within the Internet Engineering Task Force by Kevin Kelley, who leads standards work at Anthropic. OpenAI ties the effort to the IETF's AI Preferences working group and says it is joining other members of that group in announcing the signal.
The token lives in robots.txt, the file sites have long used to tell crawlers what they may fetch, whether for search indexing, archiving or AI training. OpenAI describes the arrangement as voluntary and frames the signal as covering the use of content for training foundational generative AI models.
Neither source spells out the exact syntax of the token, so site owners should wait for the published specification and each provider's documentation before editing their files.
Who supports it, and when
The four companies say their AI crawlers will stay away from a site once they find the shared token, but the timelines differ. The Verge reports that Anthropic already uses the signal. Google and Microsoft expect to add support next year, and OpenAI is aiming for the "coming months", promising a further update describing its implementation once it ships.
Meta's communications head, Andy Stone, voiced support for a common signal, according to The Verge, but did not say whether Meta will actually honor it. That leaves at least one large AI company outside the firm commitments for now, and any provider that has not signed on is free to ignore the token.
What changes for existing controls
OpenAI says the new token does not replace the crawler-specific robots.txt controls it already offers, and that those rules will keep applying. Sites that already block individual AI crawlers by name therefore lose nothing by leaving those rules in place.
Microsoft says it will offer finer-grained choices, letting publishers opt out of AI training or of AI grounding, which The Verge describes as pointing a model to a verified source of information to cut down on hallucinations. Microsoft adds that a site's preferences will not affect whether it shows up in Bing search results. The Verge notes that Google already lets sites decide separately whether their content feeds its AI search features or its training, without changing search visibility.
The announcement arrives while publishers are still dealing with a steady stream of AI crawlers. The Verge points to other responses already in use, such as the Really Simple Licensing standard, which lets publishers charge crawlers for access, and Cloudflare blocking AI crawlers by default. A shared opt-out does not compete with those approaches; it expresses a preference, and it depends on crawlers choosing to respect it.
What to do
- Keep any existing per-crawler AI rules in robots.txt, since OpenAI says its current controls will continue to apply alongside the new token.
- Watch for the IETF specification and each provider's implementation notes before adding the shared token, because support arrives at different times.
- If you rely on Bing traffic, note that Microsoft says opting out of AI training or grounding will not remove your site from Bing search results.