Site operators get one robots.txt switch to block AI training

Website operators could soon block AI training with a single robots.txt entry instead of a separate rule for each provider. Google, Microsoft, Anthropic and OpenAI plan to support the shared crawling standard, according to SiliconANGLE.

The shared token is named __usage_ai-training, according to secondary coverage. The companies have submitted a draft to the Internet Engineering Task Force (IETF), which maintains the Robots Exclusion Protocol. Per ppc.land, the draft is an individual Internet-Draft dated 30 October 2026. The IETF datatracker says it is not endorsed by the IETF, and the draft must still pass through the IETF AI Preferences working group.

Timing varies by company. Microsoft and OpenAI plan support in the coming months, and both are adding separate controls for pages an AI model reads to answer a user's question. Microsoft's control is called AI Grounding. Microsoft will honour the preferences from 1 February 2027, replacing the NoArchive and NoCache tags it has used since 2023. Google targets early 2027 and will read the token as a Google-Extended opt-out.

The standard would not apply to models that perform exclusively non-generative tasks, and the companies plan to add it to the Robots Exclusion Protocol. Meta has expressed support for the idea but has not committed to implementing it. Secondary coverage adds further limits: the token is a voluntary preference, not an access control; it does not replace GPTBot or Google-Extended rules; and it applies only going forward, so content already crawled stays in existing training sets.

Anthropic's rollout timing is unconfirmed. SiliconANGLE gives no timeline for the company, one roundup says it has already begun implementing support, and Anthropic's own statement was not found. Some coverage lists only OpenAI, Google and Microsoft as committing. Details beyond SiliconANGLE come from secondary coverage rather than the companies' own posts.

Site operators gain the most, provided crawlers honour the token. The standard is voluntary and does not reach content already in training sets, so operators who need firm protection still depend on access controls, which the token is not. Meta's position is the gap to watch: support stated, commitment absent.

Watch three things: how the IETF AI Preferences working group handles the draft, whether Anthropic confirms its timing, and whether Microsoft's 1 February 2027 switch-over and Google's early 2027 target hold.