Labs back one robots.txt opt-out from AI training
Confirmed
The takeaway
On Oct 30, 2026 OpenAI, Google, Microsoft and Anthropic said they will honour a shared robots.txt signal that lets a website block all crawlers collecting data to train generative AI models with one entry, instead of listing each company’s bots.
Status
- Claim
Confirmed
- Our reporting
- High confidence
- Importance
- 3 of 5
- Last verified
- 31 October 2026
Your AI and this story
- GPT-6.2 Astra152 days after its cutoff
- Claude Opus 5.5122 days after its cutoff
- Gemini 4 Hydrogen213 days after its cutoff
- Grok 4.7152 days after its cutoff
None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 122 days before it.
Key facts
- OpenAI (Oct 30): it ‘will honor a shared robots.txt signal to help website operators communicate their AI training preferences to participating providers’
- How it works (SiliconANGLE): a single robots.txt entry blocks all scrapers gathering training data for generative AI; it does not apply to models that ‘perform exclusively non-generative tasks’; to be incorporated into the Robots Exclusion Protocol format
- Draft submitted to the Internet Engineering Task Force by the four companies; timing: OpenAI and Microsoft ‘in the coming months’, Google ‘sometime next year’
- Microsoft also plans an ‘AI Grounding’ setting for pages fetched to answer a user’s question rather than for training, covering third-party tools built on Microsoft technology
- Meta ‘expressed support for the idea’ but has not committed to implement it (SiliconANGLE)
- Same day OpenAI published ‘Fair use benefits everyone’, excerpts from court briefs backing its position that AI training is fair use; the UK CMA opened a case on Meta’s terms for reusing publisher content
What happened
Website owners today must list every AI company’s crawler in robots.txt to keep their pages out of training data. The four largest Western AI developers proposed one shared token instead and took it to the IETF. The signal covers training only; Microsoft is adding a separate control for real-time fetching to ground answers.
Why it matters
It is the first joint technical commitment by the leading labs on training opt-outs, arriving while publishers sue, Reddit closes its site to logged-out users and regulators examine platforms’ terms. It is a voluntary signal: it does not cover data already collected, and its value depends on the many other crawlers adopting it.
Sources
4 sources from 3 sites. Numbers match the chips in the text.
4 sources: 2 primary, 2 press
Primary
- OpenAI: Making it easier for websites to express AI training preferences (Oct 30, 2026)openai.com, official
- OpenAI: Fair use benefits everyone (Oct 30)openai.com, official
Press
Changes
- Filed (news agent; OpenAI page 403, its RSS description used; SiliconANGLE read)