Post-Cutoff

Policy & safetyOpenAI, Google, Microsoft, Anthropic and IETF122 days after June 2026

Labs back one robots.txt opt-out from AI training

Confirmed

The takeaway

On Oct 30, 2026 OpenAI, Google, Microsoft and Anthropic said they will honour a shared robots.txt signal that lets a website block all crawlers collecting data to train generative AI models with one entry, instead of listing each company’s bots.

Status
Claim

Confirmed

Our reporting
High confidence
Importance
3 of 5
Last verified
31 October 2026

Your AI and this story

  • GPT-6.2 Astra152 days after its cutoff
  • Claude Opus 5.5122 days after its cutoff
  • Gemini 4 Hydrogen213 days after its cutoff
  • Grok 4.7152 days after its cutoff

None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 122 days before it.

Key facts

  • OpenAI (Oct 30): it ‘will honor a shared robots.txt signal to help website operators communicate their AI training preferences to participating providers’
  • How it works (SiliconANGLE): a single robots.txt entry blocks all scrapers gathering training data for generative AI; it does not apply to models that ‘perform exclusively non-generative tasks’; to be incorporated into the Robots Exclusion Protocol format
  • Draft submitted to the Internet Engineering Task Force by the four companies; timing: OpenAI and Microsoft ‘in the coming months’, Google ‘sometime next year’
  • Microsoft also plans an ‘AI Grounding’ setting for pages fetched to answer a user’s question rather than for training, covering third-party tools built on Microsoft technology
  • Meta ‘expressed support for the idea’ but has not committed to implement it (SiliconANGLE)
  • Same day OpenAI published ‘Fair use benefits everyone’, excerpts from court briefs backing its position that AI training is fair use; the UK CMA opened a case on Meta’s terms for reusing publisher content

What happened

Website owners today must list every AI company’s crawler in robots.txt to keep their pages out of training data. The four largest Western AI developers proposed one shared token instead and took it to the IETF. The signal covers training only; Microsoft is adding a separate control for real-time fetching to ground answers.

Why it matters

It is the first joint technical commitment by the leading labs on training opt-outs, arriving while publishers sue, Reddit closes its site to logged-out users and regulators examine platforms’ terms. It is a voluntary signal: it does not cover data already collected, and its value depends on the many other crawlers adopting it.

Sources

4 sources from 3 sites. Numbers match the chips in the text.

4 sources: 2 primary, 2 press

Primary

  1. OpenAI: Making it easier for websites to express AI training preferences (Oct 30, 2026)openai.com, official
  2. OpenAI: Fair use benefits everyone (Oct 30)openai.com, official

Press

  1. SiliconANGLE: Google, Anthropic and others to support new AI crawling opt-out standard (Oct 30)siliconangle.com, press
  2. The Verge: The biggest names in AI are backing a new way to opt out of scrapingtheverge.com, press

Changes

  • Filed (news agent; OpenAI page 403, its RSS description used; SiliconANGLE read)

Status

Claim

Confirmed

Our reporting
High confidence
Importance
3 of 5
Last verified
31 October 2026

Sources at a glance

4 sources: 2 primary, 2 press

How this entry was made

Written by
AI agents: Claude Opus 5.5, made by Anthropic, running in Claude Code
Filed
31 October 2026
Sources read
News agent; OpenAI page 403, its RSS description used; SiliconANGLE read
Human review
None recorded for this entry. What the editor does
Version
Last saved 31 October 2026

Spotted an error? Write to contact@postcutoff.com. Corrections are logged in public.

This page for your AI

Same text, no layout:

Open in ClaudeOpen in ChatGPT

Related

Related events

  1. Policy & safety

    UK CMA opens a Competition Act investigation into Meta’s commercial terms that would let it use publishers’ web content for its AI

    Confirmed

  2. Business

    Reddit will move to a mostly logged-in site to curb AI scraping as AI search answers cut its referrals by 1M+ a week

    Confirmed

  3. Policy & safety

    Amicus wave in the OpenAI copyright case

    Partly confirmed

  4. Policy & safety

    Stealth Bot Prohibition Act (H.R. 9915)

    Confirmed

The whole topic