{"schema":"postcutoff/event@1","as_of":"2026-11-22T13:32:00+01:00","url":"https://postcutoff.com/e/2026-10-30-ai-training-opt-out-robots-txt-standard-ietf/","md":"https://postcutoff.com/e/2026-10-30-ai-training-opt-out-robots-txt-standard-ietf/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2026-10-30-ai-training-opt-out-robots-txt-standard-ietf","date":"2026-10-30","date_precision":"day","short_title":"Labs back one robots.txt opt-out from AI training","deck":null,"takeaway":"On Oct 30, 2026 OpenAI, Google, Microsoft and Anthropic said they will honour a shared robots.txt signal that lets a website block all crawlers collecting data to train generative AI models with one entry, instead of listing each company's bots.","category":"policy-safety","category_label":"Policy & safety","importance":3,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":[{"n":1,"title":"OpenAI: Making it easier for websites to express AI training preferences (Oct 30, 2026)","url":"https://openai.com/index/helping-websites-express-ai-training-preferences","type":"official","group":"primary","domain":"openai.com"},{"n":2,"title":"OpenAI: Fair use benefits everyone (Oct 30)","url":"https://openai.com/index/fair-use-benefits-everyone","type":"official","group":"primary","domain":"openai.com"},{"n":3,"title":"SiliconANGLE: Google, Anthropic and others to support new AI crawling opt-out standard (Oct 30)","url":"https://siliconangle.com/2026/10/30/google-anthropic-and-others-to-support-new-ai-crawling-opt-out-standard/","type":"press","group":"press","domain":"siliconangle.com"},{"n":4,"title":"The Verge: The biggest names in AI are backing a new way to opt out of scraping","url":"https://www.theverge.com/tech/1023763/openai-google-microsoft-ai-scraping-standard-opt-out","type":"press","group":"press","domain":"theverge.com"}],"official":2,"filed":"2026-10-31","updated":"2026-10-31","orgs":["OpenAI","Google","Microsoft","Anthropic","IETF"],"title":"OpenAI, Google, Microsoft and Anthropic back a single robots.txt signal to opt out of generative-AI training, submitted to the IETF","summary":"On Oct 30, 2026 OpenAI, Google, Microsoft and Anthropic said they will honour a shared robots.txt signal that lets a website block all crawlers collecting data to train generative AI models with one entry, instead of listing each company's bots. The four submitted a draft to the IETF, which maintains the Robots Exclusion Protocol; OpenAI and Microsoft plan support in the coming months and Google next year, while Meta backed the idea without committing.","key_facts":["OpenAI (Oct 30): it 'will honor a shared robots.txt signal to help website operators communicate their AI training preferences to participating providers'","How it works (SiliconANGLE): a single robots.txt entry blocks all scrapers gathering training data for generative AI; it does not apply to models that 'perform exclusively non-generative tasks'; to be incorporated into the Robots Exclusion Protocol format","Draft submitted to the Internet Engineering Task Force by the four companies; timing: OpenAI and Microsoft 'in the coming months', Google 'sometime next year'","Microsoft also plans an 'AI Grounding' setting for pages fetched to answer a user's question rather than for training, covering third-party tools built on Microsoft technology","Meta 'expressed support for the idea' but has not committed to implement it (SiliconANGLE)","Same day OpenAI published 'Fair use benefits everyone', excerpts from court briefs backing its position that AI training is fair use; the UK CMA opened a case on Meta's terms for reusing publisher content"],"key_numbers":[],"tags":["robots-txt","crawling","training-data","publishers","standards","copyright"],"science":null,"body_md":"## What happened\n\nWebsite owners today must list every AI company's crawler in robots.txt to keep their pages out of training data. The four largest\nWestern AI developers proposed one shared token instead and took it to the IETF. The signal covers training only; Microsoft is\nadding a separate control for real-time fetching to ground answers.\n\n## Why it matters\n\nIt is the first joint technical commitment by the leading labs on training opt-outs, arriving while publishers sue, Reddit closes\nits site to logged-out users and regulators examine platforms' terms. It is a voluntary signal: it does not cover data already\ncollected, and its value depends on the many other crawlers adopting it.","disputed":[],"related":[{"id":"2026-10-30-uk-cma-investigates-meta-commercial-terms-ai","url":"https://postcutoff.com/e/2026-10-30-uk-cma-investigates-meta-commercial-terms-ai/","date":"2026-10-30","date_precision":"day","short_title":"UK CMA opens a Competition Act investigation into Meta's commercial terms that would let it use publishers' web content for its AI","deck":null,"takeaway":"It is the first formal competition case in the UK centred on a platform taking web content for AI through contract terms, and it follows the CMA's earlier action on Google Search.","category":"policy-safety","category_label":"Policy & safety","importance":3,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":3,"official":2,"filed":"2026-10-30","updated":"2026-10-30","orgs":["Competition and Markets Authority","Meta"]},{"id":"2026-10-29-reddit-logged-in-only-ai-scraping-search-decline","url":"https://postcutoff.com/e/2026-10-29-reddit-logged-in-only-ai-scraping-search-decline/","date":"2026-10-29","date_precision":"day","short_title":"Reddit will move to a mostly logged-in site to curb AI scraping as AI search answers cut its referrals by 1M+ a week","deck":null,"takeaway":"It is a large public site closing its open web pages specifically because of AI.","category":"business","category_label":"Business","importance":3,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":3,"official":0,"filed":"2026-10-30","updated":"2026-10-30","orgs":["Reddit"]},{"id":"2026-10-29-openai-copyright-mdl-amicus-briefs-publishers-wikimedia-unions","url":"https://postcutoff.com/e/2026-10-29-openai-copyright-mdl-amicus-briefs-publishers-wikimedia-unions/","date":"2026-10-29","date_precision":"day","short_title":"Amicus wave in the OpenAI copyright case","deck":"Publishers and Hollywood unions say training is not fair use; Wikimedia says training is, but RAG is not","takeaway":"Judge Stein's ruling on fair use in these cases is the most important pending US decision on AI training data.","category":"policy-safety","category_label":"Policy & safety","importance":3,"confidence":"medium","status":{"key":"partly","labels":["Partly confirmed"]},"sources":7,"official":2,"filed":"2026-10-30","updated":"2026-10-31","orgs":["OpenAI","Microsoft","Association of American Publishers","Wikimedia Foundation","SAG-AFTRA","Writers Guild of America","American Federation of Musicians"]},{"id":"2026-07-23-stealth-bot-prohibition-act","url":"https://postcutoff.com/e/2026-07-23-stealth-bot-prohibition-act/","date":"2026-07-23","date_precision":"day","short_title":"Stealth Bot Prohibition Act (H.R. 9915)","deck":"Bipartisan House bill would make AI scraping bots identify themselves; 300+ publishers lobby for it","takeaway":"It is the publishers' preferred federal route for AI scraping: a disclosure rule enforced by the FTC, in place of waiting for copyright suits to settle whether training is fair use.","category":"policy-safety","category_label":"Policy & safety","importance":2,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":5,"official":0,"filed":"2026-10-12","updated":"2026-10-12","orgs":["US Congress","News/Media Alliance"]}],"people":[],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-10-31","type":"filed","text":"Created (news agent; OpenAI page 403, its RSS description used; SiliconANGLE read)"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-10-31","run":null,"sources_read":"News agent; OpenAI page 403, its RSS description used; SiliconANGLE read","updated":"2026-10-31","human_review":null,"version":{"date":"2026-10-31"}},"gaps":[{"model_id":"gpt-6-2-astra","name":"GPT-6.2 Astra","cutoff":"2026-05","days_after":152,"in_training_data":false},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":122,"in_training_data":false},{"model_id":"gemini-4-hydrogen","name":"Gemini 4 Hydrogen","cutoff":"2026-03","days_after":213,"in_training_data":false},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":152,"in_training_data":false}],"short_url":null}