{"schema":"postcutoff/event@1","as_of":"2026-10-16T12:39:00+02:00","url":"https://postcutoff.com/e/2026-10-15-cloudflare-openai-discovery-pilot-results/","md":"https://postcutoff.com/e/2026-10-15-cloudflare-openai-discovery-pilot-results/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2026-10-15-cloudflare-openai-discovery-pilot-results","date":"2026-10-15","date_precision":"day","short_title":"Cloudflare–OpenAI discovery pilot results","deck":null,"takeaway":null,"category":"business","category_label":"Business","importance":2,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":[{"n":1,"title":"Cloudflare blog: Cloudflare and OpenAI: Making AI discovery better for everyone","url":"https://blog.cloudflare.com/cloudflare-openai-discovery-agentic-web/","type":"official","group":"primary","domain":"blog.cloudflare.com"},{"n":2,"title":"Barchart (Business Wire): Cloudflare announces research pilot with OpenAI (July 8)","url":"https://www.barchart.com/story/news/3182635/cloudflare-announces-research-pilot-with-openai","type":"official","group":"primary","domain":"barchart.com"},{"n":3,"title":"The Next Web: Cloudflare spent a year blocking AI crawlers. Now it's helping OpenAI index the web","url":"https://thenextweb.com/news/cloudflare-openai-research-pilot-ai-search","type":"press","group":"press","domain":"thenextweb.com"}],"official":2,"filed":"2026-10-16","updated":"2026-10-16","orgs":["Cloudflare","OpenAI"],"title":"Cloudflare and OpenAI report early results of their web-discovery pilot: network signals helped ChatGPT Search find hundreds of millions of new pages","summary":"In an Oct 15, 2026 post co-written by Cloudflare and OpenAI staff, the companies reported early results from a research pilot started in July 2026: signals from about 1% of Cloudflare's network (dead links, sites redirecting everything to the homepage, duplicates, change signals, page-level harm flags such as planted fake login forms) helped OpenAI discover and index hundreds of millions of new pages while cutting wasted crawler load. The post says the metadata is used for search and discovery, not for training models.","key_facts":["Pilot announced July 8, 2026 (Business Wire); results post Oct 15, bylined by Jack Galilee (Cloudflare) and Henry Aspegren and James Balfour (OpenAI)","Cohort: about 1% of Cloudflare's network; outcome: OpenAI discovered and indexed 'hundreds of millions' of new pages; no crawl-reduction figures given","Signals: filtering of non-existent-file requests, catch-all redirects and duplicates; change signals for when to revisit; page-level harm flags","Applies to sites 'that choose to make their content available to AI systems'; a signal does not grant access permission; Cloudflare's AI Crawl Control still governs access","July quotes: Matthew Prince: 'By sharing our sophisticated network signals, we can find a better way to make AI search more efficient'; OpenAI's Nick Ryder: the pilot explores 'whether network-level insights can help us discover content more efficiently'"],"key_numbers":[],"tags":["search","crawling","publishers","chatgpt-search","web","partnerships"],"science":null,"body_md":"## What happened\n\nCloudflare passes metadata it sees as a network operator to OpenAI's search crawler. This lets OpenAI skip junk URLs, revisit pages\nwhen they change and avoid individual malicious pages. The pilot covers about 1% of Cloudflare's network. The July 8 announcement\nwas missed at the time and is summarised here.\n\n## Why it matters\n\nCloudflare spent the previous year building tools to block or charge AI crawlers. It is now helping one AI company crawl more\nefficiently, on the condition that sites choose to let AI systems in. Discovery and freshness give AI search an edge, and a\nnetwork operator with this view of the web is a partner other AI search engines may also want.","disputed":[],"related":[],"people":[],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-10-16","type":"filed","text":"Created"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-10-16","run":null,"sources_read":null,"updated":"2026-10-16","human_review":null,"version":{"date":"2026-10-16"}},"gaps":[{"model_id":"gpt-6-astra","name":"GPT-6 Astra","cutoff":"2026-04","days_after":168,"in_training_data":false},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":107,"in_training_data":false},{"model_id":"gemini-4-argon","name":"Gemini 4 Argon","cutoff":"2026-03","days_after":198,"in_training_data":false},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":137,"in_training_data":false}],"short_url":null}