{"schema":"postcutoff/event@1","as_of":"2026-10-14T22:23:00+02:00","url":"https://postcutoff.com/e/2026-10-14-first-proof-third-batch/","md":"https://postcutoff.com/e/2026-10-14-first-proof-third-batch/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2026-10-14-first-proof-third-batch","date":"2026-10-14","date_precision":"day","short_title":"First Proof batch 3: frontier systems solve 9 of 10 research problems, open-weight Kimi K3 harness none","deck":null,"takeaway":"On Oct 14, 2026 the First Proof Foundation (Abouzaid, Srivastava, Ward, Williams) published its third refereed batch: ten solved but unpublished research problems, one 24-hour shot per system with no human input, graded by 30 mathematicians with negative marks for wrong answers.","category":"benchmark","category_label":"Benchmarks","importance":4,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":[{"n":1,"title":"First Proof Third Batch report (Oct 14, 2026, PDF)","url":"https://1stproof.org/assets/docs/batch3-report.pdf","type":"paper","group":"primary","domain":"1stproof.org"},{"n":2,"title":"First Proof: Third Batch results page","url":"https://1stproof.org/third-batch.html","type":"official","group":"primary","domain":"1stproof.org"},{"n":3,"title":"Anthropic math-proof plugin (Siege skill)","url":"https://github.com/anthropics/claude-plugins-official/tree/main/plugins/math-proof","type":"code","group":"primary","domain":"github.com"},{"n":4,"title":"Scientific American: New 'First Proof' exam reveals how AI-powered math is a pay-for-play sport (Oct 14)","url":"https://www.scientificamerican.com/article/new-first-proof-exam-reveals-how-ai-powered-math-is-a-pay-for-play-sport/","type":"press","group":"press","domain":"scientificamerican.com"},{"n":5,"title":"Terence Tao: First Proof, Third Batch webinar (Oct 14)","url":"https://terrytao.wordpress.com/2026/10/14/first-proof-third-batch-webinar/","type":"discussion","group":"reactions","domain":"terrytao.wordpress.com"}],"official":3,"filed":"2026-10-14","updated":"2026-10-14","orgs":["First Proof Foundation","OpenAI","Anthropic","ETH Zurich","Moonshot AI"],"title":"First Proof third batch: ProofCouncil solves 9 of 10 unpublished research problems, GPT-6 Astra 7 for $36, Anthropic's Siege 3 with no negative scores, open-weight Kimi K3 harness 0","summary":"On Oct 14, 2026 the First Proof Foundation (Abouzaid, Srivastava, Ward, Williams) published its third refereed batch: ten solved but unpublished research problems, one 24-hour shot per system with no human input, graded by 30 mathematicians with negative marks for wrong answers. The ETH-led ProofCouncil harness solved 9 of 10 ($1,888), plain GPT-6 Astra 7 ($36), MerLean 4, Anthropic's Siege workflow on Claude Opus 5.5 3 (it declined to submit five, and none of its five submissions got a negative correctness score), and the Humanize harness on open-weight Kimi K3 none (every submission had major errors, about $2,000). Every problem was solved by some system. The organisers say the open-weight gap in research maths is far wider than in general benchmarks.","key_facts":["Problems: 10, from logarithmic geometry, statistical inverse problems, quantum topology, symplectic geometry, hyperfields, low-dimensional topology, complex and other dynamics, Teichmüller theory and algebraic geometry; each had a human proof of at most 12 pages not yet public","Process: problems refined at Harvard CMSA Aug 13–14; testing Oct 2–5 (24 hours per system, no human input); 30 referees graded anonymised solutions at CMU's ICARM Oct 8–9; results and webinar Oct 14","New scoring: from −1 to 1 for correctness, exposition and attribution, so wrong solutions claimed as correct are penalised and no answer or an acknowledged partial result is better","Problems solved (correctness ≥ 0.5): ProofCouncil 9 (avg 0.87); GPT-6 Astra at max reasoning effort 7 (0.65); MerLean (GPT-6 Astra + GPT-6.1 Sol) 4 (0.23); Siege (Claude Opus 5.5, high effort) 3 (0.25); Humanize (Kimi K3) 0 (−0.86)","Cost for 10 problems: GPT-6 Astra $36; Siege $1,400; ProofCouncil $1,888; Humanize ~$2,127 at API-equivalent prices (about 400 self-hosted GPU-hours); MerLean $5,323. Batch 2 comparison: ProofCouncil $3,200 and the plain OpenAI model $117","Siege is a Claude Code skill in Anthropic's public math-proof plugin; it runs up to 14 multi-agent rounds and submits only on a 'PROVED' verdict. It declined problems 3, 4, 8, 9 and 10 (scored 0) and ran without web search; on Problem 6 no referee could determine its correctness","ProofCouncil team: Tim Gehrunger (lead), Johannes Schmitt, David Holmes, Jasper Dekoninck and others; base models GPT-6 Astra Pro, GPT-6 Astra, Claude Fable 5.1, Claude Sonnet 5, Gemini 3.1 Pro preview","Failures noted: ProofCouncil got stuck in a loop on Problem 9 and destroyed a correct Astra solution; MerLean 'cheated' on Problem 10 with a broken definition from an old arXiv version; Astra's Problem 5 solution 'essentially constituted plagiarism'; only 4 of 10 Astra solutions had adequate citations (ProofCouncil 7)","Open weights: referees found the Kimi K3 system generally weaker than GPT-5.2 Pro (Dec 2025); report: 'We urge the community to immediately invest in improving open-source/open-weight models.' Lauren Williams to Scientific American: Kimi K3 'really fell flat on its face'","Not tested: Google did not take part; OpenAI's newer internal model was excluded because entrants must be openly available (Scientific American)","Funding: unrestricted industry donations for Batches 2 and 3 (acknowledgments list Anthropic, OpenAI and Google.org), used for graders, never for editors or board. Batch 4 runs Oct 2026–Jan 2027, results in the first half of January"],"key_numbers":[],"tags":["math","benchmark","research-math","gpt-6-astra","claude-opus-5-5","kimi-k3","open-weights","harness","evaluation","refereed"],"science":null,"body_md":"## What happened\n\nFirst Proof is the refereed exam for AI in research mathematics run by Mohammed Abouzaid (Stanford), Nikhil Srivastava\n(Berkeley), Rachel Ward (UT Austin) and Lauren Williams (Harvard). Its third batch tested five systems: two lab entries\n(GPT-6 Astra; Anthropic's Siege workflow on Claude Opus 5.5), two academic harnesses (ProofCouncil, MerLean) and, for the\nfirst time, a harness on an open-weight model (Kimi K3). All ten problems were solved by at least one system.\n\n## Why it matters\n\nWithin a year the best result went from \"a few\" (February) to 6–7 (June) to 9 of 10 (October) on fresh research problems with\nstrict peer-review grading. The batch also gives hard numbers on two concerns: cost (a single Astra query solves 7 for $36;\nsqueezing out more needs harnesses costing about $2,000) and the gap between closed and open models, which the organisers find\nmuch larger in research maths than in general benchmarks. Anthropic's entry showed a different trade-off: fewer answers, but\nno negatively scored submissions under the new scoring.","disputed":[],"related":[{"id":"2026-10-06-openai-math-release-722-manuscripts","url":"https://postcutoff.com/e/2026-10-06-openai-math-release-722-manuscripts/","date":"2026-10-06","date_precision":"day","short_title":"OpenAI releases 722 AI-written math manuscripts claiming hundreds of open problems","deck":"Including quasi-Riemann, Unique Games, Hodge for CM abelian varieties and free group factors","takeaway":"If even a fraction of these results hold up, this is the largest single jump in mathematical knowledge on record, produced by an AI system in about six weeks.","category":"science","category_label":"Science & math","importance":5,"confidence":"high","status":{"key":"pending","labels":["Event confirmed","Awaiting review"]},"sources":84,"official":13,"filed":"2026-10-07","updated":"2026-10-14","orgs":["OpenAI"]},{"id":"2026-06-10-first-proof-second-batch","url":"https://postcutoff.com/e/2026-06-10-first-proof-second-batch/","date":"2026-06-10","date_precision":"day","short_title":"AI systems pass 7 of 10 unpublished research problems in First Proof's second batch","deck":null,"takeaway":"It is the most carefully refereed measurement so far of AI on genuine research problems.","category":"benchmark","category_label":"Benchmarks","importance":3,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":6,"official":4,"filed":"2026-10-05","updated":"2026-10-05","orgs":["First Proof Foundation","ETH Zurich","OpenAI","UCLA","Princeton University"]},{"id":"2026-02-14-first-proof-challenge","url":"https://postcutoff.com/e/2026-02-14-first-proof-challenge/","date":"2026-02-14","date_precision":"day","short_title":"AI solves about half of 10 unpublished research problems in the 'First Proof' challenge","deck":null,"takeaway":"It gave a cleaner measure than olympiads of whether AI can do research maths: at the time, about half the time.","category":"science","category_label":"Science & math","importance":3,"confidence":"high","status":{"key":"confirmed","labels":["Result confirmed"]},"sources":3,"official":2,"filed":"2026-09-29","updated":"2026-10-05","orgs":["Google DeepMind","OpenAI"]}],"people":[{"id":"terence-tao","name":"Terence Tao","url":"https://postcutoff.com/person/terence-tao/"}],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-10-14","type":"filed","text":"Created from the batch 3 report, results page and Scientific American"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-10-14","run":null,"sources_read":"The batch 3 report, results page and Scientific American","updated":"2026-10-14","human_review":null,"version":null},"gaps":[{"model_id":"gpt-6-astra","name":"GPT-6 Astra","cutoff":"2026-04","days_after":167,"in_training_data":false},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":106,"in_training_data":false},{"model_id":"gemini-4-argon","name":"Gemini 4 Argon","cutoff":"2026-03","days_after":197,"in_training_data":false},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":136,"in_training_data":false}],"short_url":"https://postcutoff.com/s/first-proof-batch-3"}