{"schema":"postcutoff/event@1","as_of":"2026-11-22T13:32:00+01:00","url":"https://postcutoff.com/e/2026-06-10-first-proof-second-batch/","md":"https://postcutoff.com/e/2026-06-10-first-proof-second-batch/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2026-06-10-first-proof-second-batch","date":"2026-06-10","date_precision":"day","short_title":"AI systems pass 7 of 10 unpublished research problems in First Proof's second batch","deck":null,"takeaway":"On June 10, 2026 the First Proof Foundation (Abouzaid, Srivastava, Ward, Williams) released the formal second batch of its research-math benchmark.","category":"benchmark","category_label":"Benchmarks","importance":3,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":[{"n":1,"title":"First Proof Second Batch report (June 10, 2026)","url":"https://1stproof.org/assets/docs/report.pdf","type":"paper","group":"primary","domain":"1stproof.org"},{"n":2,"title":"arXiv 2606.18119: First Proof Second Batch","url":"https://arxiv.org/abs/2606.18119","type":"paper","group":"primary","domain":"arxiv.org"},{"n":3,"title":"First Proof Project","url":"https://1stproof.org/","type":"official","group":"primary","domain":"1stproof.org"},{"n":4,"title":"arXiv 2607.09474: ProofCouncil, an LLM agent for solving open mathematical problems","url":"https://arxiv.org/abs/2607.09474","type":"paper","group":"primary","domain":"arxiv.org"},{"n":5,"title":"Harvard Math: First Proof's second batch of math problems test AI","url":"https://www.math.harvard.edu/first-proofs-second-batch-of-math-problems-test-ai/","type":"press","group":"press","domain":"math.harvard.edu"},{"n":6,"title":"Nature news (June 12, 2026): Humans outperform AI at this highly rigorous mathematics test (PDF copy)","url":"https://people.math.harvard.edu/~williams/Nature-SecondBatch.pdf","type":"press","group":"press","domain":"people.math.harvard.edu"}],"official":4,"filed":"2026-10-05","updated":"2026-10-05","orgs":["First Proof Foundation","ETH Zurich","OpenAI","UCLA","Princeton University"],"title":"First Proof second batch: refereed benchmark finds AI systems pass 7 of 10 unpublished research problems; ETH's ProofCouncil harness best with 6","summary":"On June 10, 2026 the First Proof Foundation (Abouzaid, Srivastava, Ward, Williams) released the formal second batch of its research-math benchmark. Ten solved but unpublished problems from working mathematicians were run by the organisers themselves on four public systems: ChatGPT 5.5 Pro and three academic harnesses. Each of the 39 solutions was refereed by 2–3 expert mathematicians. Seven problems got at least one passing solution (essentially flawless or minor revisions). The ETH Zurich/Aarhus ProofCouncil harness (System A, built on gpt-5.5-pro) passed 6. UCLA's harness and plain ChatGPT 5.5 Pro passed 5 each, the latter for $117 in total versus $3,186 and $4,799 for the harnesses. Princeton's Gemini-based harness passed 1, and no system made progress on the metric-geometry problem.","key_facts":["Systems: A = IMProofBench ProofCouncil (ETH Zurich/Aarhus; base gpt-5.5-pro, limited use of gpt-5.5, gemini-3.1-pro-preview, claude-opus-4-7); B = UCLA Moonshot harness (gpt-5.5-pro); C = OpenAI ChatGPT 5.5 Pro; D = Princeton Momus (gemini-3.1-pro-preview)","Passing grades per system (our tally of the report's editorial decisions): A 6 of 9 graded (P1, P2, P3, P5, P7, P9; no P6 solution due to a technical issue), B 5 (P1, P2, P6, P7, P9), C 5 (P1, P2, P6, P7, P9), D 1 (P7). Problems 1, 2, 3, 5, 6, 7, 9 had at least one pass; P8 and P10 only 'major revisions'; P4 (metric geometry) no progress","Problem 5 (stochastic PDE): System A's solution rated essentially flawless, with a novel approach different from the human solution that 'impressed the referees'; AI arguments also differed from the human ones on Problems 3 and 9","Cost and time: A $3,186 / 22.9 h; B $4,799 / 23.1 h; C $117 / ~5.8 h; D ~$1,014 (imputed) / 7.8 h. The report: harnesses can improve on their base model, 'However, this improvement comes at a substantial financial cost'","Referee themes: routine steps handled meticulously while the hardest steps were glossed as 'standard arguments'; citations to papers that do not contain the claimed results; several Problem 2 solutions copied the author's earlier paper's terminology and labels without citing it, which for a human 'would have been flagged for plagiarism'","Process: problems collected March–May; systems fixed by May 28; grading with 30 expert mathematicians at Harvard CMSA June 4–5; report and webinar June 10; arXiv 2606.18119 (June 16)","ProofCouncil paper (arXiv 2607.09474, July 10; Schmitt, Gehrunger, Dekoninck, Bérczi, Kreitner, Price, Holmes): author–critic agent, open-sourced as eth-sri/proof-council; on 30 researcher-submitted open problems, 5 judged completely correct, 2 promising pending verification, 8 with useful partial progress"],"key_numbers":[],"tags":["math","benchmark","research-math","gpt-5-5-pro","agents","harness","evaluation"],"science":null,"body_md":"## What happened\n\nAfter the informal first batch in February (2026-02-14-first-proof-challenge), the organisers set up the First Proof Foundation and ran batch 2 as a controlled benchmark. They tested only public systems, or harnesses over public models, ran every test themselves with full logs, and had each solution formally refereed. A smaller community-experimentation set was promised to follow.\n\n## Why it matters\n\nIt is the most carefully refereed measurement so far of AI on genuine research problems. By mid-2026, public systems produced publishable-quality solutions to most such problems, sometimes with new ideas. A plain chat model nearly matched harnesses that cost 30–40× more. The failure modes are the ones referees keep reporting: skipped hard steps and missing or false citations.","disputed":[],"related":[{"id":"2026-02-14-first-proof-challenge","url":"https://postcutoff.com/e/2026-02-14-first-proof-challenge/","date":"2026-02-14","date_precision":"day","short_title":"AI solves about half of 10 unpublished research problems in the 'First Proof' challenge","deck":null,"takeaway":"It gave a cleaner measure than olympiads of whether AI can do research maths: at the time, about half the time.","category":"science","category_label":"Science & math","importance":3,"confidence":"high","status":{"key":"confirmed","labels":["Result confirmed"]},"sources":3,"official":2,"filed":"2026-09-29","updated":"2026-10-05","orgs":["Google DeepMind","OpenAI"]}],"people":[],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-10-05","type":"filed","text":"Created (arXiv 2607.09474)"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-10-05","run":null,"sources_read":null,"updated":"2026-10-05","human_review":null,"version":{"date":"2026-10-07"}},"gaps":[{"model_id":"gpt-6-2-astra","name":"GPT-6.2 Astra","cutoff":"2026-05","days_after":10,"in_training_data":false},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":null,"in_training_data":true},{"model_id":"gemini-4-hydrogen","name":"Gemini 4 Hydrogen","cutoff":"2026-03","days_after":71,"in_training_data":false},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":10,"in_training_data":false}],"short_url":null}