First Proof batch 3: frontier systems solve 9 of 10 research problems, open-weight Kimi K3 harness none
Confirmed
Importance: major (4 of 5)The takeaway
On Oct 14, 2026 the First Proof Foundation (Abouzaid, Srivastava, Ward, Williams) published its third refereed batch: ten solved but unpublished research problems, one 24-hour shot per system with no human input, graded by 30 mathematicians with negative marks for wrong answers.
Status
- Claim
Confirmed
- Our reporting
- High confidence
- Importance
- Major (4 of 5)
- Last verified
- 14 October 2026
Your AI and this story
- GPT-6 Astra167 days after its cutoff
- Claude Opus 5.5106 days after its cutoff
- Gemini 4 Argon197 days after its cutoff
- Grok 4.7136 days after its cutoff
None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 106 days before it.
Key facts
- Problems: 10, from logarithmic geometry, statistical inverse problems, quantum topology, symplectic geometry, hyperfields, low-dimensional topology, complex and other dynamics, Teichmüller theory and algebraic geometry; each had a human proof of at most 12 pages not yet public
- Process: problems refined at Harvard CMSA Aug 13–14; testing Oct 2–5 (24 hours per system, no human input); 30 referees graded anonymised solutions at CMU’s ICARM Oct 8–9; results and webinar Oct 14
- New scoring: from −1 to 1 for correctness, exposition and attribution, so wrong solutions claimed as correct are penalised and no answer or an acknowledged partial result is better
- Problems solved (correctness ≥ 0.5): ProofCouncil 9 (avg 0.87); GPT-6 Astra at max reasoning effort 7 (0.65); MerLean (GPT-6 Astra + GPT-6.1 Sol) 4 (0.23); Siege (Claude Opus 5.5, high effort) 3 (0.25); Humanize (Kimi K3) 0 (−0.86)
- Cost for 10 problems: GPT-6 Astra $36; Siege $1,400; ProofCouncil $1,888; Humanize ~$2,127 at API-equivalent prices (about 400 self-hosted GPU-hours); MerLean $5,323. Batch 2 comparison: ProofCouncil $3,200 and the plain OpenAI model $117
- Siege is a Claude Code skill in Anthropic’s public math-proof plugin; it runs up to 14 multi-agent rounds and submits only on a ‘PROVED’ verdict. It declined problems 3, 4, 8, 9 and 10 (scored 0) and ran without web search; on Problem 6 no referee could determine its correctness
- ProofCouncil team: Tim Gehrunger (lead), Johannes Schmitt, David Holmes, Jasper Dekoninck and others; base models GPT-6 Astra Pro, GPT-6 Astra, Claude Fable 5.1, Claude Sonnet 5, Gemini 3.1 Pro preview
- Failures noted: ProofCouncil got stuck in a loop on Problem 9 and destroyed a correct Astra solution; MerLean ‘cheated’ on Problem 10 with a broken definition from an old arXiv version; Astra’s Problem 5 solution ‘essentially constituted plagiarism’; only 4 of 10 Astra solutions had adequate citations (ProofCouncil 7)
Show 3 more
- Open weights: referees found the Kimi K3 system generally weaker than GPT-5.2 Pro (Dec 2025); report: ‘We urge the community to immediately invest in improving open-source/open-weight models.’ Lauren Williams to Scientific American: Kimi K3 ‘really fell flat on its face’
- Not tested: Google did not take part; OpenAI’s newer internal model was excluded because entrants must be openly available (Scientific American)
- Funding: unrestricted industry donations for Batches 2 and 3 (acknowledgments list Anthropic, OpenAI and Google.org), used for graders, never for editors or board. Batch 4 runs Oct 2026–Jan 2027, results in the first half of January
What happened
First Proof is the refereed exam for AI in research mathematics run by Mohammed Abouzaid (Stanford), Nikhil Srivastava (Berkeley), Rachel Ward (UT Austin) and Lauren Williams (Harvard). Its third batch tested five systems: two lab entries (GPT-6 Astra; Anthropic’s Siege workflow on Claude Opus 5.5), two academic harnesses (ProofCouncil, MerLean) and, for the first time, a harness on an open-weight model (Kimi K3). All ten problems were solved by at least one system.
Why it matters
Within a year the best result went from “a few” (February) to 6–7 (June) to 9 of 10 (October) on fresh research problems with strict peer-review grading. The batch also gives hard numbers on two concerns: cost (a single Astra query solves 7 for $36; squeezing out more needs harnesses costing about $2,000) and the gap between closed and open models, which the organisers find much larger in research maths than in general benchmarks. Anthropic’s entry showed a different trade-off: fewer answers, but no negatively scored submissions under the new scoring.
Sources
5 sources from 4 sites. Numbers match the chips in the text.
5 sources: 3 primary, 1 press, 1 reaction
Primary
- First Proof Third Batch report (Oct 14, 2026, PDF)1stproof.org, paper
- First Proof: Third Batch results page1stproof.org, official
- Anthropic math-proof plugin (Siege skill)github.com, code
Press
- Scientific American: New ‘First Proof’ exam reveals how AI-powered math is a pay-for-play sport (Oct 14)scientificamerican.com, press
Reactions
- Terence Tao: First Proof, Third Batch webinar (Oct 14)terrytao.wordpress.com, discussion
Changes
- Filed from the batch 3 report, results page and Scientific American