Benchmarks: everything we have
Every benchmarks event we have logged, with sources, in one place.
38 events (27 after June 2026), 0 models, 1 video, 23 posts. Latest event 16 October 2026.
Download everything as one Markdown file (118 KB) Full text and every source of each event, plus models, upcoming items, videos and posts. Readable as plain text, and ready to give to an AI model.
Key moments
- ARC Prize verifies Gemini 4 Argon at 97.5% on ARC-AGI-3 with Google’s harness (49.8% standard), the second lab above 95%On Oct 15, 2026 ARC Prize published verified scores for Gemini 4 Argon: 97.5% on ARC-AGI-3 with the provider-adapter harness ($4.8K) and 49.8% with the standard harness ($5.8K), plus 91.3% on ARC-AGI-2 and 98.5% on…
- First Proof third batch: ProofCouncil solves 9 of 10 unpublished research problems, GPT-6 Astra 7 for $36, Anthropic’s Siege 3 with no negative scores, open-weight Kimi K3 harness 0On Oct 14, 2026 the First Proof Foundation (Abouzaid, Srivastava, Ward, Williams) published its third refereed batch: ten solved but unpublished research problems, one 24-hour shot per system with no human input, graded…
- GPT-6 Astra scores 62.7% on ARC-AGI-3 (99.9% with provider harness), outacting humans on 96% of levelsARC Prize reported on 2026-09-03 that OpenAI’s GPT-6 Astra scored 62.7% on ARC-AGI-3 (semi-private) with the standard harness ($26K) and 99.9% ($19K) with OpenAI’s own provider-adapter harness, using fewer actions than…
- ARC Prize launches ARC-AGI-3, an interactive game benchmark where frontier AI scored under 1%The ARC Prize Foundation launched ARC-AGI-3 on 2026-03-25: novel turn-based game environments with no instructions, measuring skill-acquisition efficiency. In the preview humans solved 100% of environments while…
- AI reaches gold-medal level at the ICPC World FinalsAt the 2025 ICPC World Finals in Baku, OpenAI’s reasoning system solved all 12 problems and Google’s Gemini 2.5 Deep Think solved 10 of 12, both at gold-medal level, under the same time limits as human teams.
- OpenAI announces o3, scoring 75.7–87.5% on ARC-AGIOn the last day of its ‘12 Days of OpenAI’, OpenAI previewed o3, which scored 75.7% on the ARC-AGI semi-private set (87.5% with high compute) — a benchmark on which earlier LLMs scored in single digits — and 25.2% on…
- ImageNet dataset presented at CVPR 2009Fei-Fei Li’s team introduced ImageNet, a large hand-labeled image database organized by the WordNet hierarchy; its annual ILSVRC challenge (from 2010) became the proving ground for deep learning.
2026
- Mercor’s ‘benchmark map of the US economy’: ~90% of US working hours go to tasks AI benchmarks barely cover
- StarSkirmish Bench v0.2: GPT-6 Astra leads nine models at writing StarCraft bots in one hour; Gemini 4 Argon, Opus 5.5 and Fable 5.1 tie
- ARC Prize verifies Gemini 4 Argon at 97.5% on ARC-AGI-3 with Google’s harness (49.8% standard), the second lab above 95%On Oct 15, 2026 ARC Prize published verified scores for Gemini 4 Argon: 97.5% on ARC-AGI-3 with the provider-adapter harness ($4.8K) and 49.8% with the standard harness ($5.8K), plus 91.3% on ARC-AGI-2 and 98.5% on…
- Design Creativity Bench: frontier models’ UI designs converge (‘diversity collapse’); best model GPT-6 Astra scores 65.5 vs 92.3 for the human reference
- First Proof third batch: ProofCouncil solves 9 of 10 unpublished research problems, GPT-6 Astra 7 for $36, Anthropic’s Siege 3 with no negative scores, open-weight Kimi K3 harness 0On Oct 14, 2026 the First Proof Foundation (Abouzaid, Srivastava, Ward, Williams) published its third refereed batch: ten solved but unpublished research problems, one 24-hour shot per system with no human input, graded…
- Jack Clark moves his forecast for ‘full RSI’ to 2027, citing TasteVal: Opus 5.5 beats ML experts at designing experiments (2.3x compute multiplier)In Import AI 476 (Oct 12, 2026) Anthropic co-founder Jack Clark wrote “I now think it’s more likely we’ll see full RSI become possible in 2027, whereas before I thought it was more likely in 2028”. His evidence is…
- TheoPhysEval: GPT-6 Astra in Codex solves 49 of 50 unpublished research-level theoretical-physics problems (98%), Opus 5.5 in Claude Code 92%arXiv 2610.16733 (9 Oct 2026; Suvrat Raju, Praneeth Netrapalli and 35 co-authors from ICTS, TIFR, Harvard, Penn, UCSD, Wisconsin, KU Leuven, Google DeepMind and others) introduces TheoPhysEval: 50 unpublished…
- Scale AI’s DistressBench: 25 frontier models spot users in crisis but the median one fails to hand off to help in ~35% of casesOn Oct 9, 2026 Scale AI published DistressBench (shared first with TIME): 19 licensed clinicians and crisis counselors wrote 718 realistic crisis conversations, and 25 frontier models from OpenAI, Anthropic, Google and…
- OpenProblemBench: 82 unresolved math and theoretical-physics problems; GPT-6 Astra has the highest model-judged solve rate (14%)
- Humanity’s Sixth Sense (Scale AI and Elorian): people score 93.1% on intuitive visual reasoning, the best model (GPT-6 Astra) 53.6%
- Arena (ex-LMArena) raises $200M at $3.1B and launches an AI Alignment Index for agents; OpenAI models lead, Claude Opus 5.5 is 6thOn Oct 8, 2026 Arena, the crowdsourced model-leaderboard company that began as UC Berkeley’s Chatbot Arena, raised a $200M Series B at a $3.1B valuation (up from $1.7B in January), led by Lightspeed and Khosla. With it…
- Epoch AI: GPT-6 Astra nearly saturates the EBR-bench board-game benchmark; poll finds no rise in US cyber incidents since Claude Fable 5
- Epoch AI’s InnovationEval: Claude Fable 5 and GPT-5.6 Sol fail to reinvent a recent ML method and overstate their own resultsOn Oct 7, 2026 Epoch AI published “Can AI automate AI R&D yet?”, the first results of InnovationEval, an end-to-end test of whether an agent can invent a post-training method as good as a recent human one (SDPO) that it…
- ARC Prize 2026 Milestone #2: open-source ARC-AGI-3 agents top out at 27.9% under Kaggle compute limits
- Artificial Analysis launches the Cyber Index and an industry alliance (IBM, NVIDIA, Vercel, Collinear) for AI vulnerability-fixing evals
- SWE-Game benchmark: coding agents building Godot games from briefs score at most 50/100; Claude Opus 5 leads all five task types
- C5R opens Facility-0, an AI-run wet lab, and the SciUniverse benchmark: Claude Fable 5.1 leads with 45.3%On Sept 24, 2026 San Francisco startup C5R (founder Michael Akilian) launched Facility-0, a physical lab where frontier models plan experiments and run them by controlling instruments and instructing human operators. It…
- DrivingBench: GPT-6 Astra is the first frontier LLM to complete a cone course driving a real Toyota Corolla
- OpenAI releases MentalHealthBench, an open benchmark for AI in mental-health conversationsOn Sept 23, 2026 OpenAI released MentalHealthBench: 1,215 synthetic mental-health conversations with 5,262 rubric criteria written with 80+ licensed clinicians from 22 countries. It scores safety, context-seeking, user…
- CAIS releases HLE-Diamond, a refined 1,000-question Humanity’s Last Exam; GPT-6 Astra scores 82.9% with toolsAround Sept 22, 2026 the Center for AI Safety released HLE-Diamond, a cleaned 1,000-question subset of Humanity’s Last Exam (500 reasoning, 500 knowledge). GPT-6 Astra scored 82.9% with web and code tools (59.9%…
- MLPerf Inference v6.1: record 30 submitters, first peer-reviewed Vera Rubin NVL72 results, new RAG and edge-agentic testsOn Sept 16, 2026 MLCommons published MLPerf Inference v6.1 results with a record 30 submitting organizations, two new tests (End-to-End RAG and Edge Agentic Inference) and the first peer-reviewed numbers for NVIDIA’s…
- Qwen and Taobao release E-Commerce Bench: 18 models run online stores for a simulated year
- GPT-6 Astra scores 62.7% on ARC-AGI-3 (99.9% with provider harness), outacting humans on 96% of levelsARC Prize reported on 2026-09-03 that OpenAI’s GPT-6 Astra scored 62.7% on ARC-AGI-3 (semi-private) with the standard harness ($26K) and 99.9% ($19K) with OpenAI’s own provider-adapter harness, using fewer actions than…
- Booz Allen’s first Cyber Weapon Index: of 18 US and Chinese models run as autonomous attackers, only Claude Mythos completes the full kill chainIn early September 2026 Booz Allen Hamilton published its first Cyber Weapon Index, which ran 18 frontier models (nine US, nine Chinese) as fully autonomous attackers against a production-grade enterprise network.…
- BixBench3 tests AI agents on whole computational-biology studies from raw data; best score 0.48 (GPT-5.6 Sol)
- Artificial Analysis launches the Speech Agent Arena for speech-to-speech voice agents
- Project Pilot: Anthropic and Andon Labs test whether AI models can fly a surveillance drone (Drone-Bench)On July 24, 2026 Anthropic’s Frontier Red Team and Andon Labs published Project Pilot, which tested 15 models from three developers on autonomously coding and flying an indoor drone to find and follow a person. Claude…
- OpenAI releases LifeSciBench, 750 expert-written life-science research tasks; GPT-Rosalind leads with a 36% pass rate
- First Proof second batch: refereed benchmark finds AI systems pass 7 of 10 unpublished research problems; ETH’s ProofCouncil harness best with 6On June 10, 2026 the First Proof Foundation (Abouzaid, Srivastava, Ward, Williams) released the formal second batch of its research-math benchmark. Ten solved but unpublished problems from working mathematicians were…
- ARC Prize launches ARC-AGI-3, an interactive game benchmark where frontier AI scored under 1%The ARC Prize Foundation launched ARC-AGI-3 on 2026-03-25: novel turn-based game environments with no instructions, measuring skill-acquisition efficiency. In the preview humans solved 100% of environments while…
- Ai2 launches MolmoSpaces, an open simulation ecosystem and leaderboard for generalist robot policies
- Anthropic: infrastructure settings alone move Terminal-Bench 2.0 scores by 6 points
- METR releases Time Horizon 1.1 with expanded long-task suiteMETR updated its task-completion time-horizon methodology on 2026-01-29 (TH1.1), adding 34% more tasks (228 vs 170) and doubling 8h+ tasks (31 vs 14), tightening confidence intervals for frontier models; METR notes…
2025
- Arc Institute announces first Virtual Cell Challenge winners; a 2026 zero-shot round follows
- AI reaches gold-medal level at the ICPC World FinalsAt the 2025 ICPC World Finals in Baku, OpenAI’s reasoning system solved all 12 problems and Google’s Gemini 2.5 Deep Think solved 10 of 12, both at gold-medal level, under the same time limits as human teams.
- RoboArena: crowd-sourced, double-blind real-world evaluation of generalist robot policies
2024
- OpenAI announces o3, scoring 75.7–87.5% on ARC-AGIOn the last day of its ‘12 Days of OpenAI’, OpenAI previewed o3, which scored 75.7% on the ARC-AGI semi-private set (87.5% with high compute) — a benchmark on which earlier LLMs scored in single digits — and 25.2% on…
2009
- ImageNet dataset presented at CVPR 2009Fei-Fei Li’s team introduced ImageNet, a large hand-labeled image database organized by the WordNet hierarchy; its annual ILSVRC challenge (from 2010) became the proving ground for deep learning.
Videos
- Anthropic’s Co-Founder Just Moved the Timeline… AI Copium, 2026-10-14