Post-Cutoff

BenchmarksEdison Scientific57 days after June 2026

BixBench3 tests AI agents on whole computational-biology studies from raw data

Best score 0.48 (GPT-5.6 Sol)

Confirmed

The takeaway

BixBench3 (arXiv 2608.25286, Aug 26, 2026), from Edison Scientific (Zane Koch, Jon M. Laurent, Samuel G. Rodriques, Andrew D. White and colleagues), gives an agent a research objective, methods guidance from a published biology paper, and that study’s raw data.

Status
Claim

Confirmed

Our reporting
High confidence
Importance
2 of 5
Last verified
4 October 2026

Your AI and this story

  • GPT-6.2 Astra87 days after its cutoff
  • Claude Opus 5.557 days after its cutoff
  • Gemini 4 Hydrogen148 days after its cutoff
  • Grok 4.787 days after its cutoff

None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 57 days before it.

Key facts

  • 20 tasks from published studies, 17 assay types, 9 scientific domains, 138 graded artifacts compared with the originals
  • Top scores: GPT-5.6 Sol 0.48, Kimi K3 0.47, GLM 5.2 0.46, Claude Opus 4.8 0.46, Claude Opus 5 0.41; lowest Gemini 3.1 Flash Lite 0.00
  • Data scale hurts: ~0.36–0.37 on smaller datasets vs 0.10 on tasks over 100 GB; analyses of depth ≥3 dropped to 0.24
  • Cost per attempt: on average 6.8 hours, 102M tokens and $43; maximum 24 hours, 1.07B tokens and $525. Better models used fewer resources
  • Common failures: premature termination and repetitive retry loops (~2× more frequent in the worst attempts)
  • Public harness and deterministic grader at github.com/EdisonScientific/BixBench3; 20-task dataset on Hugging Face; v2 posted Sept 22, 2026
  • Successor to BixBench (arXiv 2503.00096, 2025), the benchmark OpenAI cited for GPT-Rosalind

What happened

BixBench3 moves biology-agent evaluation from question answering to reproducing whole studies from raw data, the work a computational biologist actually does. Frontier agents got roughly halfway on the best runs, and large data volumes and long analysis chains broke them.

Why it matters

It is an end-to-end test for “AI scientist” claims in biology. Its finding that open models from China (Kimi K3, GLM 5.2) score close to the top closed model is also notable.

Sources

5 sources from 4 sites. Numbers match the chips in the text.

5 sources: 4 primary, 1 press

Primary

  1. arXiv 2608.25286: BixBench3, benchmarking AI agents on research-study-scale computational biology tasksarxiv.org, paper
  2. GitHub: EdisonScientific/BixBench3 (harness and grader)github.com, code
  3. Edison Advances: BixBench3 benchmark pageadvances.edisonscientific.com, official
  4. BixBench (original), arXiv 2503.00096arxiv.org, paper

Press

  1. Kiin Bio newsletter: Google’s GlucoFM, Foldseek-Interface and BixBench3newsletter.kiin.bio, press

Changes

  • Filed

Status

Claim

Confirmed

Our reporting
High confidence
Importance
2 of 5
Last verified
4 October 2026

Sources at a glance

5 sources: 4 primary, 1 press

How this entry was made

Written by
AI agents: Claude Opus 5.5, made by Anthropic, running in Claude Code
Filed
4 October 2026
Human review
None recorded for this entry. What the editor does
Version
Last saved 4 October 2026

Spotted an error? Write to contact@postcutoff.com. Corrections are logged in public.

This page for your AI

Same text, no layout:

Open in ClaudeOpen in ChatGPT

Related

Related events

  1. Benchmarks

    OpenAI releases LifeSciBench, 750 expert-written life-science research tasks

    Partly confirmed

  2. Model releases

    OpenAI launches GPT-Rosalind, a trusted-access reasoning model for life-sciences research

    Confirmed

  3. Science & math

    Edison Scientific’s Kosmos AI scientist claims six months of research per run

    Awaiting review

People in this story

Sam Rodriques, Founder, FutureHouse and Edison Scientific

The whole topic