{"schema":"postcutoff/event@1","as_of":"2026-11-19T22:11:00+01:00","url":"https://postcutoff.com/e/2026-08-26-bixbench3-computational-biology-agents/","md":"https://postcutoff.com/e/2026-08-26-bixbench3-computational-biology-agents/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2026-08-26-bixbench3-computational-biology-agents","date":"2026-08-26","date_precision":"day","short_title":"BixBench3 tests AI agents on whole computational-biology studies from raw data","deck":"Best score 0.48 (GPT-5.6 Sol)","takeaway":"BixBench3 (arXiv 2608.25286, Aug 26, 2026), from Edison Scientific (Zane Koch, Jon M. Laurent, Samuel G. Rodriques, Andrew D. White and colleagues), gives an agent a research objective, methods guidance from a published biology paper, and that study's raw data.","category":"benchmark","category_label":"Benchmarks","importance":2,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":[{"n":1,"title":"arXiv 2608.25286: BixBench3, benchmarking AI agents on research-study-scale computational biology tasks","url":"https://arxiv.org/abs/2608.25286","type":"paper","group":"primary","domain":"arxiv.org"},{"n":2,"title":"GitHub: EdisonScientific/BixBench3 (harness and grader)","url":"https://github.com/EdisonScientific/BixBench3","type":"code","group":"primary","domain":"github.com"},{"n":3,"title":"Edison Advances: BixBench3 benchmark page","url":"https://advances.edisonscientific.com/benchmarks/bixbench3/","type":"official","group":"primary","domain":"advances.edisonscientific.com"},{"n":4,"title":"BixBench (original), arXiv 2503.00096","url":"https://arxiv.org/abs/2503.00096","type":"paper","group":"primary","domain":"arxiv.org"},{"n":5,"title":"Kiin Bio newsletter: Google's GlucoFM, Foldseek-Interface and BixBench3","url":"https://newsletter.kiin.bio/p/googles-glucofm-seoul-national-universitys","type":"press","group":"press","domain":"newsletter.kiin.bio"}],"official":4,"filed":"2026-10-04","updated":"2026-10-04","orgs":["Edison Scientific"],"title":"BixBench3 tests AI agents on whole computational-biology studies from raw data; best score 0.48 (GPT-5.6 Sol)","summary":"BixBench3 (arXiv 2608.25286, Aug 26, 2026), from Edison Scientific (Zane Koch, Jon M. Laurent, Samuel G. Rodriques, Andrew D. White and colleagues), gives an agent a research objective, methods guidance from a published biology paper, and that study's raw data. The agent must reproduce the paper's results. Across 20 studies and 138 graded artifacts, 13 frontier models scored between 0.00 and 0.48, and they did much worse on datasets over 100 GB.","key_facts":["20 tasks from published studies, 17 assay types, 9 scientific domains, 138 graded artifacts compared with the originals","Top scores: GPT-5.6 Sol 0.48, Kimi K3 0.47, GLM 5.2 0.46, Claude Opus 4.8 0.46, Claude Opus 5 0.41; lowest Gemini 3.1 Flash Lite 0.00","Data scale hurts: ~0.36–0.37 on smaller datasets vs 0.10 on tasks over 100 GB; analyses of depth ≥3 dropped to 0.24","Cost per attempt: on average 6.8 hours, 102M tokens and $43; maximum 24 hours, 1.07B tokens and $525. Better models used fewer resources","Common failures: premature termination and repetitive retry loops (~2× more frequent in the worst attempts)","Public harness and deterministic grader at github.com/EdisonScientific/BixBench3; 20-task dataset on Hugging Face; v2 posted Sept 22, 2026","Successor to BixBench (arXiv 2503.00096, 2025), the benchmark OpenAI cited for GPT-Rosalind"],"key_numbers":[],"tags":["benchmark","biology","bioinformatics","science-agents","evaluation","open-source"],"science":null,"body_md":"## What happened\n\nBixBench3 moves biology-agent evaluation from question answering to reproducing whole studies from raw data, the work a computational biologist\nactually does. Frontier agents got roughly halfway on the best runs, and large data volumes and long analysis chains broke them.\n\n## Why it matters\n\nIt is an end-to-end test for \"AI scientist\" claims in biology. Its finding that open models from China (Kimi K3, GLM 5.2) score close to\nthe top closed model is also notable.","disputed":[],"related":[{"id":"2026-06-17-openai-lifescibench","url":"https://postcutoff.com/e/2026-06-17-openai-lifescibench/","date":"2026-06-17","date_precision":"day","short_title":"OpenAI releases LifeSciBench, 750 expert-written life-science research tasks","deck":"GPT-Rosalind leads with a 36% pass rate","takeaway":"It is a large, expert-graded measure of AI for biology research.","category":"benchmark","category_label":"Benchmarks","importance":2,"confidence":"medium","status":{"key":"partly","labels":["Partly confirmed"]},"sources":4,"official":1,"filed":"2026-10-04","updated":"2026-10-04","orgs":["OpenAI"]},{"id":"2026-04-17-openai-gpt-rosalind","url":"https://postcutoff.com/e/2026-04-17-openai-gpt-rosalind/","date":"2026-04-17","date_precision":"day","short_title":"OpenAI launches GPT-Rosalind, a trusted-access reasoning model for life-sciences research","deck":null,"takeaway":"It is part of the 2026 race among frontier labs for AI-for-science products (Anthropic's Claude Science, Google's Gemini for Science).","category":"model-release","category_label":"Model releases","importance":3,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":10,"official":6,"filed":"2026-09-29","updated":"2026-10-13","orgs":["OpenAI"]},{"id":"2025-11-05-kosmos-ai-scientist","url":"https://postcutoff.com/e/2025-11-05-kosmos-ai-scientist/","date":"2025-11-05","date_precision":"month","short_title":"Edison Scientific's Kosmos AI scientist claims six months of research per run","deck":null,"takeaway":"It is an early commercial \"AI scientist\" whose headline value is reproducing months-long analyses overnight.","category":"science","category_label":"Science & math","importance":3,"confidence":"medium","status":{"key":"pending","labels":["Awaiting review"]},"sources":3,"official":2,"filed":"2026-09-29","updated":"2026-09-29","orgs":["Edison Scientific","FutureHouse"]}],"people":[{"id":"sam-rodriques","name":"Sam Rodriques","url":"https://postcutoff.com/person/sam-rodriques/"}],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-10-04","type":"filed","text":"Created"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-10-04","run":null,"sources_read":null,"updated":"2026-10-04","human_review":null,"version":{"date":"2026-10-04"}},"gaps":[{"model_id":"gpt-6-2-astra","name":"GPT-6.2 Astra","cutoff":"2026-05","days_after":87,"in_training_data":false},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":57,"in_training_data":false},{"model_id":"gemini-4-hydrogen","name":"Gemini 4 Hydrogen","cutoff":"2026-03","days_after":148,"in_training_data":false},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":87,"in_training_data":false}],"short_url":null}