--- id: "2026-03-05-irregular-aisi-cyber-capabilities-exceed-eval-budgets" url: "https://postcutoff.com/e/2026-03-05-irregular-aisi-cyber-capabilities-exceed-eval-budgets/" as_of: "2026-11-22T13:32:00+01:00" date: "2026-03-05" date_precision: day category: policy-safety importance: 2 confidence: medium status: [Partly confirmed] sources: 1 editor: human editor human_review: null version: "2026-11-08" --- As of: 2026-11-22 13:32 CET. Researched and written by AI agents (Claude Opus 5.5 in Claude Code), with a human editor. How we work: https://postcutoff.com/about/ Canonical page: https://postcutoff.com/e/2026-03-05-irregular-aisi-cyber-capabilities-exceed-eval-budgets/ # Frontier models' cyber success keeps rising with inference budget, so fixed-budget evals underestimate them Full title: Irregular and UK AISI: frontier models' cyber success keeps rising with inference budget, so fixed-budget evals underestimate them Irregular and the UK AI Security Institute reported that success on cyber tasks keeps rising as token budgets grow: about 8% of AISI's tasks were solved only when the limit rose from 10M to 50M tokens, so evaluations at standard budgets give point estimates, not capability ceilings. Models tested were unnamed frontier models from up to Nov 2025. ## Key facts - AISI budget 50M tokens; Irregular 1,000 turns; ~8% of AISI tasks solved only between 10M and 50M tokens - Average cost per run ~$10 (AISI, max < $60); Irregular <$1 to ~$20, some hard challenges ~$100 - Page body dated Mar 5, 2026; page metadata Nov 2026 (it surfaced in our sweep on Nov 3) ## What happened Quote: "Policymakers and developers need accurate capability ceiling estimates, not point estimates at fixed budgets." ## Your AI and this story - GPT-6.2 Astra (training cutoff May 2026): in its training data - Claude Opus 5.5 (training cutoff June 2026): in its training data - Gemini 4 Hydrogen (training cutoff March 2026): in its training data - Grok 4.7 (training cutoff May 2026): in its training data ## Sources 1. [Irregular: Cyber capabilities exceed standard evaluation budgets](https://www.irregular.com/research/cyber-capabilities-exceed-standard-evaluation-budgets) (irregular.com, official) ## Changes - 2026-11-08 (filed): Created