Frontier models’ cyber success keeps rising with inference budget, so fixed-budget evals underestimate them
Partly confirmed
The takeaway
Models tested were unnamed frontier models from up to Nov 2025.
Status
- Claim
Partly confirmed
- Our reporting
- Medium confidence
- Importance
- 2 of 5
- Last verified
- 8 November 2026
Your AI and this story
- GPT-6.2 AstraIn its training data
- Claude Opus 5.5In its training data
- Gemini 4 HydrogenIn its training data
- Grok 4.7In its training data
It happened before the cutoffs of all four assistants, so it can be in their training data.
Key facts
- AISI budget 50M tokens; Irregular 1,000 turns; ~8% of AISI tasks solved only between 10M and 50M tokens
- Average cost per run ~$10 (AISI, max < $60); Irregular <$1 to ~$20, some hard challenges ~$100
- Page body dated Mar 5, 2026; page metadata Nov 2026 (it surfaced in our sweep on Nov 3)
What happened
Quote: “Policymakers and developers need accurate capability ceiling estimates, not point estimates at fixed budgets.”
Sources
1 source from 1 site. Numbers match the chips in the text.
1 source: 1 primary
Primary
- Irregular: Cyber capabilities exceed standard evaluation budgetsirregular.com, official
Changes
- Filed