{"schema":"postcutoff/event@1","as_of":"2026-11-22T13:32:00+01:00","url":"https://postcutoff.com/e/2026-03-05-irregular-aisi-cyber-capabilities-exceed-eval-budgets/","md":"https://postcutoff.com/e/2026-03-05-irregular-aisi-cyber-capabilities-exceed-eval-budgets/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2026-03-05-irregular-aisi-cyber-capabilities-exceed-eval-budgets","date":"2026-03-05","date_precision":"day","short_title":"Irregular and UK AISI","deck":"Frontier models' cyber success keeps rising with inference budget, so fixed-budget evals underestimate them","takeaway":"Models tested were unnamed frontier models from up to Nov 2025.","category":"policy-safety","category_label":"Policy & safety","importance":2,"confidence":"medium","status":{"key":"partly","labels":["Partly confirmed"]},"sources":[{"n":1,"title":"Irregular: Cyber capabilities exceed standard evaluation budgets","url":"https://www.irregular.com/research/cyber-capabilities-exceed-standard-evaluation-budgets","type":"official","group":"primary","domain":"irregular.com"}],"official":1,"filed":"2026-11-08","updated":"2026-11-08","orgs":["Irregular","UK AI Security Institute"],"title":"Irregular and UK AISI: frontier models' cyber success keeps rising with inference budget, so fixed-budget evals underestimate them","summary":"Irregular and the UK AI Security Institute reported that success on cyber tasks keeps rising as token budgets grow: about 8% of AISI's tasks were solved only when the limit rose from 10M to 50M tokens, so evaluations at standard budgets give point estimates, not capability ceilings. Models tested were unnamed frontier models from up to Nov 2025.","key_facts":["AISI budget 50M tokens; Irregular 1,000 turns; ~8% of AISI tasks solved only between 10M and 50M tokens","Average cost per run ~$10 (AISI, max < $60); Irregular <$1 to ~$20, some hard challenges ~$100","Page body dated Mar 5, 2026; page metadata Nov 2026 (it surfaced in our sweep on Nov 3)"],"key_numbers":[],"tags":["cyber","evaluations","inference-scaling"],"science":null,"body_md":"## What happened\n\nQuote: \"Policymakers and developers need accurate capability ceiling estimates, not point estimates at fixed budgets.\"","disputed":[],"related":[],"people":[],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-11-08","type":"filed","text":"Created"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-11-08","run":null,"sources_read":null,"updated":"2026-11-08","human_review":null,"version":{"date":"2026-11-08"}},"gaps":[{"model_id":"gpt-6-2-astra","name":"GPT-6.2 Astra","cutoff":"2026-05","days_after":null,"in_training_data":true},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":null,"in_training_data":true},{"model_id":"gemini-4-hydrogen","name":"Gemini 4 Hydrogen","cutoff":"2026-03","days_after":null,"in_training_data":true},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":null,"in_training_data":true}],"short_url":null}