Artificial Analysis launches the Cyber Index and an industry alliance (IBM, NVIDIA, Vercel, Collinear) for AI vulnerability-fixing evals
Confirmed
Status
- Claim
Confirmed
- Our reporting
- High confidence
- Importance
- 2 of 5
- Last verified
- 30 September 2026
Your AI and this story
- GPT-6 Astra151 days after its cutoff
- Claude Opus 5.590 days after its cutoff
- Gemini 4 Hydrogen181 days after its cutoff
- Grok 4.7120 days after its cutoff
None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 90 days before it.
Key facts
- Components: CWE-Bench-AA (120 tasks: audit open-source repos and patch OWASP Top 10 flaws), DeepsecBench-AA (find vulnerabilities vs expert-verified findings), CyberGym-E2E-AA (131 tasks: discover, reproduce and patch C/C++ memory-safety bugs)
- All runs use Stirrup, an open-source agent harness
- Best models solve 41% of expert-verified vulnerabilities in DeepsecBench-AA; ‘55% of failed attempts fixed the primary issue but left a related one open’
- Refusal rates above 98% for frontier models on CyberGym-E2E-AA tasks
- Launch partners: Collinear AI, IBM, NVIDIA, Vercel
What happened
Artificial Analysis, known for its model leaderboards, added a defensive-cyber index and recruited vendors to agree on how such work is measured.
Why it matters
Cyber capability is now the main axis of frontier-model risk debates. A shared, defense-oriented benchmark from an independent evaluator gives buyers and policymakers a common yardstick. The high refusal rates show that safety filters strongly shape these scores.
Sources
1 source from 1 site. Numbers match the chips in the text.
1 source: 1 primary
Primary
- Artificial Analysis: Cyber Index and Cyber Index Allianceartificialanalysis.ai, official
Changes
- Filed