{"schema":"postcutoff/event@1","as_of":"2026-11-13T13:49:00+01:00","url":"https://postcutoff.com/e/2026-09-28-artificial-analysis-cyber-index/","md":"https://postcutoff.com/e/2026-09-28-artificial-analysis-cyber-index/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2026-09-28-artificial-analysis-cyber-index","date":"2026-09-28","date_precision":"day","short_title":"Artificial Analysis launches the Cyber Index and an industry alliance (IBM, NVIDIA, Vercel, Collinear) for AI vulnerability-fixing evals","deck":null,"takeaway":null,"category":"benchmark","category_label":"Benchmarks","importance":2,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":[{"n":1,"title":"Artificial Analysis: Cyber Index and Cyber Index Alliance","url":"https://artificialanalysis.ai/articles/artificial-analysis-cyber-index","type":"official","group":"primary","domain":"artificialanalysis.ai"}],"official":1,"filed":"2026-09-30","updated":"2026-09-30","orgs":["Artificial Analysis","IBM","NVIDIA","Vercel","Collinear AI"],"title":"Artificial Analysis launches the Cyber Index and an industry alliance (IBM, NVIDIA, Vercel, Collinear) for AI vulnerability-fixing evals","summary":"On Sept 28, 2026 Artificial Analysis launched the Cyber Index, which combines three benchmarks of how well AI agents find and patch vulnerabilities, and the Cyber Index Alliance with Collinear AI, IBM, NVIDIA and Vercel to set shared evaluation standards for defensive cyber tasks. The best models solved only 41% of expert-verified vulnerabilities on its DeepsecBench-AA component.","key_facts":["Components: CWE-Bench-AA (120 tasks: audit open-source repos and patch OWASP Top 10 flaws), DeepsecBench-AA (find vulnerabilities vs expert-verified findings), CyberGym-E2E-AA (131 tasks: discover, reproduce and patch C/C++ memory-safety bugs)","All runs use Stirrup, an open-source agent harness","Best models solve 41% of expert-verified vulnerabilities in DeepsecBench-AA; '55% of failed attempts fixed the primary issue but left a related one open'","Refusal rates above 98% for frontier models on CyberGym-E2E-AA tasks","Launch partners: Collinear AI, IBM, NVIDIA, Vercel"],"key_numbers":[],"tags":["benchmark","cybersecurity","agents","evaluation","vulnerability-detection"],"science":null,"body_md":"## What happened\n\nArtificial Analysis, known for its model leaderboards, added a defensive-cyber index and recruited vendors to agree on how such work is measured.\n\n## Why it matters\n\nCyber capability is now the main axis of frontier-model risk debates. A shared, defense-oriented benchmark from an independent evaluator gives buyers and\npolicymakers a common yardstick. The high refusal rates show that safety filters strongly shape these scores.","disputed":[],"related":[{"id":"2026-09-29-anthropic-glm-5-3-spread-of-cyber-capabilities","url":"https://postcutoff.com/e/2026-09-29-anthropic-glm-5-3-spread-of-cyber-capabilities/","date":"2026-09-29","date_precision":"day","short_title":"Anthropic: open-weights GLM-5.3 nearly matches Mythos Preview at exploit development","deck":null,"takeaway":"It is the first time a frontier lab has published evidence that an open-weights model reached the level of exploit capability it had judged too risky to release widely.","category":"policy-safety","category_label":"Policy & safety","importance":4,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":13,"official":5,"filed":"2026-09-30","updated":"2026-10-07","orgs":["Anthropic","Zhipu AI"]}],"people":[],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-09-30","type":"filed","text":"Created"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-09-30","run":null,"sources_read":null,"updated":"2026-09-30","human_review":null,"version":{"date":"2026-10-04"}},"gaps":[{"model_id":"gpt-6-astra","name":"GPT-6 Astra","cutoff":"2026-04","days_after":151,"in_training_data":false},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":90,"in_training_data":false},{"model_id":"gemini-4-hydrogen","name":"Gemini 4 Hydrogen","cutoff":"2026-03","days_after":181,"in_training_data":false},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":120,"in_training_data":false}],"short_url":null}