Anthropic: building strong safety evals
Confirmed
Status
- Claim
Confirmed
- Our reporting
- High confidence
- Importance
- 2 of 5
- Last verified
- 15 November 2026
Your AI and this story
- GPT-6.2 Astra166 days after its cutoff
- Claude Opus 5.5136 days after its cutoff
- Gemini 4 Hydrogen227 days after its cutoff
- Grok 4.7166 days after its cutoff
None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 136 days before it.
Key facts
- Authors: Ruth Appel, Evan Frondorf, Jeremy Hadfield, Rebecca Hiscott, Kamya Jagadish, Shayne Longpre, Madeleine Tucker, Brett Wittmershaus
- Single-turn harmless response rates on the API: 94–98% across recent models, i.e. saturated
- Sample sizes: at least 100 items for complex multi-turn evals, 1,000 for simple single-turn ones; multi-turn confidence intervals can exceed ±10 points
- Examples: on a romance-scam eval Opus 5 scored 98% and Opus 5.5 94%, not a significant difference; Claude Fable 5.1 showed up to a 34-point gap between API and claude.ai configurations; system-prompt mitigations raised appropriate suicide/self-harm responses from 60% (API) to 94% (claude.ai)
- ‘a safety claim is only as sound as the eval behind it, and good ones are hard to build’; announced on X by @AnthropicAI (220k views)
What happened
Methodology post from Anthropic’s engineering team with concrete numbers from its own evaluations.
Why it matters
Deployment decisions rest on these evaluations; the post concedes that common single-turn tests no longer distinguish models and that product configuration can shift results by tens of points.
Sources
2 sources from 2 sites. Numbers match the chips in the text.
2 sources: 2 primary
Primary
- Anthropic: Building strong safety evaluations (Nov 13)anthropic.com, official
- Anthropic on Xx.com, official
Changes
- Filed (news agent)