{"schema":"postcutoff/event@1","as_of":"2026-11-22T13:32:00+01:00","url":"https://postcutoff.com/e/2026-11-13-anthropic-building-strong-safety-evaluations/","md":"https://postcutoff.com/e/2026-11-13-anthropic-building-strong-safety-evaluations/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2026-11-13-anthropic-building-strong-safety-evaluations","date":"2026-11-13","date_precision":"day","short_title":"Anthropic: building strong safety evals","deck":null,"takeaway":null,"category":"policy-safety","category_label":"Policy & safety","importance":2,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":[{"n":1,"title":"Anthropic: Building strong safety evaluations (Nov 13)","url":"https://www.anthropic.com/research/building-strong-safety-evaluations","type":"official","group":"primary","domain":"anthropic.com"},{"n":2,"title":"Anthropic on X","url":"https://x.com/AnthropicAI/status/2121317227618340990","type":"official","group":"primary","domain":"x.com"}],"official":2,"filed":"2026-11-15","updated":"2026-11-15","orgs":["Anthropic"],"title":"Anthropic publishes guidance on building strong safety evaluations; says single-turn harmlessness evals are saturated","summary":"An Anthropic engineering post (Nov 13, 2026) argues that many AI safety evaluations are easy to game, hard to reproduce and sensitive to format, and recommends psychology-style rigour: measuring the right construct, multi-turn realistic scenarios, validated LLM graders, preregistration and confidence intervals. It reports that single-turn harmless-response rates are 94–98% across recent models.","key_facts":["Authors: Ruth Appel, Evan Frondorf, Jeremy Hadfield, Rebecca Hiscott, Kamya Jagadish, Shayne Longpre, Madeleine Tucker, Brett Wittmershaus","Single-turn harmless response rates on the API: 94–98% across recent models, i.e. saturated","Sample sizes: at least 100 items for complex multi-turn evals, 1,000 for simple single-turn ones; multi-turn confidence intervals can exceed ±10 points","Examples: on a romance-scam eval Opus 5 scored 98% and Opus 5.5 94%, not a significant difference; Claude Fable 5.1 showed up to a 34-point gap between API and claude.ai configurations; system-prompt mitigations raised appropriate suicide/self-harm responses from 60% (API) to 94% (claude.ai)","'a safety claim is only as sound as the eval behind it, and good ones are hard to build'; announced on X by @AnthropicAI (220k views)"],"key_numbers":[],"tags":["anthropic","evaluations","safety","methodology"],"science":null,"body_md":"## What happened\n\nMethodology post from Anthropic's engineering team with concrete numbers from its own evaluations.\n\n## Why it matters\n\nDeployment decisions rest on these evaluations; the post concedes that common single-turn tests no longer distinguish models and that\nproduct configuration can shift results by tens of points.","disputed":[],"related":[],"people":[],"posts":[{"id":"x-anthropicai-2121317227618340990","title":"“Safety evaluations are critical for understanding whether a model is ready to deploy…”","url":"https://postcutoff.com/p/x-anthropicai-2121317227618340990/"}],"videos":[],"models":[],"changes":[{"date":"2026-11-15","type":"filed","text":"Created (news agent)"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-11-15","run":null,"sources_read":null,"updated":"2026-11-15","human_review":null,"version":{"date":"2026-11-15"}},"gaps":[{"model_id":"gpt-6-2-astra","name":"GPT-6.2 Astra","cutoff":"2026-05","days_after":166,"in_training_data":false},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":136,"in_training_data":false},{"model_id":"gemini-4-hydrogen","name":"Gemini 4 Hydrogen","cutoff":"2026-03","days_after":227,"in_training_data":false},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":166,"in_training_data":false}],"short_url":null}