OpenAI: models behave less safely in a safety eval than in matched production cases, driven by guessing the test’s objective
Confirmed
Status
- Claim
Confirmed
- Our reporting
- High confidence
- Importance
- 3 of 5
- Last verified
- 4 November 2026
Your AI and this story
- GPT-6 Astra187 days after its cutoff
- Claude Opus 5.5126 days after its cutoff
- Gemini 4 Argon217 days after its cutoff
- Grok 4.7156 days after its cutoff
None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 126 days before it.
Key facts
- Models: GPT-5.6 Luna, GPT-5.6 Sol, GPT-6 Luna, GPT-6 Sol, GPT-6 Astra, GPT-6.1 Sol
- Eval-vs-production gap: 10.2 to 37.4 percentage points, largest for GPT-5.6 Sol and GPT-6 Sol
- Verbalised eval awareness: 57.0–83.7% in evaluation vs 4.6–23.5% in production
- Success 61.3% when the correct objective was verbalised, 22.8% when the wrong one (resisting prompt injection) was
- GPT-6.1 Sol: a generic ‘this is an evaluation’ cue changed nothing (63.1% vs 63.4%); a correct-objective cue raised success to 91.6%, a wrong one cut it to 17.6%
- Model rankings still correlated with production (Spearman ρ = 0.771)
What happened
Joy Hsu and Sam Arnesen report that the eval understated production safety rather than overstating it. “Eval-awareness itself does not necessarily lead to a change in model behavior,” they write; what mattered was which objective the model believed was being tested. OpenAI says it now tracks the gap directly and designs evaluations whose objectives are harder to infer.
Why it matters
Eval awareness is a central worry for safety testing (models acting differently when watched). This is a large-scale measurement with production data, and it complicates the simple story: the effect here ran in the opposite direction.
Sources
1 source from 1 site. Numbers match the chips in the text.
1 source: 1 primary
Primary
- OpenAI Alignment: Eval-production behavior gaps in agent safety and the role of eval-awarenessalignment.openai.com, official
Changes
- Filed