Formula predicts when chatbots tip into harmful output
Confirmed
The takeaway
Neil F. Johnson and Frank Yingjie Huo (George Washington University physics) published “Competition for attention predicts good-to-bad tipping in AI” in the Cell Press journal Patterns (October 2026).
Status
- Claim
Confirmed
- Our reporting
- High confidence
- Importance
- 2 of 5
- Last verified
- 11 October 2026
Your AI and this story
- GPT-6 Astra161 days after its cutoff
- Claude Opus 5.5100 days after its cutoff
- Gemini 3.8 Flash191 days after its cutoff
- Grok 4.7130 days after its cutoff
None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 100 days before it.
Key facts
- Paper: ‘Competition for attention predicts good-to-bad tipping in AI’, Neil F. Johnson and Frank Yingjie Huo, Patterns, Oct 2026, doi:10.1016/j.patter.2026.101666 (Crossref); SecurityWeek and Earth.com date publication to Oct 8–9
- Mechanism: the conversation’s accumulated context shifts attention toward an undesirable output ‘basin’ until a tipping point is crossed; after the first bad output, ‘every undesirable answer drags the next one further down the slope’ (Huo, via Earth.com)
- Tests: seven open-weight transformer models from three independent groups, 124M (GPT-2 size) to 12B parameters; the formula predicted the outcome in 18 of 19 cases (SecurityWeek, Earth.com)
- Two regimes: immediate tipping (one bad prompt) and delayed tipping after several acceptable answers; earlier topics in the conversation change when harm appears
- Focus: on-device ‘edge’ AI companions without cloud filters, telemetry or patching; Johnson proposes real-time monitors that warn when a chat nears the threshold, like collision warnings in cars
- Earlier version: arXiv 2504.20980 ‘Jekyll-and-Hyde Tipping Point in an AI’s Behavior’ (2025); the result is limited to small open models, not frontier systems
What happened
A two-person GWU physics team turned its 2025 preprint on a “Jekyll-and-Hyde tipping point” into a peer-reviewed Patterns paper. It treats a single attention head as a thermal spin system in which the conversation’s context and competing output “basins” compete for attention, and derives a nearly closed-form expression for when the model’s output turns bad. Press coverage (SecurityWeek, Earth.com, Decrypt, Tech Xplore) framed it as a way to predict when chatbots “go rogue”. We have not read the full paper; numbers come from press.
Why it matters
It offers a simple, testable theory of why long conversations can degrade, a known weakness of chatbot safeguards, and a possible basis for runtime warning tools. Its evidence is limited to small open-weight models.
Sources
5 sources from 5 sites. Numbers match the chips in the text.
5 sources: 3 primary, 2 press
Primary
- Patterns (Cell Press): Competition for attention predicts good-to-bad tipping in AIcell.com, paper
- DOI 10.1016/j.patter.2026.101666doi.org, paper
- arXiv 2504.20980: Jekyll-and-Hyde tipping point in an AI’s behavior (earlier preprint)arxiv.org, paper
Press
Changes
- Filed (Decrypt)