Anthropic study: experts defer to agent’s lean
Confirmed
The takeaway
An Anthropic randomized trial published Oct 27, 2026 gave 50 professional data analysts a causal-analysis task in Claude Code (Claude Opus 4.8), with the agent’s preferred option on each decision set by a hidden coin flip.
Status
- Claim
Confirmed
- Our reporting
- High confidence
- Importance
- 3 of 5
- Last verified
- 29 October 2026
Your AI and this story
- GPT-6.2 Astra149 days after its cutoff
- Claude Opus 5.5119 days after its cutoff
- Gemini 4 Hydrogen210 days after its cutoff
- Grok 4.7149 days after its cutoff
None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 119 days before it.
Key facts
- Design (paper): 61 analysts with causal-inference experience recruited, 11 chose not to use the agent, 50 in the main analysis; fictional farming cooperative deciding whether to expand an advisory programme; five decisions (four analytical choices and a final recommendation) where a coin flip set which option the agent favoured; a no-agent comparison group
- Analysts matched the agent’s lean on 86% of the four analytical decisions (92%, 82%, 81%, 90%) versus ~50% expected with no influence
- Final recommendation: 54% recommended rollout when the agent leaned toward it vs 36% when it leaned against
- Of 38 analysts with a prior preference, 60% switched when the agent disagreed vs 6% when it agreed; experience and expertise did not predict deferral; self-rated confidence was similar whether they followed the lean or not (71 vs 73 of 100)
- 77% repeated the agent’s arguments in their write-ups; 15% credited the agent; 52% rated the recommendation as more their own than the agent’s
- Quality: 6 of 11 no-agent analysts (55%) made serious methodological errors vs 3 of 50 agent users (6%); median time 90 vs 65 minutes
- Caveats (authors): the agent’s leans were deliberately strengthened, one task, one domain and one model; the no-agent group was less experienced; the switching estimate rests on 38 cases
- Authors: Amaç Herdağdelen, Hailey Joren, Alex Tamkin (Anthropic). Covered by Al Jazeera (Oct 28: ‘Experts defer to AI in decision-making’)
What happened
Anthropic researchers ran a randomized controlled trial of how a coding agent’s preferences shape expert decisions. Analysts worked on an observational causal-inference task with Claude Code. For each key decision the agent had a lean that a coin flip had set in advance, so its preference carried no information about the right answer. Analysts still followed it 86% of the time. Those who started with their own view changed it in 60% of cases where the agent disagreed. Most wrote the agent’s reasoning into their explanations without saying it came from the agent, and they felt as confident when following it as when not.
The same agent also helped. Agent users made serious methodological errors far less often than the small no-agent group and finished faster.
Why it matters
Many safety and governance plans count on a “human in the loop” to check AI work. This study measures that check failing quietly: the human looks independent, feels independent and attributes the decision to themselves, while the agent’s lean decides the outcome. The authors say real-world effects may be smaller because the leans were strengthened for the experiment.
Sources
4 sources from 4 sites. Numbers match the chips in the text.
4 sources: 2 primary, 2 press
Primary
- Anthropic: How AI agents shape experts’ judgment callsanthropic.com, official
- Paper (PDF)www-cdn.anthropic.com, paper
Press
Changes
- Added Gizmodo coverage link
- Filed (Anthropic post read directly; Al Jazeera read)