A GPT-4o respiratory chatbot beats web search for lay diagnosis in a randomized trial
Published in Nature Health: 70.0% vs 55.4% correct, with the chatbot running in WeChat.
Partly confirmed
The takeaway
A preregistered randomized study published in Nature Health on Oct 5, 2026 tested LungDiag, a GPT-4o-based respiratory chatbot inside WeChat, against ordinary mobile web search (with AI search features disabled).
Status
- Claim
Partly confirmed
- Our reporting
- Medium confidence
- Importance
- 2 of 5
- Last verified
- 5 October 2026
Your AI and this story
- GPT-6.2 Astra127 days after its cutoff
- Claude Opus 5.597 days after its cutoff
- Gemini 4 Hydrogen188 days after its cutoff
- Grok 4.7127 days after its cutoff
None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 97 days before it.
Key facts
- Paper: ‘Accuracy of a respiratory assessment chatbot in a nationwide messaging system for layperson diagnosis and triage: a randomized preregistered study’, Nature Health, online Oct 5, 2026 (DOI 10.1038/s44360-026-00189-9); first authors Liang, Wang, Ni et al.
- Design: single-blind, multicentre, 1:1 randomized; 2,400 adults recruited April–June 2025 across 24 healthcare systems in China; 2,176 analysed (1,088 per arm)
- Diagnostic accuracy: chatbot 70.0% (95% CI 65.2–74.8) vs web search 55.4% (49.7–61.1); risk difference 14.6 points, P < 0.001 (as reported by Bioengineer.org)
- Caveats reported: simulated vignettes rather than real patients; moderate triage specificity (62.1%); chatbot tasks took about 66 seconds longer; no evidence yet on real care-seeking or outcomes
What happened
The study compares a task-specific, retrieval-augmented chatbot built on GPT-4o and embedded in WeChat with what lay people usually do, which is to search the web. Search-engine AI answers were switched off in the control arm, so the comparison is with plain search.
Why it matters
It is one of the larger prospective randomized comparisons of a consumer health chatbot against web search, and it is peer-reviewed. The outcome is accuracy on case vignettes, not patient outcomes. The model, GPT-4o, was released in May 2024, more than two years before publication. The numbers come from the Bioengineer.org summary. The Nature Health page was confirmed to exist, but its full text was not read, hence confidence: medium.
Sources
2 sources from 2 sites. Numbers match the chips in the text.
2 sources: 1 primary, 1 press
Primary
- Nature Health: Accuracy of a respiratory assessment chatbot … a randomized preregistered studynature.com, paper
Press
Changes
- Filed