Dermatologist’s test
A fine-tuned 2015 CNN (Inception v3) beats GPT-6 Astra and Claude Fable 5.1 at diagnosing biopsied skin lesions
Partly confirmed
Status
- Claim
Partly confirmed
- Our reporting
- Medium confidence
- Importance
- 3 of 5
- Last verified
- 26 October 2026
Your AI and this story
- GPT-6.2 Astra147 days after its cutoff
- Claude Opus 5.5117 days after its cutoff
- Gemini 4 Hydrogen208 days after its cutoff
- Grok 4.7147 days after its cutoff
None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 117 days before it.
Key facts
- Test set: 200 histopathology-confirmed lesions from MILK10k (ISIC, 2025), each with a clinical and a dermoscopic photo: 100 not safe to discharge (30 melanomas, 25 BCCs, 25 SCCs, 20 actinic keratoses) and 100 safe (50 naevi, 50 seborrhoeic keratoses)
- CNN: Inception v3 (Google, 2015) fine-tuned on 68,011 ISIC Archive images on a single H100; AUROC 0.86 on this biopsied set, 0.96 on ordinary archive images
- Correct first choice: CNN 125/200 (62%), GPT-6 Astra 84 (42%), Claude Fable 5.1 85 (42%); first or second choice: 173 / 134 / 138; melanoma named first: 13 / 8 / 11 of 30
- Triage: the LLMs were cautious (kept 97 and 96 of 100 unsafe lesions) but discharged only 32 and 22 of 100 safe ones; the CNN at a matched threshold kept 98–99 unsafe lesions
- Patient-style question (clinical photo only, ‘do I need to see a doctor?’): both LLMs advised seeing a doctor for 99 and 98 of 100 further biopsy-proven melanomas
- Author: ‘A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs’
- Caveats (author): small sample (30 melanomas), one run per LLM case, possible contamination since MILK10k is public, Inception v3 is a baseline not state of the art; no preprint or peer review. Disclosure: analysis and first drafts were done with Claude Code
What happened
Magnus Lynch, a consultant dermatologist and Mohs surgeon, compared two frontier multimodal models with a small specialised image classifier on hard, biopsied cases and published the cases, prompts, model replies and scoring code. The general-purpose models were safe in the sense of rarely discharging cancers, but far less accurate at naming the lesion and much more likely to send benign lesions for review.
Why it matters
In the same week Microsoft reported that an LLM orchestrator beat doctors on text-based case puzzles, a clinician’s image test found frontier models well behind a decade-old narrow network on visual diagnosis. It is a single small study, but it supports keeping specialised models in medical imaging and checking LLM medical claims modality by modality.
Sources
2 sources from 2 sites. Numbers match the chips in the text.
2 sources: 1 primary, 1 reaction
Primary
- Lynch Lab: A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis (Oct 25)lynchlab.net, official
Reactions
- Hacker News discussion (649 points)news.ycombinator.com, discussion
Changes
- Filed (Hacker News 649 pts)