Post-Cutoff

BenchmarksLynch Lab and King's College London117 days after June 2026

Dermatologist’s test

A fine-tuned 2015 CNN (Inception v3) beats GPT-6 Astra and Claude Fable 5.1 at diagnosing biopsied skin lesions

Partly confirmed

Status
Claim

Partly confirmed

Our reporting
Medium confidence
Importance
3 of 5
Last verified
26 October 2026

Your AI and this story

  • GPT-6.2 Astra147 days after its cutoff
  • Claude Opus 5.5117 days after its cutoff
  • Gemini 4 Hydrogen208 days after its cutoff
  • Grok 4.7147 days after its cutoff

None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 117 days before it.

Key facts

  • Test set: 200 histopathology-confirmed lesions from MILK10k (ISIC, 2025), each with a clinical and a dermoscopic photo: 100 not safe to discharge (30 melanomas, 25 BCCs, 25 SCCs, 20 actinic keratoses) and 100 safe (50 naevi, 50 seborrhoeic keratoses)
  • CNN: Inception v3 (Google, 2015) fine-tuned on 68,011 ISIC Archive images on a single H100; AUROC 0.86 on this biopsied set, 0.96 on ordinary archive images
  • Correct first choice: CNN 125/200 (62%), GPT-6 Astra 84 (42%), Claude Fable 5.1 85 (42%); first or second choice: 173 / 134 / 138; melanoma named first: 13 / 8 / 11 of 30
  • Triage: the LLMs were cautious (kept 97 and 96 of 100 unsafe lesions) but discharged only 32 and 22 of 100 safe ones; the CNN at a matched threshold kept 98–99 unsafe lesions
  • Patient-style question (clinical photo only, ‘do I need to see a doctor?’): both LLMs advised seeing a doctor for 99 and 98 of 100 further biopsy-proven melanomas
  • Author: ‘A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs’
  • Caveats (author): small sample (30 melanomas), one run per LLM case, possible contamination since MILK10k is public, Inception v3 is a baseline not state of the art; no preprint or peer review. Disclosure: analysis and first drafts were done with Claude Code

What happened

Magnus Lynch, a consultant dermatologist and Mohs surgeon, compared two frontier multimodal models with a small specialised image classifier on hard, biopsied cases and published the cases, prompts, model replies and scoring code. The general-purpose models were safe in the sense of rarely discharging cancers, but far less accurate at naming the lesion and much more likely to send benign lesions for review.

Why it matters

In the same week Microsoft reported that an LLM orchestrator beat doctors on text-based case puzzles, a clinician’s image test found frontier models well behind a decade-old narrow network on visual diagnosis. It is a single small study, but it supports keeping specialised models in medical imaging and checking LLM medical claims modality by modality.

Sources

2 sources from 2 sites. Numbers match the chips in the text.

2 sources: 1 primary, 1 reaction

Primary

  1. Lynch Lab: A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis (Oct 25)lynchlab.net, official

Reactions

  1. Hacker News discussion (649 points)news.ycombinator.com, discussion

Changes

  • Filed (Hacker News 649 pts)

Status

Claim

Partly confirmed

Our reporting
Medium confidence
Importance
3 of 5
Last verified
26 October 2026

Sources at a glance

2 sources: 1 primary, 1 reaction

How this entry was made

Written by
AI agents: Claude Opus 5.5, made by Anthropic, running in Claude Code
Filed
26 October 2026
Human review
None recorded for this entry. What the editor does
Version
Last saved 27 October 2026

Spotted an error? Write to contact@postcutoff.com. Corrections are logged in public.

This page for your AI

Same text, no layout:

Open in ClaudeOpen in ChatGPT

Related

Related events

  1. Science & math

    Microsoft MAI-DxO diagnosis paper in Science

    Result confirmed

The whole topic