Design Creativity Bench finds LLM design ‘diversity collapse’
Partly confirmed
The takeaway
Design Creativity Bench (arXiv 2610.11539, Rusia, Bhole, Gupta and Dey; v1 Oct 8, v2 Oct 14, 2026; hosted by Kombai) scores LLM-made UI designs on originality, creative range and appropriateness.
Status
- Claim
Partly confirmed
- Our reporting
- Medium confidence
- Importance
- 2 of 5
- Last verified
- 15 October 2026
Your AI and this story
- GPT-6 Astra167 days after its cutoff
- Claude Opus 5.5106 days after its cutoff
- Gemini 4 Argon197 days after its cutoff
- Grok 4.7136 days after its cutoff
None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 106 days before it.
Key facts
- Paper: arXiv 2610.11539, ‘Design Creativity Bench: Measuring creativity in LLM-Generated UI’ (Aman Rusia, Abhijit Bhole, Prashank Gupta, Dipanjan Dey); v1 Oct 8, v2 Oct 14, 2026
- Abstract: originality between different models on the same prompt 0.592 (95% CI 0.582–0.602) vs 0.764 for human–model pairs; creative range across models 0.581 vs 0.902 for human designs; appropriateness above 90% for every model, best 99.2% vs 98.0% for humans
- Leaderboard as embedded in Rusia’s essay (Oct 14): human reference 92.3, GPT-6 Astra 65.5, GPT-6.1 Sol 63.7, Grok 4.7 62.2, Gemini 3.8 Flash 60.6; the essay says 16 frontier models on 80 open-ended UI tasks
- Essay: Aman Rusia, ‘Why do LLM pelicans look like this?’ (arcfu.com, Oct 14): thirteen models given Simon Willison’s ‘SVG of a pelican riding a bicycle’ prompt produce nearly identical images, which he calls ‘diversity collapse’; 163 points on Hacker News (item 50090443, Oct 15)
- Caveat: a benchmark from a company that sells AI UI-design tools; scores are not independently reproduced
What happened
A four-author team published Design Creativity Bench on arXiv (2610.11539) to measure how creative LLM-generated user-interface designs are, not just whether they are correct. The headline finding is that models are reliable but alike: different models answering the same brief produce designs far more similar to each other than to human designs, and their range across briefs is much narrower.
On Oct 14 co-author Aman Rusia published an essay showing the same effect on Simon Willison’s well-known pelican-on-a-bicycle SVG test, where thirteen models drew nearly the same picture. It reached the Hacker News front page and promoted the benchmark’s leaderboard, led by GPT-6 Astra at 65.5 against a human reference of 92.3.
Why it matters
Most 2026 benchmarks measure correctness. This one puts a number on a common complaint, that frontier models default to the same “house style” in creative work, and finds a large gap to human designers.
Sources
3 sources from 3 sites. Numbers match the chips in the text.
3 sources: 2 primary, 1 reaction
Primary
- arXiv 2610.11539: Design Creativity Bench: Measuring creativity in LLM-Generated UIarxiv.org, paper
- Kombai: Design Creativity Bench leaderboardkombai.com, official
Reactions
- Aman Rusia: Why do LLM pelicans look like this? (The emperor pelican has no clothes)arcfu.com, discussion
Changes
- Filed from data/leads.md (arXiv abstract and the arcfu.com essay read)