{"schema":"postcutoff/event@1","as_of":"2026-10-15T23:34:00+02:00","url":"https://postcutoff.com/e/2026-10-14-design-creativity-bench-llm-diversity-collapse/","md":"https://postcutoff.com/e/2026-10-14-design-creativity-bench-llm-diversity-collapse/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2026-10-14-design-creativity-bench-llm-diversity-collapse","date":"2026-10-14","date_precision":"day","short_title":"Design Creativity Bench finds LLM design 'diversity collapse'","deck":null,"takeaway":"Design Creativity Bench (arXiv 2610.11539, Rusia, Bhole, Gupta and Dey; v1 Oct 8, v2 Oct 14, 2026; hosted by Kombai) scores LLM-made UI designs on originality, creative range and appropriateness.","category":"benchmark","category_label":"Benchmarks","importance":2,"confidence":"medium","status":{"key":"partly","labels":["Partly confirmed"]},"sources":[{"n":1,"title":"arXiv 2610.11539: Design Creativity Bench: Measuring creativity in LLM-Generated UI","url":"https://arxiv.org/abs/2610.11539","type":"paper","group":"primary","domain":"arxiv.org"},{"n":2,"title":"Kombai: Design Creativity Bench leaderboard","url":"https://kombai.com/benchmark","type":"official","group":"primary","domain":"kombai.com"},{"n":3,"title":"Aman Rusia: Why do LLM pelicans look like this? (The emperor pelican has no clothes)","url":"https://arcfu.com/post/the-emperor-pelican-has-no-clothes","type":"discussion","group":"reactions","domain":"arcfu.com"}],"official":2,"filed":"2026-10-15","updated":"2026-10-15","orgs":["Kombai"],"title":"Design Creativity Bench: frontier models' UI designs converge ('diversity collapse'); best model GPT-6 Astra scores 65.5 vs 92.3 for the human reference","summary":"Design Creativity Bench (arXiv 2610.11539, Rusia, Bhole, Gupta and Dey; v1 Oct 8, v2 Oct 14, 2026; hosted by Kombai) scores LLM-made UI designs on originality, creative range and appropriateness. Models are almost always appropriate (best 99.2%, humans 98.0%) but converge on similar designs: originality across models 0.592 vs 0.764 for human–model pairs, creative range 0.581 vs 0.902 for human designs. A companion essay on near-identical \"pelican riding a bicycle\" SVGs reached the Hacker News front page.","key_facts":["Paper: arXiv 2610.11539, 'Design Creativity Bench: Measuring creativity in LLM-Generated UI' (Aman Rusia, Abhijit Bhole, Prashank Gupta, Dipanjan Dey); v1 Oct 8, v2 Oct 14, 2026","Abstract: originality between different models on the same prompt 0.592 (95% CI 0.582–0.602) vs 0.764 for human–model pairs; creative range across models 0.581 vs 0.902 for human designs; appropriateness above 90% for every model, best 99.2% vs 98.0% for humans","Leaderboard as embedded in Rusia's essay (Oct 14): human reference 92.3, GPT-6 Astra 65.5, GPT-6.1 Sol 63.7, Grok 4.7 62.2, Gemini 3.8 Flash 60.6; the essay says 16 frontier models on 80 open-ended UI tasks","Essay: Aman Rusia, 'Why do LLM pelicans look like this?' (arcfu.com, Oct 14): thirteen models given Simon Willison's 'SVG of a pelican riding a bicycle' prompt produce nearly identical images, which he calls 'diversity collapse'; 163 points on Hacker News (item 50090443, Oct 15)","Caveat: a benchmark from a company that sells AI UI-design tools; scores are not independently reproduced"],"key_numbers":[],"tags":["benchmark","creativity","ui-design","diversity","gpt-6-astra","pelican"],"science":null,"body_md":"## What happened\n\nA four-author team published Design Creativity Bench on arXiv (2610.11539) to measure how creative LLM-generated user-interface designs\nare, not just whether they are correct. The headline finding is that models are reliable but alike: different models answering the same\nbrief produce designs far more similar to each other than to human designs, and their range across briefs is much narrower.\n\nOn Oct 14 co-author Aman Rusia published an essay showing the same effect on Simon Willison's well-known pelican-on-a-bicycle SVG test,\nwhere thirteen models drew nearly the same picture. It reached the Hacker News front page and promoted the benchmark's leaderboard, led\nby GPT-6 Astra at 65.5 against a human reference of 92.3.\n\n## Why it matters\n\nMost 2026 benchmarks measure correctness. This one puts a number on a common complaint, that frontier models default to the same\n\"house style\" in creative work, and finds a large gap to human designers.","disputed":[],"related":[{"id":"2026-09-03-gpt-6-astra","url":"https://postcutoff.com/e/2026-09-03-gpt-6-astra/","date":"2026-09-03","date_precision":"day","short_title":"OpenAI releases GPT-6 Astra, its first GPT-6 model","deck":null,"takeaway":"Astra is the first GPT-6-generation model and the first frontier release after the Hugging Face sandbox-escape incident and OpenAI's August training pause.","category":"model-release","category_label":"Model releases","importance":5,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":20,"official":11,"filed":"2026-09-29","updated":"2026-09-30","orgs":["OpenAI"]}],"people":[{"id":"simon-willison","name":"Simon Willison","url":"https://postcutoff.com/person/simon-willison/"}],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-10-15","type":"filed","text":"Created from data/leads.md (arXiv abstract and the arcfu.com essay read)"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-10-15","run":null,"sources_read":"Data/leads.md (arXiv abstract and the arcfu.com essay read)","updated":"2026-10-15","human_review":null,"version":{"date":"2026-10-15"}},"gaps":[{"model_id":"gpt-6-astra","name":"GPT-6 Astra","cutoff":"2026-04","days_after":167,"in_training_data":false},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":106,"in_training_data":false},{"model_id":"gemini-4-argon","name":"Gemini 4 Argon","cutoff":"2026-03","days_after":197,"in_training_data":false},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":136,"in_training_data":false}],"short_url":null}