{"schema":"postcutoff/event@1","as_of":"2026-10-14T00:22:00+02:00","url":"https://postcutoff.com/e/2026-10-08-humanitys-sixth-sense-visual-reasoning-benchmark/","md":"https://postcutoff.com/e/2026-10-08-humanitys-sixth-sense-visual-reasoning-benchmark/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2026-10-08-humanitys-sixth-sense-visual-reasoning-benchmark","date":"2026-10-08","date_precision":"day","short_title":"Humanity's Sixth Sense (Scale AI and Elorian)","deck":"People score 93.1% on intuitive visual reasoning, the best model (GPT-6 Astra) 53.6%","takeaway":"On Oct 8, 2026 Scale AI and the startup Elorian published Humanity's Sixth Sense (HSS), a benchmark of 522 open-ended questions about what an image or short video implies without showing it.","category":"benchmark","category_label":"Benchmarks","importance":2,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":[{"n":1,"title":"Elorian: Humanity's Sixth Sense (Oct 8, 2026)","url":"https://elorian.ai/blog/humanitys-sixth-sense","type":"official","group":"primary","domain":"elorian.ai"},{"n":2,"title":"Scale AI Labs: Humanity's Sixth Sense (paper)","url":"https://labs.scale.com/papers/humanitys-sixth-sense","type":"paper","group":"primary","domain":"labs.scale.com"},{"n":3,"title":"Scale AI: HSS leaderboard","url":"https://scale.com/leaderboard/hss","type":"docs","group":"primary","domain":"scale.com"},{"n":4,"title":"Hugging Face: ScaleAI/HSS dataset","url":"https://huggingface.co/datasets/ScaleAI/HSS","type":"code","group":"primary","domain":"huggingface.co"},{"n":5,"title":"Import AI 476 (Jack Clark, Oct 12, 2026)","url":"https://importai.substack.com/p/import-ai-476-creativity-and-rsi","type":"press","group":"press","domain":"importai.substack.com"}],"official":4,"filed":"2026-10-13","updated":"2026-10-13","orgs":["Scale AI","Elorian"],"title":"Humanity's Sixth Sense (Scale AI and Elorian): people score 93.1% on intuitive visual reasoning, the best model (GPT-6 Astra) 53.6%","summary":"On Oct 8, 2026 Scale AI and the startup Elorian published Humanity's Sixth Sense (HSS), a benchmark of 522 open-ended questions about what an image or short video implies without showing it. Humans pass 93.1% of tasks at the first attempt; the best of 25 models, GPT-6 Astra, passes 53.6%, and the median model 30.9%. Most failures come from seeing and inferring, not from logic. It is one of the few autumn-2026 benchmarks on which frontier models are far below ordinary people.","key_facts":["Paper: 'Humanity's Sixth Sense: Benchmarking Intuitive Visual Reasoning in Multimodal Models' (Guo, Gu, Jang et al.), a joint paper by Scale AI and Elorian; blog post dated Oct 8, 2026; public leaderboard and dataset","Content: 522 open-ended tasks on 288 images and 234 short video clips (17.6 hours of video) in four areas: temporal and causal dynamics, physical and spatial logic, social understanding, abstract and contextual inference. Graded against human-written rubrics; an answer passes only if it meets every rubric point","Human baseline: 93.1% pass@1 (94.1% on images, 91.9% on video)","Top scores (pass@1, vendor-chosen effort settings): GPT-6 Astra 53.6, GPT-6.1 Sol 46.6, Claude Opus 5.5 44.6, Gemini 3.8 Flash 41.6, Claude Fable 5.1 40.8, Gemini 3.7 Flash 39.8, Muse Spark 1.3 37.4, Claude Fable 5 34.5, Qwen 3.8 Max 33.0. Lower: GPT-6 Sol 31.2, GLM-5.3 Flash 26.8, Kimi K3 25.5, GPT-6 Luna 21.0. Margins are about ±3 to ±4 points","Not tested: Gemini 4 Argon and Meta's Muse Wonder (both released to the public after the paper)","Control: GPT-6 Astra with the image or video removed scores 6.6%, so the questions cannot be answered from text alone","Failure analysis over 8,573 labeled failures: 53% misread the image, 41% failed to infer what was implied, 5% faulty logic. Most common single mistakes: missing a decisive detail (21%) and misidentifying an object, person or role (20%)","Social understanding was the weakest area for 21 of 25 models (average 24.4% vs 34.1% in the other three); video was harder than images for 23 of 25 models, by 7.3 points on average","More reasoning did not always help: GPT-6 Astra lost 14 points on retrodiction tasks going from high to xhigh effort. With crop-and-zoom tools an agent reached 59.3% on a 388-task subset"],"key_numbers":[],"tags":["benchmark","multimodal","visual-reasoning","video-understanding","social-understanding","gpt-6-astra","claude-opus-5-5","human-baseline"],"science":null,"body_md":"## What happened\n\nScale AI and Elorian, a startup that describes itself as building \"the foundation of visual thinking\", released a benchmark of\nquestions whose answers depend on what a scene implies: what happened just before, who is in charge, what will fall next. People\nanswer almost all of them correctly at the first try. The best model, GPT-6 Astra at maximum reasoning effort, passes just over half,\nand most of the 25 models tested pass fewer than 40%.\n\nThe failure analysis says the gap is perceptual: in 94% of labeled failures the model misread the picture or did not infer what it\nimplied, and in only 5% was the reasoning itself wrong. Social scenes were the hardest area and video harder than still images.\n\nOnly Elorian's blog post was read for this entry (scores are those of Table 1 as reproduced there); the paper PDF, the leaderboard\nand the dataset page were not opened. We found no description of Elorian's founders or funding on the page.\n\n## Why it matters\n\nIn the weeks when models were posting research-level mathematics and beating experts at ML experiment design, this benchmark shows\na plain human skill where the best system is 40 points below an average person. The public dataset means the gap can be\nre-measured on Gemini 4 Argon and Muse Wonder.","disputed":[],"related":[{"id":"2026-10-12-jack-clark-full-rsi-2027-tasteval","url":"https://postcutoff.com/e/2026-10-12-jack-clark-full-rsi-2027-tasteval/","date":"2026-10-12","date_precision":"day","short_title":"Jack Clark moves his forecast for 'full RSI' to 2027, citing TasteVal","deck":"Opus 5.5 beats ML experts at designing experiments (2.3x compute multiplier)","takeaway":"Clark co-wrote Anthropic's June 2026 paper \"When AI builds itself\" and is one of the lab's main public voices on policy.","category":"benchmark","category_label":"Benchmarks","importance":3,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":6,"official":4,"filed":"2026-10-13","updated":"2026-10-13","orgs":["Anthropic","P-Zero Research"]},{"id":"2026-09-29-gpt-6-1-sol","url":"https://postcutoff.com/e/2026-09-29-gpt-6-1-sol/","date":"2026-09-29","date_precision":"day","short_title":"OpenAI releases GPT-6.1 Sol","deck":"Near-Astra performance at one-fifth of Astra's price","takeaway":"Near-flagship capability is now available at mid-tier prices, a week after the previous mid-tier model.","category":"model-release","category_label":"Model releases","importance":4,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":15,"official":11,"filed":"2026-09-29","updated":"2026-10-10","orgs":["OpenAI"]},{"id":"2026-09-22-claude-opus-5-5","url":"https://postcutoff.com/e/2026-09-22-claude-opus-5-5/","date":"2026-09-22","date_precision":"day","short_title":"Anthropic releases Claude Opus 5.5","deck":"Fable-5.1-level performance at $4/$20, first model of the Claude 5.5 family","takeaway":"Opus 5.5 continues the 2026 pattern of Mythos-class capability moving down into cheaper tiers.","category":"model-release","category_label":"Model releases","importance":5,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":36,"official":16,"filed":"2026-09-29","updated":"2026-10-05","orgs":["Anthropic"]},{"id":"2026-09-03-gpt-6-astra","url":"https://postcutoff.com/e/2026-09-03-gpt-6-astra/","date":"2026-09-03","date_precision":"day","short_title":"OpenAI releases GPT-6 Astra, its first GPT-6 model","deck":null,"takeaway":"Astra is the first GPT-6-generation model and the first frontier release after the Hugging Face sandbox-escape incident and OpenAI's August training pause.","category":"model-release","category_label":"Model releases","importance":5,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":20,"official":11,"filed":"2026-09-29","updated":"2026-09-30","orgs":["OpenAI"]}],"people":[{"id":"jack-clark","name":"Jack Clark","url":"https://postcutoff.com/person/jack-clark/"}],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-10-13","type":"filed","text":"Created (found through Import AI 476)"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-10-13","run":null,"sources_read":null,"updated":"2026-10-13","human_review":null,"version":{"date":"2026-10-13"}},"gaps":[{"model_id":"gpt-6-astra","name":"GPT-6 Astra","cutoff":"2026-04","days_after":161,"in_training_data":false},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":100,"in_training_data":false},{"model_id":"gemini-4-argon","name":"Gemini 4 Argon","cutoff":"2026-03","days_after":191,"in_training_data":false},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":130,"in_training_data":false}],"short_url":null}