{"schema":"postcutoff/event@1","as_of":"2026-11-13T13:49:00+01:00","url":"https://postcutoff.com/e/2025-06-22-roboarena/","md":"https://postcutoff.com/e/2025-06-22-roboarena/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2025-06-22-roboarena","date":"2025-06-22","date_precision":"day","short_title":"RoboArena: crowd-sourced, double-blind real-world evaluation of generalist robot policies","deck":null,"takeaway":"RoboArena (arXiv 2506.18123, 2025-06-22) ranks generalist robot policies through double-blind pairwise comparisons run by a distributed network of evaluators on the DROID platform, who pick their own tasks and scenes.","category":"benchmark","category_label":"Benchmarks","importance":2,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":[{"n":1,"title":"arXiv 2506.18123: RoboArena","url":"https://arxiv.org/abs/2506.18123","type":"paper","group":"primary","domain":"arxiv.org"},{"n":2,"title":"PMLR (CoRL 2025): RoboArena","url":"https://proceedings.mlr.press/v305/atreya25a.html","type":"paper","group":"primary","domain":"proceedings.mlr.press"}],"official":2,"filed":"2026-09-29","updated":"2026-09-29","orgs":["RoboArena consortium"],"title":"RoboArena: crowd-sourced, double-blind real-world evaluation of generalist robot policies","summary":"RoboArena (arXiv 2506.18123, 2025-06-22) ranks generalist robot policies through double-blind pairwise comparisons run by a distributed network of evaluators on the DROID platform, who pick their own tasks and scenes. The first round covered 600+ real-robot episodes over 7 policies at 7 academic institutions; its open leaderboard became a standard reference, e.g. NVIDIA's GR00T N2 and Cosmos 3 claims in 2026.","key_facts":["Paper: 'RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies' (Atreya, Pertsch, Lee, Kim et al.), arXiv 2506.18123; published at CoRL 2025 (PMLR v305)","612 pairwise real-robot comparisons, 7 generalist policies, 7 universities, DROID Franka setup","Authors show this ranks policies more accurately than centralized fixed-task evaluation","Evaluation network opened to the community"],"key_numbers":[],"tags":["robotics","evaluation","leaderboard","droid","vla"],"science":null,"body_md":"## What happened\n\nRoboArena borrowed the idea behind Chatbot Arena, pairwise preference votes aggregated into a ranking, and applied it to physical robots. Evaluators at partner universities run two anonymous policies on a task of their choice and record which did better.\n\n## Why it matters\n\nReal-world robot evaluation is expensive and hard to standardize. A distributed arena gives a scalable, harder-to-game ranking of VLAs, and labs now cite it in model launches.","disputed":[],"related":[{"id":"2026-06-01-nvidia-cosmos-3-open-release","url":"https://postcutoff.com/e/2026-06-01-nvidia-cosmos-3-open-release/","date":"2026-06-01","date_precision":"day","short_title":"NVIDIA releases Cosmos 3, an open omni-model for physical AI","deck":null,"takeaway":"It is the largest open world model aimed at robotics and autonomous vehicles.","category":"open-source","category_label":"Open source","importance":3,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":5,"official":4,"filed":"2026-09-29","updated":"2026-09-29","orgs":["NVIDIA"]},{"id":"2026-02-11-ai2-molmospaces","url":"https://postcutoff.com/e/2026-02-11-ai2-molmospaces/","date":"2026-02-11","date_precision":"day","short_title":"Ai2 launches MolmoSpaces, an open simulation ecosystem and leaderboard for generalist robot policies","deck":null,"takeaway":"Robot learning lacked shared benchmarks like those language models have.","category":"benchmark","category_label":"Benchmarks","importance":2,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":3,"official":3,"filed":"2026-09-29","updated":"2026-09-29","orgs":["Ai2"]}],"people":[],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-09-29","type":"filed","text":"Created (author affiliations not verified; the paper lists Atreya, Pertsch, Lee, Kim among the authors)"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-09-29","run":null,"sources_read":null,"updated":"2026-09-29","human_review":null,"version":{"date":"2026-10-04"}},"gaps":[{"model_id":"gpt-6-astra","name":"GPT-6 Astra","cutoff":"2026-04","days_after":null,"in_training_data":true},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":null,"in_training_data":true},{"model_id":"gemini-4-hydrogen","name":"Gemini 4 Hydrogen","cutoff":"2026-03","days_after":null,"in_training_data":true},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":null,"in_training_data":true}],"short_url":null}