Muse Spark Open Is Useless

Summary
AI engineer Matt Johnston reviews Meta’s newly released open-weight model, Muse Spark Open, running it through his nine-task “Benchmark Workbench” suite. He evaluates its coding and multimodal output across interactive 3D games, UI web pages, 3D printing models, and autonomous code audits, concluding that while it performs decently, it fails to outperform existing open-weight alternatives like GLM-5.3, Kimi K3, and MiMo-V2.6 Pro.

What is shown

• Meta announcement & pricing [00:00–01:42]: Johnston displays Meta’s release post on Facebook detailing Muse Spark Open’s architecture and official benchmark comparisons against rival open models, followed by hosted API pricing ($1.25/M input, $4.25/M output).
• Halo Arena test [02:26–04:24]: Johnston plays a browser-based 3D FPS vertical slice generated by the model, testing shooting, weapon swapping, and collision before awarding it an 89/100.
• Tactical Grid test [04:25–06:44]: An isometric turn-based tactical grid game is tested; Johnston encounters a game-breaking UI loop between shoot and move states, scoring it 92/100.
• Printable Object test [06:47–07:40]: Johnston inspects a 3D-printable Doctor Who TARDIS phone stand rendered inside a Bambu Studio slicing environment, rating it 76/100 due to inaccurate geometry and styling.
• Sales Page test [07:41–09:37]: Johnston navigates an interactive SaaS landing page (“LumenOps”) generated by the model, noting clean layout and functional cost sliders but missing requested animated SVGs, scoring it 93/100.
• Diablo Arena test [09:38–11:22]: An isometric ARPG demo is tested for character controls and combat abilities; Johnston notes basic geometry and lack of a boss encounter, scoring it 82/100.
• Fifth Avenue Drone test [11:23–13:04]: A Three.js 3D flight simulation down NYC’s 5th Avenue past landmark buildings; Johnston encounters camera-dependent control inversion bugs and scores it 84/100.
• Penguin Meltdown test [13:05–13:54]: An animated SVG of a penguin dropping an ice cream cone is reviewed, noted for comical and bizarre visual flaws, and scored 85/100.
• Leaderboard evaluation [13:55–16:11]: Johnston shows the final auto-evaluated scores for “Q3 Close” (100/100) and “Skepticism Audit” (90/100), placing Muse Spark Open at an overall composite score of 87, ranking 25th on the Stone Labs leaderboard.

Claims & numbers

• Meta claims Muse Spark Open is the most capable US open-weight model to date (referenced by Johnston at [00:11]).
• Muse Spark Open has 816 billion total parameters and 42 billion active parameters (MoE), a 1-million-token context window, and native multimodality (read from Meta’s post at [00:21]).
• The model is optimized to run locally on two Apple M5 Ultra Mac Studios using Apple’s MLX framework, which Johnston estimates costs around $20,000 in hardware [00:41].
• Official benchmarks shown include 76.4 on Terminal-Bench 2.1 and 60 on DeepSWE [01:15].
• API pricing shown on Together AI and Fireworks is $1.25 per million input tokens and $4.25 per million output tokens [01:38].
• On Johnston’s Stone Labs leaderboard, Muse Spark Open finishes with a score of 87, placing 25th overall—tied with GLM-5.3 and behind open models MiMo-V2.6 Pro (11th), GLM-5.3-Flash (12th), Kimi K3 (13th), and DeepSeek-V4.1-Flash [14:26, 15:18].

Notable quotes

• “Muse Spark Open is optimized to be run on two Apple M5 Ultra Mac Studios with the latest MLX framework. So, that’s about like $20,000 in computers you would need to run this.” [00:39]
• “This is the same bug that Muse Wonder models had, and those models did not do well on my bench.” [02:42]
• “I just can’t think of a reason to use them. I honestly can’t think of one... there’s cheaper, better solutions across the board.” [15:58]

Assessment
This is an authentic hands-on evaluation and review of Meta’s Muse Spark Open model using the creator’s standardized interactive benchmarking suite. The testing and gameplay are conducted live on-screen, capturing authentic model failures, bugs, and leaderboard comparisons.

Described by gemini-3.8-flash on 2026-10-16 from the video’s audio and frames.