Post-Cutoff

Review

The Sol 6.1 Benchmarks Are STUPID, So I Tested It vs Sonnet 5.5

Chase AI YouTube 172,944 views as of 22 November 2026

Watch on YouTubePlay loads YouTube’s player from youtube-nocookie.com.

Description

Description written by Gemini from the videoGemini 3.8 Flash, 1 October 2026

Summary
Chase AI reviews OpenAI’s newly announced GPT-6.1 Sol following OpenAI DevDay, analyzing its benchmark scores and head-to-head performance against Anthropic’s Claude Sonnet 5.5 across four real-world coding and generation challenges. Through side-by-side evaluations of automated canvas explainer videos, frontend web design, interactive 3D web graphics, and a browser-based 3D tank game, the presenter evaluates whether GPT-6.1 Sol’s dramatic token efficiency and lower task costs offset Claude Sonnet 5.5’s advantages in speed and visual design execution.

What is shown

  • [00:00] OpenAI DevDay announcement and benchmark slides: OpenAI’s launch page for GPT-6.1 Sol, showing DeepSWE cost vs. score graphs and OSWorld 2.0 offline set comparisons between GPT-6.1 Sol, GPT-6 Sol, and GPT-6 Astra.
  • [01:13] Benchmark breakdown: Artificial Analysis index comparisons across Intelligence Index, Finance & Accounting, Strategy & Ops, Legal, Healthcare & Medical, Engineering, and Economics, alongside lab benchmarks (DeepSWE v1.1 and AutomationBench 1.0.6) and effort-level task costs.
  • [03:23] Test 1 — JavaScript Explainer Video: Prompting both models to generate a self-contained 15-second vertical animated canvas video explaining Claude subagents with synchronized audio. GPT-6.1 Sol’s output is reviewed ([03:48]), followed by Claude Sonnet 5.5’s output ([04:20]).
  • [04:56] Test 2 — Landing Page Design (“Dune House”): Prompting both models to create a responsive landing page for a fictional boutique hotel named “Dune House”. GPT-6.1 Sol generated images natively ([06:01]), while Claude Sonnet 5.5 generated images via a Higgsfield MCP integration ([05:08], [07:17]).
  • [08:28] Test 3 — 3D Interactive Travel Globe: Testing Three.js interactive data visualizations. GPT-6.1 Sol created “ORBIT” ([08:28]), an interactive 3D globe with flight routes, while Claude Sonnet 5.5 produced “MERIDIAN” ([09:33]), featuring audio, cinematic fly-in camera animations, and destination cards.
  • [10:55] Test 4 — 3D Browser Game: Prompting both models to clone a World of Tanks-style game using Three.js. GPT-6.1 Sol produced “IronFront: Armored Warfare” ([11:04]), and Claude Sonnet 5.5 generated “Iron Vanguard” ([12:06]), both showing playable tanks, aiming, HUDs, and destructible elements.

Claims & numbers

  • The presenter says GPT-6.1 Sol was announced at OpenAI DevDay immediately following the underwhelming release of GPT-6.
  • On DeepSWE v1.1, the presenter states GPT-6.1 Sol scored 75.2% (High effort, $0.65 cost per task), beating GPT-6 Astra’s 73.2% (Max effort, $7.50 cost per task) and Sonnet 5.5’s score by +4.2 points.
  • On AutomationBench 1.0.6, the presenter reports Sonnet 5.5 leads Sol by +8.7 points (44.8% max with fallbacks vs. 36.1% max).
  • On token API list pricing, both models share identical rates: $2 per million input tokens and $10 per million output tokens.
  • The presenter claims Sonnet 5.5 generates tokens roughly twice as fast as GPT-6.1 Sol (138 tokens/second vs. 69 tokens/second).
  • The presenter shows Artificial Analysis cost-per-task data indicating Sol is up to 10x cheaper at maximum effort ($0.72 vs. $7.60), 7x cheaper at extra high ($0.39 vs. $2.74), 3.4x cheaper at high ($0.32 vs. $1.08), and 2.8x cheaper at medium ($0.21 vs. $0.59); when normalized to 52% benchmark accuracy, Sol was 4x cheaper.
  • The presenter states OpenAI recently slashed usage allowances on its $200/month plan by 50% (from 20x to 10x), while noting Anthropic’s 20x tier had a 5-hour rolling limit rather than an uncapped weekly allowance.
  • Explainer video test token usage: Sol used 70,000 tokens versus Sonnet 5.5’s ~300,000 tokens.
  • Landing page test token usage: Sol used 109,000 tokens versus Sonnet 5.5’s ~200,000 tokens.
  • 3D globe test token usage: Sol used ~150,000 tokens versus Sonnet 5.5’s ~400,000 tokens.
  • 3D tank game test: Sol took over 2 hours and ~350,000 tokens; Sonnet 5.5 took ~1 hour and ~700,000 tokens.

Notable quotes

  • [02:39]: “Basically, Sol: super, super, super token-efficient.”
  • [07:05]: “Which probably is a good thing, because this model is so dang cheap that if this is giving me Astra performance, what’s there to complain about?”
  • [13:28]: “6.1 Sol kind of feels like just a slight step below Sonnet 5.5, yet it’s way more efficient with tokens. It’s cheaper, but it’s slower.”

Assessment
This is a hands-on review and comparative benchmark demonstration by an independent AI creator. The presenter directly demonstrates genuine browser-rendered outputs (HTML5 canvas, Three.js applications, and front-end layouts) and provides full transparency regarding token consumption, runtime, and qualitative shortcomings for both models.

Described by gemini-3.8-flash on 2026-10-01 from the video’s audio and frames.

How AI was used to make it

AI made a substantial part of this video: subject built by ai.

  • demonstrating browser-based 3D games, Three.js graphics, landing pages, and canvas animations generated by AI models

Tools named: GPT-6.1 Sol, Claude Sonnet 5.5, Higgsfield MCP

Classified by gemini-3.8-flash from our description of the video (high confidence), 25 October 2026. How AI is used to make videos, across the whole log

Related

  1. Model releases 91 days after the cutoff

    OpenAI releases GPT-6.1 Sol

  2. Model releases 90 days after the cutoff

    Anthropic releases Claude Sonnet 5.5

More videos on this

  1. X

    “Build what you think you look like”: Claude Sonnet 5.5 vs GPT-6 Astra

    Siya (@siyabuilt)7,259,229 views as of 27 Oct

  2. OfficialYouTube

    Introducing Claude Sonnet 5.5

    Claude4,403,927 views as of 27 Oct

  3. OfficialYouTube

    Live from OpenAI DevDay 2026: Keynote

    OpenAI685,790 views as of 24 Oct