So I’ve been testing Gemini 4...

Summary
Theo Browne (t3.gg) reviews Google DeepMind’s newly released flagship frontier model, Gemini 4 Argon, testing its performance across frontend design, 3D browser generation, and complex real-world coding tasks. While praising its multimodal capabilities, YouTube integration, and frontend generation with Anthropic design skills, he sharply criticizes its extreme hallucination and failure rates in production codebases, token inefficiency, and poor cache hit rates. He concludes that Gemini 4 Argon is unreliable for serious software engineering compared to competitors like Claude Opus 5.5 and GPT-6.1 Sol.

---

What is shown

• Google Announcement & Vals Index [00:00–03:49]: The official Google blog post by Koray Kavukcuoglu announcing Gemini 4 Argon and the Vals Index GDP-weighted benchmark showing Argon ranked #1.
• Slopalytics Benchmarks [03:52–04:32]: The Artificial Analysis Intelligence Index vs. cost chart, where Gemini 4 Argon scores 52.6 at $1.99 per task, trailing Claude Opus 5.5.
• Pricing & Caching Details [04:33–05:35, 10:10–10:50]: Google AI Studio API documentation showing introductory pricing of $2.00 input / $10.00 output per million tokens (through November 30, 2026), doubling on December 1, 2026 to $4.00 / $20.00, alongside low cache hit rates (70–80%).
• Skatebench v2 Benchmark [08:04–09:25]: Terminal view of Theo’s private Skatebench benchmark where Gemini 4 Argon scores a perfect 100% (300/300), outperforming Opus 5.5 (79%) and GPT-6 Astra (84%).
• Frontend Design on Which AI Made This? [10:53–13:35]: Generations of the “Marrow” and “Loam” note-taking web interfaces tested on whichai.dev, comparing Gemini 4 Argon with and without Anthropic’s design skill.
• Fishslop 3D Game Implementation [17:21–21:20]: A Three.js 3D submarine aquarium simulator game generated by Gemini 4 Argon, showing gameplay bugs (fish starvation pathing, broken food expiry, and an attack power-up expiring before enemies spawn).
• Artificial Analysis AA-Omniscience Benchmark [21:40–22:35]: Chart showing Argon’s 15% hallucination rate, counterbalanced by heavy refusals placing it 4th overall on the Omniscience Index.
• Codebase Audit Benchmarks [25:39–29:16]:
  • TS-Rust Analysis Eval: Gemini 4 Argon places 11th with a score of 4.38/10, making 24 false claims out of 36.
  • T3 Code Update Audit Eval: Argon scores 5.31/10 with 14 wrong claims out of 40.
  • Lakebed Deep Architecture Analysis: Argon scores 2.25/10 with 56 of 78 checked claims refuted (70% failure rate).
  • T3 Code Architectural Analysis Scoreboard: Argon scores 3.95/10 with 47 false claims out of 92.
• Live Agent Coding Failure in T3 Code [29:17–34:50]: Gemini 4 Argon attempts to fix an SQLite connection instability issue, running for 31 minutes to generate a 1,200-line PR using regex routing that fails CI and lint checks; Claude Opus 5.5 reviews and rejects it, subsequently implementing a working 21-line fix in 2 minutes.
• Automated PR Filtering Prompt [36:27–38:09]: Theo instructs Claude Opus 5.5 to implement a workflow rule in T3 Code that automatically detects and closes pull requests authored by Gemini 4 Argon.

---

Claims & numbers

• The presenter says it had been eight months since Google released a full-sized frontier model (Gemini 3.1 Pro in February 2026), compared to 1–2 months between OpenAI releases like GPT-6.1 Sol and GPT-6 Astra [00:10–00:25].
• Google lists introductory API pricing at $2.00/M input tokens and $10.00/M output tokens with a 95% discount on cached inputs, but presenter points out prices double to $4.00/M input and $20.00/M output on December 1, 2026 [04:33–05:22, 10:15–10:25].
• The presenter claims Argon achieves only a 70–80% prompt cache hit rate in practice compared to 99%+ on competing models, significantly driving up actual inference costs [05:04–05:25].
• On Slopalytics, Gemini 4 Argon achieves an Intelligence Index of 52.6, higher than GPT-6.1 Sol (51.8) and GPT-6 Astra (52.4), but below Claude Opus 5.5 High (53.6) while costing more ($1.99 vs $1.82 per task) [04:05–04:20].
• On Theo’s private Skatebench v2 benchmark, Gemini 4 Argon scored 100% (300/300 correct), beating Claude Opus 5.5 Max (79%) and Muse Spark 1.3 (91%) [08:04–08:18].
• On Artificial Analysis AA-Omniscience, Argon recorded a 15% hallucination rate, the lowest among models scoring 45+ on the Intelligence Index, but fell to 4th place on the overall index due to refusals [21:40–22:25].
• In empirical codebase analysis tests across Theo’s repositories, Argon achieved:
  • TS-Rust analysis: 4.38/10 score, 24 false claims out of 36, taking 43m 37s [25:53–26:35].
  • T3 Code update audit: 5.31/10 score, 14 wrong claims out of 40 [26:40–26:50].
  • Lakebed deep architecture: 2.25/10 score, 56 wrong claims out of 78 (70% error rate) [27:31–27:46].
  • T3 Code architectural analysis: 3.95/10 score, 47 false claims out of 92 [28:20–28:50].

---

Notable quotes

• [06:21] “The way it was described to me is: ‘I find myself reaching for Opus 5 less.’ Not 5.5, Opus 5. The one that I notoriously panned...”
• [24:36] “Do you understand how egregious that is? ... It means that when this model is doing real work in a real codebase, it’s as likely, if not more likely, to be wrong than right.”
• [37:31] “This is one of the worst releases of the year. I don’t care what the benchmarks say, I have been testing it constantly in real-world work, and I have not merged a thing that this model has shipped.”

---

Assessment
This is an independent hands-on developer review and benchmark evaluation. Theo tests Gemini 4 Argon through multiple custom benchmarks, external leaderboards, and production repositories, backing his strong criticisms with full side-by-side comparative scoring, pull requests, and CI logs against Claude Opus 5.5, GPT-6.1 Sol, and Muse Wonder.

Described by gemini-3.8-flash on 2026-10-14 from the video’s audio and frames.