Bloomberg’s report on internal Gemini 4 skepticism goes viral
tae kim @firstadopterX
Why it matters
Most-viewed relay (~320K views) of Bloomberg’s report that Google employees found Gemini 4 weaker in real coding work than on benchmarks.
Summary
Quotes Bloomberg: Google ‘is grappling with internal skepticism’ over Gemini 4; it ‘does less well when employees actually put it to work’ and ‘struggles to handle certain coding tasks’, per anonymous people with direct access.
Archived text
HOLY CRAP. Bloomberg has the goods on Gemini 4!
“it does less well when employees actually put it to work .. The model struggles to handle certain coding tasks”
Bloomberg: “As Alphabet Inc.’s Google prepares for the coming launch of Gemini 4, it’s grappling with internal skepticism over how well the flagship artificial intelligence model performs in key areas, such as coding.
While Gemini 4 has performed well on benchmarks the industry uses to gauge model efficacy, it does less well when employees actually put it to work, according to people with direct access to the effort. The model struggles to handle certain coding tasks, said the people, who requested anonymity to discuss an internal matter.”
Quoting @firstadopter: Gemini 4! We won’t know how good it really is until developers actually use it and see what they say.
Google’s models have historically tended to be bench-maxxed, performing better on benchmarks than in real world use