DeepMind’s Chinchilla revises scaling laws toward more data
Confirmed
Importance: major (4 of 5)The takeaway
Hoffmann et al. found that for compute-optimal training, parameters and training tokens should scale equally (~20 tokens per parameter); 70B Chinchilla outperformed the 280B Gopher.
Status
- Claim
Confirmed
- Our reporting
- High confidence
- Importance
- Major (4 of 5)
- Last verified
- 29 September 2026
Your AI and this story
- GPT-6 AstraIn its training data
- Claude Opus 5.5In its training data
- Gemini 4 HydrogenIn its training data
- Grok 4.7In its training data
It happened before the cutoffs of all four assistants, so it can be in their training data.
Key facts
- arXiv 2203.15556 ‘Training Compute-Optimal Large Language Models’
- Chinchilla: 70B parameters trained on 1.4 trillion tokens
- Beat Gopher (280B), GPT-3 (175B) and Megatron-Turing NLG (530B) on many benchmarks
- Implied most prior LLMs were undertrained
What happened
Over 400 training runs showed earlier scaling laws had over-weighted parameter count relative to data.
Why it matters
Reshaped how every lab trains LLMs, pushing toward far larger datasets and smaller, cheaper-to-serve models (e.g. LLaMA).
Sources
2 sources from 2 sites. Numbers match the chips in the text.
2 sources: 1 primary, 1 reaction
Primary
- Training Compute-Optimal Large Language Models (arXiv)arxiv.org, paper
Reactions
- Wikipedia: Chinchilla (language model)en.wikipedia.org, discussion
Changes
- Filed