Google releases BERT, bidirectional Transformer pre-training
Confirmed
Importance: major (4 of 5)The takeaway
BERT pre-trained a bidirectional Transformer encoder with masked language modeling and set new records on 11 NLP tasks; it was open-sourced and soon deployed in Google Search.
Status
- Claim
Confirmed
- Our reporting
- High confidence
- Importance
- Major (4 of 5)
- Last verified
- 29 September 2026
Your AI and this story
- GPT-6 AstraIn its training data
- Claude Opus 5.5In its training data
- Gemini 4 HydrogenIn its training data
- Grok 4.7In its training data
It happened before the cutoffs of all four assistants, so it can be in their training data.
Key facts
- arXiv 1810.04805 (October 2018); NAACL 2019 best paper
- BERT-Large: 340M parameters
- Pre-training objectives: masked LM + next sentence prediction
- Google said in October 2019 that BERT was used in Search ranking
What happened
Google published and open-sourced BERT, which fine-tuned easily to classification, QA and tagging tasks.
Why it matters
Triggered the ‘ImageNet moment’ of NLP: pre-trained Transformers became the default for all language tasks.
Sources
2 sources from 2 sites. Numbers match the chips in the text.
2 sources: 2 primary
Primary
- BERT: Pre-training of Deep Bidirectional Transformers (arXiv)arxiv.org, paper
- google-research/bert (code)github.com, code
Changes
- Filed