{"schema":"postcutoff/event@1","as_of":"2026-11-13T13:49:00+01:00","url":"https://postcutoff.com/e/2017-06-12-transformer/","md":"https://postcutoff.com/e/2017-06-12-transformer/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2017-06-12-transformer","date":"2017-06-12","date_precision":"day","short_title":"'Attention Is All You Need' introduces the Transformer","deck":null,"takeaway":"Vaswani et al. proposed the Transformer, an architecture built entirely on self-attention without recurrence; it became the foundation of BERT, GPT and virtually every modern large AI model.","category":"research","category_label":"Research","importance":5,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":[{"n":1,"title":"Attention Is All You Need (arXiv)","url":"https://arxiv.org/abs/1706.03762","type":"paper","group":"primary","domain":"arxiv.org"},{"n":2,"title":"Google Research blog: Transformer","url":"https://research.google/blog/transformer-a-novel-neural-network-architecture-for-language-understanding/","type":"official","group":"primary","domain":"research.google"},{"n":3,"title":"Wikipedia: Attention Is All You Need","url":"https://en.wikipedia.org/wiki/Attention_Is_All_You_Need","type":"discussion","group":"reactions","domain":"en.wikipedia.org"}],"official":2,"filed":"2026-09-29","updated":"2026-09-29","orgs":["Google Brain","Google Research"],"title":"'Attention Is All You Need' introduces the Transformer","summary":"Vaswani et al. proposed the Transformer, an architecture built entirely on self-attention without recurrence; it became the foundation of BERT, GPT and virtually every modern large AI model.","key_facts":["arXiv 1706.03762, posted 12 June 2017; NeurIPS 2017","Eight co-authors from Google Brain/Research","WMT 2014 English–German: 28.4 BLEU, a new state of the art","Highly parallelizable training vs. RNNs, enabling scale","The 'T' in GPT stands for Transformer"],"key_numbers":[],"tags":["transformer","attention","architecture","nlp"],"science":null,"body_md":"## What happened\n\nThe paper introduced multi-head self-attention, positional encodings and an encoder–decoder stack, beating recurrent models on translation while training much faster.\n\n## Why it matters\n\nArguably the most consequential AI paper of the century so far: the Transformer's scalability made LLMs, multimodal models and AlphaFold 2 possible.","disputed":[],"related":[{"id":"2018-10-11-bert","url":"https://postcutoff.com/e/2018-10-11-bert/","date":"2018-10-11","date_precision":"day","short_title":"Google releases BERT, bidirectional Transformer pre-training","deck":null,"takeaway":"Triggered the 'ImageNet moment' of NLP: pre-trained Transformers became the default for all language tasks.","category":"model-release","category_label":"Model releases","importance":4,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":2,"official":2,"filed":"2026-09-29","updated":"2026-09-29","orgs":["Google AI Language"]},{"id":"2018-06-11-gpt-1","url":"https://postcutoff.com/e/2018-06-11-gpt-1/","date":"2018-06-11","date_precision":"day","short_title":"OpenAI's GPT-1: generative pre-training of Transformers","deck":null,"takeaway":"Established the pre-train-then-adapt paradigm and the decoder-only Transformer lineage that led to ChatGPT.","category":"model-release","category_label":"Model releases","importance":4,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":2,"official":2,"filed":"2026-09-29","updated":"2026-09-29","orgs":["OpenAI"]},{"id":"2014-09-10-seq2seq-attention","url":"https://postcutoff.com/e/2014-09-10-seq2seq-attention/","date":"2014-09-10","date_precision":"day","short_title":"Sequence-to-sequence learning and neural attention","deck":null,"takeaway":"Established the encoder–decoder paradigm and attention — the direct precursors of the Transformer and modern LLMs.","category":"research","category_label":"Research","importance":4,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":3,"official":3,"filed":"2026-09-29","updated":"2026-09-29","orgs":["Google","Université de Montréal"]}],"people":[],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-09-29","type":"filed","text":"Created"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-09-29","run":null,"sources_read":null,"updated":"2026-09-29","human_review":null,"version":{"date":"2026-10-04"}},"gaps":[{"model_id":"gpt-6-astra","name":"GPT-6 Astra","cutoff":"2026-04","days_after":null,"in_training_data":true},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":null,"in_training_data":true},{"model_id":"gemini-4-hydrogen","name":"Gemini 4 Hydrogen","cutoff":"2026-03","days_after":null,"in_training_data":true},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":null,"in_training_data":true}],"short_url":"https://postcutoff.com/s/attention-all-you-need"}