{"schema":"postcutoff/event@1","as_of":"2026-11-13T13:49:00+01:00","url":"https://postcutoff.com/e/2022-01-27-instructgpt/","md":"https://postcutoff.com/e/2022-01-27-instructgpt/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2022-01-27-instructgpt","date":"2022-01-27","date_precision":"day","short_title":"InstructGPT: RLHF aligns language models to follow instructions","deck":null,"takeaway":"OpenAI fine-tuned GPT-3 with reinforcement learning from human feedback (RLHF); labelers preferred outputs of the 1.3B InstructGPT over the 175B GPT-3, and the method became the recipe for ChatGPT.","category":"research","category_label":"Research","importance":5,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":[{"n":1,"title":"Aligning language models to follow instructions (OpenAI)","url":"https://openai.com/index/instruction-following/","type":"official","group":"primary","domain":"openai.com"},{"n":2,"title":"Training language models to follow instructions with human feedback (arXiv)","url":"https://arxiv.org/abs/2203.02155","type":"paper","group":"primary","domain":"arxiv.org"},{"n":3,"title":"Deep reinforcement learning from human preferences (arXiv)","url":"https://arxiv.org/abs/1706.03741","type":"paper","group":"primary","domain":"arxiv.org"}],"official":3,"filed":"2026-09-29","updated":"2026-09-29","orgs":["OpenAI"],"title":"InstructGPT: RLHF aligns language models to follow instructions","summary":"OpenAI fine-tuned GPT-3 with reinforcement learning from human feedback (RLHF); labelers preferred outputs of the 1.3B InstructGPT over the 175B GPT-3, and the method became the recipe for ChatGPT.","key_facts":["Announced 27 January 2022; paper arXiv 2203.02155","Three steps: supervised fine-tuning, reward model, PPO optimization","1.3B InstructGPT outputs preferred over 175B GPT-3","Built on 'Deep RL from Human Preferences' (Christiano et al., 2017, arXiv 1706.03741)"],"key_numbers":[],"tags":["rlhf","alignment","llm"],"science":null,"body_md":"## What happened\n\nOpenAI made InstructGPT models the default in its API, showing that human-preference fine-tuning made models more helpful and truthful.\n\n## Why it matters\n\nRLHF turned raw LLMs into usable assistants and underlies ChatGPT, Claude and nearly all chat models.","disputed":[],"related":[{"id":"2022-12-15-constitutional-ai","url":"https://postcutoff.com/e/2022-12-15-constitutional-ai/","date":"2022-12-15","date_precision":"day","short_title":"Anthropic introduces Constitutional AI (RLAIF)","deck":null,"takeaway":"Showed alignment could scale with AI supervision, making values explicit and auditable; RLAIF is now widespread.","category":"policy-safety","category_label":"Policy & safety","importance":4,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":2,"official":2,"filed":"2026-09-29","updated":"2026-09-29","orgs":["Anthropic"]},{"id":"2022-11-30-chatgpt","url":"https://postcutoff.com/e/2022-11-30-chatgpt/","date":"2022-11-30","date_precision":"day","short_title":"OpenAI launches ChatGPT","deck":null,"takeaway":"The moment AI became a mass-market technology — the start of the current era of AI investment, adoption and policy attention.","category":"product","category_label":"Products","importance":5,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":2,"official":1,"filed":"2026-09-29","updated":"2026-09-29","orgs":["OpenAI"]},{"id":"2020-05-28-gpt-3","url":"https://postcutoff.com/e/2020-05-28-gpt-3/","date":"2020-05-28","date_precision":"day","short_title":"GPT-3 (175B) shows in-context few-shot learning","deck":null,"takeaway":"Turned LLMs into a platform; many startups were built on its API, and it directly preceded InstructGPT and ChatGPT.","category":"model-release","category_label":"Model releases","importance":5,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":2,"official":2,"filed":"2026-09-29","updated":"2026-09-29","orgs":["OpenAI"]}],"people":[{"id":"paul-christiano","name":"Paul Christiano","url":"https://postcutoff.com/person/paul-christiano/"}],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-09-29","type":"filed","text":"Created"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-09-29","run":null,"sources_read":null,"updated":"2026-09-29","human_review":null,"version":{"date":"2026-10-04"}},"gaps":[{"model_id":"gpt-6-astra","name":"GPT-6 Astra","cutoff":"2026-04","days_after":null,"in_training_data":true},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":null,"in_training_data":true},{"model_id":"gemini-4-hydrogen","name":"Gemini 4 Hydrogen","cutoff":"2026-03","days_after":null,"in_training_data":true},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":null,"in_training_data":true}],"short_url":"https://postcutoff.com/s/instructgpt-rlhf-aligns-language"}