InstructGPT: RLHF aligns language models to follow instructions
Confirmed
Importance: historic (5 of 5)The takeaway
OpenAI fine-tuned GPT-3 with reinforcement learning from human feedback (RLHF); labelers preferred outputs of the 1.3B InstructGPT over the 175B GPT-3, and the method became the recipe for ChatGPT.
Status
- Claim
Confirmed
- Our reporting
- High confidence
- Importance
- Historic (5 of 5)
- Last verified
- 29 September 2026
Your AI and this story
- GPT-6 AstraIn its training data
- Claude Opus 5.5In its training data
- Gemini 4 HydrogenIn its training data
- Grok 4.7In its training data
It happened before the cutoffs of all four assistants, so it can be in their training data.
Key facts
- Announced 27 January 2022; paper arXiv 2203.02155
- Three steps: supervised fine-tuning, reward model, PPO optimization
- 1.3B InstructGPT outputs preferred over 175B GPT-3
- Built on ‘Deep RL from Human Preferences’ (Christiano et al., 2017, arXiv 1706.03741)
What happened
OpenAI made InstructGPT models the default in its API, showing that human-preference fine-tuning made models more helpful and truthful.
Why it matters
RLHF turned raw LLMs into usable assistants and underlies ChatGPT, Claude and nearly all chat models.
Sources
3 sources from 2 sites. Numbers match the chips in the text.
3 sources: 3 primary
Primary
- Aligning language models to follow instructions (OpenAI)openai.com, official
- Training language models to follow instructions with human feedback (arXiv)arxiv.org, paper
- Deep reinforcement learning from human preferences (arXiv)arxiv.org, paper
Changes
- Filed