---
id: "2022-01-27-instructgpt"
url: "https://postcutoff.com/e/2022-01-27-instructgpt/"
as_of: "2026-11-13T13:49:00+01:00"
date: "2022-01-27"
date_precision: day
category: research
importance: 5
confidence: high
status: [Confirmed]
sources: 3
editor: human editor
human_review: null
version: "2026-10-04"
---

As of: 2026-11-13 13:49 CET.
Researched and written by AI agents (Claude Opus 5.5 in Claude Code), with a human editor. How we work: https://postcutoff.com/about/
Canonical page: https://postcutoff.com/e/2022-01-27-instructgpt/

# InstructGPT: RLHF aligns language models to follow instructions

OpenAI fine-tuned GPT-3 with reinforcement learning from human feedback (RLHF); labelers preferred outputs of the 1.3B InstructGPT over the 175B GPT-3, and the method became the recipe for ChatGPT.

## Key facts
- Announced 27 January 2022; paper arXiv 2203.02155
- Three steps: supervised fine-tuning, reward model, PPO optimization
- 1.3B InstructGPT outputs preferred over 175B GPT-3
- Built on 'Deep RL from Human Preferences' (Christiano et al., 2017, arXiv 1706.03741)

## What happened

OpenAI made InstructGPT models the default in its API, showing that human-preference fine-tuning made models more helpful and truthful.

## Why it matters

RLHF turned raw LLMs into usable assistants and underlies ChatGPT, Claude and nearly all chat models.

## Your AI and this story

- GPT-6 Astra (training cutoff April 2026): in its training data
- Claude Opus 5.5 (training cutoff June 2026): in its training data
- Gemini 4 Hydrogen (training cutoff March 2026): in its training data
- Grok 4.7 (training cutoff May 2026): in its training data

## Sources

1. [Aligning language models to follow instructions (OpenAI)](https://openai.com/index/instruction-following/) (openai.com, official)
2. [Training language models to follow instructions with human feedback (arXiv)](https://arxiv.org/abs/2203.02155) (arxiv.org, paper)
3. [Deep reinforcement learning from human preferences (arXiv)](https://arxiv.org/abs/1706.03741) (arxiv.org, paper)

## Changes

- 2026-09-29 (filed): Created

## Related

- 2022-12-15: [Anthropic introduces Constitutional AI (RLAIF)](https://postcutoff.com/e/2022-12-15-constitutional-ai/index.md)
- 2022-11-30: [OpenAI launches ChatGPT](https://postcutoff.com/e/2022-11-30-chatgpt/index.md)
- 2020-05-28: [GPT-3 (175B) shows in-context few-shot learning](https://postcutoff.com/e/2020-05-28-gpt-3/index.md)
- People: [Paul Christiano](https://postcutoff.com/person/paul-christiano/)