Training language models to follow instructions with human feedback

Long Ouyang et al. · OpenAI · submitted 4 March 2022 · arXiv:2203.02155 · PDF

Takeaway

Three things to remember

• A bigger language model is not automatically a better assistant. Predicting likely internet text and following a person's intent are different goals.
• InstructGPT adds human guidance in three stages: people demonstrate good answers, rank model answers, and provide the preference signal used to improve the model.
• The gain is striking but bounded: human evaluators preferred a 1.3-billion-parameter InstructGPT model to the 175-billion-parameter GPT-3 baseline, yet the system still made factual, safety, and instruction-following mistakes.

The paper's central move is to train for the behavior users actually want, not merely for fluent continuation. Its strongest evidence is direct human comparison on held-out customer prompts. Its largest unresolved risk is that the learned target reflects a small, selected group of labelers—not a universal definition of good behavior. Source: abstract; Sections 1, 4.1, and 5.2–5.3, pp. 1–3 and 17–19 of the paper PDF.

Problem

The objective gap

Standard language-model pretraining rewards the model for predicting the next token in internet text. A user, however, wants an answer that helps with a task, stays truthful, and avoids harm. More scale can improve language prediction without closing that gap; a fluent continuation may still ignore an instruction, invent a fact, or produce toxic material. The paper describes the desired behavior as helpful, honest, and harmless. Source: Section 1, pp. 1–2.

Visual 1. A new teaching illustration of the mismatch the paper identifies. Pretraining optimizes next-token prediction; an assistant is judged against a user's broader intent.

Why existing benchmarks were not enough

Earlier instruction-tuning work used collections of public NLP tasks. Those datasets are useful, but the authors found that they did not match how customers used the API: classification and question answering were about 18% of their labeled prompt data, while open-ended generation and brainstorming were about 57%. Their versions of FLAN and T0 also performed worse than the supervised InstructGPT baseline on the API prompt distribution. Source: Sections 2 and 4.1, pp. 4 and 12.

Method

1. Learn from demonstrations

The team began with GPT-3 and collected examples of desired responses, mostly to prompts from the OpenAI API Playground plus prompts written by labelers. Supervised fine-tuning taught the model to imitate those demonstrations. The training split contained 11,295 labeler prompts and 1,430 customer prompts. Prompts were split by user ID, capped at 200 per user, deduplicated heuristically, and filtered for personally identifiable information. Source: Sections 3.1–3.2 and Appendix A.1, pp. 6–7 and 32–34.

2. Turn rankings into a reward

For another set of prompts, the model produced several candidate answers. Labelers ranked those answers from best to worst. A separate reward model learned to predict those preferences, converting a human comparison into a numerical training signal. The reward-model training data contained 6,623 labeler prompts and 26,584 customer prompts. Source: Section 3.1 and Appendix A.1, pp. 6 and 32.

3. Optimize without forgetting everything else

The researchers then used proximal policy optimization (PPO) to update the supervised model so that its answers earned higher reward-model scores. Their main variant, PPO-ptx, mixed this process with ordinary pretraining updates. That mix reduced losses on some public NLP benchmarks—the paper calls this loss an “alignment tax”—without giving up the human-preference gains. The PPO training split contained 31,144 customer prompts. Source: Sections 3.1 and 3.5, pp. 6 and 8–9; Appendix A.1, p. 32.

Visual 2. Demonstrations create a supervised model; rankings train a reward model; PPO uses that learned reward to produce InstructGPT. This is a new teaching diagram, not the paper's Figure 2.

Evidence and limits

What the evaluations found

On held-out API prompts, evaluators preferred outputs from the 175B InstructGPT model to plain 175B GPT-3 85 ± 3% of the time, and to few-shot-prompted 175B GPT-3 71 ± 4% of the time. Even the 1.3B InstructGPT model beat the 175B GPT-3 baseline in preference comparisons. These are judgments on the paper's prompt distribution, not a claim that the smaller model is universally more capable. Source: Section 1, p. 3; Figure 1 and Section 4.1, pp. 2 and 11.

The paper also reports that on closed-domain tasks, hallucinations fell from 41% for GPT-3 to 21% for InstructGPT. On TruthfulQA, truthful and informative answers appeared about twice as often. With a respectful prompt, InstructGPT produced about 25% fewer toxic outputs than GPT-3. Bias results on Winogender and CrowS-Pairs did not improve significantly. Source: Sections 1 and 4.2, pp. 3 and 13–15.

The preference bottleneck

The feedback came from about 40 contractors chosen through a screening process. They were primarily English-speaking, and roughly 96% of the broader dataset was classified as English. Most comparisons were labeled by one person for cost reasons. The paper is unusually direct about the consequence: the procedure aligns the model to the stated preferences of these labelers and researchers, not to “human values” in general. Source: Sections 3.4 and 5.2–5.3, pp. 7–8 and 17–19; Appendix A.1, p. 34.

Visual 3. The learned preference signal passes through a narrow human sample. The paper's own limitation is representativeness, not a failure to use human judgment at all.

The resulting models still failed to follow instructions, invented facts, gave hedged answers, and missed false premises. More seriously, they often followed harmful instructions; when explicitly asked to be maximally biased, InstructGPT could be more toxic than a same-sized GPT-3 model. The evaluations therefore support improved behavior under the tested conditions, not solved alignment or guaranteed safety. Source: Sections 1 and 5.3, pp. 4 and 19.

Practical meaning

Better behavior can beat more scale

For a product team, the paper suggests that response quality is not only a model-size problem. A smaller base model, tuned against representative examples and preferences, can feel more useful than a much larger untuned model on the target traffic. That is an interpretation of the experiments—not a promise that RLHF always beats scaling.

The feedback design becomes part of the product

The people chosen to label, the instructions they receive, the prompts sampled, and the tradeoffs they are told to make all shape the resulting assistant. A practical deployment should therefore measure quality on its real users and high-risk cases, audit disagreements across groups, and keep factuality and safety checks separate from a single preference score. The paper itself notes that training prioritized helpfulness, while final evaluation prioritized truthfulness and harmlessness, showing that even the definition of “best” can shift. Source: Appendix B.2, pp. 35–36.

Reading notes and verification questions

Terms. Supervised fine-tuning (SFT) means learning from example answers. A reward model (RM) predicts which answer a labeler would prefer. Reinforcement learning from human feedback (RLHF) uses that learned preference signal to update the response model. PPO-ptx is the paper's PPO variant that also mixes in pretraining updates.

Questions to carry forward. Would the result hold for different languages, cultures, and expert domains? How stable are preferences when several people label the same comparison? Can safety constraints override a harmful request without causing excessive refusals? How well do preference gains track factual accuracy outside closed-domain tasks? These remain open because the paper's data and evaluations do not settle them.

Primary source. Ouyang, L. et al. (2022), “Training language models to follow instructions with human feedback,” arXiv:2203.02155, especially Figures 1–3, Table 6, Sections 3–5, and Appendices A–B.