Language Models are Few-Shot Learners

Tom B. Brown et al. · OpenAI · 2020 · arXiv:2005.14165

Takeaway

The paper in one sentence

GPT-3 shows that a sufficiently large language model can often perform a new text task from an instruction or a handful of examples placed in its prompt, without retraining the model for that task. The paper tests this idea with a 175-billion-parameter model across more than two dozen datasets. Source: Abstract and §1, pp. 1, 5

Three points to remember

1. The prompt becomes the task interface. Instead of changing the model's weights for every task, the user describes or demonstrates the task in text.
2. Scale improves this ability, but not uniformly. Larger models generally use in-prompt examples better; strong results on some tasks sit beside large failures on others.
3. Broad usefulness is not the same as dependable understanding. Test-set overlap, social bias, weak reasoning, cost, and uncertainty about what the model is actually learning remain central limits.

Visual 1 — Fine-tuning changes model weights using a task dataset. In-context learning leaves the weights fixed and changes only the text supplied at inference time. This is a new teaching diagram based on §2, not a reproduction of a paper figure. Source: §2, pp. 6–7

Problem

A new task used to require a new training job

The dominant workflow was to pretrain a general language model and then fine-tune it on thousands to hundreds of thousands of labeled examples for each downstream task. That can produce strong benchmark scores, but collecting a fresh dataset and running a fresh optimization process for every use case is slow and expensive. It can also encourage a model to exploit quirks in a narrow task dataset rather than learn a rule that travels well. Source: §1 and §2, pp. 3, 6

The research question

Can one general model infer the intended task from ordinary text at the moment of use? Earlier language models had shown hints of this behavior, but their in-context results were far behind fine-tuned systems. The paper asks whether scaling the model, data, and training would make this form of adaptation genuinely useful. Source: §1, pp. 4–5

Method

One large next-token predictor

GPT-3 is an autoregressive Transformer: it reads the text so far and predicts what comes next. The largest model has 175 billion parameters, 96 layers, and a 2,048-token context window. The authors also train seven smaller models, from 125 million to 13 billion parameters, so they can study how behavior changes with scale. Every model is trained for 300 billion tokens. Source: Table 2.1 and §2.1, p. 8 in the PDF viewer

Broad pretraining, then fixed weights

Training draws from filtered Common Crawl, WebText2, two book corpora, and English Wikipedia. The mixture contains 410 billion Common Crawl tokens, 19 billion WebText2 tokens, 12 billion Books1 tokens, 55 billion Books2 tokens, and 3 billion Wikipedia tokens; higher-quality sources are sampled more often than their raw size would imply. After pretraining, task evaluations do not update GPT-3's weights. Source: Table 2.2 and §2.2, pp. 8–9

Three ways to specify a task

In zero-shot evaluation, the prompt contains an instruction but no demonstration. One-shot adds one worked example. Few-shot adds as many demonstrations as fit in the context window—typically 10 to 100 in this paper. The model then continues the text. Calling this “learning” can be misleading: the immediate adaptation happens in the model's activations, not through a training update. Source: §1 and §2, pp. 5–7

Visual 2 — The model stays fixed; only the amount of task information in the prompt changes. Source: §2, pp. 6–7

Evidence and limits

Strong evidence: examples help more as models grow

Across 42 benchmarks reported with accuracy, aggregate zero-shot performance rises with model size and few-shot performance rises faster. Concrete results show both the promise and the spread: on closed-book TriviaQA, GPT-3 scores 64.3% zero-shot, 68.0% one-shot, and 71.2% few-shot; on CoQA reading comprehension it scores 81.5, 84.0, and 85.0 F1. The latter remains below the fine-tuned state of the art at 90.7 F1. The authors explicitly warn that their aggregate score is a heuristic, not a rigorous benchmark. Source: §1, p. 5; Table 3.7, p. 18

The qualitative probes are also revealing. Given demonstrations, GPT-3 can unscramble words, perform some arithmetic, use a newly defined nonsense word in a sentence, and correct grammar. These examples show fast pattern use, but they are not proof of a general reasoning engine. Source: §3.9, pp. 22–29

Uneven performance is the main empirical limit

Performance varies sharply by task and format. Few-shot GPT-3 reaches 85.0 F1 on CoQA but only 36.5 on DROP and 44.3 on QuAC; on RACE it remains roughly 45 percentage points behind the then state of the art. The paper also reports near-chance behavior on some comparison and natural-language-inference tasks, plus repetition, contradiction, and lost coherence in longer generated passages. Source: Table 3.7 and §3.6, pp. 18–19; §5, p. 33

Four cautions around the headline result

• Benchmark overlap: web-scale training data may contain evaluation examples. A filtering bug left some overlaps; the authors flag or omit affected results and say the remaining analysis cannot fully rule out distortion. Source: §2.2, p. 9; §4, pp. 29–33
• What “learning” means is unresolved: the model may be learning a new rule in context, recognizing a task seen during pretraining, or doing something between those extremes. The paper does not settle this. Source: §5, p. 34
• Bias and misuse: the authors find stereotyped associations involving gender, race, and religion, and warn that convincing synthetic text can lower barriers to misuse. Their bias study is described as preliminary. Source: §6.1–6.2, pp. 35–39
• Compute and deployment cost: training the 175B model used several thousand petaflop/s-days, and the paper calls models of this size expensive and inconvenient to run. Source: §5, p. 34; §6.3, p. 39

Visual 3 — The paper's capability results and cautions belong in the same picture. The arrows show areas evaluated or discussed, not guaranteed performance. This is a new teaching diagram. Source: §§3–6, pp. 11–39

Practical meaning

What changed for builders

The paper introduces a powerful product pattern: treat natural-language instructions and examples as a lightweight way to program one general model. That can shorten prototyping and make low-data tasks approachable. This is a practical interpretation of the results, not a claim that the authors proved prompts can replace task-specific systems.

The safe reading is: use in-context learning when tasks change often, examples are easy to provide, and mistakes can be checked. Do not treat a fluent answer as evidence that the model understands the world, and do not assume a prompt that works on a few examples will remain reliable across formats, groups, or edge cases.

Questions to ask before using the idea

• Does the task improve with clear demonstrations, or does it require stable rules, current facts, tools, or external verification?
• Are tests separated from likely training material, and do they cover format changes rather than one benchmark template?
• What happens on the worst-performing slices, not just the average?
• Can a human review failures, and is the model's cost justified by the value of avoiding task-specific training?

Source note and terminology

The primary source is the authors' arXiv record and version 4 PDF, revised 22 July 2020. Page references above use the PDF viewer's page numbering.

In-context learning means conditioning a fixed model with task text and examples inside its context. Fine-tuning means changing model weights using task-specific training data. Autoregressive means generating or scoring text one next token at a time. Few-shot in this paper usually means 10–100 prompt demonstrations, limited by the 2,048-token context window.

What remains uncertain

The experiments do not establish that scale alone produces robust reasoning, that in-context learning is equivalent to human learning, or that the reported behavior will generalize to every real deployment. They also do not isolate which training examples produced each capability. The most important open question is still mechanistic: when GPT-3 appears to learn from a prompt, how much is new adaptation and how much is recognition of patterns absorbed during pretraining?