Scaling Laws for Neural Language Models

Jared Kaplan and colleagues, 2020 · arXiv:2001.08361

Takeaway

Three things to remember

1. Scale behaved predictably. In these experiments, test loss fell along smooth power laws as model size, dataset size, or training compute increased—provided the other two were not the bottleneck. The fitted trends covered six orders of magnitude in model size and eight in compute. Paper §1.1–1.2, pp. 3–5
2. The three resources must stay in balance. A larger model eventually runs out of useful data; more data cannot rescue a model that is too small; and both need enough compute. The paper summarizes this balance with fitted equations for model size, data, training steps, and loss. Paper §1.1, pp. 3–4
3. For a fixed training budget, the paper favored a bigger model trained for fewer steps. Its compute-efficient recipe stops well before full convergence. That is an empirical claim about cross-entropy loss on the study's setup—not proof that every larger model has better real-world abilities. Paper §6 and Appendix B, pp. 15–17, 21–22

Visual 1. The paper studies three separate levers. Each follows a smooth power law only while the other levers are large enough not to limit the run. This is a new teaching diagram, not a reproduction of a paper figure.

Problem

Planning expensive training runs

Training a language model forces several linked choices: how many parameters to use, how much text to collect, and how long to train. Testing every combination at the largest scale is too expensive. The paper asks whether smaller experiments reveal stable rules that can forecast the test loss of larger runs and identify a better use of a fixed compute budget. Paper §1, pp. 2–5

What “better” means here

The outcome is cross-entropy test loss, measured in nats: how surprised the model is by the next token. Lower is better. The main experiments use autoregressive, decoder-only Transformers trained on WebText2, a byte-pair-tokenized text dataset, with loss averaged over 1,024-token contexts. “Model size” excludes embedding parameters, and training compute is estimated as roughly 6 × parameters × batch size × steps. Paper §1.3 and §2, pp. 6–7

Method

Change one bottleneck at a time

The researchers trained many models while varying non-embedding parameter count, dataset tokens, and training compute. To measure the effect of one resource, they made the others sufficiently large. They also varied Transformer depth, width, and attention heads; within the tested range, overall scale mattered much more than these shape choices. LSTMs and recurrent Transformers appeared as comparisons, but the main fitted laws concern standard Transformers. Paper §2–3, pp. 6–10

Fit simple power laws

A power law says improvement continues at a steady percentage rate on log–log axes, rather than by a fixed amount. The fitted loss exponents were about 0.076 for model size, 0.095 for dataset size, and 0.050 for optimally allocated compute. These small exponents mean returns diminish: for example, doubling parameters reduced loss by only about 5% in the parameter-limited fit. The scale constants depend on vocabulary and tokenization, so they are not universal physical constants. Paper §1.2, pp. 4–5; Appendix A, p. 20

Search the compute-efficient frontier

The fitted learning curves connect loss to model size and the number of parameter updates. The authors then solve for the model size, batch size, and step count that minimize loss at each fixed compute budget. Their empirical frontier scales roughly as model size C^0.73, batch size C^0.24, and serial steps C^0.03. In plain English: as the budget rises, the fit sends most of the growth into parameters, much less into batch size, and almost none into extra sequential training. Paper §1.2 and §6, pp. 5, 15–17

Visual 2. The lower lane captures the paper's counterintuitive training recommendation. The tiny curve sketches are conceptual, not data plots; the quantitative evidence remains in the text.

Evidence and limits

Strongest evidence

The strongest result is consistency across a wide sweep of experiments: the combined fits covered eight orders of magnitude in optimally allocated compute, six in model size, and more than two in dataset size. The overfitting fit says data should grow approximately as N^0.74; the authors translate that into about 5× more data for an 8× larger model. Their appendix comparison predicts that, at the same target loss, a compute-efficient run uses 2.7× more parameters, 7.7× fewer updates, and 65% less training compute than their chosen near-convergence baseline. These last three values are model-derived comparisons, not a separate production benchmark. Paper §1.1–1.2, pp. 3–5; Appendix B.3, p. 22

Where the claim stops

The authors say they lack a solid theory for the laws, making extrapolation hard to trust. Fits were poor in the smallest-data regime; regularization and data augmentation were not explored; the compute estimate omits context-dependent work that can matter for very long contexts; and some hyperparameters may be undertuned. The critical-batch prediction is also uncertain outside the measured loss range. Most importantly, the paper explicitly leaves open whether lower loss becomes better performance on relevant language tasks, and whether the relations extend to images, audio, video, or other settings. Paper §8, p. 19; Appendix C, pp. 22–23

Visual 3. The arrow stops at the evidence boundary: the experiments support claims about measured loss, while capability and cross-domain claims remain questions.

Practical meaning

A planning tool, not a guarantee

Interpretation: the paper offers a way to turn cheap pilot runs into a first-pass training plan. A team can fit its own curves, check which resource is limiting, and compare candidate allocations before committing its largest budget. But the 2020 coefficients should not be copied blindly to a different tokenizer, dataset, architecture, context length, optimizer, hardware regime, or target metric. Inference cost also matters: the appendix notes that a smaller-than-training-optimal model may be preferable when serving cost is included. Paper §1.2, p. 5; Appendix B.4, pp. 22

Decisions this paper can inform

Use the framework to ask whether a run is parameter-limited, data-limited, or compute-limited; whether training closer to convergence is worth more than increasing model size; and how much extra data is needed when scaling parameters. Before acting, remeasure the curves on the actual system and evaluate downstream quality, safety, latency, memory, energy, and serving cost separately. Those additional criteria are practical recommendations from this article, not claims established by the paper.

Source grounding and terms

• N: non-embedding model parameters. D: dataset tokens. C: estimated training floating-point operations. Paper §1.3, p. 6
• Test loss: average next-token cross-entropy on held-out text; lower is better. Paper §1.3 and §2, pp. 6–7
• Compute-efficient frontier: the fitted combination of model size and training duration that reaches the lowest loss for a fixed training-compute budget. Paper §6 and Appendix B, pp. 15–17, 21
• Uncertainty: WebText2 is described only as an extended WebText dataset in this paper; its precise composition is not documented here. The study does not establish a causal theory behind the fitted exponents. Paper §2, p. 6; Appendix C, pp. 22–23

Verification questions

1. Do the fitted exponents still hold on the intended data, tokenizer, architecture, optimizer, and context length?
2. Does lower held-out loss improve the downstream tasks that matter, rather than only the aggregate next-token metric?
3. Does the training-efficient choice remain economical after inference traffic, latency, memory, and energy are counted?
4. Where do the power laws bend or break when runs leave the scale and loss ranges measured in this study?