Training Compute-Optimal Large Language Models

Jordan Hoffmann and colleagues · arXiv:2203.15556 · submitted 29 March 2022

Takeaway

The paper in one sentence

When the training-compute budget is fixed, this paper asks how much should go into a model’s parameter count and how much should go into the number of training tokens; its answer is that the two should grow at roughly the same rate. In plain terms: do not spend nearly all the extra compute on making the model wider or deeper—also let it learn from proportionally more text. Source: abstract; Sections 1 and 3.4; Table 2.

Three points to remember:

1. Bigger is not automatically a better use of the same training budget. A very large model can be undertrained if it sees too little data.
2. The paper’s rule is balanced scaling. When model size doubles, the number of training tokens should also roughly double.
3. Chinchilla is the real-world check. With the same training compute as Gopher, a 70-billion-parameter model trained on 1.4 trillion tokens beat the 280-billion-parameter Gopher on most reported tasks.

Teaching visual, newly generated for this article. It explains the allocation problem; it is not a data chart or a reproduction of a paper figure.

Problem

A budget allocation problem

Training compute is usually constrained: a team has a certain number of accelerators for a certain amount of time. For a standard dense Transformer, that budget is spent jointly on the size of the model and on how many tokens pass through it. A larger model can represent more complex patterns, while more tokens give it more chances to learn. The practical question is where the best trade-off lies. Source: Section 1, especially the fixed-FLOP formulation around Equation 1.

Why the old recipe could waste compute

Earlier scaling guidance, especially Kaplan et al. (2020), favored growing model size faster than training data as compute increased. Many prominent models then clustered near roughly 300 billion training tokens even as parameter counts rose sharply: GPT-3 used 175B parameters and 300B tokens; Gopher used 280B and 300B; MT-NLG used 530B and 270B. The authors agree that a compute-optimal model need not be trained all the way to its lowest possible loss, but argue that large models still need far more tokens than the earlier rule suggested. Source: Sections 1 and 2; Table 1.

Method

More than 400 training runs

The authors trained more than 400 autoregressive language models, ranging from 70 million to over 16 billion parameters and from 5 billion to 500 billion training tokens. They observed how final training loss changes with parameter count, token count, and training compute. They then extrapolated a compute-efficient frontier: for each compute budget, which parameter–token combination should produce the lowest loss? Source: abstract; Sections 1 and 3.

Three ways to estimate the same frontier

The study uses three estimation routes. First, it follows complete training curves for fixed model sizes and selects the lowest-loss run at each compute level. Second, it holds compute fixed, varies model size, and locates the bottom of each resulting IsoFLOP loss profile. Third, it fits one parametric loss function to the final losses, separating irreducible text uncertainty, limited model capacity, and incomplete training. Despite their different assumptions, all three routes predict near-equal growth in parameters and tokens as compute grows; the earlier Kaplan rule reported exponents of 0.73 for parameters and 0.27 for tokens. Source: Sections 3.1–3.4; Table 2.

Teaching visual, newly generated for this article. The small curve shapes are conceptual and carry no numeric values.

A large-scale test, not just a fitted curve

For roughly Gopher’s training budget, the estimates suggested a model between 40B and 70B parameters trained on much more data. The team chose the upper end and trained Chinchilla: 70B parameters on 1.4T tokens. Gopher had 280B parameters and 300B tokens. The two used the same total training FLOPs, so the comparison tests the allocation recipe rather than simply spending more on Chinchilla. The setup was not perfectly controlled, however: Chinchilla also used AdamW instead of Adam, a slightly changed data mixture, and a modified tokenizer. Source: Sections 3.4, 4, and 4.1; Tables 1 and 4.

Evidence and limits

What the head-to-head result shows

Chinchilla reached 67.6% five-shot accuracy on MMLU, versus 60.0% for Gopher, and performed better on 51 of the 57 MMLU tasks, tied on two, and was worse on four. On the 62 reported BIG-bench tasks, its mean accuracy was 65.1% versus 54.4%, with worse performance on four tasks. It also reduced WikiText-103 perplexity from 7.75 to 7.16 and improved over Gopher across every reported Pile subset. Taken together, these experiments are strong evidence that Gopher’s budget was better spent on a smaller model trained longer. They do not prove that the exact same ratio is optimal for every architecture, dataset, or compute scale. Source: Sections 4.2.1, 4.2.2, and 4.2.4; Tables 6 and A7; Figures 5–7.

Teaching visual, newly generated from values in Table 1 and Section 4. It is a conceptual comparison, not a plot of measured performance.

What remains uncertain

The authors had only two comparable runs at the largest scale—Gopher and Chinchilla—and no intermediate large-scale checks. Their projection assumes a power-law frontier, yet they observed curvature at high compute, which may mean even smaller models are optimal there. The scaling experiments used less than one epoch of data, so repeated-data training was left for future work. More data also raises leakage risk: because Chinchilla saw about four times as many tokens, overlap with evaluation data could inflate language-modeling scores. Finally, better loss did not remove social harms. The paper reports remaining gender bias and almost no change in unprompted toxicity scores, and warns that web-scale data can contain toxic language, bias, and private information. Source: Sections 4.2.1, 4.2.7, and 5; Appendix A and the model card.

Practical meaning

Plan the model and dataset together

For a team planning pre-training, the paper’s useful change is a budgeting habit: choose parameter count, token count, and learning-rate schedule together before the expensive run. Under this paper’s assumptions, extra compute should fund both a larger model and more training data, rather than parameter count alone. A smaller, more fully trained model can also be cheaper to serve and fine-tune because inference cost and memory depend heavily on parameter count. That operational benefit is a reasonable product interpretation of the paper’s result, not proof about a particular deployment. Source for the underlying model claim: Sections 1, 4, and 5.

Do not turn one scaling law into a universal recipe

The result is most directly grounded in dense autoregressive Transformers trained on the paper’s data mixtures and schedules. Before applying it elsewhere, a team should rerun smaller IsoFLOP experiments on its own architecture and data, check whether repeated epochs change the frontier, and audit benchmark contamination, privacy, bias, and toxicity. The paper itself calls extension to other modalities an expectation, not an established finding. Source: Sections 3, 4.2.7, and 5.

Sources and verification notes

• Primary source: arXiv abstract and metadata; full manuscript in HTML.
• Strongest evidence: the same-compute Chinchilla–Gopher comparison, especially Section 4.2 and Tables 6–9.
• Main verification questions: Would the near-equal rule survive other architectures and cleaner datasets? Does it hold when data must be repeated for multiple epochs? How much of the downstream gain remains after stricter contamination checks? How much did the optimizer, tokenizer, and data-mixture changes contribute?
• Editorial distinction: statements about measured losses and benchmark scores are author results; statements about planning or deployment are interpretations drawn from those results.