Curriculum · Unit 7 · Scale

Why did everyone bet on bigger, and how much bigger is the right size?

Around 2020, researchers noticed that language models get better in a strikingly regular way as you feed them more computing power. That regularity made "build it bigger" a plan you could forecast and budget for. Two years later, a second study showed that most labs had been spending that budget a little lopsidedly.

Nib says: Two toys here. In the first, drag the slider (or press +10×) to pour more compute into training and watch the loss fall by the same step every time. In the second, the budget is fixed: slide between a big model with little data and a small model with lots of data, and hunt for the spot where the beam balances. Curves marked illustrative show the shape of the idea, not a real model's numbers.

Toy 1 · The scaling curve

Every 10× buys the same step down

Loss is how wrong the model's next-token guesses are on text it hasn't seen. Lower is better. Both axes are logarithmic, so each gridline is ten times the last.

10¹⁸
FLOPs 10²⁶
FLOPs

Illustrative power law: loss = 4 × (compute ÷ 10¹⁸)−0.05. Real curves are measured on many models and are noisier. The GPT-3 compute mark is real; the loss values are not.

Toy 2 · The Chinchilla balancer

Same budget, different split

The compute budget is fixed at about 6 × 10²³ FLOPs, roughly what DeepMind spent on each of Gopher and Chinchilla. Since compute ≈ 6 × parameters × tokens, a bigger model means fewer training tokens, and the reverse.

Bigger model,
less data Smaller model,
more data

Illustrative curve with the form Hoffmann et al. fit, L = E + A/Nα + B/Dβ, with constants chosen so the low point sits at 20 tokens per parameter. The parameter and token counts follow from C = 6ND; the loss values are made up, and the bottom of the curve is zoomed in.

The idea in plain words

In 2020, Jared Kaplan and colleagues at OpenAI measured something unusually tidy. As they trained bigger language models, on more text, with more computing power, the test loss fell along a smooth power law: a straight line on a log-log chart, holding across a huge range of sizes. These scaling laws meant you could train a few small models, draw the line, and predict roughly how good a model a hundred times bigger would be before you spent the money.

Later that year, GPT-3 showed what the bet could buy. With 175 billion parameters, it could often pick up a new task from a few examples typed into the prompt, with no extra training at all. The paper called this few-shot learning, and it made many people take "just make it bigger" seriously.

A handy rule of thumb ties the pieces together: training compute is about 6 × parameters × training tokens, counted in FLOPs. So once your budget is set, you face a choice. Spend it on a bigger brain that reads less, or a smaller brain that reads more?

In 2022, Jordan Hoffmann and colleagues at DeepMind trained hundreds of models to answer that, and found that model size and data should grow together, roughly in step. The rule people quote is about 20 training tokens per parameter. Their 70-billion-parameter Chinchilla, trained on about 1.4 trillion tokens, beat their own 280-billion-parameter Gopher, which used the same compute but read only about 300 billion tokens. Many earlier giants, it turned out, were oversized and underfed.

Scale also changed what AI is as an industry. Bigger runs need warehouses of GPUs, fast memory, power lines and cooling, mostly rented from hyperscalers. That's why AI news is now also chip, data center and electricity news. Nib's Mountain and the Goblin Guide to AI Infrastructure show you the hardware behind those FLOPs.

The back-of-the-envelope formula

Compute ≈ 6 × N × D

N is the number of parameters and D is the number of training tokens. The 6 is roughly 2 operations per parameter per token to run the model forward, plus about 4 more to work out how to adjust it.

Model (year)ND6NDD ÷ N
GPT-3 (2020)175B300B3.2×10²³≈2
Gopher (2021)280B300B5.0×10²³≈1
Chinchilla (2022)70B1.4T5.9×10²³20

Rough estimates from the formula; the papers report slightly different totals. Many labs now deliberately train smaller models on far more than 20 tokens per parameter, because a smaller model is cheaper to run for millions of users. Chinchilla's rule is about the cheapest way to reach a given loss during training, not the whole bill.

Three things to keep

Check yourself

A lab multiplies its training compute by 10. What does a scaling law let it predict?

Roughly how much the test loss will drop: about the same fraction as the last 10× bought. It doesn't promise a model that is "ten times smarter," and it says little about exactly which new skills will appear.

Same budget, two plans: a 280B model on 300B tokens, or a 70B model on 1.4T tokens. Which did Chinchilla find works better, and why?

The smaller model trained on more data. The big model was undertrained for its size: with a fixed budget, it's better to balance parameters and tokens, roughly 20 tokens per parameter, than to pour everything into size.

Using compute ≈ 6ND, about how much compute does it take to train a 1-billion-parameter model on 20 billion tokens?

6 × 10⁹ × 2×10¹⁰ = 1.2 × 10²⁰ FLOPs. That's about five thousand times less than Chinchilla's run, and it also happens to be exactly 20 tokens per parameter.

Go deeper