The scaling law shape: loss falls predictably with compute
The popular story about AI progress is that it's unpredictable, a string of surprise breakthroughs nobody saw coming. Model behavior can genuinely surprise you. But the number that drives all of it, the loss a model reaches after training, does not. Plot loss against the amount of compute spent training the model, on a log-log scale, and it doesn't scatter or plateau unpredictably. It forms a straight line, holding across many orders of magnitude of compute, from tiny runs you could do on a laptop to runs costing tens of millions of dollars.
A straight line on a log-log plot means one specific thing mathematically: a power law. Loss shrinks as compute raised to some small negative exponent, roughly loss ≈ A × compute^(−α), where A and α are constants you fit from actual training runs, not guessed. The Pretraining at Scale article gave you the compute formula this α term feeds into, total_flops ≈ 6 × num_parameters × total_tokens. This article is about the other half: what happens to loss once you know that FLOPs number, and the Evaluating LLMs article's cross-entropy loss is the exact y-axis quantity being plotted here.
Why this matters practically: a straight line is extrapolatable. Run a handful of small, cheap training jobs, say at 1018 to 1020 FLOPs, plot the resulting losses, fit the line. That line now predicts the loss at 1024 or 1025 FLOPs, a run that hasn't happened yet and would cost millions of dollars if it did. This is exactly how frontier labs decide whether a proposed large run is likely to be worth its cost, months before committing the hardware, and it's the empirical foundation under everything the rest of this article covers.
Rule: a straight line on a log-log plot is a power law, not a coincidence of the axes. Move the slider above across orders of magnitude of compute and the point never jumps off the line, that predictability is the whole reason scaling laws are useful for planning.
One caveat worth flagging early: this relationship holds for loss, the raw next-token prediction quality the model was actually trained to minimize. It does not automatically hold for every downstream capability or benchmark score you might care about, and the gap between “loss went down smoothly” and “this specific skill improved smoothly” is exactly the subject of the emergent abilities section later in this article.
Chinchilla: parameters versus data, the compute-optimal tradeoff
The scaling law in the previous section tells you loss falls as compute grows. It does not tell you how to spend that compute. Given a fixed FLOPs budget, you can buy a bigger model trained on less data, or a smaller model trained on more data, and total_flops ≈ 6 × num_parameters × total_tokenslets both trade off against each other freely. For years, the industry's answer, set by GPT-3 and the models that followed its lead, was: spend it on parameters. Scale the model up aggressively, keep dataset size roughly fixed or growing much slower.
In 2022, DeepMind's Chinchilla paper checked that assumption directly and it broke. The team trained over 400 models across a wide range of sizes and token counts, fit the scaling law from actual data, and found that GPT-3-era models were undertrained for their size, not undertrained in some vague sense, undertrained in a specific, quantifiable way: for the compute they cost, a substantially smaller model trained on far more data would have reached lower loss. Their own 70-billion-parameter Chinchilla model, trained on 1.4 trillion tokens, beat Gopher, a 280-billion-parameter model from the same lab trained on far fewer tokens, on the same compute budget.
Rule: the Chinchilla-optimal ratio is roughly 20 training tokens per parameter. A 10-billion-parameter model wants on the order of 200 billion tokens to be compute-optimal, not 30 billion, not 2 trillion. Below that ratio you're leaving loss on the table with a model too big for its diet, above it you're past the point of useful returns for this fixed budget.
The derivation is short enough to run yourself. Take the compute formula from Pretraining at Scale, substitute the 20-tokens-per-parameter ratio for total_tokens, and solve for the parameter count that's optimal at a given compute budget.
Compute-optimal split, derived from C ≈ 6 × N × D and D ≈ 20 × N
1def compute_optimal_split(flops_budget):2 # C ~= 6 * N * D, and Chinchilla found D ~= 20 * N3 # substituting: C ~= 6 * N * (20 * N) = 120 * N^24 n_opt = (flops_budget / 120) ** 0.55 d_opt = 20 * n_opt6 return n_opt, d_opt78n, d = compute_optimal_split(1e24)9print(f"{n:.2e} parameters, {d:.2e} tokens")
That single result reshaped how frontier labs plan training runs. Instead of asking “how many parameters can we afford to serve” and then training whatever data was on hand, labs now start from a compute budget, solve for the compute-optimal parameter and token counts, and then go build a dataset large enough to hit that token target, which is why the years after Chinchilla saw a scramble for more training data, not just more GPUs. Later models like LLaMA deliberately trained smaller models on far more tokens than the Chinchilla ratio strictly required, because a smaller model that's cheaper to run at inference time can be worth slightly worse compute efficiency during training. That's a real tradeoff labs make on top of the compute-optimal point, not a rejection of it.
Notice what this section and the last one have in common: both come from fitting a curve to real training runs and reading off a prediction, not from theory alone. The next section looks at a place where reading a curve incorrectly produced a conclusion that looked dramatic and turned out to be mostly an illusion of the metric being used.
What “emergent” abilities are, and reason for skepticism about the term
Around 2021 and 2022, researchers noticed something that looked genuinely strange. Test models of increasing size on tasks like three-digit multiplication or certain multi-step reasoning benchmarks, and small and mid-sized models score at essentially random chance, flat near zero no matter how much bigger you make them within that range. Then, at some threshold of scale, accuracy jumps, not gradually, sharply, from near-zero to well above chance within a relatively narrow band of model size. That was named an emergent ability: a capability the smaller versions of the model simply did not have, that the larger version does, with no smooth ramp in between.
It's an unsettling idea if you take it at face value. The scaling law from the first section of this article says loss falls smoothly and predictably with scale, no jumps anywhere. If specific abilities can appear suddenly and unpredictably at some scale you haven't reached yet, that means you cannot actually predict what a bigger model will be able to do just by extrapolating a curve, which undercuts a lot of what makes scaling laws useful for planning in the first place.
Gotcha:a 2023 paper, “Are Emergent Abilities of Large Language Models a Mirage?”, re-ran the same experiments with a different metric and the sharp jumps mostly disappeared. The underlying model wasn't doing anything different. The metric was.
Here's the mechanism. Most of the tasks that showed “emergence” were scored with a discontinuous metric: exact-match accuracy on a multi-step problem, right or wrong, nothing in between. Multiply two three-digit numbers and get every single digit correct or the whole answer counts as a miss, there's no partial credit for getting four out of five digits right. A model that's slowly, smoothly improving its per-token accuracy will look completely flat on that metric for a long stretch, because getting one digit wrong in a five-digit answer still scores zero, right up until it crosses the point where it reliably gets all five digits right, at which point the score jumps from near-zero to near-100% almost immediately. The underlying skill was improving the entire time. The ruler just couldn't show it.
Swap exact-match accuracy for a continuous metric on the exact same tasks, like per-token cross-entropy loss or the probability the model assigned to each correct digit, and the curve stops looking like a cliff. It looks like a smooth, gradual improvement across model scale, the same shape as the loss-versus-compute line from the first section of this article. The capability wasn't emerging out of nowhere. It was there the whole time, moving gradually, and a metric with a hard pass or fail threshold was hiding the gradient from you.
This doesn't mean nothing surprising ever happens at scale, and it doesn't mean every claimed emergent ability is purely a metric artifact, some cases are genuinely more subtle than a single discontinuous benchmark. But the more skeptical reading, that a lot of “sudden capability jumps” are the predictable output of a smooth underlying trend run through an unforgiving pass or fail scorer, is now the better-supported one. It's also a useful habit generally: before concluding a system did something qualitatively new, check whether the metric you're reading is capable of showing you a gradual change in the first place.
Using scaling laws to predict a model before training it
Put the last three sections together and you get an actual planning process, not just three independently interesting facts. A lab deciding whether to attempt a frontier training run doesn't start by training it and seeing what happens, that would mean risking tens of millions of dollars on a guess. It starts by running a series of much smaller, much cheaper training jobs across a range of compute budgets, exactly the kind of runs that produced the straight line in the first section, and fitting a scaling law to the results.
From there the process is mechanical. Pick the compute budget you're willing to spend. Use the Chinchilla-style tradeoff to solve for the compute-optimal split between parameter count and token count, the same arithmetic from the second section. Plug that compute figure into the fitted scaling law and read off a predicted loss, before a single dollar goes toward the actual full-scale run. If that predicted loss isn't meaningfully better than the lab's last model, the run doesn't get greenlit, no matter how much hardware is sitting idle. If it is, the prediction becomes the number the training team is held to, and a real run that lands far off that line is treated as a sign something went wrong in training, not just a disappointing result to shrug off.
This is also where the emergent abilities caveat from the previous section earns its keep. The scaling law predicts loss reliably. It does not, by itself, promise that a specific downstream capability will appear at a specific scale, because as you just saw, capability benchmarks can hide or exaggerate what the loss curve is actually doing. A careful prediction distinguishes the two: “loss will land around here” is a claim scaling laws are good at, “the model will suddenly be able to do X” is a claim that needs its own evidence, not just an extrapolated line.
Predicting a model's loss before spending the money to train it used to be closer to a research curiosity than an operational tool. It's now a standard input into how large training runs get planned and approved at every lab running them, sitting right alongside the engineering questions from the Pretraining at Scale article about datasets and batches and GPU-time. That closes out the theory side of this series. The next and final article, Build a Tiny LLM From Scratch, takes every mechanism covered across the fourteen articles before it, tokenization, embeddings, attention, the transformer block, loss, sampling, and everything in between, and wires them together into working code you can actually run.