Perplexity: cross-entropy loss, made interpretable
It's tempting to treat perplexity as some separate evaluation metric, invented specifically to score language models after the fact. It isn't. The Loss and Backpropagation article covered cross-entropy loss as the number a model is trained to minimize: the negative log probability it assigned to the actual next token, averaged across every token it sees. Perplexity is that exact same number, unchanged, run through one transformation. Exponentiate it.
perplexity = e ^ (average cross-entropy loss). That is the whole formula. Nothing new gets measured, nothing new gets computed during evaluation that wasn't already being computed during training. The only reason this transformation exists is that a raw loss value like 2.0 doesn't mean anything to a human on its own. Perplexity turns it into a count you can actually picture.
Take a model with an average cross-entropy loss of 2.0 nats on some test set. e^2 is about 7.4. That means, on average, across this test set, the model was about as unsure at each step as if it had to choose uniformly among roughly 7 equally likely next tokens, then happened to guess right. Push the loss down to 0.5 and perplexity drops to about 1.6, close to certain. Push it up to 5.0 and perplexity jumps to about 148, badly lost. Same underlying number every time, one version a person can reason about at a glance.
Perplexity from the per-token losses already computed during training
1import math23def perplexity(losses):4 # losses: per-token cross-entropy loss on a held-out5 # test set, the same quantity minimized during training6 avg_loss = sum(losses) / len(losses)7 return math.exp(avg_loss)89perplexity([1.8, 2.1, 1.9, 2.2, 2.0]) # average loss 2.0
Rule: perplexity and cross-entropy loss move together by definition, not by empirical correlation. A model with lower loss has lower perplexity, always. There is no scenario where one drops and the other doesn't follow.
Try it directly below. Five toy predictions, one slider per token, each one the model's actual assigned probability for the real next word in “The cat sat on the mat.” Pull a slider down toward an unconfident guess and watch both numbers move together, the per-token loss climbs and the perplexity across all five climbs right along with it.
Drag any one slider down toward an unconfident guess and watch perplexity climb. One weak prediction is enough to raise the average uncertainty across the whole sequence, the same way one bad step drags down the model's score on the full test set.
Real perplexity numbers get computed over a held-out test set of thousands or millions of tokens, not five, and they are only ever meaningfully compared between models scored on the exact same test set with the exact same tokenizer. A perplexity of 12 on one dataset and 15 on a different one says nothing about which model is better. The comparison only holds once the test set is held fixed.
What perplexity misses
A lower perplexity number feels like it should mean a better model, full stop. It doesn't. Perplexity measures exactly one thing: how well a model predicts the next token in text that looks statistically like whatever it was trained and tested on. That's a narrow question, and a model can answer it well while failing at nearly everything a user actually wants from it.
Perplexity says nothing about whether a model can follow an instruction phrased a way it's never seen before. It says nothing about whether a multi-step answer is correct or merely fluent. It says nothing about whether the model just invented a citation that doesn't exist. None of that is next-token prediction quality on held-out text, it is downstream behavior that a single scalar loss was never designed to capture.
Two models can land within a fraction of a point of each other on perplexity and diverge wildly in practice. One follows instructions cleanly, the other ignores half of them. One works through a multi-step problem correctly, the other confidently goes wrong at step two. This is close to the exact gap between a base model and the instruction-tuned assistant built from it, covered in the From Base Model to Assistant article: fine-tuning barely moves perplexity on general text, and yet it is the entire difference between a model that completes your sentence and one that does what you actually asked.
If perplexity can't answer “is this model actually useful,” something else has to. That something is a benchmark suite: a fixed set of tasks with known right answers, scored directly on whether the model gets them right.
Benchmark suites: MMLU and friends
MMLU, short for Massive Multitask Language Understanding, is not a clever new evaluation technique. It's a fixed quiz: roughly 14,000 multiple-choice questions spread across 57 subjects, high school chemistry, US law, abstract algebra, professional medicine, world religions. Each question has four answer choices and exactly one correct one. Run a model over the whole set, count what fraction it got right, and that percentage is the score. Nothing more sophisticated is happening underneath.
Other benchmarks follow the same shape with a different focus. HellaSwag scores whether a model picks the sensible ending to a scenario. GSM8K scores grade-school math word problems. HumanEval scores whether generated code actually passes its test cases. Different content, same idea: a fixed set of questions with known right answers, scored automatically, no human judgment required per question.
That fixed, automatic scoring is exactly why benchmarks matter. Perplexity can't be compared across different test sets, but a benchmark score can be compared across every model that ever gets run against it, on the same questions, every time. That's why a model release announcement leads with “improved from 68% to 74% on MMLU” instead of a perplexity number nobody outside the lab can interpret.
But a fixed quiz has a fixed set of questions, and those questions live on the internet, which is also where pretraining data comes from. Benchmark contamination is what happens when MMLU questions, or close paraphrases of them, end up scraped into a model's training data along with the rest of the web. The model isn't reasoning its way to the right answer anymore, it's recalling one it already saw, and the score goes up without the underlying capability improving at all.
There's a subtler version of the same problem even without direct contamination. When every lab reports the same handful of benchmark numbers in every release, there's pressure, explicit or not, to optimize specifically for those numbers. A model can genuinely improve on MMLU without getting any better at the messy, unbenchmarked tasks a real user actually cares about. The yardstick is useful precisely because it's fixed, and that same fixedness is what makes it gameable.
Why hallucination resists easy detection
A hallucinationis a fluent, confident, wrong statement: a case citation that doesn't exist, a function argument that was never in the library, a historical date invented wholesale. The instinct is to treat this as a bug, some faulty code path that fires under specific conditions and could, in principle, get patched. It isn't. There is no separate hallucination code path to patch.
The Sampling and Generation article covered how every single token a model outputs, correct or not, gets produced the same way: sampled from a probability distribution over the whole vocabulary, shaped by everything the model learned during training. A hallucinated fact and a correct fact come out of the identical mechanism, run the identical arithmetic, at the identical layer. There is no internal flag anywhere in the network that reads “I am making this up right now,” because generation never learned to distinguish the two cases in the first place.
That is what makes hallucination fundamentally harder to catch than an ordinary software bug. A null pointer exception announces itself, a stack trace points at a line number, a malformed response fails to parse. A hallucinated citation is syntactically perfect, grammatically fluent, formatted exactly like a real one, and produced with the same distribution of token probabilities a true statement would have used. There is no error signal to catch, because from the model's perspective nothing went wrong.
Current mitigations reduce the problem without closing it. Retrieval grounding hands the model real source documents at generation time so it has actual text to draw from instead of only its trained-in weights, cutting down invented facts on questions those documents actually cover. Asking a model to cite its sources makes fabrication easier to check afterward, though the citations themselves can be invented just as fluently as anything else. Uncertainty estimation tries to surface cases where the model's output distribution is unusually flat, a rough proxy for “not confident,” but a model can be catastrophically, uniformly wrong with a sharply peaked distribution too.
Gotcha: none of these mitigations fully solve hallucination, and treating any one of them as a solved problem is a mistake worth avoiding on its own. They shift the odds. None of them restores the missing internal signal, because that signal was never there to restore.
This is also why evaluation is genuinely hard, not just tedious. Perplexity measures the wrong thing precisely, benchmarks measure the right thing approximately and are gameable, and the failure mode users care about most, confident fabrication, doesn't show up as an error signal anywhere in the pipeline. None of that is a reason to stop measuring. It is a reason to stay skeptical of any single number, including the ones in this article, and to keep asking what a given metric can't see.