Sampling and Generation

From probability distribution to one word: the decoding problem

It's tempting to think of the model as “deciding” the next word, the same way you'd decide the next word in a sentence you're writing. It doesn't. A forward pass through the network ends at a fixed point every single time: a raw score for every entry in the vocabulary, tens of thousands of them, run through softmax to become a proper probability distribution that sums to 1. The model's job stops exactly there. Nothing in the weights picks a winner.

Turning that distribution into one emitted token is a separate, deliberate algorithm called decoding. It runs after the network, on top of the network's output, and it's where every question you've ever had about LLM behavior, why the same prompt gives different answers, why some models sound repetitive and others ramble, actually lives. Same model, same weights, same prompt, same probability distribution, and the decoding step still gets to choose differently every time.

1logits = model(tokens)          # one raw score per vocabulary entry
2distribution = softmax(logits)  # now sums to 1, a real probability distribution
3
4next_token = decode(distribution)  # greedy, temperature, top-k, top-p, or beam search
5                                    # this line is a different algorithm each time

This matters because it means the “personality” of a model's output, careful and consistent versus loose and varied, isn't fixed at training time the way its knowledge is. It's a knob you can turn after the fact, on the exact same set of weights. The rest of this article is about what that knob actually does, one decoding strategy at a time.

Temperature: turning the randomness dial

“Temperature” sounds like a mood setting, calm versus wild. It's actually one line of arithmetic applied before softmax runs: divide every logit by a number T, then softmax the result as usual.

1import math
2
3def softmax_with_temperature(logits, temperature):
4    scaled = [logit / temperature for logit in logits]
5    m = max(scaled)
6    exps = [math.exp(x - m) for x in scaled]
7    total = sum(exps)
8    return [e / total for e in exps]

You saw the temperature slider reshape this same kind of distribution back in the first article of this series, the exponential in softmax is what makes dividing the logits do something non-obvious to the output shape. Here's the mechanism in full. Dividing by T less than 1 makes the logits bigger in magnitude before the exponential runs, which stretches the gap between the top candidate and everything else. The distribution sharpens: probability mass concentrates on the tokens that were already ahead. Push T toward 0 and it approaches picking the single highest-probability token every time, deterministic, and at the extreme, repetitive, the model loops on its own safest continuation.

Dividing by T greater than 1 does the opposite: it shrinks the logits toward each other before the exponential runs, so the gap between the best token and the rest narrows. The distribution flattens, tokens that were barely plausible get a real shot at being sampled. Push T high enough and the model is close to picking uniformly at random from the vocabulary, more variety, and at the extreme, word salad. Neither end of the dial is usually where you want to sit. Most chat models default somewhere around 0.7 to 1.0.

Temperature alone reshapes the whole distribution, but it never removes a token from consideration entirely, only softmax-adjacent math ever assigns something exactly zero probability. That turns out to be a problem, because the vocabulary is huge and the tail of barely-plausible tokens is long. Reshaping isn't the same as cutting it off.

Top-k and top-p (nucleus) sampling: cutting off the long tail

You'd think a well-tuned temperature is enough. It isn't, and the reason is scale. A real vocabulary has tens of thousands of entries, and even a confident, well-trained distribution assigns some sliver of probability to thousands of tokens that are, in context, nonsense. Individually those slivers are tiny. Collectively, across a long generation, sampling from that whole tail eventually pulls one of them. Generation is autoregressive, every later token is conditioned on everything emitted so far, so one garbage token early in a sentence doesn't just look bad, it drags the rest of the sentence down with it. Temperature reshapes the distribution. It never removes anything from it.

Top-k is the blunt fix: keep only the khighest-probability tokens, discard the rest outright, renormalize what's left so it sums back to 1. If kis 5, only 5 tokens are ever eligible, no matter how the rest of the probability mass is distributed among the thousands you cut. Simple, cheap, and its exact weakness is that fixed number. A model that's extremely confident, 90% of its mass on one token, still gets 4 other tokens forced into contention. A model that's genuinely torn between a dozen reasonable continuations gets artificially capped at 5, and some of those reasonable options never get a chance.

Top-p, also called nucleus sampling, fixes exactly that. Instead of a fixed count, sort tokens by probability descending and keep adding them to the eligible set until their cumulative probability crosses a threshold, commonly 0.9. That set might be 2 tokens wide or 40, it adapts to the shape of the distribution at that specific step. A peaked, confident distribution needs only a couple of tokens to reach 0.9 of the mass. A flat, genuinely uncertain distribution needs many more, and top-p keeps all of them eligible instead of arbitrarily lopping off at some fixed rank.

Switch between the two presets above and watch the survivor count. On the peaked “capital of France” distribution, top-p at 0.9 keeps around 4 tokens, everything past “located” is already redundant. Fix top-k at 5 instead on that same distribution and a token like “actually” sneaks in purely because 5 was the number you picked, not because the model thought it was plausible. Now switch to the flatter “favorite hobby” distribution: top-p at 0.9 has to keep 8 tokens to reach the threshold, but a fixed top-k of 5 would cut “gardening,” “writing,” and “cycling,” three answers that were nearly as likely as the ones kept. Same k, opposite outcome, because k doesn't know or care how confident the model is at that step.

Temperature decides how sharp or flat the distribution is. Top-k and top-p decide how much of that distribution's tail is even in play before sampling happens. All three still leave one thing unanswered: why sample at all, instead of always taking the safest, highest-probability path.

Greedy decoding, beam search, and why chatbots use neither

The obvious answer to “why sample at all” is: don't. Always take the single highest-probability token, every step, no dice roll. That's greedy decoding, and it is deterministic, the same prompt really does give the same output every time. It also produces some of the dullest text an LLM can generate. Greedy decoding has no mechanism for escaping a locally safe choice, so it tends to fall into loops, the same phrase, or a close variant of it, repeated because at every step repeating was still the single most probable next token given what it just said.

1def greedy_decode(distribution):
2    return max(range(len(distribution)), key=lambda i: distribution[i])

Beam search: hedging by tracking several sequences at once

Greedy decoding's real flaw isn't any single step, it's that a token which looks slightly suboptimal right now can lead to a much better sentence two words later, and greedy has already committed and can't go back. Beam searchhedges against that by tracking several candidate sequences in parallel, called beams, instead of one. At each step, every beam gets extended by its most likely next tokens, all of those extensions get scored by their running total probability, and only the top few survive into the next step. It's a breadth-limited search for the highest-scoring whole sequence, not just the highest-scoring next token.

That makes beam search genuinely better than greedy for tasks with a fairly well-defined correct answer, translation, summarization, where there's a right ballpark and the job is finding the best-scoring sequence inside it. It costs a lot more, though. Tracking b beams multiplies the forward passes per step by b, and for open-ended conversation it doesn't even pay off: beam search is still, at its core, maximizing probability. It converges on the same kind of bland, generically-safe phrasing greedy decoding does, just with a wider search around it. A tie between five polished, forgettable sentences is still a forgettable sentence.

This is why essentially every production chat model samples instead, temperature to shape the distribution, top-k and top-p to cut off the tail before rolling the dice, exactly the stack covered in this article. Not because sampling is fancier than search, but because the goal is different. A translation has a target to converge on. A conversation doesn't, it has a wide range of reasonable next things to say, and the whole point of sampling is refusing to collapse that range down to one “best” answer every single time.

Every one of these strategies, greedy, beam, temperature, top-k, top-p, still runs the model fresh at every generated token, recomputing attention over the entire sequence so far. That recomputation is expensive, and it's avoidable in a specific, clever way. That's the KV cache, and what it caches, and why it exists, is next.