What Is a Language Model

What a language model actually is

Most explanations describe a language model as something that "understands" text, or a database of facts it "remembers." Both are wrong in ways that will confuse you later, when the model confidently states something false or forgets what you told it two messages ago. Neither behavior makes sense if you think of it as a knowledge store. Both make complete sense once you know what it actually is.

A language model is a function. Give it a sequence of tokens, it gives you back a probability distribution over what token comes next. That is the entire definition. Everything else, chat interfaces, code generation, reasoning-looking output, is built on top of calling that function over and over.

Making that concrete

Take the sentence "The cat sat on the ___". A trained model does not know, in any factual sense, what the cat sat on. It has seen enormous amounts of text where sentences like this one are completed with words like mat, floor, or couch, far more often than with words like moon or equation. So it assigns mat a high probability, floor a lower but still meaningful one, and moon something close to zero. That ranked list of probabilities is the model's entire output for this step.

Try it below. Pick a prompt and watch the distribution. Notice that "2 + 2 =" gets a sharply confident answer, not because the model computed the sum, but because virtually every training example that looks like that sentence was completed the same way. Rule: confidence in a language model's output reflects how consistently the training data completed a similar pattern, not whether the model verified anything.

The distribution, the sample, the loop

A single forward pass through the model produces one probability distribution over the entire vocabulary, tens of thousands of possible next tokens, each with a probability. Generating an entire paragraph is that same step, repeated. Pick one token from the distribution, append it to the input, run the model again on the new, slightly longer sequence. This is called autoregressive generation: each output becomes part of the input for the next step.

These probabilities are hand-authored to illustrate the shape of a real distribution, not pulled from a running model. Adjust the temperature slider and watch the bars sharpen toward one token or spread out across many, that reshaping is a real, verifiable calculation (softmax with temperature scaling), covered in full in the Sampling and Generation article later in this series.

Why "pick one" and not "pick the best"

You might expect the model to always output the single highest-probability token. Some systems do exactly that (it's called greedy decoding), but it tends to produce dull, repetitive text. Most production systems sample: they roll a weighted die over the distribution, so a token with 20% probability gets chosen roughly one time in five. Same underlying model, different generations, every time you run it. That's why asking the same question twice can give you two different answers with plausible-sounding confidence in both.

1tokens = tokenize(prompt)
2
3for _ in range(max_new_tokens):
4    distribution = model(tokens)      # probabilities over the whole vocabulary
5    next_token = sample(distribution) # weighted random pick, not always the top one
6    tokens.append(next_token)
7    if next_token == END_OF_TEXT:
8        break
9
10return detokenize(tokens)

Every capability that feels intelligent, writing a function, explaining a concept, carrying on a conversation, is this loop running thousands of times, one token guessed after another, each guess conditioned on everything generated so far.

What this explains, and what it doesn't

Once "next-token prediction" is the model you actually hold in your head, a lot of confusing LLM behavior stops being confusing. It becomes predictable, because you can ask a sharper question: "what would a plausible continuation of this text look like," instead of "does the model know this."

Then why does this feel like more than autocomplete?

Fair question. Your phone's keyboard also predicts the next word, and nobody confuses it for something intelligent. The difference is scale and depth: a phone keyboard looks at the last one or two words and a tiny model. A large language model looks at thousands of tokens of context, was trained on a meaningful fraction of publicly available text, and runs that prediction through a network with billions of tuned parameters. The objective stayed exactly the same, predict the next token, but at sufficient scale, getting that objective right on enough diverse text requires the model to internalize grammar, facts, style, and multi-step patterns well enough to reproduce them. Nobody designed those capabilities in directly. They fell out of the objective once the scale was large enough.

Where this series goes from here

Fifteen articles, each one building directly on the last. No article assumes anything beyond what came before it. By the end, you will have built a working, if small, GPT-style model yourself, and understood every line of it.