Why a raw pretrained model doesn't chat
A common assumption: train a language model on enough of the internet and it naturally learns to be helpful, because helpfulness is what a smart, well-read entity would do. That's wrong, and the previous article explains why. Pretraining, as covered in Pretraining at Scale, has exactly one objective: predict the next token given the tokens before it, over a huge scrape of web pages, books, and code. Nothing in that objective ever asked the model to be helpful. It asked the model to be a good guesser of what text comes next.
Feed a base model the prompt “How do I bake bread?” and it doesn't reach for an answer. It reaches for whatever continuation looks statistically plausible given everything it saw during training that started that way. Sometimes that's a direct answer, because plenty of web text is direct answers. Just as often it's a list of similar questions, because that prompt looks exactly like the first line of a forum thread or a FAQ page, and those get followed by more questions, not answers.
Completion of “How do I bake bread?”
How do I bake bread? How do I bake cookies without a stand mixer? How do I make pizza dough at home? Reply Report Share this thread Related questions from last week...
Toggle between the two modes above. The base model completion isn't broken or buggy, it's doing exactly what it was trained to do: continue the text in a way that matches the distribution of similar text it saw. A prompt that starts a forum thread gets continued like a forum thread. A prompt that reads like a worksheet prompt gets continued like a worksheet, with more prompts underneath it. The model has no notion of “the user asked me something, so I should answer it,” because pretraining never drew that boundary. There is no user, no assistant, just one long stream of tokens to keep going.
Rule: a base model completes text plausibly. An assistant answers a question directly, in a consistent voice, and stops when it's done. Those are different skills, and pretraining only teaches the first one.
Getting from one to the other isn't a matter of prompting harder or waiting for a bigger model. It requires more training, on different data, pointed at a different goal. That's the rest of this article: three techniques, each one layered on top of the last, that turn a fluent text-continuer into something that behaves like an assistant.
Supervised fine-tuning: teaching the format
The fix for the problem above isn't a new architecture and isn't a new loss function. It's the exact same cross-entropy next-token loss from Loss and Backpropagation, computing the exact same gradients through the exact same transformer blocks. What changes is the data. Instead of a scrape of the open web, the model keeps training on a curated set of (instruction, ideal response) pairs, written or vetted by humans specifically to demonstrate what a good answer looks like.
This step is called supervised fine-tuning, SFT for short, and “fine” is doing real work in that name. Pretraining runs over trillions of tokens of whatever text was available. SFT runs over a dataset that might be tens of thousands to a few million examples, tiny by comparison, but built for exactly one purpose: show the model, over and over, what “answer the question directly” looks like instead of “continue the text however the web tends to continue it.” Each example is typically wrapped with role markers, something like a user turn followed by an assistant turn, so the model also picks up the conversational structure itself: where a turn starts, where it ends, whose voice is speaking.
Rule: SFT doesn't teach the model new facts or new capabilities, it teaches the model a new default behavior for old capabilities it already has. The knowledge came from pretraining. The habit of answering directly comes from SFT.
After SFT, a prompt like “How do I bake bread?” reliably gets a direct recipe instead of a list of related forum questions, because every training example the model saw at this stage ended a question with a direct answer, never with more questions. That's imitation learning: the model is copying the shape of good behavior from examples of it.
But imitation has a ceiling. SFT data has one ideal response per prompt, written by one labeler on one day. It has nothing to say about the harder question: given two responses that are both reasonable, both correctly formatted, both factually fine, which one is actually better. There's no single “correct” token sequence to imitate for a judgment call like that, and next-token loss has no way to express “this one, but only slightly more than that one.” That gap is what the next technique closes.
RLHF: learning from human preference rankings
SFT teaches format. It can't teach taste. Whether a response is appropriately concise, whether its tone lands right, whether it should refuse a request instead of attempting it, none of those are things you can write down as one ground-truth token sequence and train a next-token loss against. They're comparative judgments: given two candidate answers, which one is better. Reinforcement learning from human feedback, RLHF, is built specifically to learn from comparisons instead of single correct answers.
The process runs in two stages. First, take the SFT model and generate multiple candidate responses to the same prompt. Human raters look at pairs of those responses and pick which one they prefer, thousands of times over, across thousands of prompts. That preference data trains a second, separate model called the reward model: given a prompt and a response, it outputs a single number, its estimate of how much a human would like that response. The reward model never generates text, its only job is scoring it.
Second, use that reward model to actually update the language model, through reinforcement learning. The language model generates a response, the reward model scores it, and an algorithm called PPO, proximal policy optimization, adjusts the language model's weights to make highly-scored outputs more likely and poorly-scored ones less likely. Do this over millions of generated responses and the model's behavior shifts toward whatever the reward model has learned humans tend to prefer, tone, thoroughness, appropriate refusals, all of it, without any of those qualities ever being written down as an explicit rule.
Gotcha: a reward model is a proxy, not the real thing, and PPO is very good at finding cracks in a proxy. Left unchecked, this produces reward hacking, responses that score high on the reward model without actually being better, padding an answer with unnecessary length because the reward model has a slight bias toward verbosity, for example. Real RLHF pipelines add a penalty that keeps the updated model's outputs statistically close to the original SFT model, to keep this kind of drift in check.
RLHF works, and it's the process behind the first wave of assistant-style models that felt meaningfully different from a raw completion engine. It's also expensive and fiddly to run correctly: you're training and maintaining two separate models, the reward model and the policy, and PPO has a reputation for being sensitive to its own hyperparameters, unstable in ways that are hard to diagnose. That cost is exactly what the next technique was built to avoid.
DPO: getting RLHF's result without the reinforcement learning
Look at what RLHF actually needs as input: pairs of responses to the same prompt, with a label saying which one humans preferred. Look at what it produces: a language model that's more likely to generate the preferred kind of response. RLHF gets from one to the other through a reward model and a reinforcement learning loop, but that path is not the only way to connect those two points. Direct Preference Optimization, DPO, takes the same chosen-versus-rejected preference data and skips straight to the destination.
DPO reformulates the preference as a direct loss function on the language model itself. For each training example, that's a prompt, a preferred response, and a rejected one, the loss pushes the model to increase the relative probability it assigns to the preferred response versus the rejected one. No reward model gets trained. No PPO rollouts get sampled. No separate scoring pass happens at all. It's still gradient descent on a loss function computed from the model's own output probabilities, the same mechanics as every other training step in this series, just shaped around pairs of outputs instead of pointed at one ideal target per prompt like SFT.
Rule: RLHF trains a judge, then runs a separate, unstable optimization process to satisfy the judge. DPO proves you can skip training the judge and update the model directly from the verdicts instead, mathematically, and get a comparable result.
This is why DPO became the more common choice for most teams doing preference tuning. There's one model to train instead of two, one stable supervised-style loss instead of an RL loop that needs careful tuning to avoid collapsing, and the infrastructure looks much closer to the SFT step that already precedes it. RLHF and PPO haven't disappeared, some frontier labs still lean on the extra flexibility a learned reward model provides, but for the large majority of instruction-tuned models released since DPO's introduction, it's the default path from preference data to a finished model.
Put the three stages together and the picture is complete. Pretraining teaches the model language and knowledge. SFT teaches it the shape of a direct answer. RLHF or DPO teaches it which of several correct-shaped answers humans actually prefer, tone, safety, refusals included. Three different training runs, three different datasets, and in every case, at the bottom, the same gradient descent machinery from the Loss and Backpropagation article, just pointed at a different objective each time.