A token ID is not a meaningful number
The previous article ended with text turned into a sequence of token IDs, integers like 464 or 2368. It is tempting to think the model can now just go to work on those numbers. It can't, not yet, because a token ID is only an index into a lookup table. It carries no information about what the token means. Token 464 is not “less than” token 2368 in any sense that matters, and two tokens with nearby IDs are not more related than two tokens with distant ones. The ID is assigned by whatever order the vocabulary happened to be built in.
Feeding a raw ID straight into a neural network would be actively harmful. Networks learn by adjusting weights based on numeric relationships: bigger numbers, closer numbers, and proportional numbers all carry implicit signal to a network doing arithmetic on them. An ID has none of the relationships it would imply. The network needs, instead, a representation where distance and direction actually mean something. That representation is an embedding: a vector of real numbers assigned to each token, positioned in space so that similar tokens end up near each other.
A language model is a next-word guesser, extraordinarily well trained. An embedding table is the part that turns “which word” into “where that word sits, relative to every other word the model knows.”
What the embedding table actually is
Mechanically, it is a single matrix. One row per vocabulary entry, so a 50,000-token vocabulary has 50,000 rows. Looking up a token's embedding is nothing more than reading row number token_id out of that matrix. Real models use hundreds to a few thousand columns per row, meaning each token becomes a vector of that many numbers. Those numbers are not hand-assigned by anyone. They start out random and are learned during training, the same way every other weight in the network is learned, covered in the Loss and Backpropagation article later in this series.
For now, the important shift is conceptual: stop thinking of a token as a symbol, and start thinking of it as a point in space. Every operation the rest of the network performs, from here through the final probability distribution, operates on that point, never on the original word or ID again.
Distance and direction become meaning
Once every token is a point in space, two geometric properties of that space start doing real work. The first is distance: words used in similar contexts during training end up positioned near each other, because the training process repeatedly nudges a word's vector toward the vectors of words it tends to appear alongside. “Cat” and “kitten” show up in similar sentences constantly, so their vectors converge. “Cat” and “refrigerator” almost never do, so theirs stay far apart. Nobody tells the model cats and kittens are related, it falls out of which words keep appearing near which other words.
The second property is direction, and it is stranger: consistent relationships between words tend to show up as consistent directions in the space, not just consistent neighbors. The most famous illustration, from the original word2vec research in 2013, is that subtracting the vector for “man” from “king,” then adding “woman,” lands close to the vector for “queen.” The “royal to commoner” step and the “male to female” step behave like directions you can apply, not just labels on individual points.
The space above is hand-built in two dimensions so it fits on screen, real embeddings live in hundreds of dimensions and are never this tidy. But the two properties it demonstrates are real: click through the words and notice each one's nearest neighbors come from its own conceptual cluster, animals near animals, fruits near fruits, royalty terms near each other, without anyone labeling the clusters by hand. Then switch to vector arithmetic and watch king - man + woman land on queen.
In practice: analogy arithmetic like this is a genuinely observed property of trained embedding spaces, not a hand-scripted demo trick, but it is also imperfect and inconsistent across word pairs and across different trained models. Treat it as evidence that embeddings capture real structure, not as something you can rely on for every analogy you try.
How "close" gets measured
Saying two vectors are “close” needs a precise definition, and the one almost everyone uses for embeddings is cosine similarity: the cosine of the angle between two vectors, ignoring their length entirely. Two vectors pointing in exactly the same direction score 1. Perpendicular vectors score 0. Opposite directions score -1.
Why ignore length? Because a word that appears far more often in training tends to develop a longer vector, purely as a side effect of how many times it got nudged during training, not because it is “more meaningful.” Cosine similarity cares only about direction, so it compares meaning without being thrown off by how frequent a word happened to be.
Cosine similarity, computed directly
1def cosine_similarity(a, b):2 dot_product = sum(x * y for x, y in zip(a, b))3 magnitude_a = sum(x ** 2 for x in a) ** 0.54 magnitude_b = sum(y ** 2 for y in b) ** 0.55 return dot_product / (magnitude_a * magnitude_b)67cosine_similarity(cat_vector, kitten_vector) # close to 1: similar8cosine_similarity(cat_vector, refrigerator_vector) # close to 0: unrelated
Look closely at that formula and note the numerator: a plain dot product between the two vectors. That single operation, multiply matching positions and sum the results, is the same core computation that shows up again, doing much more sophisticated work, when the Attention article covers how a model decides which earlier tokens are relevant to the one it is currently processing. Embeddings and attention are not separate mechanisms bolted together, attention is built directly on top of the geometry embeddings create.
Rule: cosine similarity measures direction, not magnitude. Two synonyms rarely have identical vector lengths, and that's expected, not a bug.
Learned, not designed
The toy space earlier in this article was hand-placed, so the clustering could be guaranteed for the demo. Real embedding tables are not designed by anyone. Every one of those numbers starts as a small random value, and training slowly adjusts them, millions of times, using the exact same next-token-prediction objective from the first article in this series. If moving “cat” slightly closer to “kitten” helps the model predict training text more accurately, that adjustment happens. If it doesn't help, it doesn't. The clustering you saw is an emergent side effect of optimizing prediction accuracy, nobody writes a rule that says animals belong near other animals.
Real embeddings are also far larger than the two dimensions used above. GPT-2's smallest version uses 768 numbers per token. GPT-3's largest version uses 12,288. More dimensions mean more independent directions available to encode independent relationships, gender, tense, sentiment, topic, formality, and thousands of others, all at once, in the same vector, without those relationships interfering with each other the way they would if forced into two dimensions.
In practice:the embedding table is usually the single largest chunk of parameters in a small language model. A 50,000-token vocabulary at 768 dimensions is 38.4 million numbers, just to represent “which token is this” before any actual reasoning happens.
None of this explains yet how a vector actually gets adjusted, what “training” does mechanically to a number inside it, or how a network built from these vectors makes a decision at all. That requires stepping back from language specifically and looking at the machinery underneath: neurons, weights, and the matrix multiplications that turn one vector into another. That's next.