Why raw text can't be fed to a neural network
The previous article treated the model as a function from “tokens so far” to “probability of the next token.” That glossed over a real problem: a neural network is a stack of matrix multiplications. It only accepts numbers. The word “cat” is not a number, so before any of the math from the rest of this series can happen, every piece of text has to become a sequence of integers first. That conversion step is tokenization, and the choices made here quietly shape almost every weird thing a language model does later.
The obvious approaches, and why both fail
The first idea most people have is: assign every whole word a number. “cat” is 1, “dog” is 2, and so on. This is called word-level tokenization, and it breaks immediately for two reasons. First, English has hundreds of thousands of words, plus names, typos, slang, and made-up words a model will see for the first time at inference. Any word not in the fixed list has no number to become, that's the out-of-vocabulary problem. Second, “run”, “runs”, “running”, and “runner” would each need their own separate entry, with no way for the model to know they share a root.
The opposite idea is: assign every individual character a number. “c” is 1, “a” is 2, “t” is 3. This fixes the out-of-vocabulary problem completely, there are only a few dozen characters, so nothing is ever unseen. But it creates a new one: sequences get very long. A model has a fixed limit on how many tokens it can look at once, spending that entire budget one character at a time means far less actual content fits, and the model has to work much harder to learn that “c”-“a”-“t” means the same thing every time it appears, instead of just being handed “cat” as a single unit.
Rule: tokenization is a tradeoff between vocabulary size and sequence length. Fewer, larger tokens means shorter sequences but more out-of-vocabulary risk. More, smaller tokens means no out-of-vocabulary risk but much longer sequences. Every production tokenizer picks a point between the two extremes.
The answer every modern LLM converged on sits in between: break words into common subwordpieces. “running” might become “run” plus “ning”. Common whole words stay as one token, rare or unseen words fall back to smaller, still-meaningful chunks, and there's never a word the tokenizer simply cannot represent. The algorithm that builds this middle-ground vocabulary is called byte-pair encoding, covered next.
Byte-pair encoding: building a vocabulary from scratch
Byte-pair encoding, BPE, is the algorithm behind the tokenizer in GPT, Llama, and most other modern LLMs. The idea is almost embarrassingly simple. Start by treating every piece of text as individual characters. Then repeatedly find the pair of adjacent symbols that occurs most often across the training text, and merge that pair into a single new symbol. Repeat that, thousands of times. Whatever chunks survive as frequent, recurring pairs become entries in the vocabulary.
Nobody hand-writes rules like “-ing is a suffix.” The algorithm never sees grammar. It only ever counts how often two symbols sit next to each other and merges the winner. Run it on enough English text and “-ing”, “-tion”, and “un-” end up as vocabulary entries anyway, because they really do recur that often. The linguistic structure falls out of frequency counting, it isn't built in.
The training loop, stripped to its essence
1# Start: every word is a sequence of individual characters2vocab = set(all_characters_in_corpus)3sequences = [list(word) for word in corpus]45for _ in range(num_merges):6 pair_counts = count_adjacent_pairs(sequences)7 most_common_pair = max(pair_counts, key=pair_counts.get)8 sequences = merge_everywhere(sequences, most_common_pair)9 vocab.add("".join(most_common_pair))1011# After num_merges rounds, vocab contains characters,12# common subwords, and whole common words, all together
Two things matter about the result. First, the number of merge rounds is a hyperparameter chosen before training. GPT-style tokenizers typically run on the order of tens of thousands of merges, ending with a vocabulary around 50,000 to 100,000 entries. Second, every merge rule is applied in the exact order it was learned, highest-frequency merges first. That ordering is itself part of the tokenizer, applying the same merges in a different order produces a different split.
In practice: because the vocabulary is built by merging whatever is frequent in the training corpus, a tokenizer trained mostly on English text ends up with common English morphemes as single tokens, while less-represented languages and scripts get chopped into far more, smaller pieces. That difference in token efficiency is real and has real cost implications, more on that at the end of this article.
Watching the merges happen
Below is a real byte-pair encoding algorithm, running with a small, hand-built set of 32 merge rules instead of the tens of thousands a real tokenizer would learn from a full training corpus. Pick a word, step through the merges one at a time, and watch individual characters fuse into the subword chunks the algorithm has learned to recognize.
Notice that strawberry collapses cleanly into straw and berry, and tokenization collapses into token, iz, ation. Now try internationalization. It comes apart into a messier mix, several single characters mixed in with real chunks like er and ation. That is not a bug in the demo. This toy vocabulary only knows 32 merge rules. A real tokenizer, with tens of thousands of learned merges built from a training corpus containing that exact word many times over, would very likely have a dedicated chunk for it. The lesson generalizes: how cleanly a word splits depends entirely on how well-represented it was in whatever text the tokenizer's merges were learned from.
Rule: a word does not have one true tokenization. It has whatever tokenization falls out of applying one specific tokenizer's specific learned merge rules, in order. Change the tokenizer, change the split.
What tokenization quietly explains
A surprising amount of LLM behavior that gets attributed to "reasoning failures" is actually a tokenization artifact. Once you know the model operates on tokens, not characters or words, several famous quirks stop being mysterious.
The model never sees "s-t-r-a-w-b-e-r-r-y" as ten letters. It sees whatever tokens its tokenizer produced, maybe "straw" and "berry", maybe something else entirely depending on the tokenizer. Counting letters inside a token requires the model to have memorized that specific token's spelling from training data, it cannot just look. That is why letter-counting questions are unreliable in a way that feels bizarre until you know what the model is actually looking at.
Numbers get tokenized too, and not always digit by digit. Depending on the tokenizer, "1234" might be one token, or split as "12" and "34", or some other chunking entirely, and that chunking is not guaranteed to be consistent across every number. Arithmetic requires reliably tracking place value across digits. If the tokenizer hands the model an inconsistent chunking of the same-shaped number from one occurrence to the next, arithmetic gets harder for reasons that have nothing to do with the model's reasoning ability.
For common English words, they are close. For rare English words, code, or non-English text, they are not. A word that is a single token in a well-represented language can require several tokens in a less-represented one, because the tokenizer's merges were learned from a training corpus that contained far less of that language. Same sentence, same meaning, different token cost depending on what language it happens to be written in.
The token, not the word, is the unit of cost
Every model has a fixed context window, a maximum number of tokens it can process at once, and API pricing is charged per token, not per word or character. Both limits are defined in terms of whatever the tokenizer produces. This is why a paragraph of dense code or a paragraph in a less-represented language can eat noticeably more of a context window than a paragraph of plain English of similar length. The tokenizer decided that up front, before the model ever saw a single number.
In practice: when a model does something that looks like a reasoning failure on short, symbolic, or non-English input, tokenization is worth ruling out first. Ask what tokens the input actually became, not what words it started as.
Once text is a sequence of token IDs, the next question is what the model actually does with those IDs internally. An ID is just an index into a lookup table, it carries no meaning by itself. Turning that bare integer into something the network can reason over is the job of embeddings, next in this series.