Positional Encoding

Why attention alone can't tell word order

“The dog bites the man” and “the man bites the dog” use the exact same five tokens. As a bag of words they are identical. As sentences they describe two different events, and any reader knows immediately which one is the ordinary Tuesday and which one is the headline. A language model needs to know the difference too, and it is reasonable to assume the attention mechanism from the previous two articles already handles this. It doesn't.

Recall the attention computation: softmax(QK^T / sqrt(d_k)) weighting a sum of value vectors. That formula runs on whichever embeddings sit in the query, key, and value matrices, it never looks at where in the array a row came from. Feed it the embeddings for “dog,” “bites,” and “man” in that order, or feed it the same three embeddings in the order “man,” “bites,” “dog,” and the dot products between any two specific embeddings come out identical either way. Swapping two tokens just permutes which row of the value matrix gets summed with which attention weight. The arithmetic itself doesn't change, only the labels attached to the rows do.

This isn't a theoretical nitpick, it's the difference between a model that understands subject and object and one that treats a sentence as a shuffled deck of words. Without some signal telling the model “dog” came before “bites” and “bites” came before “man,” there is nothing in the attention computation itself that could recover which noun did the biting. Set aside self-attention for a moment and the problem gets even starker: a feedforward layer processes one position at a time and has no visibility into other positions at all, so if position isn't encoded into the token representation itself, it is gone by the time any layer sees it.

The fix has to happen before attention ever runs. Since the transformer block covered in the previous article never mentions position anywhere inside it, attention, feedforward, residuals, normalization, the injection point has to be earlier: right at the input embedding, once, before the first block even starts. That injected signal is a positional encoding, and the rest of this article covers the two ways production models build it.

Sinusoidal positional encoding: a unique fingerprint per position

The original transformer paper's fix is almost embarrassingly simple: build a fixed vector for each position and add itto that token's embedding before anything else happens. Position 0 gets one vector, position 1 gets a different vector, position 2 a different one again, all the way out. Add position's vector into the embedding and every downstream computation, attention included, now has access to “where,” because it's baked directly into the numbers it's already operating on.

The question is how to build a vector per position that's actually useful. A naive option, just use the integer position value itself, breaks immediately: raw position numbers grow unbounded and swamp the embedding at long sequence lengths. The paper's answer is sine and cosine waves at a different frequency for every pair of dimensions in the embedding. Think of each dimension pair as a clock hand: the first pair spins once almost every position, a fast hand. The last pair barely moves at all across thousands of positions, a slow hand. Read off every hand's position at once and you get a reading that never repeats for a very long time, exactly like how the hour, minute, and second hand together tell you the exact time, not just the second.

1import math
2
3def positional_encoding(pos, d_model):
4    pe = [0.0] * d_model
5    for i in range(0, d_model, 2):
6        freq = 1 / (10000 ** (i / d_model))
7        pe[i] = math.sin(pos * freq)
8        if i + 1 < d_model:
9            pe[i + 1] = math.cos(pos * freq)
10    return pe
11
12positional_encoding(0, 8)
13positional_encoding(1, 8)
14positional_encoding(2, 8)

Look at those three outputs. Every one is a different pattern, and the early dimensions (fast hands, high frequency) swing wildly between positions 0, 1, and 2, while the late dimensions (slow hands, low frequency) barely budge. That's deliberate. It also means the encoding needs no training at all, it's pure math, computed once and reused for every sequence the model ever sees.

Slide through the positions below and watch both views update: the wave across one position's dimensions on top, and every position from 0 to 20 stacked as a grid on the bottom. No two rows in that grid ever match.

This scheme shipped in the original transformer and worked well enough to prove the architecture. But “in principle” is doing real work in that rule above, a model has to learn to exploit that relative structure, it isn't handed a relative distance directly. The next section covers a newer approach that builds relative position into the attention arithmetic itself.

Rotary position embeddings (RoPE): encoding position as rotation

Most current open-weight models, Llama, Mistral, and plenty of others, don't add a positional vector at all. They use rotary position embeddings, RoPE for short, which throws out the “add a fixed vector” idea entirely and instead rotates the query and key vectors by an angle proportional to their position, right before the dot product in attention runs.

Picture each pair of dimensions in a query vector as a point on a 2D plane. RoPE spins that point around the origin by an angle equal to position × θ, for some fixed frequency θ, the same frequency schedule idea as the sine and cosine waves above, just applied as a rotation instead of an addition. A token at position 0 doesn't rotate at all. A token at position 10 rotates ten steps around. Different dimension pairs rotate at different speeds, fast pairs and slow pairs, exactly like the clock hands from the previous section.

1import math
2
3def rotate(v, pos, theta):
4    angle = pos * theta
5    c, s = math.cos(angle), math.sin(angle)
6    return (v[0] * c - v[1] * s, v[0] * s + v[1] * c)
7
8def dot(a, b):
9    return a[0] * b[0] + a[1] * b[1]
10
11q, k, theta = (1.0, 0.0), (0.0, 1.0), 0.5
12
13# same relative distance, three different absolute positions
14print(dot(rotate(q, 0, theta),   rotate(k, 3, theta)))
15print(dot(rotate(q, 5, theta),   rotate(k, 8, theta)))
16print(dot(rotate(q, 105, theta), rotate(k, 108, theta)))

That's the entire point of RoPE, made concrete. Query at position 0 against key at position 3, query at position 5 against key at position 8, query at position 105 against key at position 108: three completely different absolute positions, but every pair sits exactly 3 apart, and every one produces the identical dot product. Rotating both vectors by their own position and then taking the dot product cancels out the absolute positions algebraically and leaves only the distance between them. Sinusoidal encoding hands the model relative structure it has to learn to use. RoPE makes relative distance the only thing the attention score can see in the first place.

This relative-distance property is also why RoPE tolerates longer sequences better than sinusoidal or learned encodings, a claim the next section puts to the test.

What happens past the trained context length

Train a model on sequences up to 4,096 tokens and something predictable happens at token 4,097: quality falls off, sometimes gently, sometimes into outright incoherence. This isn't a bug, it's the direct consequence of how the positional encodings above get learned and used.

Sinusoidal encoding is computed by a fixed formula, so technically nothing stops you from plugging in position 5,000 even if training never went past 4,096. But the weight matrices that turn those encoded vectors into useful attention behavior were only ever adjusted against encoding patterns the model actually saw during training. Position 5,000's specific sine and cosine values are, mathematically, valid outputs the formula can produce, but they're patterns the attention weights never learned to interpret. It's the same failure mode as asking a model to classify a type of image it never saw in training, the function still runs, the output is still a number, it's just not a meaningful one. Learned positional embeddings, an alternative some models use instead of the sinusoidal formula, fail even more bluntly: there is no vector at all for a position past the trained range, nothing to look up.

RoPE degrades too, but more gracefully, and the reason traces straight back to the relative distance property from the previous section. Because attention scores depend on the distance between two positions rather than their absolute values, a RoPE model that trained on distances up to 4,096 has at least seen most of the relative distances it will encounter even in a longer sequence, a pair of tokens 50 apart look the same to it whether they sit at positions 10 and 60 or positions 4,000 and 4,050. But the largest distances, and the sheer volume of tokens competing for attention at once, still push the model outside what it trained on, and quality still slips.

Production teams do stretch models past their trained length, interpolating the rotation angles so a longer sequence maps back into the range of distances the model already knows, or rescaling frequencies so the fastest and slowest “clock hands” still land in familiar territory. Both are patches on the positional scheme, not a different mechanism. The full story of context length, what actually gets stored and recomputed token by token, and why it's expensive, is covered in full later in this series, in the Context Windows and the KV Cache article.