Why Position Must Be Encoded

The claim in one sentence

Self-attention treats its input as a set of vectors, not a sequence. Shuffle the rows of the input matrix and the output rows shuffle identically — nothing else changes. So without extra help, a transformer literally cannot distinguish "dog bites man" from "man bites dog".

Let's prove it, then fix it.

1. Setting up the notation

You've seen this in Part 1, but pin it down. A batch of $n$ tokens, each embedded into $d$ dimensions, is a matrix

$X \in \mathbb{R}^{n \times d}, \quad \text{row } i = \text{embedding of token } i.$

Single-head attention is:

$Q = XW_Q,\quad K = XW_K,\quad V = XW_V$ $S = \frac{QK^\top}{\sqrt{d_k}},\qquad A = \mathrm{softmax}_{\text{row}}(S),\qquad \mathrm{Attn}(X) = AV$

Notice what's not in there: no index $i$, no counter, no "this token came third". The only thing determining $A_{ij}$ is the content of rows $i$ and $j$.

Permutation matrices

A permutation matrix $P$ is an identity matrix with its rows shuffled. Left-multiplying reorders rows: $PX$ is $X$ with its token rows permuted. Two facts we'll use:

• $P^\top P = I$ (permuting and un-permuting is a no-op)
• $P M P^\top$ applies the same permutation to the rows and columns of $M$

For "dog bites man" → "man bites dog" (swap positions 1 and 3):

$P = \begin{pmatrix} 0&0&1\\0&1&0\\1&0&0 \end{pmatrix}$

2. The proof (four lines)

Feed in $PX$ instead of $X$:

$Q' = PXW_Q = PQ, \qquad K' = PK, \qquad V' = PV$

The projections are applied per row, so they just come along for the ride. Now the scores:

$S' = \frac{(PQ)(PK)^\top}{\sqrt{d_k}} = \frac{PQK^\top P^\top}{\sqrt{d_k}} = P S P^\top$

Softmax is applied independently to each row, and within a row it's invariant to relabeling which column is which (the denominator is a sum over all entries of that row — order doesn't matter). So softmax commutes with the shuffle:

$A' = \mathrm{softmax}(PSP^\top) = P\,\mathrm{softmax}(S)\,P^\top = PAP^\top$

Finally:

$\mathrm{Attn}(PX) = A'V' = PAP^\top PV = PA V = P\,\mathrm{Attn}(X)$

$\boxed{\ \mathrm{Attn}(PX) = P\cdot\mathrm{Attn}(X)\ }$

Read that carefully. It doesn't say the output is unchanged — it says the output is the same rows, in the shuffled order. That property is called permutation equivariance. The practical consequence is the damning part: the output vector for the token "dog" is bit-for-bit identical whether "dog" appeared first or last.

And the rest of the transformer doesn't save you. LayerNorm and the feed-forward MLP act on each token row independently, so they're equivariant too. Residual connections add equivariant things to equivariant things. Stack 96 layers: still equivariant. If you then mean-pool for a classification head, pooling is order-insensitive, so the whole model becomes fully permutation-invariant — a very expensive bag-of-words.

3. Watching it happen with real numbers

Three tokens in $d = 2$, with $W_Q = W_K = W_V = I$ for simplicity (so $\sqrt{d_k} = \sqrt{2} \approx 1.414$):

$\text{dog} = [1, 0],\quad \text{bites} = [0,1],\quad \text{man} = [1,1]$

Sentence A: "dog bites man" — compute the output row for dog.

Scores (dot products with dog, bites, man): $1,\ 0,\ 1$, divided by $\sqrt2$: $0.707,\ 0,\ 0.707$.

Softmax: $\frac{(2.028,\ 1.000,\ 2.028)}{5.056} = (0.401,\ 0.198,\ 0.401)$

Output: $0.401[1,0] + 0.198[0,1] + 0.401[1,1] = [0.802,\ 0.599]$

Sentence B: "man bites dog" — dog is now row 3.

Scores against (man, bites, dog): $\text{dog}\cdot\text{man}=1$, $\text{dog}\cdot\text{bites}=0$, $\text{dog}\cdot\text{dog}=1$ → the same multiset $\{1, 0, 1\}$, just reordered, and each weight stays glued to the same token.

Output: $0.401[1,1] + 0.198[0,1] + 0.401[1,0] = [0.802,\ 0.599]$

Identical. The model has no idea whether the dog was the biter or the bitten.

import numpy as np

def attn(X):
    S = X @ X.T / np.sqrt(X.shape[1])
    A = np.exp(S) / np.exp(S).sum(axis=1, keepdims=True)
    return A @ X

dog, bites, man = np.array([1,0]), np.array([0,1]), np.array([1,1])
A = attn(np.stack([dog, bites, man]))   # dog is row 0
B = attn(np.stack([man, bites, dog]))   # dog is row 2
print(A[0], B[2])   # [0.802 0.599] [0.802 0.599]

4. The fix: make the input rows position-dependent

The proof breaks the moment the input row for a token depends on where it sits. So we build a matrix $PE \in \mathbb{R}^{n \times d}$ where row $t$ encodes "position $t$", and feed the model

$X' = X + PE$

Now "dog at position 0" and "dog at position 2" are genuinely different vectors, and everything downstream can tell them apart. Two standard ways to build $PE$.

Scheme A: Sinusoidal (fixed, from the original paper)

For position $t$ and dimension pair $i$:

$PE_{t,\,2i} = \sin\!\left(\frac{t}{10000^{2i/d}}\right), \qquad PE_{t,\,2i+1} = \cos\!\left(\frac{t}{10000^{2i/d}}\right)$

Think of it as $d/2$ clock hands rotating at geometrically-spaced frequencies. The fastest hand (small $i$) flips every couple of tokens; the slowest (large $i$) barely moves across thousands of tokens. Together they give every position a unique fingerprint — it's essentially a smooth, continuous binary counter.

def sinusoidal(n, d, base=10000.0):
    t = np.arange(n)[:, None]                          # (n, 1)
    i = np.arange(d // 2)[None, :]                      # (1, d/2)
    freq = t / base ** (2 * i / d)                      # (n, d/2)
    pe = np.zeros((n, d))
    pe[:, 0::2], pe[:, 1::2] = np.sin(freq), np.cos(freq)
    return pe

The elegant property: because $(\sin\omega t, \cos\omega t)$ is a point on a circle, shifting by $k$ steps is a fixed linear map, independent of $t$:

$\begin{pmatrix}\sin\omega(t+k)\\ \cos\omega(t+k)\end{pmatrix} = \begin{pmatrix}\cos\omega k & \sin\omega k\\ -\sin\omega k & \cos\omega k\end{pmatrix}\begin{pmatrix}\sin\omega t\\ \cos\omega t\end{pmatrix}$

So a linear layer can learn to compute "is this token 5 to my left?" without caring about absolute position. (This observation is exactly the seed of RoPE, the rotary encoding used by Llama and friends — instead of adding rotations to the embeddings, it applies them directly to $Q$ and $K$.) A second perk: it's a formula, not a parameter table, so you can evaluate it at positions never seen during training.

Scheme B: Learned

pos_emb = nn.Embedding(max_seq_len, d_model)   # just a lookup table
x = tok_emb(ids) + pos_emb(torch.arange(len(ids)))

One trainable $d$-vector per position, learned by gradient descent alongside everything else. This is what GPT-2 and BERT do. Simpler, works about as well in-distribution, but the table has a hard size limit: at position max_seq_len you fall off a cliff, because there's no row there and no rule for inventing one.

Redo the example with sinusoidal encoding

With $d=2$ there's one frequency pair, $10000^0 = 1$, so $PE_t = [\sin t, \cos t]$:

$PE_0 = [0,\ 1], \quad PE_1 = [0.841,\ 0.540], \quad PE_2 = [0.909,\ -0.416]$

Sentence A ("dog bites man"), after adding position:

$\text{dog@0} = [1,1],\quad \text{bites@1} = [0.841,\ 1.540],\quad \text{man@2} = [1.909,\ 0.584]$

Dog's scores: $2/\sqrt2 = 1.414$, $2.381/\sqrt2 = 1.684$, $2.493/\sqrt2 = 1.763$ → weights $(0.268,\ 0.351,\ 0.380)$ → output $\approx [1.290,\ 1.032]$.

Sentence B ("man bites dog"):

$\text{man@0} = [1,2],\quad \text{bites@1} = [0.841,\ 1.540],\quad \text{dog@2} = [1.909,\ -0.416]$

Dog's scores: $0.762,\ 0.683,\ 2.699$ → weights $(0.113,\ 0.104,\ 0.783)$ → output $\approx [1.695,\ 0.061]$.

Different vectors, and different attention patterns. In sentence B, dog attends overwhelmingly to itself; in sentence A it spreads attention across all three. Word order now changes the computation.

5. Two objections your colleague will raise

"Doesn't adding position clobber the word meaning?" It feels wrong, but in $d = 768$ or $4096$ dimensions there's plenty of room. Random high-dimensional directions are nearly orthogonal, and the $W_Q, W_K, W_V$ projections are free to read the "content subspace" and "position subspace" through different rows. Training sorts out the allocation. Concatenating instead would work too, but costs dimensions you'd rather spend on meaning.

"Doesn't the causal mask already tell the model about order?" Partially, yes — good catch, and worth the nuance. The causal mask zeroes out $A_{ij}$ for $j > i$, and that mask depends on the index, so it is not permutation-equivariant. A decoder-only model can infer position from "how many tokens am I allowed to see?" (this is the basis of "NoPE" research results). But that signal is weak and indirect, and it doesn't exist at all in encoders like BERT, which attend bidirectionally and are therefore exactly as order-blind as the proof above says. Every production model ships explicit position information.

Try it yourself

Take the 10-line attn function above and:

1. Extend it to a 4-token sentence and verify the equivariance identity numerically: build a permutation matrix P, then check np.allclose(attn(P @ X), P @ attn(X)). Convince yourself the residual is machine epsilon, not luck.
2. Add sinusoidal(n, d) to X, rerun, and confirm the identity now fails — print the magnitude of the difference.
3. Set d = 8 and plot the 8 columns of sinusoidal(64, 8) against position. Identify which columns are the "fast hands" and which are the "slow hands."
4. Then write down, in two sentences and no equations, how you'd explain the "attention sees a set, not a sequence" idea to your colleague. If you can land that plus one concrete example like dog/bites/man, you own this lesson.