Dot Products as Similarity Scores

Attention, at its core, answers one question over and over: when processing this word, how much should I pay attention to that other word? The answer is a number. That number comes from a dot product. Everything else in the attention formula — the softmax, the value vectors, the scaling factor — is plumbing around this one operation.

So it's worth getting the dot product into your fingers, not just your notes.

1. The mechanic

Given two vectors of the same length, multiply them element-wise and add up the results.

$\mathbf{a} \cdot \mathbf{b} = \sum_{i} a_i b_i$

For $\mathbf{a} = [3, 4]$ and $\mathbf{b} = [4, 3]$:

3 × 4 = 12
4 × 3 = 12
        ----
        24

That's it. In Python, without NumPy:

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

dot([3, 4], [4, 3])  # 24

Two vectors in → one scalar out. Note that the output is a single number no matter how long the vectors are. A 768-dimensional dot product still collapses to one score. That's exactly what you want for "how relevant is this token to that one."

2. Why the number means "similar direction"

The definition above is the computational one. There's an equivalent geometric one:

$\mathbf{a} \cdot \mathbf{b} = \|\mathbf{a}\| \, \|\mathbf{b}\| \cos\theta$

where $\|\mathbf{a}\|$ is the length of $\mathbf{a}$ (i.e. $\sqrt{\sum a_i^2}$, Pythagoras generalized) and $\theta$ is the angle between the two vectors.

You don't need to derive this, but you do need to trust it, so here's the intuition. Imagine shining a light straight down onto $\mathbf{b}$ and seeing where $\mathbf{a}$'s shadow falls. That shadow — the projection of $\mathbf{a}$ onto $\mathbf{b}$ — has length $\|\mathbf{a}\|\cos\theta$. The dot product is that shadow length, scaled by how long $\mathbf{b}$ is.

• If $\mathbf{a}$ and $\mathbf{b}$ point the same way, $\theta = 0°$, $\cos\theta = 1$, the shadow is as long as $\mathbf{a}$ itself. Dot product is maximal.
• If they're perpendicular, $\theta = 90°$, $\cos\theta = 0$, $\mathbf{a}$ casts no shadow at all. Dot product is zero.
• If they point opposite ways, $\theta = 180°$, $\cos\theta = -1$. Dot product is maximally negative.

Check that against $\mathbf{a} = [3,4]$, $\mathbf{c} = [-4, 3]$:

3 × (-4) = -12
4 ×   3  =  12
           ----
             0

Zero — and indeed those two vectors are at right angles (rotate $[3,4]$ by 90° and you get $[-4,3]$).

3. Where normalization comes in

The geometric formula has a nuisance in it: $\|\mathbf{a}\|\,\|\mathbf{b}\|$. A long vector produces big dot products with everything, regardless of direction. If you're ranking candidates by similarity, a merely-adjacent-but-very-long vector can outscore a perfectly-aligned-but-short one.

Normalizing removes that confound. To normalize, divide a vector by its own length:

$\hat{\mathbf{a}} = \frac{\mathbf{a}}{\|\mathbf{a}\|}$

$\mathbf{a} = [3,4]$ has length $\sqrt{9+16} = 5$, so $\hat{\mathbf{a}} = [0.6, 0.8]$. Check: $0.6^2 + 0.8^2 = 0.36 + 0.64 = 1$. ✓

Now, if both vectors have length 1, the geometric formula collapses:

$\hat{\mathbf{a}} \cdot \hat{\mathbf{b}} = 1 \cdot 1 \cdot \cos\theta = \cos\theta$

The dot product of two normalized vectors is the cosine of the angle between them. It lives in $[-1, 1]$, and it depends on nothing but direction. That's the sentence to say to your colleague:

A bigger dot product between unit vectors means a smaller angle between them — the length is factored out, so the only thing left that the number can be reporting is alignment.

This quantity has a name you'll see everywhere: cosine similarity.

| $\hat{\mathbf{a}} \cdot \hat{\mathbf{b}}$ | Angle | Reading | |---|---|---| | 1.0 | 0° | identical direction | | ~0.9 | ~25° | strongly related | | 0.0 | 90° | unrelated / orthogonal | | −1.0 | 180° | opposite |

4. Worked example: a toy attention score

Forget learned embeddings for a moment. Suppose we hand-build a 4-dimensional space where the axes mean:

[ animal-ness, machine-ness, edible-ness, emotion-ness ]

Here are five word vectors, all chosen with length exactly 5 to keep the arithmetic clean:

| word | vector | length | |---|---|---| | puppy (our query) | $[4, 0, 0, 3]$ | $\sqrt{16+9}=5$ | | kitten | $[3, 0, 0, 4]$ | $\sqrt{9+16}=5$ | | engine | $[0, 5, 0, 0]$ | $5$ | | steak | $[3, 0, 4, 0]$ | $\sqrt{9+16}=5$ | | algorithm | $[0, 3, 0, -4]$ | $\sqrt{9+16}=5$ |

Compute puppy · each key by hand:

puppy · kitten

4×3 = 12
0×0 =  0
0×0 =  0
3×4 = 12
      ---
       24

puppy · engine

4×0 = 0
0×5 = 0
0×0 = 0
3×0 = 0
      ---
        0

puppy · steak

4×3 = 12
0×0 =  0
0×4 =  0
3×0 =  0
      ---
       12

puppy · algorithm

4×0  =   0
0×3  =   0
0×0  =   0
3×(-4) = -12
        ----
         -12

Since every vector here has length 5, the cosine similarity is just the dot product divided by $5 \times 5 = 25$:

| key | dot | cosine | angle | |---|---|---|---| | kitten | 24 | 0.96 | ≈ 16° | | steak | 12 | 0.48 | ≈ 61° | | engine | 0 | 0.00 | 90° | | algorithm | −12 | −0.48 | ≈ 119° |

The ranking matches intuition: kitten shares both animal-ness and emotion-ness with puppy, so it's nearly aligned. Steak shares animal-ness only — related, but at a wide angle. Engine shares nothing; it lives on an axis puppy has no component along, so it's exactly orthogonal. Algorithm is anti-aligned on the emotion axis, which drags the score below zero.

This is, structurally, one row of an attention score matrix. A real transformer computes puppy's query vector and every other token's key vector, dot-products the query against all the keys, and gets exactly this kind of ranked list of numbers — which a softmax then turns into weights that sum to 1.

Verify the arithmetic:

import numpy as np

puppy = np.array([4, 0, 0, 3])
keys = {
    "kitten":    np.array([3, 0, 0, 4]),
    "engine":    np.array([0, 5, 0, 0]),
    "steak":     np.array([3, 0, 4, 0]),
    "algorithm": np.array([0, 3, 0, -4]),
}

for name, k in keys.items():
    raw = puppy @ k
    cos = raw / (np.linalg.norm(puppy) * np.linalg.norm(k))
    print(f"{name:10s} dot={raw:4d}  cos={cos:+.2f}  angle={np.degrees(np.arccos(cos)):.0f}°")

5. One honest caveat to keep in your pocket

Transformers do not normalize queries and keys before taking the dot product. They compute raw $\mathbf{q} \cdot \mathbf{k}$, then divide by $\sqrt{d_k}$ (the square root of the vector dimension). So the scores aren't strictly cosines — magnitude does contribute.

That's fine, and we'll unpack the $\sqrt{d_k}$ later in the week. The reason to learn the normalized case first is that it isolates the why: it proves the dot product is fundamentally an alignment measure, with magnitude riding along as a separate, optional signal. If a colleague asks "why does multiplying and summing tell you anything about meaning?", the unit-vector story is the answer.

Try it yourself

Using the same 4-axis space ([animal, machine, edible, emotion]), take the query vector

$\mathbf{q} = \text{robot} = [0, 6, 0, 8]$

and these three keys:

• $\mathbf{k}_1 = \text{drone} = [0, 5, 0, 0]$
• $\mathbf{k}_2 = \text{puppy} = [4, 0, 0, 3]$
• $\mathbf{k}_3 = \text{toaster} = [0, 4, 0, -3]$

1. Compute all three dot products by hand, showing the element-wise products.
2. Compute $\|\mathbf{q}\|$ and each $\|\mathbf{k}\|$, then convert each dot product to a cosine similarity.
3. Notice that drone gets a lower raw dot product than one of the others but a competitive cosine — or vice versa. Which key benefits most from not normalizing, and why?
4. Write two sentences you'd actually say out loud to a colleague explaining why a large dot product between unit vectors means "similar direction." Don't use the word "cosine" in the first sentence.

Then check yourself with NumPy.