AI, machine learning and deep learning · Advanced

Self-attention: queries, keys, values

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Self-attention lets every token in a sequence build a new representation as a weighted average of all tokens' value vectors. The weights come from comparing that token's query with every token's key, scaled and passed through a softmax. This gives content-based, data-dependent mixing of information across positions in a single parallel step.

Intuition

Think of each token as asking a question (query), advertising what it contains (key), and carrying a payload (value). A token compares its question with every advertisement using a dot product. High similarity means high attention weight. The token then collects a blend of payloads, weighted by those similarities. The queries, keys and values are all learned linear projections of the same input, so the network learns what to ask, what to advertise and what to send. Dividing by the square root of the key dimension keeps dot products from growing with dimension, which would push softmax into saturated regions with tiny gradients.

Key equations

Q=XWQ,K=XWK,V=XWV
(1)

Project the input matrix X (n tokens by d_model) into queries, keys and values.

S=QK⊤dk
(2)

Scaled similarity scores between every query and every key; S is n by n.

A=softmax(S)(row-wise)
(3)

Each row becomes a probability distribution over positions to attend to.

Attn(X)=AV
(4)

Output for each token is the attention-weighted sum of value vectors.

Derivation

  1. Start with token embeddings X of shape n by d_model, one row per token.
  2. Compute Q, K, V with three learned matrices so each token has a query, key and value vector.
  3. The raw match between token i and token j is the dot product q_i · k_j; stacking all pairs gives QK^T.
  4. If components of q and k have zero mean and unit variance, q⋅k has variance dk, so divide by dk to restore unit variance.
  5. Apply softmax across each row so weights a_ij are positive and sum to 1 over j.
  6. Output for token i is z_i = sum_j a_ij v_j, which in matrix form is AV.
  7. For causal models, add minus infinity to scores where j > i before softmax so future tokens get zero weight.

Worked example

Problem

Two tokens with d_k = 2. Queries: q1=(1,0), q2=(0,1). Keys: k1=(1,0), k2=(0,2). Values: v1=(1,2), v2=(3,4). Compute the output for token 1.

  1. Scores for token 1: q1·k1 = 1, q1·k2 = 0.
  2. Scale by 2≈1.414: scores are 0.707 and 0.
  3. Softmax: e0.707≈2.028, e0=1, sum 3.028, so weights ≈0.670 and 0.330.
  4. Output = 0.670·(1,2) + 0.330·(3,4) = (0.670+0.990, 1.340+1.320).
  5. This gives (1.660, 2.660).

Answer Output for token 1 ≈ (1.66, 2.66), with attention weights ≈ (0.67, 0.33).

Common mistakes

  • Applying softmax over columns instead of over keys for each query (rows).
  • Forgetting the 1/dk scaling and getting saturated softmax.
  • Thinking attention weights are fixed parameters; they are computed from the input each time.
  • Confusing keys and values: keys determine the weights, values determine what is mixed.
  • Forgetting that self-attention has no notion of order without positional information.
  • Omitting the causal mask in autoregressive training, which leaks future tokens.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(82 lines)
import sys
from manim import *
import numpy as np
from blocks import *
from blocks import _fit


def attention_trace(X, Wq, Wk, Wv):
    """Scaled dot-product self-attention; records each stage."""
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    d = K.shape[1]
    S = Q @ K.T / np.sqrt(d)
    E = np.exp(S - S.max(axis=1, keepdims=True))
    A = E / E.sum(axis=1, keepdims=True)
    O = A @ V
    return {"Q": Q, "K": K, "V": V, "S": S, "A": A, "O": O}


def make_data():
    rng = np.random.default_rng(3)
    X = rng.normal(size=(3, 4))
    Wq, Wk, Wv = [rng.normal(size=(4, 2)) for _ in range(3)]
    return X, Wq, Wk, Wv


class Main(Scene):
    def construct(self):
        self.camera.background_color = "#101418"
        X, Wq, Wk, Wv = make_data()
        tr = attention_trace(X, Wq, Wk, Wv)
        A = np.round(tr["A"], 2)
        S = np.round(tr["S"], 2)

        TitleCard(self, "Self-attention", "queries, keys, values")

        # Step 1: tokens and Q/K/V
        tokens = VGroup(*[Text(t, font_size=28, color=C_MAIN) for t in ["the", "cat", "sat"]])
        tokens.arrange(RIGHT, buff=1.5).shift(UP * 1.6)
        eq = VGroup(
            MathTex(r"Q=XW_Q", font_size=36),
            MathTex(r"K=XW_K", font_size=36),
            MathTex(r"V=XW_V", font_size=36),
        ).arrange(RIGHT, buff=1.0).next_to(tokens, DOWN, buff=0.6)
        _fit(eq, 11.0)
        self.play(FadeIn(tokens))
        self.play(Write(eq))
        StepCaption(self, "Each token is projected to a query, key and value")
        self.wait(1)
        self.play(FadeOut(tokens), FadeOut(eq))

        # Step 2: scores
        g1 = GridView([[str(x) for x in r] for r in S], cell_size=0.9).shift(UP * 0.3)
        lab1 = MathTex(r"QK^T/\sqrt{d_k}", font_size=34).next_to(g1, UP, buff=0.4)
        self.play(FadeIn(g1), Write(lab1))
        StepCaption(self, "Scores: how well each query matches each key")
        self.wait(1)

        # Step 3: softmax
        g2 = GridView([[str(x) for x in r] for r in A], cell_size=0.9).shift(UP * 0.3)
        lab2 = MathTex(r"A=\mathrm{softmax}(\cdot)", font_size=34).next_to(g2, UP, buff=0.4)
        self.play(FadeOut(g1), FadeOut(lab1))
        self.play(FadeIn(g2), Write(lab2))
        top = [(i, int(np.argmax(A[i]))) for i in range(3)]
        self.play(g2.mark(top, C_GOOD))
        StepCaption(self, "Softmax turns each row into weights summing to 1")
        self.wait(1)

        # Step 4: output
        out = MathTex(r"O=AV", font_size=44, color=C_ACCENT).next_to(g2, RIGHT, buff=0.8)
        self.play(Write(out))
        StepCaption(self, "Output = attention-weighted mix of the values")
        self.wait(1)


if __name__ == "__main__" and "--selftest" in sys.argv:
    X, Wq, Wk, Wv = make_data()
    tr = attention_trace(X, Wq, Wk, Wv)
    assert np.allclose(tr["A"].sum(axis=1), 1.0)
    assert (tr["A"] >= 0).all()
    assert tr["O"].shape == (3, 2)
    assert np.allclose(tr["O"], tr["A"] @ (X @ Wv))
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.