Content
Summary
Self-attention lets every token in a sequence build a new representation as a weighted average of all tokens' value vectors. The weights come from comparing that token's query with every token's key, scaled and passed through a softmax. This gives content-based, data-dependent mixing of information across positions in a single parallel step.
Intuition
Think of each token as asking a question (query), advertising what it contains (key), and carrying a payload (value). A token compares its question with every advertisement using a dot product. High similarity means high attention weight. The token then collects a blend of payloads, weighted by those similarities. The queries, keys and values are all learned linear projections of the same input, so the network learns what to ask, what to advertise and what to send. Dividing by the square root of the key dimension keeps dot products from growing with dimension, which would push softmax into saturated regions with tiny gradients.
Key equations
Project the input matrix X (n tokens by d_model) into queries, keys and values.
Scaled similarity scores between every query and every key; S is n by n.
Each row becomes a probability distribution over positions to attend to.
Output for each token is the attention-weighted sum of value vectors.
Derivation
- Start with token embeddings X of shape n by d_model, one row per token.
- Compute Q, K, V with three learned matrices so each token has a query, key and value vector.
- The raw match between token i and token j is the dot product q_i · k_j; stacking all pairs gives QK^T.
- If components of and have zero mean and unit variance, has variance , so divide by to restore unit variance.
- Apply softmax across each row so weights a_ij are positive and sum to 1 over j.
- Output for token i is z_i = sum_j a_ij v_j, which in matrix form is AV.
- For causal models, add minus infinity to scores where j > i before softmax so future tokens get zero weight.
Worked example
Two tokens with d_k = 2. Queries: q1=(1,0), q2=(0,1). Keys: k1=(1,0), k2=(0,2). Values: v1=(1,2), v2=(3,4). Compute the output for token 1.
- Scores for token 1: q1·k1 = 1, q1·k2 = 0.
- Scale by : scores are and .
- Softmax: , , sum , so weights and .
- Output = 0.670·(1,2) + 0.330·(3,4) = (0.670+0.990, 1.340+1.320).
- This gives (1.660, 2.660).
Answer Output for token 1 ≈ (1.66, 2.66), with attention weights ≈ (0.67, 0.33).
Common mistakes
- Applying softmax over columns instead of over keys for each query (rows).
- Forgetting the scaling and getting saturated softmax.
- Thinking attention weights are fixed parameters; they are computed from the input each time.
- Confusing keys and values: keys determine the weights, values determine what is mixed.
- Forgetting that self-attention has no notion of order without positional information.
- Omitting the causal mask in autoregressive training, which leaks future tokens.