Content
Summary
Softmax turns a vector of raw scores (logits) into a probability distribution that is positive and sums to one. Cross-entropy then measures how much probability the model assigned to the true class, giving a loss that is small when the model is confidently correct and large when it is confidently wrong.
Intuition
A classifier outputs raw scores, one per class, which can be negative or huge and are not probabilities. Softmax exponentiates each score, which makes everything positive and exaggerates differences, then divides by the total so the numbers sum to one. Think of it as a smooth, differentiable 'winner takes most' vote. Cross-entropy asks: how surprised would we be by the true answer, given the model's probabilities? Surprise is the negative log of the probability given to the correct class. If the model gave it 0.9, the surprise is small; if it gave 0.01, the surprise is large. Training lowers average surprise. Combined, softmax plus cross-entropy has a very clean gradient: predicted probability minus true label.
Key equations
Softmax: converts logits z into probabilities p over K classes.
Cross-entropy loss for a one-hot true label y.
With one-hot labels this reduces to the negative log probability of the correct class c.
Gradient of the loss with respect to each logit.
Derivation
- Start with logits z_1..z_K. Exponentiate: e^{z_i} > 0 for every i.
- Divide by the sum S = sum_j e^{z_j} so that the p_i add to 1.
- For a one-hot label y with correct class c, L = -sum y_i log p_i keeps only the term i = c, so L = -log p_c.
- Write log p_c = z_c - log S.
- Then L = -z_c + log S.
- Differentiate with respect to z_i: the first term gives -y_i (1 if i=c else 0).
- The derivative of log S with respect to z_i is e^{z_i}/S = p_i.
- Therefore dL/dz_i = p_i - y_i.
Worked example
Logits for three classes are z = (2, 1, 0). The true class is class 1. Compute the softmax probabilities, the cross-entropy loss, and the gradient on the logits.
- Exponentiate: , , .
- Sum S ≈ 7.389 + 2.718 + 1 = 11.107.
- Probabilities: p1 ≈ 0.665, p2 ≈ 0.245, p3 ≈ 0.090 (sum ≈ 1.000).
- Loss L = -ln(0.665) ≈ 0.408.
- Gradient = p - y = (0.665 - 1, 0.245 - 0, 0.090 - 0) = (-0.335, 0.245, 0.090).
Answer p ≈ (0.665, 0.245, 0.090); L ≈ 0.41; gradient ≈ (-0.335, 0.245, 0.090), so training raises z1 and lowers z2 and z3.
Common mistakes
- Applying softmax and then feeding the result into a loss function that already applies softmax internally, such as PyTorch's CrossEntropyLoss, which expects raw logits.
- Computing softmax naively with large logits, causing overflow; subtract the maximum logit first, which does not change the result.
- Thinking softmax outputs are calibrated confidence; they are often overconfident.
- Using softmax for multi-label problems where classes are not mutually exclusive; use independent sigmoids instead.
- Forgetting that the log of zero is undefined; use the log-sum-exp form for stability.