AI, machine learning and deep learning · Introductory

Softmax and cross-entropy

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Softmax turns a vector of raw scores (logits) into a probability distribution that is positive and sums to one. Cross-entropy then measures how much probability the model assigned to the true class, giving a loss that is small when the model is confidently correct and large when it is confidently wrong.

Intuition

A classifier outputs raw scores, one per class, which can be negative or huge and are not probabilities. Softmax exponentiates each score, which makes everything positive and exaggerates differences, then divides by the total so the numbers sum to one. Think of it as a smooth, differentiable 'winner takes most' vote. Cross-entropy asks: how surprised would we be by the true answer, given the model's probabilities? Surprise is the negative log of the probability given to the correct class. If the model gave it 0.9, the surprise is small; if it gave 0.01, the surprise is large. Training lowers average surprise. Combined, softmax plus cross-entropy has a very clean gradient: predicted probability minus true label.

Key equations

pi=ezi∑j=1Kezj
(1)

Softmax: converts logits z into probabilities p over K classes.

L=−∑i=1Kyilog⁡pi
(2)

Cross-entropy loss for a one-hot true label y.

L=−log⁡pc
(3)

With one-hot labels this reduces to the negative log probability of the correct class c.

∂L∂zi=pi−yi
(4)

Gradient of the loss with respect to each logit.

Derivation

  1. Start with logits z_1..z_K. Exponentiate: e^{z_i} > 0 for every i.
  2. Divide by the sum S = sum_j e^{z_j} so that the p_i add to 1.
  3. For a one-hot label y with correct class c, L = -sum y_i log p_i keeps only the term i = c, so L = -log p_c.
  4. Write log p_c = z_c - log S.
  5. Then L = -z_c + log S.
  6. Differentiate with respect to z_i: the first term gives -y_i (1 if i=c else 0).
  7. The derivative of log S with respect to z_i is e^{z_i}/S = p_i.
  8. Therefore dL/dz_i = p_i - y_i.

Worked example

Problem

Logits for three classes are z = (2, 1, 0). The true class is class 1. Compute the softmax probabilities, the cross-entropy loss, and the gradient on the logits.

  1. Exponentiate: e2≈7.389, e1≈2.718, e0=1.
  2. Sum S ≈ 7.389 + 2.718 + 1 = 11.107.
  3. Probabilities: p1 ≈ 0.665, p2 ≈ 0.245, p3 ≈ 0.090 (sum ≈ 1.000).
  4. Loss L = -ln(0.665) ≈ 0.408.
  5. Gradient = p - y = (0.665 - 1, 0.245 - 0, 0.090 - 0) = (-0.335, 0.245, 0.090).

Answer p ≈ (0.665, 0.245, 0.090); L ≈ 0.41; gradient ≈ (-0.335, 0.245, 0.090), so training raises z1 and lowers z2 and z3.

Common mistakes

  • Applying softmax and then feeding the result into a loss function that already applies softmax internally, such as PyTorch's CrossEntropyLoss, which expects raw logits.
  • Computing softmax naively with large logits, causing overflow; subtract the maximum logit first, which does not change the result.
  • Thinking softmax outputs are calibrated confidence; they are often overconfident.
  • Using softmax for multi-label problems where classes are not mutually exclusive; use independent sigmoids instead.
  • Forgetting that the log of zero is undefined; use the log-sum-exp form for stability.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(70 lines)
import sys
import numpy as np
from manim import *
from blocks import *


def softmax(z):
    """Numerically stable softmax."""
    z = np.asarray(z, dtype=float)
    e = np.exp(z - z.max())
    return e / e.sum()


def cross_entropy(p, k):
    """Cross-entropy for true class index k."""
    return float(-np.log(p[k]))


LOGITS = [2.0, 1.0, 0.1]
PROBS = softmax(LOGITS)
TRUE = 0
LOSS = cross_entropy(PROBS, TRUE)


class Main(Scene):
    def construct(self):
        TitleCard(self, "Softmax and cross-entropy")

        # logits as bars
        logits_bars = ArrayBars(LOGITS, max_width=4.0, max_height=2.5)
        logits_bars.to_edge(LEFT, buff=1.0).shift(UP * 0.2)
        names = VGroup(*[Text(n, font_size=24, color=C_DIM).next_to(b, DOWN, buff=0.15)
                         for n, b in zip(["cat", "dog", "bird"], logits_bars.bars)])
        head1 = Text("Logits z", font_size=26, color=C_MAIN).next_to(logits_bars, UP, buff=0.4)
        self.play(FadeIn(logits_bars), FadeIn(names), FadeIn(head1))
        StepCaption(self, "Raw scores (logits) from a network")

        # softmax probabilities
        probs_r = [round(float(p), 2) for p in PROBS]
        prob_bars = ArrayBars(probs_r, max_width=4.0, max_height=2.5)
        prob_bars.to_edge(RIGHT, buff=1.0).shift(UP * 0.2)
        names2 = VGroup(*[Text(n, font_size=24, color=C_DIM).next_to(b, DOWN, buff=0.15)
                          for n, b in zip(["cat", "dog", "bird"], prob_bars.bars)])
        head2 = Text("Probabilities p", font_size=26, color=C_GOOD).next_to(prob_bars, UP, buff=0.4)
        eq = MathTex(r"p_i=\frac{e^{z_i}}{\sum_j e^{z_j}}", font_size=34)
        eq.move_to([0, 1.2, 0])
        arrow = Arrow(LEFT * 0.9 + UP * 0.2, RIGHT * 0.9 + UP * 0.2, color=C_ACCENT)
        eq.next_to(arrow, UP, buff=0.1)
        self.play(Write(eq), Create(arrow))
        self.play(FadeIn(prob_bars), FadeIn(names2), FadeIn(head2))
        StepCaption(self, "Softmax: positive values that sum to 1")

        # cross-entropy for true class
        self.play(prob_bars.mark([TRUE], C_GOOD))
        ce = MathTex(r"L=-\log p_{\mathrm{true}}=%.3f" % LOSS, font_size=36, color=C_ACCENT)
        ce.move_to([0, -1.9, 0])
        self.play(Write(ce))
        self.wait(1)
        StepCaption(self, "Loss is small when the true class gets high probability")
        self.wait(1)


if __name__ == "__main__" and "--selftest" in sys.argv:
    assert abs(PROBS.sum() - 1) < 1e-12
    assert all(PROBS > 0)
    assert PROBS.argmax() == 0
    assert abs(softmax([1, 1, 1])[0] - 1 / 3) < 1e-12
    assert abs(LOSS + np.log(PROBS[0])) < 1e-12
    assert cross_entropy(np.array([1.0, 0.0]), 0) == 0
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.