AI, machine learning and deep learning · Intermediate

Backpropagation step by step

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Backpropagation computes the gradient of a loss with respect to every weight in a neural network by applying the chain rule backwards from the output. It reuses intermediate results from the forward pass, making gradient computation about as cheap as one extra forward pass. These gradients drive gradient descent updates.

Intuition

Think of the network as a chain of simple functions. The forward pass pushes an input through the chain to produce a prediction and a loss. To learn, we ask: if I nudge this weight slightly, how much does the loss change? The answer is a product of local sensitivities along the path from that weight to the loss. Backpropagation computes these products efficiently by starting at the loss and moving backwards, carrying an error signal called delta. At each layer, delta is multiplied by the local derivative and passed to the previous layer, so shared work is never repeated. Each weight's gradient is then its incoming activation times the delta at its output neuron.

Key equations

z(l)=W(l)a(l−1)+b(l),a(l)=σ(z(l))
(1)

Forward pass for layer l: linear step then nonlinearity.

δ(L)=∂L∂a(L)⊙σ′(z(L))
(2)

Error signal at the output layer.

δ(l)=((W(l+1))Tδ(l+1))⊙σ′(z(l))
(3)

Error signal propagated backwards one layer.

∂L∂W(l)=δ(l)(a(l−1))T,∂L∂b(l)=δ(l)
(4)

Gradients for weights and biases.

W←W−η∂L∂W
(5)

Gradient descent update with learning rate eta.

Derivation

  1. Define δ at layer l as the derivative of the loss with respect to z at layer l.
  2. At the output, the chain rule gives δ=dLdaσ′(z).
  3. Since z at layer l+1 depends on a at layer l through W, dLda at layer l equals W transpose of layer l+1 times δ of layer l+1.
  4. Multiply by σ′(z) at layer l to get δ at layer l.
  5. Because z at layer l equals Wa(prev) plus b, dzdW contributes the previous activation, so dLdW equals δ times a(prev) transpose.
  6. Because dz/db equals 1, dL/db equals delta.
  7. Repeat from the last layer to the first, storing activations from the forward pass.

Worked example

Problem

One neuron: x=2, w=0.5, b=0, sigmoid activation, target y=1, loss L=0.5(a-y)^2. Compute dL/dw.

  1. Forward: z = 0.5*2 + 0 = 1.
  2. a = sigmoid(1) ≈ 0.7311.
  3. L = 0.5*(0.7311-1)^2 ≈ 0.0362.
  4. dL/da = a - y = -0.2689.
  5. σ′(z)=a(1−a)=0.7311×0.2689≈0.1966.
  6. δ=−0.2689×0.1966≈−0.0529.
  7. dLdw=δ×x=−0.0529×2≈−0.1057.

Answer dL/dw ≈ -0.106, so gradient descent increases w.

Common mistakes

  • Forgetting to store forward activations and z values needed in the backward pass.
  • Mixing up the transpose of W when propagating delta backwards.
  • Using the elementwise product incorrectly, or omitting the derivative of the activation.
  • Updating weights before all gradients are computed, corrupting later layers' gradients.
  • Ignoring vanishing gradients from saturated sigmoids in deep networks.
  • Confusing backpropagation (gradient computation) with gradient descent (the update).

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(71 lines)
import sys
import numpy as np
from manim import *
from blocks import *


def sigmoid(z):
    """Logistic function."""
    return 1.0 / (1.0 + np.exp(-z))


def train_trace(steps=3, lr=0.5):
    """Tiny 1-1-1 network, squared loss; record forward/backward states."""
    x, y = 1.0, 0.0
    w1, w2 = 0.8, 1.2
    trace = []
    for _ in range(steps):
        a1 = sigmoid(w1 * x)
        a2 = sigmoid(w2 * a1)
        loss = 0.5 * (a2 - y) ** 2
        d2 = (a2 - y) * a2 * (1 - a2)
        g2 = d2 * a1
        d1 = d2 * w2 * a1 * (1 - a1)
        g1 = d1 * x
        trace.append(dict(w1=w1, w2=w2, a1=a1, a2=a2, loss=loss, g1=g1, g2=g2))
        w1 -= lr * g1
        w2 -= lr * g2
    return trace


TRACE = train_trace()


class Main(Scene):
    def construct(self):
        self.camera.background_color = "#101418"
        TitleCard(self, "Backpropagation", "chain rule, layer by layer")
        net = NetworkLayers([1, 1, 1], max_width=6, max_height=2).shift(UP * 0.6)
        self.play(FadeIn(net))
        info = Text("", font_size=24)
        for k, s in enumerate(TRACE):
            # forward pass
            self.play(net.activate(0, [1.0]), run_time=0.4)
            self.play(net.activate(1, [s["a1"]]), run_time=0.5)
            self.play(net.activate(2, [s["a2"]]), run_time=0.5)
            # backward pass: show gradients
            new = Text(
                f"Step {k+1}  loss={s['loss']:.3f}  dL/dw2={s['g2']:.3f}  dL/dw1={s['g1']:.3f}",
                font_size=24, color=C_ACCENT).next_to(net, DOWN, buff=0.6)
            back = [net.layers[2][0].animate.set_stroke(C_BAD),
                    net.layers[1][0].animate.set_stroke(C_BAD)]
            self.play(FadeIn(new) if k == 0 else Transform(info, new),
                      *back, run_time=0.8)
            if k == 0:
                info = new
            self.play(net.layers[2][0].animate.set_stroke(C_MAIN),
                      net.layers[1][0].animate.set_stroke(C_MAIN), run_time=0.3)
        StepCaption(self, "Gradients flow backward; weights step downhill, so loss falls.")
        self.wait(1)


if __name__ == "__main__" and "--selftest" in sys.argv:
    losses = [s["loss"] for s in TRACE]
    assert losses[0] > losses[1] > losses[2]
    # numeric gradient check on w2 at step 0
    e = 1e-6
    w1, w2 = 0.8, 1.2
    f = lambda b: 0.5 * sigmoid(b * sigmoid(w1)) ** 2
    num = (f(w2 + e) - f(w2 - e)) / (2 * e)
    assert abs(num - TRACE[0]["g2"]) < 1e-6
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.