Content
Summary
Backpropagation computes the gradient of a loss with respect to every weight in a neural network by applying the chain rule backwards from the output. It reuses intermediate results from the forward pass, making gradient computation about as cheap as one extra forward pass. These gradients drive gradient descent updates.
Intuition
Think of the network as a chain of simple functions. The forward pass pushes an input through the chain to produce a prediction and a loss. To learn, we ask: if I nudge this weight slightly, how much does the loss change? The answer is a product of local sensitivities along the path from that weight to the loss. Backpropagation computes these products efficiently by starting at the loss and moving backwards, carrying an error signal called delta. At each layer, delta is multiplied by the local derivative and passed to the previous layer, so shared work is never repeated. Each weight's gradient is then its incoming activation times the delta at its output neuron.
Key equations
Forward pass for layer l: linear step then nonlinearity.
Error signal at the output layer.
Error signal propagated backwards one layer.
Gradients for weights and biases.
Gradient descent update with learning rate eta.
Derivation
- Define at layer as the derivative of the loss with respect to at layer .
- At the output, the chain rule gives .
- Since at layer depends on at layer through , at layer equals transpose of layer times of layer .
- Multiply by at layer to get at layer .
- Because at layer equals plus , contributes the previous activation, so equals times transpose.
- Because equals 1, equals delta.
- Repeat from the last layer to the first, storing activations from the forward pass.
Worked example
One neuron: x=2, w=0.5, b=0, sigmoid activation, target y=1, loss L=0.5(a-y)^2. Compute dL/dw.
- Forward: z = 0.5*2 + 0 = 1.
- a = sigmoid(1) ≈ 0.7311.
- L = 0.5*(0.7311-1)^2 ≈ 0.0362.
- dL/da = a - y = -0.2689.
- .
- .
- .
Answer dL/dw ≈ -0.106, so gradient descent increases w.
Common mistakes
- Forgetting to store forward activations and z values needed in the backward pass.
- Mixing up the transpose of when propagating delta backwards.
- Using the elementwise product incorrectly, or omitting the derivative of the activation.
- Updating weights before all gradients are computed, corrupting later layers' gradients.
- Ignoring vanishing gradients from saturated sigmoids in deep networks.
- Confusing backpropagation (gradient computation) with gradient descent (the update).