Content
Summary
Batch normalization standardizes each feature of a layer's pre-activations using the mean and variance computed over the current mini-batch, then applies a learned scale and shift. This stabilizes the distribution of inputs to later layers, allows larger learning rates, and speeds up training.
Intuition
Imagine each layer receiving inputs whose scale and center keep drifting as earlier layers update. The layer must constantly chase a moving target. Batch normalization pins each feature to zero mean and unit variance within the batch, so downstream layers see a steady distribution. Because forcing zero mean and unit variance might remove useful information, two learnable parameters, and , let the network restore any scale and offset it needs. At test time there is no batch to rely on, so running averages of the mean and variance collected during training are used instead.
Key equations
Batch mean and variance of one feature over m examples.
Normalized value; epsilon avoids division by zero.
Learned scale gamma and shift beta restore representational freedom.
Running average used at inference.
Derivation
- Take a feature x across a mini-batch of size m.
- Compute the batch mean, then subtract it so the feature is centered at zero.
- Compute the batch variance and divide by the square root of variance plus epsilon, giving mean about 0 and variance about 1.
- Check: the mean of is zero, and its variance is , approximately 1.
- Apply . If equals the square root of the variance and equals the mean, the original input is recovered, so the layer loses no expressive power.
- Backpropagation flows through and as well, since they depend on every in the batch; and get gradients as sum of times , and sum of .
- At inference, replace mu_B and sigma_B squared with running estimates so outputs are deterministic.
Worked example
A feature has batch values 2, 4, 6, 8. Use epsilon of 0, , . Compute the outputs.
- Mean = (2+4+6+8)/4 = 5.
- Variance = ((-3)^2+(-1)^2+1^2+3^2)/4 = 20/4 = 5, so the standard deviation is about 2.236.
- Normalized values: -3/2.236 = -1.342, -0.447, 0.447, 1.342.
- Apply y = 2x-hat + 1: -1.683, 0.106, 1.894, 3.683.
Answer Outputs are approximately -1.68, 0.11, 1.89, 3.68.
Common mistakes
- Using batch statistics at test time instead of running averages.
- Forgetting epsilon, causing division by zero for constant features.
- Normalizing across the batch for each example rather than per feature (that is layer normalization).
- Assuming it works well with very small batches, where statistics are noisy.
- Thinking gamma and beta are unnecessary; they are what keep the layer expressive.