AI, machine learning and deep learning · Intermediate

Batch normalization

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Batch normalization standardizes each feature of a layer's pre-activations using the mean and variance computed over the current mini-batch, then applies a learned scale and shift. This stabilizes the distribution of inputs to later layers, allows larger learning rates, and speeds up training.

Intuition

Imagine each layer receiving inputs whose scale and center keep drifting as earlier layers update. The layer must constantly chase a moving target. Batch normalization pins each feature to zero mean and unit variance within the batch, so downstream layers see a steady distribution. Because forcing zero mean and unit variance might remove useful information, two learnable parameters, γ and β, let the network restore any scale and offset it needs. At test time there is no batch to rely on, so running averages of the mean and variance collected during training are used instead.

Key equations

μB=1m∑i=1mxi,σB2=1m∑i=1m(xi−μB)2
(1)

Batch mean and variance of one feature over m examples.

x^i=xi−μBσB2+ϵ
(2)

Normalized value; epsilon avoids division by zero.

yi=γx^i+β
(3)

Learned scale gamma and shift beta restore representational freedom.

μrun←(1−α)μrun+αμB
(4)

Running average used at inference.

Derivation

  1. Take a feature x across a mini-batch of size m.
  2. Compute the batch mean, then subtract it so the feature is centered at zero.
  3. Compute the batch variance and divide by the square root of variance plus epsilon, giving mean about 0 and variance about 1.
  4. Check: the mean of x^ is zero, and its variance is σ2σ2+ϵ, approximately 1.
  5. Apply y=γx-hat+β. If γ equals the square root of the variance and β equals the mean, the original input is recovered, so the layer loses no expressive power.
  6. Backpropagation flows through μ and σ as well, since they depend on every x in the batch; γ and β get gradients as sum of dL/dy times x^, and sum of dL/dy.
  7. At inference, replace mu_B and sigma_B squared with running estimates so outputs are deterministic.

Worked example

Problem

A feature has batch values 2, 4, 6, 8. Use epsilon of 0, γ=2, β=1. Compute the outputs.

  1. Mean = (2+4+6+8)/4 = 5.
  2. Variance = ((-3)^2+(-1)^2+1^2+3^2)/4 = 20/4 = 5, so the standard deviation is about 2.236.
  3. Normalized values: -3/2.236 = -1.342, -0.447, 0.447, 1.342.
  4. Apply y = 2x-hat + 1: -1.683, 0.106, 1.894, 3.683.

Answer Outputs are approximately -1.68, 0.11, 1.89, 3.68.

Common mistakes

  • Using batch statistics at test time instead of running averages.
  • Forgetting epsilon, causing division by zero for constant features.
  • Normalizing across the batch for each example rather than per feature (that is layer normalization).
  • Assuming it works well with very small batches, where statistics are noisy.
  • Thinking gamma and beta are unnecessary; they are what keep the layer expressive.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(70 lines)
import sys
from manim import *
from blocks import *
import numpy as np


def make_batch():
    """Fixed-seed batch of 1D activations with shifted mean and large spread."""
    rng = np.random.default_rng(3)
    return rng.normal(4.0, 2.0, 12)


def batchnorm(x, gamma=1.0, beta=0.0, eps=1e-5):
    """Return normalized and scaled values with statistics."""
    mu = float(np.mean(x))
    var = float(np.var(x))
    xh = (x - mu) / np.sqrt(var + eps)
    return xh, gamma * xh + beta, mu, var


X = make_batch()
XH, Y, MU, VAR = batchnorm(X, 1.5, 0.5)


class Main(Scene):
    def construct(self):
        TitleCard(self, "Batch normalization", "normalize each feature over the batch")

        # number line with the raw batch
        line = NumberLine(x_range=[-4, 8, 1], length=11, include_numbers=True,
                          font_size=24, color=C_DIM).shift(UP * 0.3)
        self.play(Create(line))
        dots = VGroup(*[Dot(line.n2p(v), color=C_MAIN, radius=0.1) for v in X])
        self.play(FadeIn(dots))
        mean_mark = Triangle(color=C_ACCENT, fill_opacity=1).scale(0.12)
        mean_mark.next_to(line.n2p(MU), DOWN, buff=0.45).rotate(PI)
        mean_lab = Text("mean", font_size=24, color=C_ACCENT).next_to(mean_mark, DOWN, buff=0.1)
        self.play(FadeIn(mean_mark), FadeIn(mean_lab))
        StepCaption(self, "Raw activations: mean %.1f, variance %.1f" % (MU, VAR))

        # normalize
        eq1 = MathTex(r"\hat{x}=\frac{x-\mu}{\sqrt{\sigma^2+\epsilon}}", font_size=34)
        eq1.to_edge(UP, buff=1.2)
        self.play(Write(eq1))
        new_mean = Triangle(color=C_GOOD, fill_opacity=1).scale(0.12)
        new_mean.next_to(line.n2p(0), DOWN, buff=0.45).rotate(PI)
        self.play(*[d.animate.move_to(line.n2p(v)) for d, v in zip(dots, XH)],
                  Transform(mean_mark, new_mean),
                  mean_lab.animate.next_to(line.n2p(0), DOWN, buff=0.85),
                  run_time=2)
        StepCaption(self, "Normalized: mean 0, variance 1")

        # scale and shift
        eq2 = MathTex(r"y=\gamma\hat{x}+\beta", font_size=34).next_to(eq1, DOWN, buff=0.3)
        self.play(Write(eq2))
        self.play(*[d.animate.move_to(line.n2p(v)).set_color(C_ACCENT)
                    for d, v in zip(dots, Y)], run_time=2)
        StepCaption(self, "Learnable $\\gamma,\\beta$ restore scale and shift".replace("$", "") if False else "Learnable gamma and beta rescale and shift the result")
        StepCaption(self, "Takeaway: normalize per batch, then let the network rescale")
        self.wait(1)


if __name__ == "__main__" and "--selftest" in sys.argv:
    xh, y, mu, var = batchnorm(X)
    assert abs(np.mean(xh)) < 1e-9
    assert abs(np.var(xh) - 1) < 1e-3
    assert abs(mu - np.mean(X)) < 1e-12
    assert abs(np.mean(Y) - 0.5) < 1e-9
    assert abs(np.std(Y) - 1.5) < 1e-3
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.