AI, machine learning and deep learning · Intermediate

L2 regularization and overfitting

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

L2 regularization adds a penalty on the squared size of the weights to the loss function. This discourages the model from using large weights to fit noise, which reduces overfitting and improves performance on unseen data. It trades a small increase in bias for a larger decrease in variance.

Intuition

An overfit model wiggles through every training point, and to make sharp wiggles it needs large weights. L2 regularization puts a price on large weights, like a spring pulling every weight toward zero. The data loss pushes weights toward fitting the training set, while the penalty pulls them back. The balance is set by the strength λ. Small λ: the model is nearly unconstrained and may overfit. Large λ: weights are squashed and the model underfits. In gradient descent the penalty appears as weight decay: each step first shrinks every weight by a small fraction, then applies the usual data gradient. For linear regression this has a closed form, ridge regression, where adding λ times the identity makes the matrix more stable to invert.

Key equations

J(𝐰)=1n∑i=1n(yi−𝐰⊤𝐱i)2+λ‖𝐰‖22
(1)

Regularized loss: data-fit error plus λ times the sum of squared weights.

𝐰←(1−2ηλ)𝐰−η∇𝐰Ldata
(2)

Gradient descent update showing weight decay: weights shrink by a factor each step.

𝐰^ridge=(X⊤X+nλI)−1X⊤𝐲
(3)

Closed-form ridge solution for linear regression with the loss above.

Derivation

  1. Start with the data loss L(𝐰) and add the penalty: J(𝐰)=L(𝐰)+λ𝐰T𝐰.
  2. Differentiate the penalty: the gradient of λ×𝐰T𝐰 is 2×λ×𝐰.
  3. So the gradient of J is ∇L+2λw.
  4. Gradient descent step: w←w−η(∇L+2λw).
  5. Rearrange: w←(1−2ηλ)w−η∇L. The first factor is less than 1, so weights decay toward zero.
  6. For linear regression, with L=1n‖y−Xw‖2, set the gradient to zero: −2nXT(y−Xw)+2λw=0.
  7. Solve: (XTX+nλI)w=XTy, giving the ridge solution. The added term makes the matrix invertible even with correlated features.

Worked example

Problem

A single weight w has data loss L(w)=(w−4)2. Add an L2 penalty λw2 with λ=1. Find the regularized minimizer.

  1. J(w)=(w−4)2+w2.
  2. Differentiate: J'(w) = 2(w - 4) + 2w = 4w - 8.
  3. Set to zero: 4w - 8 = 0, so w = 2.
  4. Compare with the unregularized minimum w = 4: the penalty shrank the weight by half.
  5. General case: 2(w−4)+2λw=0 gives w=41+λ. For λ=1 this is 2, and as λ grows w tends to 0.

Answer w=2 (in general w=4/(1+λ))

Common mistakes

  • Penalizing the bias term; usually only weights are regularized.
  • Not standardizing features first, so the penalty affects features unequally depending on their scale.
  • Thinking larger λ is always better; too large causes underfitting.
  • Choosing λ by training error rather than a validation set or cross-validation.
  • Confusing L2 with L1: L2 shrinks weights smoothly but rarely makes them exactly zero, while L1 produces sparsity.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(186 lines)
import sys
import numpy as np
from manim import *
from blocks import *

# --- Data Generation (Fixed Seed) ---
np.random.seed(42)
N = 20
# True underlying function: sin(x)
X_true = np.linspace(-3, 3, 100)
Y_true = np.sin(X_true)

# Noisy observations
X_obs = np.linspace(-3, 3, N)
Y_obs = np.sin(X_obs) + np.random.normal(0, 0.3, N)

# --- Model Fitting ---
# We fit polynomials of degree 1, 3, 9, 15
# L2 regularization (Ridge) with lambda = 1.0
LAMBDA = 1.0

def fit_poly(degree, X, Y, lam=LAMBDA):
    """Fit polynomial using closed-form Ridge Regression: w = (X^T X + lambda I)^-1 X^T Y"""
    # Design matrix
    X_mat = np.vander(X, degree + 1, increasing=True)
    # Regularization matrix (don't regularize bias term usually, but for simplicity here we regularize all or just weights)
    # Standard Ridge: lambda * diag(0, 1, 1, ...)
    reg = lam * np.eye(degree + 1)
    reg[0, 0] = 0  # Don't regularize intercept
    
    # Solve
    try:
        A = X_mat.T @ X_mat + reg
        b = X_mat.T @ Y
        w = np.linalg.solve(A, b)
    except np.linalg.LinAlgError:
        w = np.zeros(degree + 1)
    return w

def predict_poly(w, x):
    """Evaluate polynomial at x"""
    return np.polyval(w[::-1], x)

# Compute coefficients
degrees = [1, 3, 9, 15]
models = {}
for d in degrees:
    w = fit_poly(d, X_obs, Y_obs)
    models[d] = w

# Compute Metrics
def mse(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)

train_mses = []
test_mses = []
for d in degrees:
    y_pred_train = predict_poly(models[d], X_obs)
    y_pred_test = predict_poly(models[d], X_true)
    train_mses.append(mse(Y_obs, y_pred_train))
    test_mses.append(mse(Y_true, y_pred_test))

class Main(Scene):
    def construct(self):
        TitleCard(self, "L2 Regularization", "Preventing Overfitting")
        
        # --- Part 1: The Curves ---
        # Setup Axes
        axes = Axes(
            x_range=[-3.5, 3.5, 1],
            y_range=[-2, 2, 1],
            x_length=8,
            y_length=4.5,
            axis_config={"color": C_DIM},
            tips=False
        )
        axes.to_edge(DOWN, buff=0.2)
        axes_labels = axes.get_axis_labels(MathTex("x"), MathTex("y"))
        
        # True Curve
        true_curve = axes.plot(lambda x: np.sin(x), color=C_DIM, stroke_width=2)
        
        # Data Points
        dots = VGroup(*[
            Dot(axes.c2p(x, y), color=C_MAIN, radius=0.08)
            for x, y in zip(X_obs, Y_obs)
        ])
        
        # Model Curves
        line_colors = [C_ACCENT, C_GOOD, C_BAD, C_BAD] # 1, 3, 9, 15
        model_curves = []
        for i, d in enumerate(degrees):
            w = models[d]
            # Plot using numpy polyval via a lambda
            # Note: np.polyval expects coefficients from highest degree
            curve = axes.plot(
                lambda x, w=w: np.polyval(w[::-1], x),
                color=line_colors[i],
                stroke_width=3
            )
            model_curves.append(curve)
            
        # Legend/Labels for curves
        labels = VGroup(
            Text("Degree 1 (Underfit)", font_size=20, color=line_colors[0]),
            Text("Degree 3 (Good)", font_size=20, color=line_colors[1]),
            Text("Degree 9 (Overfit)", font_size=20, color=line_colors[2]),
            Text("Degree 15 (Overfit)", font_size=20, color=line_colors[3])
        )
        labels.arrange(DOWN, aligned_edge=LEFT, buff=0.2)
        labels.to_edge(LEFT, buff=0.5).shift(UP * 1.5)
        
        # Animations
        self.play(Create(axes), Write(axes_labels))
        self.play(Create(true_curve))
        self.play(FadeIn(dots))
        self.wait(0.5)
        
        # Animate curves appearing one by one
        for i, d in enumerate(degrees):
            self.play(Create(model_curves[i]))
            self.play(Write(labels[i]))
            self.wait(0.3)
            
        StepCaption(self, "High degree fits noise; L2 keeps curves smooth")
        
        # --- Part 2: The Bias-Variance Tradeoff (Bar Chart) ---
        # Move curves away or fade them? Let's keep them but add a chart below or beside.
        # To avoid overlap, let's move the graph up slightly and add a chart below?
        # Or just replace the view. Let's replace the view for clarity.
        
        self.play(FadeOut(axes), FadeOut(true_curve), FadeOut(dots), FadeOut(VGroup(*model_curves)), FadeOut(labels))
        
        # Create Bar Chart for MSE
        # X-axis: Degrees, Y-axis: MSE
        chart_axes = Axes(
            x_range=[0, 4, 1],
            y_range=[0, 2, 0.5],
            x_length=6,
            y_length=3,
            axis_config={"color": C_DIM}
        )
        chart_axes.to_edge(DOWN, buff=0.2)
        chart_labels = chart_axes.get_axis_labels(
            Text("Model Complexity", font_size=24),
            Text("MSE", font_size=24)
        )
        
        # Bars
        bar_width = 0.6
        bars_group = VGroup()
        test_bar_colors = [C_ACCENT, C_GOOD, C_BAD, C_BAD]
        
        for i, d in enumerate(degrees):
            h_train = train_mses[i]
            h_test = test_mses[i]
            
            # Train Bar (Left)
            b_train = Rectangle(width=bar_width/2, height=h_train, fill_color=C_DIM, fill_opacity=0.5, stroke_color=C_DIM)
            b_train.move_to(chart_axes.c2p(i * 1.0 - 0.15, h_train/2)) # Approx centering
            
            # Test Bar (Right)
            b_test = Rectangle(width=bar_width/2, height=h_test, fill_color=test_bar_colors[i], fill_opacity=0.8, stroke_color=test_bar_colors[i])
            b_test.move_to(chart_axes.c2p(i * 1.0 + 0.15, h_test/2))
            
            bars_group.add(b_train, b_test)
            
        # Animate Chart
        self.play(Create(chart_axes), Write(chart_labels))
        self.play(Create(bars_group))
        
        # Highlight the gap
        # The gap between train and test error indicates overfitting
        # We can't easily animate a "gap" box without complex geometry, so we use StepCaption
        
        StepCaption(self, "Gap between Train and Test error indicates Overfitting")
        self.wait(2)

if __name__ == "__main__" and "--selftest" in sys.argv:
    # Check that degree 1 has higher test error than degree 3 (underfitting)
    assert test_mses[0] > test_mses[1], "Degree 1 should overfit less than 3? No, underfit more."
    # Check that degree 15 has higher test error than degree 3 (overfitting)
    assert test_mses[3] > test_mses[1], "Degree 15 should have higher test error than 3"
    # Check that train error decreases with complexity
    assert train_mses[0] > train_mses[1] > train_mses[2] > train_mses[3]
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.