AI, machine learning and deep learning · Intermediate
L2 regularization and overfitting
Written by SignalsLab
✎ Early edition: these lessons are being reviewed.
Content
Summary
L2 regularization adds a penalty on the squared size of the weights to the loss function. This discourages the model from using large weights to fit noise, which reduces overfitting and improves performance on unseen data. It trades a small increase in bias for a larger decrease in variance.
Intuition
An overfit model wiggles through every training point, and to make sharp wiggles it needs large weights. L2 regularization puts a price on large weights, like a spring pulling every weight toward zero. The data loss pushes weights toward fitting the training set, while the penalty pulls them back. The balance is set by the strength . Small : the model is nearly unconstrained and may overfit. Large : weights are squashed and the model underfits. In gradient descent the penalty appears as weight decay: each step first shrinks every weight by a small fraction, then applies the usual data gradient. For linear regression this has a closed form, ridge regression, where adding times the identity makes the matrix more stable to invert.
Key equations
(1)
Regularized loss: data-fit error plus times the sum of squared weights.
(2)
Gradient descent update showing weight decay: weights shrink by a factor each step.
(3)
Closed-form ridge solution for linear regression with the loss above.
Derivation
Start with the data loss and add the penalty: .
Differentiate the penalty: the gradient of is .
So the gradient of is .
Gradient descent step: .
Rearrange: . The first factor is less than 1, so weights decay toward zero.
For linear regression, with , set the gradient to zero: .
Solve: , giving the ridge solution. The added term makes the matrix invertible even with correlated features.
Worked example
Problem
A single weight has data loss . Add an L2 penalty with . Find the regularized minimizer.
.
Differentiate: J'(w) = 2(w - 4) + 2w = 4w - 8.
Set to zero: 4w - 8 = 0, so w = 2.
Compare with the unregularized minimum w = 4: the penalty shrank the weight by half.
General case: gives . For this is 2, and as grows tends to 0.
Answer (in general )
Common mistakes
Penalizing the bias term; usually only weights are regularized.
Not standardizing features first, so the penalty affects features unequally depending on their scale.
Thinking larger is always better; too large causes underfitting.
Choosing by training error rather than a validation set or cross-validation.
Confusing L2 with L1: L2 shrinks weights smoothly but rarely makes them exactly zero, while L1 produces sparsity.
Animation
Animation coming soon
The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.
Code
scene.py(186 lines)
import sys
import numpy as np
from manim import *
from blocks import *
# --- Data Generation (Fixed Seed) ---
np.random.seed(42)
N = 20
# True underlying function: sin(x)
X_true = np.linspace(-3, 3, 100)
Y_true = np.sin(X_true)
# Noisy observations
X_obs = np.linspace(-3, 3, N)
Y_obs = np.sin(X_obs) + np.random.normal(0, 0.3, N)
# --- Model Fitting ---
# We fit polynomials of degree 1, 3, 9, 15
# L2 regularization (Ridge) with lambda = 1.0
LAMBDA = 1.0
def fit_poly(degree, X, Y, lam=LAMBDA):
"""Fit polynomial using closed-form Ridge Regression: w = (X^T X + lambda I)^-1 X^T Y"""
# Design matrix
X_mat = np.vander(X, degree + 1, increasing=True)
# Regularization matrix (don't regularize bias term usually, but for simplicity here we regularize all or just weights)
# Standard Ridge: lambda * diag(0, 1, 1, ...)
reg = lam * np.eye(degree + 1)
reg[0, 0] = 0 # Don't regularize intercept
# Solve
try:
A = X_mat.T @ X_mat + reg
b = X_mat.T @ Y
w = np.linalg.solve(A, b)
except np.linalg.LinAlgError:
w = np.zeros(degree + 1)
return w
def predict_poly(w, x):
"""Evaluate polynomial at x"""
return np.polyval(w[::-1], x)
# Compute coefficients
degrees = [1, 3, 9, 15]
models = {}
for d in degrees:
w = fit_poly(d, X_obs, Y_obs)
models[d] = w
# Compute Metrics
def mse(y_true, y_pred):
return np.mean((y_true - y_pred) ** 2)
train_mses = []
test_mses = []
for d in degrees:
y_pred_train = predict_poly(models[d], X_obs)
y_pred_test = predict_poly(models[d], X_true)
train_mses.append(mse(Y_obs, y_pred_train))
test_mses.append(mse(Y_true, y_pred_test))
class Main(Scene):
def construct(self):
TitleCard(self, "L2 Regularization", "Preventing Overfitting")
# --- Part 1: The Curves ---
# Setup Axes
axes = Axes(
x_range=[-3.5, 3.5, 1],
y_range=[-2, 2, 1],
x_length=8,
y_length=4.5,
axis_config={"color": C_DIM},
tips=False
)
axes.to_edge(DOWN, buff=0.2)
axes_labels = axes.get_axis_labels(MathTex("x"), MathTex("y"))
# True Curve
true_curve = axes.plot(lambda x: np.sin(x), color=C_DIM, stroke_width=2)
# Data Points
dots = VGroup(*[
Dot(axes.c2p(x, y), color=C_MAIN, radius=0.08)
for x, y in zip(X_obs, Y_obs)
])
# Model Curves
line_colors = [C_ACCENT, C_GOOD, C_BAD, C_BAD] # 1, 3, 9, 15
model_curves = []
for i, d in enumerate(degrees):
w = models[d]
# Plot using numpy polyval via a lambda
# Note: np.polyval expects coefficients from highest degree
curve = axes.plot(
lambda x, w=w: np.polyval(w[::-1], x),
color=line_colors[i],
stroke_width=3
)
model_curves.append(curve)
# Legend/Labels for curves
labels = VGroup(
Text("Degree 1 (Underfit)", font_size=20, color=line_colors[0]),
Text("Degree 3 (Good)", font_size=20, color=line_colors[1]),
Text("Degree 9 (Overfit)", font_size=20, color=line_colors[2]),
Text("Degree 15 (Overfit)", font_size=20, color=line_colors[3])
)
labels.arrange(DOWN, aligned_edge=LEFT, buff=0.2)
labels.to_edge(LEFT, buff=0.5).shift(UP * 1.5)
# Animations
self.play(Create(axes), Write(axes_labels))
self.play(Create(true_curve))
self.play(FadeIn(dots))
self.wait(0.5)
# Animate curves appearing one by one
for i, d in enumerate(degrees):
self.play(Create(model_curves[i]))
self.play(Write(labels[i]))
self.wait(0.3)
StepCaption(self, "High degree fits noise; L2 keeps curves smooth")
# --- Part 2: The Bias-Variance Tradeoff (Bar Chart) ---
# Move curves away or fade them? Let's keep them but add a chart below or beside.
# To avoid overlap, let's move the graph up slightly and add a chart below?
# Or just replace the view. Let's replace the view for clarity.
self.play(FadeOut(axes), FadeOut(true_curve), FadeOut(dots), FadeOut(VGroup(*model_curves)), FadeOut(labels))
# Create Bar Chart for MSE
# X-axis: Degrees, Y-axis: MSE
chart_axes = Axes(
x_range=[0, 4, 1],
y_range=[0, 2, 0.5],
x_length=6,
y_length=3,
axis_config={"color": C_DIM}
)
chart_axes.to_edge(DOWN, buff=0.2)
chart_labels = chart_axes.get_axis_labels(
Text("Model Complexity", font_size=24),
Text("MSE", font_size=24)
)
# Bars
bar_width = 0.6
bars_group = VGroup()
test_bar_colors = [C_ACCENT, C_GOOD, C_BAD, C_BAD]
for i, d in enumerate(degrees):
h_train = train_mses[i]
h_test = test_mses[i]
# Train Bar (Left)
b_train = Rectangle(width=bar_width/2, height=h_train, fill_color=C_DIM, fill_opacity=0.5, stroke_color=C_DIM)
b_train.move_to(chart_axes.c2p(i * 1.0 - 0.15, h_train/2)) # Approx centering
# Test Bar (Right)
b_test = Rectangle(width=bar_width/2, height=h_test, fill_color=test_bar_colors[i], fill_opacity=0.8, stroke_color=test_bar_colors[i])
b_test.move_to(chart_axes.c2p(i * 1.0 + 0.15, h_test/2))
bars_group.add(b_train, b_test)
# Animate Chart
self.play(Create(chart_axes), Write(chart_labels))
self.play(Create(bars_group))
# Highlight the gap
# The gap between train and test error indicates overfitting
# We can't easily animate a "gap" box without complex geometry, so we use StepCaption
StepCaption(self, "Gap between Train and Test error indicates Overfitting")
self.wait(2)
if __name__ == "__main__" and "--selftest" in sys.argv:
# Check that degree 1 has higher test error than degree 3 (underfitting)
assert test_mses[0] > test_mses[1], "Degree 1 should overfit less than 3? No, underfit more."
# Check that degree 15 has higher test error than degree 3 (overfitting)
assert test_mses[3] > test_mses[1], "Degree 15 should have higher test error than 3"
# Check that train error decreases with complexity
assert train_mses[0] > train_mses[1] > train_mses[2] > train_mses[3]
print("ok")
The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).