AI, machine learning and deep learning · Intermediate

Principal component analysis

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Principal component analysis (PCA) finds new orthogonal axes along which a centered dataset varies most. These axes are the eigenvectors of the covariance matrix, and the variance along each is its eigenvalue. Projecting onto the top few axes gives a compact, low-dimensional summary that loses as little variance as possible.

Intuition

Imagine a cloud of points shaped like a tilted ellipse. The original x and y axes describe it poorly, because both coordinates change together. PCA rotates the axes to line up with the ellipse: the first axis runs along the long direction, the second across the short one. If the ellipse is very thin, the position along the long axis tells you almost everything, so you can drop the short axis and keep one number per point. Geometrically, you are projecting each point onto the line that keeps the spread as large as possible, which is the same as making the squared distances from points to the line as small as possible.

Key equations

S=1n−1Xc⊤Xc
(1)

Covariance matrix of the centered data matrix X_c (n samples by d features).

Svk=λkvk
(2)

Principal directions v_k are eigenvectors of S; eigenvalue λ_k is the variance along v_k.

zi=Vk⊤(xi−x‾)
(3)

Coordinates (scores) of sample i in the first k principal components.

EVRk=∑j=1kλj∑j=1dλj
(4)

Fraction of total variance retained by the first k components.

Derivation

  1. Center the data by subtracting the mean so that the projections have mean zero.
  2. Project onto a unit vector v: the projected variance is v^T S v.
  3. Maximize v^T S v subject to v^T v = 1 using a Lagrange multiplier: L = v^T S v - λ(v^T v - 1).
  4. Setting the gradient to zero gives S v = λ v, so v is an eigenvector of S.
  5. Then the variance equals v^T S v = λ, so choose the eigenvector with the largest eigenvalue first.
  6. Later components maximize variance subject to being orthogonal to the earlier ones, giving the next eigenvectors in order of decreasing eigenvalue.
  7. Because S is symmetric, its eigenvectors are orthogonal, so the components form a rotated coordinate system.

Worked example

Problem

Centered data points are (2,2), (-2,-2), (1,-1), (-1,1). Find the principal components and the variance explained by the first.

  1. Compute sums: Σx² = 4+4+1+1 = 10, Σy² = 10, Σxy = 4+4-1-1 = 6.
  2. With n-1 = 3: S = (1/3)[[10,6],[6,10]].
  3. Eigenvalues of [[10,6],[6,10]] are 16 and 4, so λ1 = 16/3 ≈ 5.33 and λ2 = 4/3 ≈ 1.33.
  4. Eigenvector for 16 is (1,1)/√2; for 4 it is (1,-1)/√2.
  5. Explained variance ratio = 16/(16+4) = 0.8.

Answer PC1 is the direction (1,1)/√2 with variance 16/3, explaining 80% of the total variance; PC2 is (1,-1)/√2.

Common mistakes

  • Forgetting to center the data before computing the covariance.
  • Not standardizing features measured on different scales, so large-unit features dominate.
  • Believing that principal components are original features or are always interpretable.
  • Confusing the eigenvalue (variance) with the eigenvector (direction).
  • Assuming high variance means high relevance for a prediction task.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(79 lines)
import sys
from manim import *
from blocks import *
import numpy as np


def make_data():
    """Fixed-seed correlated 2D points (centered)."""
    rng = np.random.default_rng(3)
    t = rng.normal(0, 1.6, 40)
    n = rng.normal(0, 0.4, 40)
    ang = np.deg2rad(30)
    x = t * np.cos(ang) - n * np.sin(ang)
    y = t * np.sin(ang) + n * np.cos(ang)
    pts = np.stack([x, y], axis=1)
    return pts - pts.mean(axis=0)


def pca(pts):
    """Eigen-decomposition of the covariance matrix."""
    cov = np.cov(pts.T)
    vals, vecs = np.linalg.eigh(cov)
    order = np.argsort(vals)[::-1]
    return vals[order], vecs[:, order]


class Main(Scene):
    def construct(self):
        # title
        TitleCard(self, "Principal component analysis", "Find the direction of most variance")
        pts = make_data()
        vals, vecs = pca(pts)
        # axes and points, kept well above the caption area
        axes = Axes(x_range=[-5, 5, 1], y_range=[-4, 4, 1], x_length=7.0, y_length=3.8,
                    axis_config={"color": C_DIM, "include_tip": False})
        axes.move_to([-2.0, 0.2, 0])
        dots = VGroup(*[Dot(axes.c2p(p[0], p[1]), radius=0.06, color=C_MAIN) for p in pts])
        self.play(Create(axes), FadeIn(dots))
        StepCaption(self, "Centered data cloud")
        self.wait(0.8)
        # principal axes
        v1, v2 = vecs[:, 0], vecs[:, 1]
        L1 = 2 * np.sqrt(vals[0])
        L2 = 2 * np.sqrt(vals[1])
        a1 = Arrow(axes.c2p(0, 0), axes.c2p(*(v1 * L1)), buff=0, color=C_ACCENT)
        a2 = Arrow(axes.c2p(0, 0), axes.c2p(*(v2 * L2)), buff=0, color=C_GOOD)
        l1 = Text("PC1", font_size=24, color=C_ACCENT).next_to(a1.get_end(), UP, buff=0.1)
        l2 = Text("PC2", font_size=24, color=C_GOOD).next_to(a2.get_end(), LEFT, buff=0.1)
        self.play(GrowArrow(a1), FadeIn(l1))
        self.play(GrowArrow(a2), FadeIn(l2))
        StepCaption(self, "PC1: direction of largest variance")
        self.wait(0.8)
        # variance explained
        ratio = vals / vals.sum()
        info = VGroup(
            Text(f"PC1 variance: {ratio[0]*100:.0f}%", font_size=26, color=C_ACCENT),
            Text(f"PC2 variance: {ratio[1]*100:.0f}%", font_size=26, color=C_GOOD),
        ).arrange(DOWN, aligned_edge=LEFT)
        info.to_edge(RIGHT, buff=0.4).shift(UP * 0.3)
        self.play(FadeIn(info))
        # project onto PC1
        proj = [np.dot(p, v1) * v1 for p in pts]
        self.play(*[d.animate.move_to(axes.c2p(q[0], q[1])) for d, q in zip(dots, proj)],
                  run_time=2)
        StepCaption(self, "Projecting onto PC1 keeps most of the information")
        self.wait(1)


if __name__ == "__main__" and "--selftest" in sys.argv:
    p = make_data()
    assert np.allclose(p.mean(axis=0), 0, atol=1e-9)
    w, V = pca(p)
    assert w[0] >= w[1] > 0
    assert abs(np.dot(V[:, 0], V[:, 1])) < 1e-9
    assert np.isclose(np.linalg.norm(V[:, 0]), 1.0)
    assert np.isclose(w.sum(), np.trace(np.cov(p.T)))
    pr = p @ V[:, 0]
    assert np.isclose(pr.var(ddof=1), w[0])
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.