Content
Summary
Principal component analysis (PCA) finds new orthogonal axes along which a centered dataset varies most. These axes are the eigenvectors of the covariance matrix, and the variance along each is its eigenvalue. Projecting onto the top few axes gives a compact, low-dimensional summary that loses as little variance as possible.
Intuition
Imagine a cloud of points shaped like a tilted ellipse. The original x and y axes describe it poorly, because both coordinates change together. PCA rotates the axes to line up with the ellipse: the first axis runs along the long direction, the second across the short one. If the ellipse is very thin, the position along the long axis tells you almost everything, so you can drop the short axis and keep one number per point. Geometrically, you are projecting each point onto the line that keeps the spread as large as possible, which is the same as making the squared distances from points to the line as small as possible.
Key equations
Covariance matrix of the centered data matrix X_c (n samples by d features).
Principal directions v_k are eigenvectors of S; eigenvalue λ_k is the variance along v_k.
Coordinates (scores) of sample i in the first k principal components.
Fraction of total variance retained by the first k components.
Derivation
- Center the data by subtracting the mean so that the projections have mean zero.
- Project onto a unit vector v: the projected variance is v^T S v.
- Maximize v^T S v subject to v^T v = 1 using a Lagrange multiplier: L = v^T S v - λ(v^T v - 1).
- Setting the gradient to zero gives S v = λ v, so v is an eigenvector of S.
- Then the variance equals v^T S v = λ, so choose the eigenvector with the largest eigenvalue first.
- Later components maximize variance subject to being orthogonal to the earlier ones, giving the next eigenvectors in order of decreasing eigenvalue.
- Because S is symmetric, its eigenvectors are orthogonal, so the components form a rotated coordinate system.
Worked example
Centered data points are (2,2), (-2,-2), (1,-1), (-1,1). Find the principal components and the variance explained by the first.
- Compute sums: Σx² = 4+4+1+1 = 10, Σy² = 10, Σxy = 4+4-1-1 = 6.
- With n-1 = 3: S = (1/3)[[10,6],[6,10]].
- Eigenvalues of [[10,6],[6,10]] are 16 and 4, so λ1 = 16/3 ≈ 5.33 and λ2 = 4/3 ≈ 1.33.
- Eigenvector for 16 is (1,1)/√2; for 4 it is (1,-1)/√2.
- Explained variance ratio = 16/(16+4) = 0.8.
Answer PC1 is the direction (1,1)/√2 with variance 16/3, explaining 80% of the total variance; PC2 is (1,-1)/√2.
Common mistakes
- Forgetting to center the data before computing the covariance.
- Not standardizing features measured on different scales, so large-unit features dominate.
- Believing that principal components are original features or are always interpretable.
- Confusing the eigenvalue (variance) with the eigenvector (direction).
- Assuming high variance means high relevance for a prediction task.