Content
Summary
Shrinkage methods like ridge regression constrain coefficients toward zero to reduce variance, while dimension reduction techniques like PCR transform predictors into orthogonal components. Ridge regression minimizes residual sum of squares plus a penalty term, whereas PCR uses principal components derived from maximizing predictor variance.
Intuition
Shrinkage methods offer an alternative to subset selection by keeping all predictors but forcing their coefficients toward zero, thereby reducing variance at the cost of slight bias. Ridge regression achieves this by adding a penalty proportional to the squared magnitude of coefficients, controlled by a tuning parameter . Dimension reduction methods like Principal Components Regression (PCR) take a different approach by transforming the original predictors into a smaller set of uncorrelated components that capture the maximum variance, which are then used for prediction.
Key equations
The objective function minimized in ridge regression, combining residual sum of squares and a penalty on squared coefficients.
The first principal component as a linear combination of centered predictors.
The unit norm constraint on principal component loadings to prevent variance inflation.
Derivation
- Shrinkage methods fit models with all p predictors but impose constraints on coefficients to pull them toward zero, reducing variance.
- Ridge regression minimizes an objective function consisting of the residual sum of squares plus a penalty term times the sum of squared coefficients.
- The tuning parameter controls the strength of the penalty; when is zero, ridge regression equals ordinary least squares.
- Dimension reduction methods transform original predictors into M new variables, such as principal components, before fitting a regression model.
- Principal Components Regression (PCR) uses the first M principal components, which are linear combinations of centered predictors, as predictors in a least squares model.
- The first principal component is defined as the direction that maximizes the variance of the projected observations.
- Principal component loadings are constrained to have a sum of squares equal to one to ensure the variance is not arbitrarily inflated.
Worked example
Explain why PCR is considered unsupervised and how this differs from PLS.
- PCR constructs principal components by maximizing the variance of the predictor matrix X.
- This process does not reference the response variable Y.
- Therefore, the directions chosen may not be the most useful for predicting Y.
- In contrast, Partial Least Squares (PLS) is supervised and seeks directions that maximize covariance with Y.
Answer PCR is unsupervised because it ignores Y when creating components, potentially missing predictive signals in low-variance directions, whereas PLS incorporates Y to find components relevant to prediction.
Common mistakes
- Believing PCR performs feature selection; it uses linear combinations of all predictors rather than selecting a subset.
- Assuming high-variance directions in X are always best for predicting Y; PCR may perform poorly if Y is associated with low-variance directions.
- Forgetting to standardize predictors before applying PCR, which can cause variables with larger scales to disproportionately influence the components.