Content
Summary
Ordinary least squares (OLS) is a statistical procedure that estimates regression parameters by minimizing the sum of squared differences between observed and predicted values. This method provides the unique line that minimizes vertical distances from data points, ensuring an optimal fit for linear relationships. The resulting coefficients are calculated using specific formulas derived from the sums of squares and cross-products of the data.
Intuition
Think of OLS as finding the single best straight line through a cloud of data points. The goal is not just to get close, but to minimize the total squared vertical distance between every actual data point and the line. Squaring the errors is preferred because it is mathematically easier to derive and offers theoretical advantages over absolute deviations. Geometrically, this process finds the solution where the residual vector is orthogonal to the column space of the design matrix, meaning the predicted values are the closest possible approximation to the observed values in terms of Euclidean distance. This ensures that for any given value of the independent variable, the predicted value represents the mean of the dependent variable.
Key equations
The estimated regression equation, where is the estimated value, is the intercept, is the slope, and is the independent variable.
The least squares criterion, which minimizes the sum of the squared differences between observed values and estimated values .
The formula for the slope, calculated as the ratio of the corrected sum of cross products to the corrected sum of squares for x.
The formula for the intercept, calculated using the means of y and x and the estimated slope.
An alternative formula for the slope using corrected sums of cross products () and sums of squares ().
The corrected sum of squares for x, representing the sum of squared deviations of x from its mean.
The corrected sum of cross products, representing the joint variation of x and y.
Derivation
- Start with the least squares criterion: minimize .
- Substitute the linear model into the criterion.
- Use differential calculus to find the values of and that minimize this sum.
- Derive the slope as the ratio of the corrected sum of cross products () to the corrected sum of squares for x ().
- Derive the intercept using the relationship .
- Recognize that in matrix notation, the solution satisfies the condition that the residual is orthogonal to the column space of the design matrix.
Worked example
A researcher wants to predict sales based on student population. They have data where the mean student population () is 10,000, the mean sales () is 500,000, the corrected sum of squares for x () is 2,000,000, and the corrected sum of cross products () is 1,500,000. Calculate the slope and intercept.
- Calculate the slope using the formula .
- Substitute the values: .
- Calculate the intercept using the formula .
- Substitute the values: .
- The estimated regression equation is .
Answer Slope , Intercept .
Common mistakes
- Interpreting regression analysis as establishing a cause-and-effect relationship rather than just an association.
- Confusing the estimated value with the observed value ; the method minimizes the difference between them.
- Using absolute deviations instead of squared deviations, which is less theoretically convenient and harder to derive.
- Assuming that a linear fit is appropriate for all data; nonlinear relationships may require nonlinear regression techniques.
- Forgetting that the intercept depends on the means of the variables and the slope, not just the data points directly.