Content
Summary
Linear regression fits a straight line to data so that predictions are as close as possible to observed values. Least squares chooses the slope and intercept that minimize the sum of squared vertical errors, and this has a closed-form solution.
Intuition
Imagine a scatter of points, such as dose versus response. Many lines could pass through the cloud, but some are clearly better. For each point, measure the vertical gap between the point and the line; this gap is the residual. Squaring residuals makes all errors positive and punishes large misses more heavily. The best line is the one whose total squared residual is smallest. Geometrically, the line always passes through the mean point of the data, and its slope is the covariance of x and y divided by the variance of x.
Key equations
The model's prediction for observation i.
Sum of squared residuals, the quantity to minimize.
Least squares slope.
Least squares intercept; the line passes through (x-bar, y-bar).
Matrix form (normal equations solution) for many predictors.
Derivation
- Define S(b0,b1) = sum of (y_i - b0 - b1 x_i)^2.
- Set the partial derivative with respect to b0 to zero: -2 sum (y_i - b0 - b1 x_i) = 0, so sum y_i = n b0 + b1 sum x_i, giving b0 = ybar - b1 xbar.
- Set the partial derivative with respect to b1 to zero: -2 sum x_i (y_i - b0 - b1 x_i) = 0.
- Substitute b0 = ybar - b1 xbar and simplify to get b1 * sum (x_i - xbar)^2 = sum (x_i - xbar)(y_i - ybar).
- Solve for b1 to get the slope formula. S is a convex quadratic, so this stationary point is the unique minimum (if x values are not all equal).
Worked example
Fit a least squares line to the points (1,2), (2,3), (3,5), (4,6).
- xbar = (1+2+3+4)/4 = 2.5; ybar = (2+3+5+6)/4 = 4.
- Deviations in x: -1.5, -0.5, 0.5, 1.5. Deviations in y: -2, -1, 1, 2.
- Sum of products: 3 + 0.5 + 0.5 + 3 = 7.
- Sum of squared x deviations: 2.25+0.25+0.25+2.25 = 5.
- Slope = 7/5 = 1.4.
- Intercept = 4 - 1.4*2.5 = 0.5.
Answer y-hat = 0.5 + 1.4 x
Common mistakes
- Confusing correlation with causation: a good fit does not mean x causes y.
- Minimizing horizontal or perpendicular distances instead of vertical residuals.
- Extrapolating far outside the range of observed x values.
- Ignoring outliers, which strongly influence squared-error fits.
- Assuming a high R-squared means the linear model is appropriate without plotting residuals.