Biostatistics · Intermediate

Ordinary least squares

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Ordinary least squares (OLS) is a statistical procedure that estimates regression parameters by minimizing the sum of squared differences between observed and predicted values. This method provides the unique line that minimizes vertical distances from data points, ensuring an optimal fit for linear relationships. The resulting coefficients are calculated using specific formulas derived from the sums of squares and cross-products of the data.

Intuition

Think of OLS as finding the single best straight line through a cloud of data points. The goal is not just to get close, but to minimize the total squared vertical distance between every actual data point and the line. Squaring the errors is preferred because it is mathematically easier to derive and offers theoretical advantages over absolute deviations. Geometrically, this process finds the solution where the residual vector is orthogonal to the column space of the design matrix, meaning the predicted values are the closest possible approximation to the observed values in terms of Euclidean distance. This ensures that for any given value of the independent variable, the predicted value represents the mean of the dependent variable.

Key equations

y^i=b0+b1xi
(1)

The estimated regression equation, where y^i is the estimated value, b0 is the intercept, b1 is the slope, and xi is the independent variable.

min⁡∑(yi−y^i)2
(2)

The least squares criterion, which minimizes the sum of the squared differences between observed values yi and estimated values y^i.

b1=∑(xi−x‾)(yi−y‾)∑(xi−x‾)2
(3)

The formula for the slope, calculated as the ratio of the corrected sum of cross products to the corrected sum of squares for x.

b0=y‾−b1x‾
(4)

The formula for the intercept, calculated using the means of y and x and the estimated slope.

b1=LxyLxx
(5)

An alternative formula for the slope using corrected sums of cross products (Lxy) and sums of squares (Lxx).

Lxx=∑xi2−(∑xi)2n
(6)

The corrected sum of squares for x, representing the sum of squared deviations of x from its mean.

Lxy=∑xiyi−(∑xi)(∑yi)n
(7)

The corrected sum of cross products, representing the joint variation of x and y.

Derivation

  1. Start with the least squares criterion: minimize S=∑(yi−y^i)2.
  2. Substitute the linear model y^i=b0+b1xi into the criterion.
  3. Use differential calculus to find the values of b0 and b1 that minimize this sum.
  4. Derive the slope b1 as the ratio of the corrected sum of cross products (Lxy) to the corrected sum of squares for x (Lxx).
  5. Derive the intercept b0 using the relationship b0=y‾−b1x‾.
  6. Recognize that in matrix notation, the solution satisfies the condition that the residual is orthogonal to the column space of the design matrix.

Worked example

Problem

A researcher wants to predict sales based on student population. They have data where the mean student population (x‾) is 10,000, the mean sales (y‾) is 500,000, the corrected sum of squares for x (Lxx) is 2,000,000, and the corrected sum of cross products (Lxy) is 1,500,000. Calculate the slope and intercept.

  1. Calculate the slope b1 using the formula b1=LxyLxx.
  2. Substitute the values: b1=1,500,0002,000,000=0.75.
  3. Calculate the intercept b0 using the formula b0=y‾−b1x‾.
  4. Substitute the values: b0=500,000−(0.75×10,000)=500,000−7,500=492,500.
  5. The estimated regression equation is y^=492,500+0.75x.

Answer Slope b1=0.75, Intercept b0=492,500.

Common mistakes

  • Interpreting regression analysis as establishing a cause-and-effect relationship rather than just an association.
  • Confusing the estimated value y^i with the observed value yi; the method minimizes the difference between them.
  • Using absolute deviations instead of squared deviations, which is less theoretically convenient and harder to derive.
  • Assuming that a linear fit is appropriate for all data; nonlinear relationships may require nonlinear regression techniques.
  • Forgetting that the intercept b0 depends on the means of the variables and the slope, not just the data points directly.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

This is a text lesson: it has no animation code yet.

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.