AI, machine learning and deep learning · Introductory

Linear regression and least squares

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Linear regression fits a straight line to data so that predictions are as close as possible to observed values. Least squares chooses the slope and intercept that minimize the sum of squared vertical errors, and this has a closed-form solution.

Intuition

Imagine a scatter of points, such as dose versus response. Many lines could pass through the cloud, but some are clearly better. For each point, measure the vertical gap between the point and the line; this gap is the residual. Squaring residuals makes all errors positive and punishes large misses more heavily. The best line is the one whose total squared residual is smallest. Geometrically, the line always passes through the mean point of the data, and its slope is the covariance of x and y divided by the variance of x.

Key equations

y^i=β0+β1xi
(1)

The model's prediction for observation i.

S(β0,β1)=∑i=1n(yi−β0−β1xi)2
(2)

Sum of squared residuals, the quantity to minimize.

β^1=∑(xi−x‾)(yi−y‾)∑(xi−x‾)2
(3)

Least squares slope.

β^0=y‾−β^1x‾
(4)

Least squares intercept; the line passes through (x-bar, y-bar).

𝜷^=(XTX)−1XTy
(5)

Matrix form (normal equations solution) for many predictors.

Derivation

  1. Define S(b0,b1) = sum of (y_i - b0 - b1 x_i)^2.
  2. Set the partial derivative with respect to b0 to zero: -2 sum (y_i - b0 - b1 x_i) = 0, so sum y_i = n b0 + b1 sum x_i, giving b0 = ybar - b1 xbar.
  3. Set the partial derivative with respect to b1 to zero: -2 sum x_i (y_i - b0 - b1 x_i) = 0.
  4. Substitute b0 = ybar - b1 xbar and simplify to get b1 * sum (x_i - xbar)^2 = sum (x_i - xbar)(y_i - ybar).
  5. Solve for b1 to get the slope formula. S is a convex quadratic, so this stationary point is the unique minimum (if x values are not all equal).

Worked example

Problem

Fit a least squares line to the points (1,2), (2,3), (3,5), (4,6).

  1. xbar = (1+2+3+4)/4 = 2.5; ybar = (2+3+5+6)/4 = 4.
  2. Deviations in x: -1.5, -0.5, 0.5, 1.5. Deviations in y: -2, -1, 1, 2.
  3. Sum of products: 3 + 0.5 + 0.5 + 3 = 7.
  4. Sum of squared x deviations: 2.25+0.25+0.25+2.25 = 5.
  5. Slope = 7/5 = 1.4.
  6. Intercept = 4 - 1.4*2.5 = 0.5.

Answer y-hat = 0.5 + 1.4 x

Common mistakes

  • Confusing correlation with causation: a good fit does not mean x causes y.
  • Minimizing horizontal or perpendicular distances instead of vertical residuals.
  • Extrapolating far outside the range of observed x values.
  • Ignoring outliers, which strongly influence squared-error fits.
  • Assuming a high R-squared means the linear model is appropriate without plotting residuals.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(73 lines)
import sys
import numpy as np
from manim import *
from blocks import *


def make_data():
    """Fixed-seed noisy linear data."""
    rng = np.random.default_rng(3)
    x = np.linspace(0.5, 5.5, 8)
    y = 0.8 * x + 1.0 + rng.normal(0, 0.5, size=x.size)
    return x, y


def fit_line(x, y):
    """Least squares slope and intercept via normal equations."""
    A = np.vstack([x, np.ones_like(x)]).T
    w = np.linalg.solve(A.T @ A, A.T @ y)
    return float(w[0]), float(w[1])


def sse(x, y, m, b):
    """Sum of squared residuals."""
    return float(np.sum((y - (m * x + b)) ** 2))


X, Y = make_data()
M, B = fit_line(X, Y)


class Main(Scene):
    def construct(self):
        self.camera.background_color = "#111118"
        TitleCard(self, "Linear regression", "Least squares fit")
        # axes and points
        plot = CurvePlot(lambda x: M * x + B, x_range=(0, 6), y_range=(0, 7),
                         x_label="x", y_label="y", x_length=7, y_length=4.2)
        plot.shift(LEFT * 1.8 + DOWN * 0.2)
        ax = plot.axes
        dots = VGroup(*[Dot(ax.c2p(a, c), color=C_ACCENT, radius=0.08)
                        for a, c in zip(X, Y)])
        self.play(Create(ax), FadeIn(plot.labels), FadeIn(dots))
        StepCaption(self, "Data points with noise")
        # a poor guess line
        m0, b0 = 0.2, 3.5
        guess = ax.plot(lambda x: m0 * x + b0, x_range=[0, 6], color=C_BAD)
        info = Text(f"SSE = {sse(X, Y, m0, b0):.2f}", font_size=28, color=C_BAD)
        info.to_edge(RIGHT, buff=0.8).shift(UP * 1.5)
        self.play(Create(guess), FadeIn(info))
        StepCaption(self, "A bad guess has large squared errors")
        # morph to the best fit with residuals
        best = ax.plot(lambda x: M * x + B, x_range=[0, 6], color=C_GOOD)
        info2 = Text(f"SSE = {sse(X, Y, M, B):.2f}", font_size=28, color=C_GOOD)
        info2.move_to(info)
        self.play(Transform(guess, best), Transform(info, info2), run_time=2)
        res = VGroup(*[Line(ax.c2p(a, c), ax.c2p(a, M * a + B), color=C_MAIN)
                       for a, c in zip(X, Y)])
        self.play(Create(res))
        eq = MathTex(r"\min_{m,b}\sum_i (y_i-mx_i-b)^2", font_size=32)
        eq.next_to(info, DOWN, buff=0.6)
        self.play(Write(eq))
        StepCaption(self, "Least squares picks the line minimizing total squared error")
        self.wait(1)


if __name__ == "__main__" and "--selftest" in sys.argv:
    m, b = fit_line(X, Y)
    ref = np.polyfit(X, Y, 1)
    assert abs(m - ref[0]) < 1e-9 and abs(b - ref[1]) < 1e-9
    assert sse(X, Y, m, b) <= sse(X, Y, m + 0.1, b)
    assert sse(X, Y, m, b) <= sse(X, Y, m, b - 0.1)
    assert abs(np.sum(Y - (m * X + b))) < 1e-9
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.