AI, machine learning and deep learning · Intermediate

Logistic regression and the decision boundary

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Logistic regression models the probability of a binary outcome by passing a linear combination of features through the sigmoid function. The decision boundary is the set of points where the predicted probability equals one half, which is exactly where the linear score is zero. This makes the boundary a straight line in two dimensions and a hyperplane in higher dimensions.

Intuition

A linear score z = w·x + b can be any real number, but a probability must lie between 0 and 1. The sigmoid squashes z smoothly into that range: large positive scores give probabilities near 1, large negative scores near 0, and a score of zero gives exactly 0.5. The weights w set the direction in which the evidence for class 1 grows, and the bias b shifts the boundary. Points far from the boundary on the positive side get confident predictions; points near it are uncertain. Training chooses w and b to maximize the likelihood of the observed labels, which is the same as minimizing cross-entropy loss. A clinical example is predicting disease from age and a biomarker level: the boundary is the line where risk is 50 percent.

Key equations

z=𝐰⊤𝐱+b
(1)

Linear score (log-odds) for input x.

p(y=1∣𝐱)=σ(z)=11+e−z
(2)

Sigmoid converts the score to a probability.

ln⁡p1−p=𝐰⊤𝐱+b
(3)

The log-odds are linear in the features.

L=−1N∑i[yiln⁡pi+(1−yi)ln⁡(1−pi)]
(4)

Cross-entropy loss minimized during training.

∇𝐰L=1N∑i(pi−yi)𝐱i
(5)

Gradient used for gradient descent.

Derivation

  1. Assume the log-odds of class 1 are linear: ln(p/(1-p)) = w·x + b = z.
  2. Exponentiate: p/(1-p) = e^z.
  3. Solve for p: p = e^z/(1+e^z) = 1/(1+e^{-z}), the sigmoid.
  4. Predict class 1 when p ≥ 0.5. Since σ(z) = 0.5 only at z = 0, and σ is increasing, this means w·x + b ≥ 0.
  5. Therefore the decision boundary is w·x + b = 0, a hyperplane with normal vector w.
  6. For the likelihood, with Bernoulli labels, the log-likelihood is Σ[y ln p + (1-y) ln(1-p)]; its negative mean is the cross-entropy L.
  7. Using dσ/dz = σ(1-σ), the gradient simplifies to (p - y)x, so the update is w ← w - η·mean((p-y)x).

Worked example

Problem

A model has w = (2, 1), b = -4 for features x1 (biomarker) and x2 (age score). Find the decision boundary and the predicted probability for the point (1, 3).

  1. Boundary: 2x1 + x2 - 4 = 0, i.e. x2 = 4 - 2x1.
  2. Compute the score at (1, 3): z = 2(1) + 1(3) - 4 = 1.
  3. Probability: σ(1) = 1/(1+e^{-1}) = 1/(1+0.368) ≈ 0.731.
  4. Since 0.731 > 0.5, predict class 1. The point lies on the positive side of the line (3 > 4 - 2 = 2).

Answer Boundary: 2x1 + x2 = 4; at (1,3) p ≈ 0.73, so class 1.

Common mistakes

  • Thinking the sigmoid output is a class label rather than a probability.
  • Believing the boundary is curved; with raw features it is linear (curves need feature transformations).
  • Using mean squared error instead of cross-entropy, which gives poor gradients when predictions are confidently wrong.
  • Forgetting that the 0.5 threshold can be changed, which shifts the boundary in z, e.g. for screening tests.
  • Interpreting weights as probability changes; they change log-odds, not probability, by a fixed amount.
  • Not scaling features or regularizing, and ignoring that perfectly separable data makes weights diverge.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(67 lines)
import sys
from manim import *
from blocks import *
import numpy as np


def sigmoid(z):
    """Logistic function."""
    return 1.0 / (1.0 + np.exp(-z))


# 1D data and a fitted model (gradient descent, fixed steps, no randomness)
XS = np.array([-3.0, -2.2, -1.5, -0.8, 0.4, 1.0, 1.8, 2.6, 3.2])
YS = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1], dtype=float)


def fit(xs, ys, steps=2000, lr=0.1):
    """Fit w, b by gradient descent on the log loss."""
    w, b = 0.0, 0.0
    for _ in range(steps):
        p = sigmoid(w * xs + b)
        w -= lr * np.mean((p - ys) * xs)
        b -= lr * np.mean(p - ys)
    return w, b


W, B = fit(XS, YS)
BOUNDARY = -B / W


class Main(Scene):
    def construct(self):
        TitleCard(self, "Logistic regression", "Sigmoid and decision boundary")
        # plot of fitted probability curve
        plot = CurvePlot(lambda x: float(sigmoid(W * x + B)), x_range=(-4, 4, 1),
                         y_range=(0, 1.2, 0.5), x_label="x", y_label="P(y=1)",
                         color=C_MAIN, x_length=8.0, y_length=4.0)
        plot.shift(UP * 0.1)
        eq = MathTex(r"P(y=1)=\sigma(wx+b)", font_size=32).to_corner(UR, buff=0.5)
        self.play(Create(plot.axes), Write(plot.labels))
        self.play(Write(eq))
        # data points
        dots = VGroup(*[Dot(plot.axes.c2p(x, y), color=C_GOOD if y else C_BAD, radius=0.09)
                        for x, y in zip(XS, YS)])
        self.play(FadeIn(dots))
        StepCaption(self, "Fit w and b by minimizing log loss")
        self.play(Create(plot.curve), run_time=2)
        # decision boundary where P = 0.5
        half = DashedLine(plot.axes.c2p(-4, 0.5), plot.axes.c2p(4, 0.5), color=C_DIM)
        vline = DashedLine(plot.axes.c2p(BOUNDARY, 0), plot.axes.c2p(BOUNDARY, 1.1),
                           color=C_ACCENT)
        lab = MathTex(r"x^*=-b/w", font_size=30, color=C_ACCENT)
        lab.next_to(vline, UP, buff=0.1)
        self.play(Create(half))
        self.play(Create(vline), Write(lab))
        StepCaption(self, "The boundary is where P = 0.5, i.e. wx + b = 0")
        self.wait(1.5)


if __name__ == "__main__" and "--selftest" in sys.argv:
    assert abs(sigmoid(0.0) - 0.5) < 1e-12
    assert W > 0
    assert abs(sigmoid(W * BOUNDARY + B) - 0.5) < 1e-9
    pred = (sigmoid(W * XS + B) > 0.5).astype(float)
    assert (pred == YS).mean() >= 0.85
    assert -4 < BOUNDARY < 4
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.