Content
Summary
Logistic regression models the probability of a binary outcome by passing a linear combination of features through the sigmoid function. The decision boundary is the set of points where the predicted probability equals one half, which is exactly where the linear score is zero. This makes the boundary a straight line in two dimensions and a hyperplane in higher dimensions.
Intuition
A linear score z = w·x + b can be any real number, but a probability must lie between 0 and 1. The sigmoid squashes z smoothly into that range: large positive scores give probabilities near 1, large negative scores near 0, and a score of zero gives exactly 0.5. The weights w set the direction in which the evidence for class 1 grows, and the bias b shifts the boundary. Points far from the boundary on the positive side get confident predictions; points near it are uncertain. Training chooses w and b to maximize the likelihood of the observed labels, which is the same as minimizing cross-entropy loss. A clinical example is predicting disease from age and a biomarker level: the boundary is the line where risk is 50 percent.
Key equations
Linear score (log-odds) for input x.
Sigmoid converts the score to a probability.
The log-odds are linear in the features.
Cross-entropy loss minimized during training.
Gradient used for gradient descent.
Derivation
- Assume the log-odds of class 1 are linear: ln(p/(1-p)) = w·x + b = z.
- Exponentiate: p/(1-p) = e^z.
- Solve for p: p = e^z/(1+e^z) = 1/(1+e^{-z}), the sigmoid.
- Predict class 1 when p ≥ 0.5. Since σ(z) = 0.5 only at z = 0, and σ is increasing, this means w·x + b ≥ 0.
- Therefore the decision boundary is w·x + b = 0, a hyperplane with normal vector w.
- For the likelihood, with Bernoulli labels, the log-likelihood is Σ[y ln p + (1-y) ln(1-p)]; its negative mean is the cross-entropy L.
- Using dσ/dz = σ(1-σ), the gradient simplifies to (p - y)x, so the update is w ← w - η·mean((p-y)x).
Worked example
A model has w = (2, 1), b = -4 for features x1 (biomarker) and x2 (age score). Find the decision boundary and the predicted probability for the point (1, 3).
- Boundary: 2x1 + x2 - 4 = 0, i.e. x2 = 4 - 2x1.
- Compute the score at (1, 3): z = 2(1) + 1(3) - 4 = 1.
- Probability: σ(1) = 1/(1+e^{-1}) = 1/(1+0.368) ≈ 0.731.
- Since 0.731 > 0.5, predict class 1. The point lies on the positive side of the line (3 > 4 - 2 = 2).
Answer Boundary: 2x1 + x2 = 4; at (1,3) p ≈ 0.73, so class 1.
Common mistakes
- Thinking the sigmoid output is a class label rather than a probability.
- Believing the boundary is curved; with raw features it is linear (curves need feature transformations).
- Using mean squared error instead of cross-entropy, which gives poor gradients when predictions are confidently wrong.
- Forgetting that the 0.5 threshold can be changed, which shifts the boundary in z, e.g. for screening tests.
- Interpreting weights as probability changes; they change log-odds, not probability, by a fixed amount.
- Not scaling features or regularizing, and ignoring that perfectly separable data makes weights diverge.