AI, machine learning and deep learning · Intermediate

Evaluating Binary Classifier Thresholds

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

The Receiver Operating Characteristic (ROC) curve evaluates binary classifiers by plotting the True Positive Rate against the False Positive Rate across various probability thresholds. The Area Under the Curve (AUC) serves as a summary metric where larger areas indicate superior model performance compared to a random classifier baseline.

Intuition

Binary classifiers output probability estimates rather than hard labels, and the default threshold is often 0.5. However, different applications require balancing the costs of errors, such as choosing a higher threshold to reduce costly false positives even if it means missing some true positives. The ROC curve visualizes this trade-off by calculating the True Positive Rate and False Positive Rate for every possible threshold, allowing analysts to select the cutoff that best suits their specific problem context.

Key equations

TPR=TruePositivesTruePositives+FalseNegatives
(1)

True Positive Rate (Sensitivity): the proportion of actual positives correctly identified.

FPR=FalsePositivesFalsePositives+TrueNegatives
(2)

False Positive Rate: the proportion of actual negatives incorrectly identified as positives.

FPR=1−Specificity
(3)

The False Positive Rate is equal to one minus the True Negative Rate (Specificity).

Specificity=TNTN+FP
(4)

Specificity (True Negative Rate): the proportion of actual negatives correctly identified.

Derivation

  1. Start with the confusion matrix categorizing predictions into True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN).
  2. Calculate the True Positive Rate (Sensitivity) as TP / (TP + FN).
  3. Calculate the False Positive Rate as FP / (FP + TN).
  4. Recognize that False Positive Rate is also equal to 1 minus Specificity (TN / (TN + FP)).
  5. Generate predicted probabilities for the test set using methods like predict_proba.
  6. Calculate TPR and FPR for every possible probability threshold to illustrate the trade-off between sensitivity and specificity.
  7. Plot FPR on the x-axis and TPR on the y-axis to create the ROC curve.

Worked example

Problem

A medical test has 90 True Positives, 10 False Negatives, 19 False Positives, and 81 True Negatives. Calculate the TPR and FPR.

  1. Identify TP=90, FN=10, FP=19, TN=81.
  2. Calculate TPR: 90 / (90 + 10) = 90 / 100 = 0.90.
  3. Calculate FPR: 19 / (19 + 81) = 19 / 100 = 0.19.

Answer TPR = 0.90, FPR = 0.19

Common mistakes

  • Confusing the ROC curve with a precision-recall curve; the ROC plots TPR and FPR, not precision and recall.
  • Assuming that a higher threshold always improves the model; while it may reduce false positives, it also reduces true positives, and the optimal threshold depends on the specific context.
  • Thinking that the diagonal line represents a bad model; it actually represents a random classifier, which is the baseline for no predictive power.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

This is a text lesson: it has no animation code yet.

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.