Content
Summary
The Receiver Operating Characteristic (ROC) curve evaluates binary classifiers by plotting the True Positive Rate against the False Positive Rate across various probability thresholds. The Area Under the Curve (AUC) serves as a summary metric where larger areas indicate superior model performance compared to a random classifier baseline.
Intuition
Binary classifiers output probability estimates rather than hard labels, and the default threshold is often 0.5. However, different applications require balancing the costs of errors, such as choosing a higher threshold to reduce costly false positives even if it means missing some true positives. The ROC curve visualizes this trade-off by calculating the True Positive Rate and False Positive Rate for every possible threshold, allowing analysts to select the cutoff that best suits their specific problem context.
Key equations
True Positive Rate (Sensitivity): the proportion of actual positives correctly identified.
False Positive Rate: the proportion of actual negatives incorrectly identified as positives.
The False Positive Rate is equal to one minus the True Negative Rate (Specificity).
Specificity (True Negative Rate): the proportion of actual negatives correctly identified.
Derivation
- Start with the confusion matrix categorizing predictions into True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN).
- Calculate the True Positive Rate (Sensitivity) as TP / (TP + FN).
- Calculate the False Positive Rate as FP / (FP + TN).
- Recognize that False Positive Rate is also equal to 1 minus Specificity (TN / (TN + FP)).
- Generate predicted probabilities for the test set using methods like predict_proba.
- Calculate TPR and FPR for every possible probability threshold to illustrate the trade-off between sensitivity and specificity.
- Plot FPR on the x-axis and TPR on the y-axis to create the ROC curve.
Worked example
A medical test has 90 True Positives, 10 False Negatives, 19 False Positives, and 81 True Negatives. Calculate the TPR and FPR.
- Identify TP=90, FN=10, FP=19, TN=81.
- Calculate TPR: 90 / (90 + 10) = 90 / 100 = 0.90.
- Calculate FPR: 19 / (19 + 81) = 19 / 100 = 0.19.
Answer TPR = 0.90, FPR = 0.19
Common mistakes
- Confusing the ROC curve with a precision-recall curve; the ROC plots TPR and FPR, not precision and recall.
- Assuming that a higher threshold always improves the model; while it may reduce false positives, it also reduces true positives, and the optimal threshold depends on the specific context.
- Thinking that the diagonal line represents a bad model; it actually represents a random classifier, which is the baseline for no predictive power.