Content
Summary
This lesson introduces the confusion matrix and key metrics like precision, recall, and specificity for evaluating binary classification models. It explains how probability cutoffs influence these metrics and introduces the ROC curve and AUC for assessing model performance across thresholds. Finally, it addresses the challenges of imbalanced data using Precision-Recall curves and lift charts.
Intuition
Evaluating a classifier is not just about counting correct guesses; it is about understanding the trade-offs between different types of errors. The confusion matrix provides the foundational counts (TP, TN, FP, FN) from which all other metrics are derived. Accuracy is often misleading in imbalanced datasets, so we rely on sensitivity (recall) and specificity to understand how well the model identifies positives and negatives respectively. Precision tells us about the reliability of a positive prediction. Since models output probabilities, we can adjust the decision threshold (cutoff) to prioritize sensitivity or specificity. The ROC curve visualizes this trade-off, and the AUC summarizes it into a single number. For rare events, Precision-Recall curves and lift charts offer more insightful evaluations than ROC/AUC.
Key equations
Proportion of predicted positives that are actually positive.
Proportion of actual positives that are correctly identified.
Proportion of actual negatives that are correctly identified.
Derivation
- Start with the confusion matrix, a 2x2 table of predicted vs. actual outcomes.
- Identify True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN).
- Calculate Precision as TP divided by the sum of TP and FP.
- Calculate Recall (Sensitivity) as TP divided by the sum of TP and FN.
- Calculate Specificity as TN divided by the sum of TN and FP.
- Note that Accuracy is the proportion of all cases classified correctly.
- Understand that changing the probability cutoff alters the balance between sensitivity and specificity.
- Plot Sensitivity vs. 1-Specificity (or Specificity) to create the ROC curve.
- Calculate AUC as the area under the ROC curve to summarize performance.
- For imbalanced data, consider Precision-Recall curves and Lift charts instead of ROC/AUC.
Worked example
A model predicts loan defaults. In a test set of 10,000 loans, 100 actually defaulted. The model identified 80 of these as defaults (TP) and incorrectly flagged 90 non-defaults as defaults (FP). Calculate Precision and Recall.
- Identify TP = 80, FP = 90, FN = 100 - 80 = 20.
- Calculate Precision: TP / (TP + FP) = 80 / (80 + 90) = 80 / 170 ≈ 0.47.
- Calculate Recall: TP / (TP + FN) = 80 / (80 + 20) = 80 / 100 = 0.80.
Answer Precision is approximately 0.47 and Recall is 0.80.
Common mistakes
- Confusing the orientation of the confusion matrix (rows vs. columns for true/predicted).
- Using Accuracy as the primary metric for imbalanced datasets.
- Misinterpreting the False Positive Rate (FPR) definition across different sources.
- Assuming a high AUC always indicates a good model for rare class problems.
- Ignoring the impact of probability cutoff adjustments on precision and recall.