Content
Summary
Calibration measures the alignment between predicted probabilities and actual observed frequencies, defined as the ratio of the sum of predictions to the sum of labels. It is critical for systems where absolute probability values matter, such as medical diagnosis, and can be visualized using calibration plots to identify over- or under-confidence.
Intuition
Think of calibration as a reality check for your model's confidence. If a model predicts a 20% chance of an event, a perfectly calibrated model means that in the long run, that event actually happens 20% of the time. This is distinct from accuracy or discrimination; a model can be very good at ranking items (high AUC) but still be poorly calibrated, meaning its probability scores are systematically too high or too low. For example, in medical diagnosis, knowing the true likelihood of a disease is vital, so understanding whether the model's output reflects reality is more important than just knowing it can separate sick from healthy patients.
Key equations
The ratio of the sum of predicted probabilities to the sum of actual positive labels (empirical CTR).
Derivation
- Identify the set of predictions made by the model for a specific group or bin.
- Identify the set of actual positive labels (ground truth) for the same group.
- Sum all predicted probabilities to get the expected number of positive events.
- Sum all actual labels to get the observed number of positive events.
- Divide the sum of predictions by the sum of labels to calculate the calibration score.
- Interpret the result: a value of 1 indicates perfect calibration, >1 indicates over-confidence, and <1 indicates under-confidence.
Worked example
A model predicts click-through rates for 5 users. The predicted probabilities are [0.8, 0.2, 0.6, 0.4, 0.0]. The actual clicks (labels) are [1, 0, 1, 0, 0]. Calculate the calibration.
- Sum the predictions: 0.8 + 0.2 + 0.6 + 0.4 + 0.0 = 2.0
- Sum the labels: 1 + 0 + 1 + 0 + 0 = 2
- Calculate calibration: 2.0 / 2 = 1.0
Answer The calibration is 1.0, indicating perfect calibration for this small sample.
Common mistakes
- Assuming a model with a higher AUC (like SVM) is always better than a model with lower AUC (like logistic regression) without checking calibration.
- Ignoring the prediction distribution and relying solely on aggregate metrics like AUC.
- Failing to account for selection bias when resampling training data.
- Not aligning the training label with the calibration label.