AI, machine learning and deep learning · Intermediate

Calibration

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Calibration measures the alignment between predicted probabilities and actual observed frequencies, defined as the ratio of the sum of predictions to the sum of labels. It is critical for systems where absolute probability values matter, such as medical diagnosis, and can be visualized using calibration plots to identify over- or under-confidence.

Intuition

Think of calibration as a reality check for your model's confidence. If a model predicts a 20% chance of an event, a perfectly calibrated model means that in the long run, that event actually happens 20% of the time. This is distinct from accuracy or discrimination; a model can be very good at ranking items (high AUC) but still be poorly calibrated, meaning its probability scores are systematically too high or too low. For example, in medical diagnosis, knowing the true likelihood of a disease is vital, so understanding whether the model's output reflects reality is more important than just knowing it can separate sick from healthy patients.

Key equations

Calibration=∑predictions∑labels
(1)

The ratio of the sum of predicted probabilities to the sum of actual positive labels (empirical CTR).

Derivation

  1. Identify the set of predictions made by the model for a specific group or bin.
  2. Identify the set of actual positive labels (ground truth) for the same group.
  3. Sum all predicted probabilities to get the expected number of positive events.
  4. Sum all actual labels to get the observed number of positive events.
  5. Divide the sum of predictions by the sum of labels to calculate the calibration score.
  6. Interpret the result: a value of 1 indicates perfect calibration, >1 indicates over-confidence, and <1 indicates under-confidence.

Worked example

Problem

A model predicts click-through rates for 5 users. The predicted probabilities are [0.8, 0.2, 0.6, 0.4, 0.0]. The actual clicks (labels) are [1, 0, 1, 0, 0]. Calculate the calibration.

  1. Sum the predictions: 0.8 + 0.2 + 0.6 + 0.4 + 0.0 = 2.0
  2. Sum the labels: 1 + 0 + 1 + 0 + 0 = 2
  3. Calculate calibration: 2.0 / 2 = 1.0

Answer The calibration is 1.0, indicating perfect calibration for this small sample.

Common mistakes

  • Assuming a model with a higher AUC (like SVM) is always better than a model with lower AUC (like logistic regression) without checking calibration.
  • Ignoring the prediction distribution and relying solely on aggregate metrics like AUC.
  • Failing to account for selection bias when resampling training data.
  • Not aligning the training label with the calibration label.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

This is a text lesson: it has no animation code yet.

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.