Calibration

Calibration measures whether predicted probabilities match observed frequencies — when a model says 80% confident, it should be right about 80% of the time. A model can be accurate yet badly calibrated, which misleads any downstream decision that uses its confidence.

Perfect calibration means:

$$\mathbb{P}(Y=\hat{Y}\mid \hat{P}=p)=p$$

More generally, for a predicted probability $\hat{p}(x)$:

$$\mathbb{P}(Y=1\mid \hat{p}(X)=p)=p$$

for all confidence levels $p$ where enough data exists.

Expected calibration error estimates the average gap between confidence and empirical accuracy:

$$\mathrm{ECE}=\sum _{b=1}^B\frac{|B_b|}{n}\left|\mathrm{acc}(B_b)-\mathrm{conf}(B_b)\right|$$

Calibration complements probability metrics such as NLL and Brier score. NLL and Brier score summarize probability quality as scalar metrics; calibration diagnostics show where confidence and empirical frequency disagree.

Discrimination vs Calibration

Property	Question	Example Metric
discrimination	can the model rank positives above negatives?	AUROC, PR-AUC, enrichment
calibration	do predicted probabilities match frequencies?	ECE, reliability diagram
sharpness	are probabilities confident when justified?	entropy, Brier decomposition
decision quality	do probabilities improve actions under a cost?	expected utility, selective risk

A ranking model can have excellent AUROC and still be poorly calibrated. A calibrated model can still be useless if it cannot separate cases.

Binning Caveat

ECE depends on bins:

$$\mathrm{acc}(B_b)=\frac{1}{|B_b|}\sum _{i\in B_b}1[{\hat{y}}_i=y_i],\mathrm{conf}(B_b)=\frac{1}{|B_b|}\sum _{i\in B_b}{\hat{p}}_i$$

Small bins, class imbalance, and distribution shift can make ECE unstable. Use reliability diagrams and per-slice checks when decisions depend on confidence.

Practical Checks

• Plot a reliability diagram; bin predictions and compare to empirical accuracy.
• Report expected calibration error (ECE) alongside accuracy.
• Calibrate on held-out data (temperature scaling, isotonic) — never on the test set.
• Re-check calibration under distribution shift; it degrades faster than accuracy.
• For ranking tasks, separate calibration from discrimination (AUC).
• For abstention or human review, evaluate selective prediction instead of reporting confidence alone.
• Check whether calibration is evaluated on the same population where decisions will be made.
• For imbalanced datasets, inspect calibration by class, target, scaffold, subgroup, or score range.

Temperature Scaling

For classification logits $z$, temperature scaling uses:

p_T(y=k\mid x) = \frac{\exp(z_k/T)} \sum_{\ell=1}^{K}\exp(z_\ell/T)}

where $T>0$ is fit on validation data. Larger $T$ softens overconfident probabilities; smaller $T$ sharpens them.

What Calibration Is Not

• High accuracy does not imply calibration.
• High AUC does not imply calibrated probabilities.
• A model can be calibrated on IID data and miscalibrated under OOD shift.
• Calibrating on the test set leaks evaluation information.

Related

• Probability distribution
• Expectation
• Evaluation protocol
• Probability metrics
• Proper scoring rule
• Brier score
• Reliability diagram
• Uncertainty estimation
• Selective prediction
• Objective-metric alignment
• OOD generalization
• Leakage
• Evaluation