Research
A Ranking Approach for Measuring Calibration
arXiv:2609.13100v1 Announce Type: cross Abstract: When providing forecasted probabilities with a predictive model, the ideal model offers perfect calibration: the true probability of the outcome (i.e.
arXiv:2609.13100v1 Announce Type: cross Abstract: When providing forecasted probabilities with a predictive model, the ideal model offers perfect calibration: the true probability of the outcome (i.e., the probability that Y=1) exactly matches the forecasted probability f(X). In practice, models inevitably exhibit calibration error, and it is therefore important to be able to measure this miscalibration to assess a model's reliability. The Expected Calibration Error (ECE) is the most widely used measure of miscalibration, but is known to be impossible to estimate the ECE with guaranteed accuracy in an assumption-free setting. In this work, we propose an alternative measure, the rankECE, that is based on comparing points with neighboring values of the predicted probability f(X). Our theoretical guarantees and empirical results establish that rankECE provides a better proxy for ECE as compared to binned approximations to ECE, which are the most commonly-used approximations in practice.
Related
- Reading Calibrated Uncertainty from Language Model Trajectories
- Adaptive Cumulative Mass Calibration with Conformal Prediction
- Enforcing tail calibration when training probabilistic forecast models
- Truthful Calibration Measures for Sequential Prediction
Source: arXiv cs.LG | 2026-09-14