GLOSSARY
What is expected calibration error (ECE)?
Calibration error (ECE)Expected calibration error (ECE) measures how far a model’s confidence is from its accuracy: answers are grouped by confidence, and the gap between average confidence and accuracy in each group is averaged, weighted by group size. Lower is better; 0 is perfect.
An ECE of 0.04 means that, on average, the stated confidence is four percentage points away from how often the model is right. The registry records it as calibration_error in systemone.yaml.
ECE depends on the number of bins and on the test set, so compare two models’ figures only when both were measured the same way.
Read more
Next term: Decision accuracy · Every System One model