GLOSSARY
What is calibrated confidence?
Calibrated confidenceA model’s confidence is calibrated when its probabilities match how often it is right: of all the answers it gives with 0.9 confidence, about 90% are correct. Calibration is what makes a threshold on the probability mean something.
A model can be accurate and still overconfident, and overconfidence breaks every rule built on its probabilities: the 0.95 you gate on might be right 70% of the time. That is why System One models report calibration error next to accuracy.
Calibration can be repaired after training by fitting a temperature on held-out data. Laya Studio refits it per question type after every fine-tuning run.
Read more
Next term: Calibration error (ECE) · Every System One model