Concept dictionary
Calibration
Calibration describes how well stated confidence matches observed correctness across comparable cases. A well-calibrated system that assigns 80 percent confidence should be correct about 80 percent of the time on cases receiving that confidence.
- Review state
- Reviewed in POC
- Freshness
- Current
- Version
- 2
Also called
- calibration
Remember it this wayConfidence should match the record.
In plain language
Calibration asks whether confidence has earned its meaning. It compares what a person or system says about uncertainty with what happens over many relevant examples.
Why it matters
People often treat confident language as evidence of reliability. Calibration provides a record-based check. It can reveal overconfidence, underconfidence, and conditions where a confidence signal should not guide a decision.
Example
A classifier labels 100 comparable cases with 90 percent confidence, but only 65 labels are correct. The classifier may still be useful, but its confidence is poorly calibrated for those cases.
Common misconception
Calibration is not the same as accuracy, and it is not the same as the simpler question of whether one answer is correct. Confidence vs. Correctness separates two properties of a single answer. Calibration measures whether confidence levels match correctness rates across a defined set of comparable cases. A system can therefore be accurate overall and still be poorly calibrated.
Related terms
Sources
- Chuan Guo and colleagues, “On Calibration of Modern Neural Networks”calibration as agreement between predicted confidence and empirical correctness, and its distinction from accuracy.