Calibration
Predicted vs observed risk.
What calibration means
Predicted vs observed risk.
Why it matters. A model can rank well and still give wrong probabilities.
Predicted 20% risk, observed 35%.
How to measure it
Use local data
Validate on your own recent patients.
Check discrimination and calibration
Both, not just AUROC.
Measure alert burden
Positive predictive value and alerts per day.
Monitor over time
Track subgroups and drift after go-live.
A strong AUROC can coexist with poor calibration and heavy alert fatigue.
Local discrimination, calibration, PPV at your prevalence, and drift tracked over time.
Report discrimination and calibration on local data, the alert rate, positive predictive value, and drift over time.
Reviewed by the Precision research team.
How strong is the evidence
E3 · Secondary or single-source report. Reported by trade press or a single source. Directional only until checked against the primary document.
What the sources say
Wong 2021 critique (secondary) (2 Minute Medicine).
- Automatable. Can be scored by software, including an AI grader that is spot-checked against clinicians.
- 2 Minute Medicine2minutemedicine.com
Found something out of date? Write to hello@precision.example.
Agents that measure this
Quick answers
What is calibration in healthcare AI?
Predicted vs observed risk. It sits in the predictive models group.
Why does calibration matter?
A model can rank well and still give wrong probabilities. Predicted 20% risk, observed 35%.
How is calibration measured?
Use local data, then Check discrimination and calibration, then Measure alert burden, then Monitor over time. Report discrimination and calibration on local data, the alert rate, positive predictive value, and drift over time.
What should I watch out for with calibration?
A strong AUROC can coexist with poor calibration and heavy alert fatigue. A good report gives: local discrimination, calibration, PPV at your prevalence, and drift tracked over time.
Can calibration be automated?
Largely yes, including AI graders that are spot-checked against clinicians. Report the grader's error rate beside every score.