Calibration

Predicted vs observed.

Who judgesAutomatable
EvidenceGrade E4
Product typeImagingID I3

What calibration means

Predicted vs observed.

Why it matters. Predicted risk should match what happens.

Example, illustrative

Patients scored 30% risk have about 30% events.

How to measure it

01

Gather local scans

Use scans from your own scanners and protocols.

02

Adjudicate labels

Radiologists agree the reference reading.

03

Test across sites and groups

Check scanners, sites and patient groups separately.

04

Check calibration

Compare predicted and observed rates.

Watch out for

Performance at the vendor's test site often doesn't hold at yours.

What a good report shows

Local sensitivity and specificity with subgroup tables, external-site results and calibration.

Report local performance with radiologist-adjudicated labels, external sites, subgroup results and calibration, against a cleared comparator.

Reviewed by the Precision research team.

How strong is the evidence

E4 · Recalled, not verified. Recalled from background knowledge and not verified. Treat it as a lead for investigation, not a finding.

What the sources say

E4.

  • Automatable. Can be scored by software, including an AI grader that is spot-checked against clinicians.

Found something out of date? Write to hello@precision.example.

Quick answers

What is calibration in healthcare AI?

Predicted vs observed. It sits in the imaging and diagnostics group.

Why does calibration matter?

Predicted risk should match what happens. Patients scored 30% risk have about 30% events.

How is calibration measured?

Gather local scans, then Adjudicate labels, then Test across sites and groups, then Check calibration. Report local performance with radiologist-adjudicated labels, external sites, subgroup results and calibration, against a cleared comparator.

What should I watch out for with calibration?

Performance at the vendor's test site often doesn't hold at yours. A good report gives: local sensitivity and specificity with subgroup tables, external-site results and calibration.

Can calibration be automated?

Largely yes, including AI graders that are spot-checked against clinicians. Report the grader's error rate beside every score.

Measure calibration on your own data