Judge calibration and abstention agent
Shows how often the AI grader itself is wrong.
Checked against clinicians
Self-evidently the validator; re-run on every judge change.
Where a person decides
Cannot ship a metric without a calibration record.
Works on
All.
Delivered inside a Quality Audit today. Reviewed .
The problem it solves
AI graders make mistakes too. A score without the grader's error rate can look precise and be wrong.
How it works
Compare with clinicians
The agent measures how often the grader agrees with clinician labels.
Add error ranges
Every score gets a confidence interval that includes the grader's own error.
Route uncertain cases
Cases where the grader is unsure go to a clinician instead of getting a guess.
An example, illustrative
Omission rate: 12%.
Omission rate: 12% (95% interval 8 to 17%). Grader agrees with clinicians 91% of the time.
A clinician-labelled sample, refreshed when the grader changes.
It can only be as good as the labelled sample. It tells you when the sample is too small.
What it helps measure
Quick answers
What does the judge calibrator do?
Maintain judge error rates, intervals, drift in judge; route uncertain cases to clinicians.
How is the judge calibrator checked against clinicians?
Self-evidently the validator; re-run on every judge change.
What is abstention?
The grader declines to score cases it isn't sure about and sends them to a person.
How often is it re-checked?
Whenever the grader's model, prompt or your workflow changes.
What can the judge calibrator not do?
Cannot ship a metric without a calibration record. No agent makes a clinical decision.