Judge calibration and abstention agent

Shows how often the AI grader itself is wrong.

In audits Layer 2 · Measure
Takes inGold set, judge outputs
The agentJudge calibrator
Hands backCalibration report, abstain list

Checked against clinicians

Self-evidently the validator; re-run on every judge change.

Where a person decides

Cannot ship a metric without a calibration record.

Works on

All.

Delivered inside a Quality Audit today. Reviewed .

The problem it solves

AI graders make mistakes too. A score without the grader's error rate can look precise and be wrong.

How it works

01

Compare with clinicians

The agent measures how often the grader agrees with clinician labels.

02

Add error ranges

Every score gets a confidence interval that includes the grader's own error.

03

Route uncertain cases

Cases where the grader is unsure go to a clinician instead of getting a guess.

An example, illustrative

Before

Omission rate: 12%.

With the agent

Omission rate: 12% (95% interval 8 to 17%). Grader agrees with clinicians 91% of the time.

What it needs from you

A clinician-labelled sample, refreshed when the grader changes.

Its limits

It can only be as good as the labelled sample. It tells you when the sample is too small.

Quick answers

What does the judge calibrator do?

Maintain judge error rates, intervals, drift in judge; route uncertain cases to clinicians.

How is the judge calibrator checked against clinicians?

Self-evidently the validator; re-run on every judge change.

What is abstention?

The grader declines to score cases it isn't sure about and sends them to a person.

How often is it re-checked?

Whenever the grader's model, prompt or your workflow changes.

What can the judge calibrator not do?

Cannot ship a metric without a calibration record. No agent makes a clinical decision.

Use the judge calibrator on your product