Judge-versus-clinician agreement (meta)

Kappa or F1 of the automated judge against clinicians, relative to clinician-versus-clinician ceiling, with CIs.

Who judgesClinician judgement
EvidenceGrade E2
Product typeCross-cuttingID X25

What judge-versus-clinician agreement (meta) means

Kappa or F1 of the automated judge against clinicians, relative to clinician-versus-clinician ceiling, with CIs.

Why it matters. If the AI grader disagrees with clinicians, its scores can't be trusted. This measures that agreement.

Example, illustrative

The grader matches clinicians 91% of the time, while two clinicians match each other 93%.

How to measure it

01

Define the unit

Decide whether you count per sentence, per note, per patient or per call.

02

Sample on purpose

Pick cases that cover specialties, languages and rare events, not only easy ones.

03

Judge with clinicians

Clinicians label a sample. Graders are then checked against those labels.

04

Report the margin

Show an interval, and the grader's own error rate beside the score.

Watch out for

A single average hides rare, serious errors and weaker subgroups.

What a good report shows

A number with a clear unit, a stated sample, a margin and the grader's agreement with clinicians.

Report the unit (per sentence, per note, per patient), the sampling method, the severity scale, and how often the automated grader agreed with clinicians compared with how often clinicians agreed with each other.

Reviewed by the Precision research team.

How strong is the evidence

E2 · Preprint, benchmark page or vendor-run result. Backed by numbers from a preprint, a benchmark page or a vendor-run test. Treat as strong but not settled.

What the sources say

HealthBench: physician-physician and model-physician agreement both 55-75%; GPT-4.1 grader macro-F1 0.709 (arXiv).

  • Clinician judgement. Needs a clinician to judge or label, because the answer depends on clinical meaning.
  1. arXivarxiv.org

Found something out of date? Write to hello@precision.example.

Quick answers

What is judge-versus-clinician agreement (meta) in healthcare AI?

Kappa or F1 of the automated judge against clinicians, relative to clinician-versus-clinician ceiling, with CIs. It sits in the every product type group.

Why does judge-versus-clinician agreement (meta) matter?

If the AI grader disagrees with clinicians, its scores can't be trusted. This measures that agreement. The grader matches clinicians 91% of the time, while two clinicians match each other 93%.

How is judge-versus-clinician agreement (meta) measured?

Define the unit, then Sample on purpose, then Judge with clinicians, then Report the margin. Report the unit (per sentence, per note, per patient), the sampling method, the severity scale, and how often the automated grader agreed with clinicians compared with how often clinicians agreed with each other.

What should I watch out for with judge-versus-clinician agreement (meta)?

A single average hides rare, serious errors and weaker subgroups. A good report gives: a number with a clear unit, a stated sample, a margin and the grader's agreement with clinicians.

Can judge-versus-clinician agreement (meta) be automated?

Only in part. It needs clinician judgement. Software can screen, but a clinician must judge the cases that matter.

Measure judge-versus-clinician agreement (meta) on your own data