Rubric accuracy and completeness (clinician-facing)

HealthBench Professional: 525 tasks from a 15,079 pool, about one-third adversarial, rubrics adjudicated by three or more physicians.

Who judgesAutomatable and clinician judgement
EvidenceGrade E2
Product typeClinical Q&AID D1

What rubric accuracy and completeness (clinician-facing) means

HealthBench Professional: 525 tasks from a 15,079 pool, about one-third adversarial, rubrics adjudicated by three or more physicians.

Why it matters. Open-ended clinical answers need scoring on what a clinician would reward or penalise.

Example, illustrative

An answer earns points for correct dosing and loses points for omitting a contraindication.

How to measure it

01

Write real questions

Specialists write questions from real decisions, including hard ones.

02

Name the reference

Fix the guideline version each answer will be judged against.

03

Grade with rubrics

Score answers on clinician-written criteria.

04

Compare

Run the same questions through a general frontier model.

Watch out for

Comparing a tool only with its earlier version flatters it.

What a good report shows

Scores with named guideline versions, a general-model baseline and clinician agreement.

Report the question set, the specialty mix, who graded the answers and against which guideline version, and compare against a general frontier model, not only against an earlier version of the same tool.

Reviewed by the Precision research team.

How strong is the evidence

E2 · Preprint, benchmark page or vendor-run result. Backed by numbers from a preprint, a benchmark page or a vendor-run test. Treat as strong but not settled. At least one figure is a vendor claim.

What the sources say

OpenAI, April 2026 (arXiv 2604.27470); Epoch AI review verdict "Flawed" because vendor-built and vendor-run (Epoch).

  • Automatable. Can be scored by software, including an AI grader that is spot-checked against clinicians.
  • Clinician judgement. Needs a clinician to judge or label, because the answer depends on clinical meaning.
  1. arXiv 2604.27470arxiv.org
  2. Epochepoch.ai

Found something out of date? Write to hello@precision.example.

Quick answers

What is rubric accuracy and completeness (clinician-facing) in healthcare AI?

HealthBench Professional: 525 tasks from a 15,079 pool, about one-third adversarial, rubrics adjudicated by three or more physicians. It sits in the clinical question answering and decision support group.

Why does rubric accuracy and completeness (clinician-facing) matter?

Open-ended clinical answers need scoring on what a clinician would reward or penalise. An answer earns points for correct dosing and loses points for omitting a contraindication.

How is rubric accuracy and completeness (clinician-facing) measured?

Write real questions, then Name the reference, then Grade with rubrics, then Compare. Report the question set, the specialty mix, who graded the answers and against which guideline version, and compare against a general frontier model, not only against an earlier version of the same tool.

What should I watch out for with rubric accuracy and completeness (clinician-facing)?

Comparing a tool only with its earlier version flatters it. A good report gives: scores with named guideline versions, a general-model baseline and clinician agreement.

Can rubric accuracy and completeness (clinician-facing) be automated?

Only in part. It needs automatable and clinician judgement. Software can screen, but a clinician must judge the cases that matter.

Measure rubric accuracy and completeness (clinician-facing) on your own data