Clinical reviewer (rubric grader)

Grades answers against rubrics your clinicians wrote.

In audits Layer 2 · Measure
Takes inOutput, source, rubric
The agentClinical reviewer
Hands backCriterion scores with rationales

Checked against clinicians

Macro-F1 or κ vs clinicians per theme and specialty.

Where a person decides

Screening only; sampled clinician audit.

Works on

Q&A, chat, notes, agents.

Delivered inside a Quality Audit today. Reviewed .

The problem it solves

Reading thousands of outputs isn't possible for clinicians. An automated grader is needed, but it has to follow your criteria and be checked.

How it works

01

Read the rubric

Each criterion is a clear, checkable statement with a weight.

02

Grade each output

The agent scores every criterion and gives a reason.

03

Report its own accuracy

Scores are shown next to how often the grader agrees with clinicians.

An example, illustrative

Before

Free-text answer with no score.

With the agent

74 of 100 points, with a reason beside each missed criterion.

What it needs from you

A clinician-written rubric and a labelled sample to check against.

Its limits

It scores against your rubric. It doesn't decide what the rubric should be.

Quick answers

What does the clinical reviewer do?

Grade outputs against clinician-authored rubrics.

How is the clinical reviewer checked against clinicians?

Macro-F1 or κ vs clinicians per theme and specialty.

Is an AI grader trustworthy?

Only as far as it's been tested. We report agreement with clinicians, relative to how often clinicians agree with each other.

Where does this approach come from?

Rubric-based grading is used in public benchmarks such as HealthBench, which used physician-written criteria.

What can the clinical reviewer not do?

Screening only; sampled clinician audit. No agent makes a clinical decision.

Use the clinical reviewer on your product