Clinical reviewer (rubric grader)
Grades answers against rubrics your clinicians wrote.
Checked against clinicians
Macro-F1 or κ vs clinicians per theme and specialty.
Where a person decides
Screening only; sampled clinician audit.
Works on
Q&A, chat, notes, agents.
Delivered inside a Quality Audit today. Reviewed .
The problem it solves
Reading thousands of outputs isn't possible for clinicians. An automated grader is needed, but it has to follow your criteria and be checked.
How it works
Read the rubric
Each criterion is a clear, checkable statement with a weight.
Grade each output
The agent scores every criterion and gives a reason.
Report its own accuracy
Scores are shown next to how often the grader agrees with clinicians.
An example, illustrative
Free-text answer with no score.
74 of 100 points, with a reason beside each missed criterion.
A clinician-written rubric and a labelled sample to check against.
It scores against your rubric. It doesn't decide what the rubric should be.
What it helps measure
Quick answers
What does the clinical reviewer do?
Grade outputs against clinician-authored rubrics.
How is the clinical reviewer checked against clinicians?
Macro-F1 or κ vs clinicians per theme and specialty.
Is an AI grader trustworthy?
Only as far as it's been tested. We report agreement with clinicians, relative to how often clinicians agree with each other.
Where does this approach come from?
Rubric-based grading is used in public benchmarks such as HealthBench, which used physician-written criteria.
What can the clinical reviewer not do?
Screening only; sampled clinician audit. No agent makes a clinical decision.