Task-category score

MedHELM: 121 tasks, 22 subcategories (decision support 0.56-0.72, note generation 0.73-0.85, patient comms 0.78-0.83 in the preprint; HAI article gives slightly different ranges).

Who judgesAutomatable
EvidenceGrade E1/E2
Product typeClinical Q&AID D2

What task-category score means

MedHELM: 121 tasks, 22 subcategories (decision support 0.56-0.72, note generation 0.73-0.85, patient comms 0.78-0.83 in the preprint; HAI article gives slightly different ranges).

Why it matters. Task categories show where a model is strong or weak, instead of one overall number.

Example, illustrative

Decision support scores differ from note generation in the same benchmark.

How to measure it

01

Write real questions

Specialists write questions from real decisions, including hard ones.

02

Name the reference

Fix the guideline version each answer will be judged against.

03

Grade with rubrics

Score answers on clinician-written criteria.

04

Compare

Run the same questions through a general frontier model.

Watch out for

Comparing a tool only with its earlier version flatters it.

What a good report shows

Scores with named guideline versions, a general-model baseline and clinician agreement.

Report the question set, the specialty mix, who graded the answers and against which guideline version, and compare against a general frontier model, not only against an earlier version of the same tool.

Reviewed by the Precision research team.

How strong is the evidence

E1 · Peer-reviewed, regulator, audit or court source. Backed by a peer-reviewed paper, a regulator, an audit or a court record. Usually read through a summary, not the full text.

What the sources say

(arXiv 2505.23802; Stanford HAI) (conflict flagged).

  • Automatable. Can be scored by software, including an AI grader that is spot-checked against clinicians.
  1. arXiv 2505.23802arxiv.org
  2. Stanford HAIhai.stanford.edu

Found something out of date? Write to hello@precision.example.

Quick answers

What is task-category score in healthcare AI?

MedHELM: 121 tasks, 22 subcategories (decision support 0.56-0.72, note generation 0.73-0.85, patient comms 0.78-0.83 in the preprint; HAI article gives slightly different ranges). It sits in the clinical question answering and decision support group.

Why does task-category score matter?

Task categories show where a model is strong or weak, instead of one overall number. Decision support scores differ from note generation in the same benchmark.

How is task-category score measured?

Write real questions, then Name the reference, then Grade with rubrics, then Compare. Report the question set, the specialty mix, who graded the answers and against which guideline version, and compare against a general frontier model, not only against an earlier version of the same tool.

What should I watch out for with task-category score?

Comparing a tool only with its earlier version flatters it. A good report gives: scores with named guideline versions, a general-model baseline and clinician agreement.

Can task-category score be automated?

Largely yes, including AI graders that are spot-checked against clinicians. Report the grader's error rate beside every score.

Measure task-category score on your own data