Instrument sensitivity (meta)

How much the measured rate moves with reviewer instruction or reviewing model.

Who judgesAutomatable
EvidenceGrade E2
Product typeCross-cuttingID X26

What instrument sensitivity (meta) means

How much the measured rate moves with reviewer instruction or reviewing model.

Why it matters. A measured rate can move depending on reviewer instructions or the grading model. This shows how fragile a number is.

Example, illustrative

The same notes score 12% omissions under one instruction and 21% under another.

How to measure it

01

Define the unit

Decide whether you count per sentence, per note, per patient or per call.

02

Sample on purpose

Pick cases that cover specialties, languages and rare events, not only easy ones.

03

Judge with clinicians

Clinicians label a sample. Graders are then checked against those labels.

04

Report the margin

Show an interval, and the grader's own error rate beside the score.

Watch out for

A single average hides rare, serious errors and weaker subgroups.

What a good report shows

A number with a clear unit, a stated sample, a margin and the grader's agreement with clinicians.

Report the unit (per sentence, per note, per patient), the sampling method, the severity scale, and how often the automated grader agreed with clinicians compared with how often clinicians agreed with each other.

Reviewed by the Precision research team.

How strong is the evidence

E2 · Preprint, benchmark page or vendor-run result. Backed by numbers from a preprint, a benchmark page or a vendor-run test. Treat as strong but not settled. At least one source is an unreviewed preprint.

What the sources say

9.3% to 79.0% from instruction change alone (summary) (unreviewed).

  • Automatable. Can be scored by software, including an AI grader that is spot-checked against clinicians.
  1. summaryblog.pebblous.ai

Found something out of date? Write to hello@precision.example.

Quick answers

What is instrument sensitivity (meta) in healthcare AI?

How much the measured rate moves with reviewer instruction or reviewing model. It sits in the every product type group.

Why does instrument sensitivity (meta) matter?

A measured rate can move depending on reviewer instructions or the grading model. This shows how fragile a number is. The same notes score 12% omissions under one instruction and 21% under another.

How is instrument sensitivity (meta) measured?

Define the unit, then Sample on purpose, then Judge with clinicians, then Report the margin. Report the unit (per sentence, per note, per patient), the sampling method, the severity scale, and how often the automated grader agreed with clinicians compared with how often clinicians agreed with each other.

What should I watch out for with instrument sensitivity (meta)?

A single average hides rare, serious errors and weaker subgroups. A good report gives: a number with a clear unit, a stated sample, a margin and the grader's agreement with clinicians.

Can instrument sensitivity (meta) be automated?

Largely yes, including AI graders that are spot-checked against clinicians. Report the grader's error rate beside every score.

Measure instrument sensitivity (meta) on your own data