Why can't you compare AI scribe error rates between studies?

Published scribe error rates run from 1.47% of sentences to 31% of notes. Unit, sampling and review instructions change the answer: what that means for buyers.

What do published scribe error rates say?

Study Reported rate Unit
CREOLA (npj Digital Medicine) 1.47% hallucination, 3.45% omission Sentences
UC Davis pilot (JMIR, 356 notes) 18% omissions, 11.5% hallucinations, 9.3% accidental inclusions, 1.1% bias Notes
Palm et al. (Frontiers 2025) 31% of ambient notes with hallucinations vs 20% of human notes Notes
565-note census (arXiv 2608.31017) 31.3% of notes with a verified failure Notes

Why can’t these be compared?

The unit differs: a rate per sentence is not a rate per note. Sampling differs, severity thresholds differ, and reviewers were told different things. The census preprint reports that changing only the review instruction moved the share of candidate errors that verified from 9.3% to 79.0%. That is not a note-level error rate, but it shows how much the instrument matters.

Does severity change the picture?

Yes. CREOLA grades errors as major or minor, with major meaning the error could change diagnosis or management if uncorrected. Counting every minor wording difference gives very different numbers than counting only those that could harm.

What should a buyer ask for?

Ask for the unit, how notes were sampled, the review instruction, the severity scale and how often an automated grader agreed with clinicians compared with how often clinicians agreed with each other. See the instrument sensitivity and hallucination rate parameters.

What did we not verify?

We read the UC Davis, CREOLA and Palm studies through search extracts, not full text. The census is a non-peer-reviewed preprint.

Frequently asked questions

What is the error rate of AI scribes?

There is no single defensible figure. Studies report 1.47% hallucination and 3.45% omission of sentences (CREOLA), omissions in 18% and hallucinations in 11.5% of notes (UC Davis), 31% of ambient notes with hallucinations (Palm), and 31.3% of notes with a verified failure (a preprint census).

Why do the numbers differ so much?

The unit differs (per sentence, per note, per patient), as do sampling, severity thresholds and the instruction given to reviewers. In one preprint, changing only the review instruction moved the share of candidate errors that verified from 9.3% to 79.0%.

How should a buyer compare scribes?

Run every vendor on the same encounters, with the same reviewers and review instruction, and report the unit, sampling method, severity scale and grader-versus-clinician agreement next to each number.

Written byPrecision research teamHealthcare AI evaluation research