Hallucination rate

Share of sentences or notes with unsupported content; sentence-level clinician annotation against transcript.

Who judgesClinician judgement and automatable
EvidenceGrade E1/E2
Product typeAmbient scribesID S1

What hallucination rate means

Share of sentences or notes with unsupported content; sentence-level clinician annotation against transcript.

Why it matters. Invented content in a note enters the medical record and can mislead later care.

Example, illustrative

A note records a normal chest exam that never took place.

How to measure it

01

Collect visits

Use recorded or simulated visits with the source transcript.

02

Compare note to source

Check every fact in the visit against the note, and every claim in the note against the visit.

03

Grade severity

Clinicians rank each error by the harm it could do.

04

Stratify

Split results by specialty, accent and visit type.

Watch out for

Rates from different studies use different definitions and aren't comparable.

What a good report shows

A per-note error rate with severity bands, specialty splits and a named review instruction.

Report whether the rate is per sentence or per note, how notes were sampled, which review instruction clinicians were given, and the severity of each error. Rates from different studies are not comparable.

Reviewed by the Precision research team.

How strong is the evidence

E1 · Peer-reviewed, regulator, audit or court source. Backed by a peer-reviewed paper, a regulator, an audit or a court record. Usually read through a summary, not the full text.

What the sources say

CREOLA 1.47% of sentences; JMIR 11.5% of notes; Palm 31% of ambient vs 20% of human notes (Frontiers) (units differ).

  • Clinician judgement. Needs a clinician to judge or label, because the answer depends on clinical meaning.
  • Automatable. Can be scored by software, including an AI grader that is spot-checked against clinicians.
  1. Frontiersfrontiersin.org

Found something out of date? Write to hello@precision.example.

Quick answers

What is hallucination rate in healthcare AI?

Share of sentences or notes with unsupported content; sentence-level clinician annotation against transcript. It sits in the ambient scribes and clinical documentation group.

Why does hallucination rate matter?

Invented content in a note enters the medical record and can mislead later care. A note records a normal chest exam that never took place.

How is hallucination rate measured?

Collect visits, then Compare note to source, then Grade severity, then Stratify. Report whether the rate is per sentence or per note, how notes were sampled, which review instruction clinicians were given, and the severity of each error. Rates from different studies are not comparable.

What should I watch out for with hallucination rate?

Rates from different studies use different definitions and aren't comparable. A good report gives: a per-note error rate with severity bands, specialty splits and a named review instruction.

Can hallucination rate be automated?

Only in part. It needs clinician judgement and automatable. Software can screen, but a clinician must judge the cases that matter.

Measure hallucination rate on your own data