Eight ways healthcare AI fails quietly.

Each one has a source and a fix.

18%of 356 reviewed notes had omissions

Scribes drop what matters

In one hospital pilot, nearly 1 in 5 AI-written notes left something out.

Source · How we measure it

<44%right action, 1,298 people

Exam scores don't survive real patients

People using chatbots picked the right action less than 44% of the time, no better than people without one.

Source · Older models · How we measure it

14–52%best published accuracy at finding the faulty step

No one knows which part broke

Microphone, transcript, prompt or model? Logs show what happened, not why.

Source · Not healthcare data · How we measure it

>95%claimed, no independent audit

Vendors grade themselves

A '95% accurate' claim for AI coding tools traced back to one vendor's own report.

Source · Secondary summary · How we measure it

63%call their approach ad hoc or still developing

Models change while you sleep

Fewer than half of hospitals have a test environment for the AI they buy.

Source · Small sample · How we measure it

9% → 79%from the instruction alone

Error rates can't be compared

Change the reviewer's instruction and the share of errors that hold up swings from 9% to 79%.

Source · Preprint · How we measure it

83%of 256 NHS digital leaders

Buyers ask for proof no one has

Most NHS digital leaders treat clinical safety assurance as a condition of buying.

Source · NHS only · How we measure it

14% → 35%word error rate, English vs Kannada

Quality drops outside English

Same system, same hospital: 14% word errors in English, 35% in Kannada.

Source · One hospital · How we measure it

Pictures show illustrative data. Figures come from the linked sources.

Sources, limits and detail

Scribes drop what matters

Omissions are the most common scribe error and the hardest to catch. In published tests, generic AI graders told complete notes from incomplete ones at close to chance (AUC 0.49 to 0.66).

UC Davis pilot, JMIR Medical Informatics 2026

Exam scores don't survive real patients

Near-identical messages produced opposite advice, and exam-style scores did not predict the failures. A model that passes a licensing exam can still fail a worried patient.

Oxford randomised trial, Nature Medicine, Feb 2026 · Limit: The trial tested older models, not current frontier systems.

No one knows which part broke

A bad note can come from the microphone, the transcript, the prompt, the model or the EHR. Counterfactual replay turns that guess into an experiment.

Failure-attribution research, arXiv 2505.00212 and 2602.23701 · Limit: General agent research. We found no healthcare study that separates speech recognition, prompt and model errors in scribes.

Vendors grade themselves

GAO found few independent studies of AI note and coding tools. Benchmarks are often built and run by the vendor being measured; one was rated 'Flawed' by an independent reviewer for exactly that reason.

GAO-26-109116 (July 2026), via HCC Buddy · Limit: Read through a secondary summary of the GAO report.

Models change while you sleep

Vendors update models on their own schedule. A system validated in March may behave differently in June, and no one can tell drift from variance.

UPMC Center for Connected Medicine and KLAS, via MedCity News, Aug 2026 · Limit: About 25 to 27 leaders surveyed; directional only.

Error rates can't be compared

Published scribe error rates run from 1.47% of sentences to 31% of notes. Unit, sampling and review instruction change the answer, so no single scribe error rate is defensible.

Census of 565 scribe notes, arXiv 2608.31017 · Limit: Unreviewed preprint, by the same authors as a companion paper on omission detection. Not independent corroboration.

Buyers ask for proof no one has

Health systems ask vendors for local validation, bias testing, a monitoring plan and an audit trail. Most vendors hand over a marketing document, and most buyers cannot check it.

Black Book poll of 256 NHS digital leaders, 2026 · Limit: Commercial pollster, NHS only. No US survey ranks the demands.

Quality drops outside English

Standard word error rate mis-scores mixed-language speech, and automated graders agree less with human raters in languages such as Bengali and Odia.

Clinical speech recognition, 162 recordings, arXiv 2512.10967 · Limit: Single hospital; the paper does not analyse medical-term errors.

Know your healthcare AI is safe to trust
before it reaches a patient.

Book a 30-minute call. We'll look at your product and tell you what we would measure first.