Five gaps in healthcare AI evaluation that nobody fills well
Calibrated graders, causal diagnosis, buyer-run testing, monitoring and language-specific evaluation: the gaps we found in healthcare AI evaluation today.
Gap 1: Do graders report their own error rates?
Most scores arrive as a single number. In published tests, LLM judges discriminated note completeness at AUC 0.49 to 0.66. A score from a grader with a known miss rate, a confidence interval and an abstain option is more useful than one without. We found no product that reports grader error bars calibrated against expert labels end to end.
Gap 2: Can anyone say which component failed?
Logs show what happened, not why. Published attribution methods find the faulty step in failed agent runs only 14% to 52% of the time, and we found no healthcare study that separates speech-recognition errors from model or prompt errors in scribes. Swapping one component on a recorded encounter and measuring the change is an experiment, not a guess.
Gap 3: Who tests your product on your own encounters?
Benchmarks rank models. Vendors grade themselves. Health systems run slow local pilots. Independent, buyer-side testing with the buyer’s clinicians as the reference is thinly served.
Gap 4: Who keeps watching after go-live?
Epic’s open-source seismometer covers predictive models. For generative AI, monitoring is largely manual. In the UPMC and KLAS survey, 63% called their strategy developing or ad hoc.
Gap 5: What about languages other than English?
Clinical benchmarks and voice testing are English-first. Indian languages and code-mixed speech are the clearest example, but the pattern holds across the UK, the Nordics and Germany, where dialects and data-sovereignty rules shape what can be tested.
| Gap | What exists | What is missing |
|---|---|---|
| Grader error rates | Generic LLM judges | Calibration against clinician labels, with error bars |
| Causal diagnosis | Tracing, clustering | Counterfactual replay in healthcare |
| Buyer-side testing | Benchmarks, vendor claims | Independent testing on your encounters |
| Monitoring | Seismometer for predictive models | Generative AI, scribes, voice agents |
| Languages | English benchmarks | Clinician-labelled sets per language |
What did we not verify?
We did not test every vendor’s claims, and “no product found” means not found in our searches. See who evaluates healthcare AI in 2026.
Frequently asked questions
What is missing in healthcare AI evaluation?
Five things stand out: graders that report their own error rates, causal diagnosis of which part of a system failed, independent testing on a buyer's own data, continuous monitoring after deployment, and evaluation outside English. Tracing and generic scores are widely available.
Why does continuous monitoring matter?
Vendors update models on their own schedule. In a UPMC and KLAS survey, fewer than half of hospitals had a dedicated AI testing environment and 63% called their approach developing or ad hoc, so a regression can pass unnoticed.
Is language a real gap?
Yes. On 162 real Indian doctor-patient recordings, the best system's word error rate rose from 14% in English to 35% in Kannada, and standard metrics mis-score mixed-language speech. Few clinical benchmarks cover Indian languages.