Who evaluates healthcare AI in 2026, and what do they miss?
Qualified Health, Vals AI, Alterra Labs, voice-testing vendors, CHAI and Epic: who evaluates healthcare AI today, what each does, and the gaps that remain.
Who builds evaluation for healthcare AI?
Six kinds of player cover parts of the problem. None covers all of it.
| Who | What they do | Evidence of scale | Limit |
|---|---|---|---|
| Qualified Health | Governance and deployment platform for health systems | $125M Series B, March 2026 (NEA); about $155M total; valuation press-estimated | Positioned as a platform, not an independent measurement lab; pricing not public |
| Vals AI | Benchmarks, including MedScribe and MedCode (February 2026) | $40M Series A at $400M, 13 August 2026 (a16z) | Benchmarks rank models; they do not test your product |
| Alterra Labs | Simulated medical environments scored by clinicians | $12.5M seed, 10 November 2026 (Oak HC/FT) | Customers are mostly frontier AI labs |
| Voice-testing vendors (Coval, Hamming, Cekura, Bluejay) | Simulation and monitoring for voice agents | Coval $28M Series A, June 2026 | Horizontal tools; healthcare is a use case, and failure rates are vendor-run |
| Epic seismometer | Open-source validation of predictive models | 272 GitHub stars | Tabular models; generative AI not covered at launch |
| Health-system in-house programmes | Local validation by informatics teams | Stanford’s FURM is documented | Slow, expensive and not shared |
Which attempts at third-party assurance failed?
CHAI planned a network of assurance labs. Fierce Healthcare reports the plan was scrapped. The Joint Commission’s 2026 certification assesses an organisation’s governance, not a product. Third-party validation of the tools themselves remains open.
What do horizontal evaluation tools miss in healthcare?
General LLM evaluation platforms gate HIPAA agreements, private hosting and data residency behind enterprise tiers, according to third-party comparison write-ups we read. They also measure generic quality, not omissions, negation or dose errors in clinical notes.
Where are the gaps?
Five stand out: calibrated graders that report their own error rates; causal diagnosis of which component failed; independent testing on a buyer’s own encounters; continuous monitoring of scribes, voice agents and other agents after deployment; and language-specific evaluation outside English.
What did we not verify?
Qualified Health’s valuation is a press estimate. Vals AI’s customers come from a secondary source. We did not search every vendor, and pricing for most evaluation services is not public.
Frequently asked questions
Is there a company that independently evaluates AI scribes?
We did not find a dedicated, funded healthcare scribe-evaluation startup. Vals AI publishes a MedScribe benchmark, and health systems run their own pilots, but independent evaluation of deployed scribes on a buyer's own encounters is largely unserved.
What does Qualified Health do?
Qualified Health raised a $125M Series B led by NEA in March 2026, about $155M in total. Press calls it AI evaluation, but it positions itself as a platform to build, deploy and govern generative AI in health systems. Pricing is not public.
Did CHAI's assurance labs work?
No. The plan for a network of assurance labs was scrapped, according to Fierce Healthcare, and replaced by assurance resource providers. CHAI's model-card registry continues, and the Joint Commission's certification covers organisations, not products.