What do health-system AI governance committees need from vendors?
Local validation, bias testing, PHI safeguards, monitoring plans and audit trails: what buyers say they require, and how little testing capacity they have.
What do buyers say they need?
CMIOs and governance committees quoted in 2026 trade coverage ask for the same five things before onboarding a vendor.
| Ask | Why |
|---|---|
| Model transparency | To know what the tool does and where it came from |
| Local validation | Performance on this system’s patients and workflows |
| Bias testing | Subgroup performance and the 45 CFR 92.210 duty |
| PHI safeguards | Data use, retention and training terms |
| A monitoring plan | Vendors update on their own schedule |
What does the NHS data add?
A Black Book poll of 256 NHS digital leaders reports that 83% treat clinical safety assurance as a procurement gate, 72% require provenance and audit trails, 59% require a formal real-world evaluation plan, and 52% say vendor evidence packs need local validation. Black Book is a commercial pollster and the sample is NHS only. We found no equivalent US survey.
How much capacity do health systems have?
In a UPMC Center for Connected Medicine and KLAS report, fewer than half of hospitals had a dedicated AI testing environment, and 63% called their approach developing or ad hoc. The sample was small, about 25 to 27 leaders.
What changed in US federal rules?
The HTI-5 final rule removes federal source-attribute requirements for predictive decision support from 28 December 2026, so buyers must ask for those disclosures contractually. See the HTI-5 page and the Joint Commission and CHAI guidance.
What should a vendor hand over?
A validation summary with the unit, sampling and severity scale; subgroup results with confidence intervals; a monitoring and change-control plan; and an audit trail. An evidence pack assembles these. See Precision’s evidence pack.
What did we not verify?
Budget owner by role, evaluation budgets, and pilot-to-purchase times were not found. The UPMC survey is small and the NHS poll is a commercial pollster’s.
Frequently asked questions
What do health systems ask AI vendors for?
CMIOs quoted in trade press ask for model transparency, local validation, bias testing, PHI safeguards and an ongoing monitoring plan before onboarding. In a Black Book poll of 256 NHS digital leaders, 83% treat clinical safety assurance as a procurement gate and 72% require provenance and audit trails.
Do hospitals have the capacity to test AI themselves?
Most do not. In a UPMC and KLAS survey of about 25 to 27 leaders, fewer than half had a dedicated AI testing environment, and 63% called their approach developing or ad hoc. The sample is small, so treat it as directional.
Did federal rules change what vendors must provide?
Yes. The HTI-5 final rule, effective 28 December 2026, removes federal source-attribute requirements for predictive decision support. Health systems that valued those disclosures must now ask for them by contract.