138 ways to measure healthcare AI.

Search by name or filter by product. Each page says what it means, who judges it and how strong the evidence is.

138 of 138 parameters shown

Cross-cutting

Clinical accuracy (rubric)

Weighted sum of physician-written positive and negative criteria (-10 to +10 each); LLM grader scores each criterion

Evidence E2
Cross-cutting

Completeness / omission

Clinically relevant content missing from output; fact-list enumeration then check

Evidence E2
Cross-cutting

Harm severity grade

Major/minor (could the error change diagnosis or management if uncorrected), then none/minor/major/critical and reached-patient status

Evidence E1
Cross-cutting

Groundedness / faithfulness

Share of claims supported by source (transcript, chart, cited reference); claim extraction plus entailment

Evidence E4
Cross-cutting

Calibration and appropriate abstention

Confidence vs correctness; rate of deferral to a clinician when uncertain

Evidence E2
Cross-cutting

Context seeking

Asks for missing information before answering

Evidence E2
Cross-cutting

Communication, empathy, readability

Rubric score; readability grades; physician vs nurse rating of bedside manner

Evidence E2
Cross-cutting

Subgroup performance

Error rates by race, language, gender, age, dialect, literacy

Evidence E3
Cross-cutting

Biased language in output

Stereotyping or meaning change by demographic

Evidence E1
Cross-cutting

Multilingual parity

Same quality across languages

Evidence E3
Cross-cutting

PHI leakage

PHI present in outputs, logs, prompts, third-party calls

Evidence E3
Cross-cutting

Prompt injection and jailbreak resistance

Attack success rate, including injections hidden in chart text

Evidence E4
Cross-cutting

Data retention and training use

Contractual and technical: retention, training on customer data

Evidence E3
Cross-cutting

Audit trail and provenance

Each output linked to model version, inputs, reviewer, edits

Evidence E3
Cross-cutting

Drift and version regression

Metric shift after model, prompt, ASR or EHR changes; black-box drift tests

Evidence E3
Cross-cutting

Wording invariance

Near-identical inputs get consistent, safe outputs

Evidence E1
Cross-cutting

Run-to-run consistency (pass^k)

Share of tasks that pass on every repeated run

Evidence E2
Cross-cutting

Latency and uptime

Turn latency, availability, time to note

Evidence E3
Cross-cutting

Override, edit and alert-acceptance rate

How often clinicians change or reject output

Evidence E4
Cross-cutting

Automation bias and deskilling

Performance of unaided clinicians after AI exposure

Evidence E1/E3
Cross-cutting

Cost and ROI

Cost per task or encounter, FTE time, denials

Evidence E3
Cross-cutting

Model card and intended-use documentation

Completeness of CHAI Applied Model Card or HTI-1-style attributes

Evidence E3
Cross-cutting

Change management and incident reporting

PCCP-style protocol for model changes; safety-event reporting

Evidence E3
Cross-cutting

AI disclosure and consent

Patients told AI was used; consent for recording

Evidence E1/E3
Cross-cutting

Judge-versus-clinician agreement (meta)

Kappa or F1 of the automated judge against clinicians, relative to clinician-versus-clinician ceiling, with CIs

Evidence E2
Cross-cutting

Instrument sensitivity (meta)

How much the measured rate moves with reviewer instruction or reviewing model

Evidence E2
Ambient scribes

Hallucination rate

Share of sentences or notes with unsupported content; sentence-level clinician annotation against transcript

Evidence E1/E2
Ambient scribes

Omission rate

Relevant transcript content missing from note

Evidence E1/E2
Ambient scribes

Accidental inclusion

Off-topic or other-person content entered

Evidence E1
Ambient scribes

Error severity (major/minor)

Could it change diagnosis or management

Evidence E1
Ambient scribes

PDQI-9 / modified PDQI-9

Nine to eleven criteria on 5-point Likert (accurate, thorough, organised, succinct, internally consistent)

Evidence E2
Ambient scribes

Targeted error classes: attribution, negation, laterality, medication and dose

Test sets built to provoke each class

Evidence E4
Ambient scribes

Interpreter and speaker-error propagation

Share of source (interpreter) errors carried into the note

Evidence E1
Ambient scribes

Patient-experience capture

Whether patient-reported experience and concerns appear

Evidence E3
Ambient scribes

Note edit burden

Edit distance, share of AI text retained, edit time

Evidence E4
Ambient scribes

Documentation time

EHR audit-log minutes per note or per week

Evidence E1/E2
Ambient scribes

Burnout and cognitive load

Validated surveys

Evidence E1/E3
Ambient scribes

ASR word error rate by accent, noise and medical term

WER stratified

Evidence E2
Ambient scribes

Specialty fit

Quality by specialty (paediatrics and cardiology rated lower agreement)

Evidence E2
Ambient scribes

Coding and billing downstream effects

Whether missing content reduces HCC capture

Evidence E3
Ambient scribes

Adoption and utilisation

Share of encounters using the scribe

Evidence E4
Ambient scribes

Patient consent and disclosure rate

Consent captured per encounter

Evidence E3
Ambient scribes

Clinical safety case and hazard log

DCB0129 (manufacturer) and DCB0160 (deployer), DPIA, clinical safety officer

Evidence E3
Ambient scribes

Copy-forward and note bloat

Repeated or stale content

Evidence E4
Clinical Q&A

Rubric accuracy and completeness (clinician-facing)

HealthBench Professional: 525 tasks from a 15,079 pool, about one-third adversarial, rubrics adjudicated by three or more physicians

Evidence E2
Clinical Q&A

Task-category score

MedHELM: 121 tasks, 22 subcategories (decision support 0.56-0.72, note generation 0.73-0.85, patient comms 0.78-0.83 in the preprint; HAI article gives slightly different ranges)

Evidence E1/E2
Clinical Q&A

Guideline concordance

Match to a named, versioned guideline

Evidence E4
Clinical Q&A

Citation and source faithfulness

Claims entailed by the cited source

Evidence E4
Clinical Q&A

Drug dose and interaction accuracy

Checked against FDA labels, RxNav

Evidence E2
Clinical Q&A

Sycophancy to clinician or patient assertions

Adversarial rubrics

Evidence E2
Clinical Q&A

Expert-panel alignment

Share of criteria clinicians judge aligned; PeerCheck review

Evidence E3
Clinical Q&A

Head-to-head versus general frontier LLMs

Same questions, physician graders

Evidence E2/E3
Clinical Q&A

Temporal validity and recency

Stale-guideline retrieval

Evidence E4
Clinical Q&A

Real-world override and alert acceptance

Clinician reliance patterns

Evidence E4
Clinical Q&A

Licensed-professional status of the function

Whether output is HCP-facing CDS within FDA enforcement discretion

Evidence E1/E3
Patient chat

Under-triage (missed emergency) and over-triage

Vignettes against clinician gold

Evidence E4
Patient chat

Emergency recognition and escalation correctness

HealthBench emergency-referral theme; GPT-4.1 grader macro-F1 0.662 vs physicians 0.647

Evidence E2
Patient chat

Real-user task accuracy

Condition identification and correct action with real people

Evidence E1
Patient chat

Simulator-to-real gap

Difference between simulated-user and real-user performance

Evidence E1
Patient chat

Unsafe-advice rate

Dangerous instructions (ECRI: wrong diagnoses, invented body parts, unsafe electrosurgical electrode placement advice)

Evidence E3
Patient chat

Scope-of-practice adherence and appropriate refusal

Declines or hands off when out of scope

Evidence E4
Patient chat

Health literacy and multilingual quality

Readability, translation fidelity

Evidence E4
Patient chat

Conversational diagnostic accuracy

CRAFT-MD, AMIE-style multi-turn consults

Evidence E4
Patient chat

AI disclosure and human-contact instruction

Required unless clinician-reviewed

Evidence E1
Patient chat

Downstream patient outcomes and safety events

Harm after advice

Evidence E4
Voice agents

Workflow pass rate (all repeats pass)

Scenario success across repeated calls

Evidence E2
Voice agents

Interruption recovery and naturalness

Composite

Evidence E2
Voice agents

ASR error and its clinical impact

WER weighted by clinical meaning; LLM judge κ ≈0.71 and 0.50 vs two clinicians (clinician-clinician κ ≈0.51)

Evidence E2
Voice agents

Turn latency, call completion, handoff-to-human rate

Operational

Evidence E3
Voice agents

Clinical-safety label error rates

Nurse-labelled errors in labs, vitals, medications, human navigation, with adjudication

Evidence E2
Voice agents

EHR write-back correctness

Scheduling, refill and intake entries match intent

Evidence E3
Voice agents

Accent, dialect and noise robustness

WER and task success by group

Evidence E4
Voice agents

AI disclosure, recording consent, identity verification

Scripted compliance checks

Evidence E4
Voice agents

Simulated-caller realism

Realism score of caller simulator against real calls

Evidence E2/E4
Coding and RCM

Code-level exact-match accuracy

Assigned codes against certified-coder gold

Evidence E1
Coding and RCM

Coding benchmark score

Vals MedCode: 2,755 ICD-10-CM samples; Claude Opus 5 63.57% (July mirror)

Evidence E2
Coding and RCM

Claim-level versus code-level accuracy

Must be specified; vendors target 95%+, no standard method

Evidence E3
Coding and RCM

Clean-claim rate and first-pass yield

Vendor target 98%+ (one vendor)

Evidence E3
Coding and RCM

First-submission denial rate

11.8% in 2024 (Experian, secondary); best-in-class under 5%

Evidence E3
Coding and RCM

Coding-related denial trend

+126% over three years (HFMA/MDaudit, secondary)

Evidence E3
Coding and RCM

Overcoding and audit-flag risk

Documentation-supported coding; compliance review

Evidence E4
Coding and RCM

Autonomy and human-in-the-loop rate

Share auto-submitted without review

Evidence E4
Coding and RCM

Payer-rule and code-validity adherence

Deterministic check against code sets and payer policy

Evidence E4
Coding and RCM

Prior-authorisation turnaround

Against CMS-0057-F decision timelines

Evidence E3
Coding and RCM

Appeal overturn rate

Denials reversed on appeal

Evidence E3
Coding and RCM

Independent validation evidence

Third-party audit exists

Evidence E3
Payer AI

Human-review compliance

AI never the sole denial decider; licensed professional signs

Evidence E3
Payer AI

Denial error rate

Wrong denials against clinician re-review

Evidence E3
Payer AI

Subgroup denial disparity

Denial rates by protected characteristics

Evidence E1/E3
Payer AI

Denial-reason quality

Specific, accurate, policy-linked reasons

Evidence E3
Payer AI

Coverage-policy consistency

Same case, same criteria, same outcome

Evidence E4
Payer AI

Decision timeline compliance

Turnaround by urgency

Evidence E3
Imaging

Sensitivity, specificity, AUROC

Against reference standard

Evidence E4
Imaging

PPV and NPV at deployment prevalence

Local prevalence adjustment

Evidence E4
Imaging

Calibration

Predicted vs observed

Evidence E4
Imaging

Subgroup performance tables and model cards

FDA draft lifecycle guidance expects them; finalisation is on the FY2027 agenda

Evidence E3
Imaging

Out-of-distribution scanner and site failure

Performance on new devices and protocols

Evidence E1
Imaging

Reader-study uplift and turnaround

Radiologist accuracy and time with and without AI

Evidence E4
Imaging

Alert burden and false-positive rate

Per read or per day

Evidence E4
Imaging

Deskilling

Unaided performance after exposure

Evidence E1
Imaging

PCCP conformance

Modifications within the authorised change plan

Evidence E3
Mental health bots

Suicide-risk detection and response

Simulated users at set risk levels, clinician rubric

Evidence E2/E3
Mental health bots

Risk exploration

Asks about plan and means

Evidence E2
Mental health bots

Validation of delusions and harmful content

Sycophancy to psychosis; psychosis extension of VERA-MH planned

Evidence E3
Mental health bots

Crisis-resource provision and handoff

Correct resource, human handoff

Evidence E4
Mental health bots

Therapeutic alliance and modality fidelity

Clinician rating

Evidence E4
Mental health bots

Clinical outcomes and dropout

PHQ-9 and GAD-7 change

Evidence E4
Mental health bots

State-law compliance map

Illinois and Nevada ban AI therapy; Utah permits with disclosure; California SB 243 and AB 489; Tennessee, Colorado, Maine, Vermont and Rhode Island acted in 2026 per one report

Evidence E3
Mental health bots

Professional impersonation

Claims to be licensed

Evidence E3
EHR agents

Task completion

Programmatic check of FHIR end state; MedAgentBench 300 tasks; <70% (2025) to >90% (2026) per secondary summary

Evidence E3
EHR agents

Retrieval versus reasoning failure share

FHIR-AgentBench: 2,931 real-world questions

Evidence E2
EHR agents

Action safety

No unauthorised writes, orders or deletions

Evidence E4
EHR agents

Order and medication accuracy

Correct resource, dose, route

Evidence E4
EHR agents

Unit and aggregation correctness

Example: 7015 ml vs 7515 ml fluid-output sum (anecdote, vendor blog)

Evidence E3
EHR agents

Human-confirmation coverage

Share of writes gated by approval

Evidence E4
EHR agents

Injection resistance in chart text

Attack success

Evidence E4
EHR agents

Access-policy adherence

Role, time and record restrictions (Cedar-style)

Evidence E3
EHR agents

Cost and latency per task

Operational

Evidence E4
Predictive models

Discrimination (AUROC)

Local validation

Evidence E1/E3
Predictive models

Calibration

Predicted vs observed risk

Evidence E3
Predictive models

PPV and alert burden

7.6% PPV in a county-ED validation

Evidence E3
Predictive models

Lead time

Hours of warning before event

Evidence E4
Predictive models

Subgroup performance and drift

Per group, over time

Evidence E3
Predictive models

Net benefit

Decision-curve analysis

Evidence E4
Predictive models

Outcome impact

Does the alert change care or outcomes

Evidence E4
Predictive models

Local validation before go-live

Required by Joint Commission/CHAI guidance

Evidence E3
Pharma and trials

Trial cycle-time and delay

53% of phase 2/3 trials delayed (Bain)

Evidence E3
Pharma and trials

Digital-twin prediction accuracy and variance reduction

Vendor claim: EMA qualification (2022), FDA supportive (2023)

Evidence E3
Pharma and trials

Model credibility assessment

FDA's Jan 2025 AI credibility draft; not opened

Evidence E4
Pharma and trials

Electronic-record audit trail (21 CFR Part 11)

Complete, attributable record

Evidence E4
Pharma and trials

Pharmacovigilance case-extraction accuracy

Adverse events extracted correctly

Evidence E4
Pharma and trials

Medical-affairs content accuracy and promotional compliance

Claims match labelling

Evidence E4
India

Indic-language and code-switching speech and note quality

WER and clinical impact in Hindi, Tamil, others

Evidence E4
India

ABDM interoperability conformance

FHIR records aligned with the national health stack

Evidence E3
India

CDSCO software-as-medical-device classification

Function-based test; final guidance dated 21 Jul 2026; no separate AI class

Evidence E3
India

Public-programme screening performance (for example TB, breast)

Sensitivity and specificity in rural settings

Evidence E3
India

BODH benchmark performance

National benchmarking platform (IIT Kanpur and National Health Authority)

Evidence E3
India

Consent under India's data-protection rules

DPDP consent and residency

Evidence E4
How to read the evidence grades
  • E1 · Peer-reviewed, regulator, audit or court source. Backed by a peer-reviewed paper, a regulator, an audit or a court record. Usually read through a summary, not the full text.
  • E2 · Preprint, benchmark page or vendor-run result. Backed by numbers from a preprint, a benchmark page or a vendor-run test. Treat as strong but not settled.
  • E3 · Secondary or single-source report. Reported by trade press or a single source. Directional only until checked against the primary document.
  • E4 · Recalled, not verified. Recalled from background knowledge and not verified. Treat it as a lead for investigation, not a finding.

Not sure which of these
matter for your product?

Tell us what you are building. We will say which parameters we would measure first and who should judge them.