138 ways to measure healthcare AI.
Search by name or filter by product. Each page says what it means, who judges it and how strong the evidence is.
Clinical accuracy (rubric)
Weighted sum of physician-written positive and negative criteria (-10 to +10 each); LLM grader scores each criterion
Completeness / omission
Clinically relevant content missing from output; fact-list enumeration then check
Harm severity grade
Major/minor (could the error change diagnosis or management if uncorrected), then none/minor/major/critical and reached-patient status
Groundedness / faithfulness
Share of claims supported by source (transcript, chart, cited reference); claim extraction plus entailment
Calibration and appropriate abstention
Confidence vs correctness; rate of deferral to a clinician when uncertain
Context seeking
Asks for missing information before answering
Communication, empathy, readability
Rubric score; readability grades; physician vs nurse rating of bedside manner
Subgroup performance
Error rates by race, language, gender, age, dialect, literacy
Biased language in output
Stereotyping or meaning change by demographic
Multilingual parity
Same quality across languages
PHI leakage
PHI present in outputs, logs, prompts, third-party calls
Prompt injection and jailbreak resistance
Attack success rate, including injections hidden in chart text
Data retention and training use
Contractual and technical: retention, training on customer data
Audit trail and provenance
Each output linked to model version, inputs, reviewer, edits
Drift and version regression
Metric shift after model, prompt, ASR or EHR changes; black-box drift tests
Wording invariance
Near-identical inputs get consistent, safe outputs
Run-to-run consistency (pass^k)
Share of tasks that pass on every repeated run
Latency and uptime
Turn latency, availability, time to note
Override, edit and alert-acceptance rate
How often clinicians change or reject output
Automation bias and deskilling
Performance of unaided clinicians after AI exposure
Cost and ROI
Cost per task or encounter, FTE time, denials
Model card and intended-use documentation
Completeness of CHAI Applied Model Card or HTI-1-style attributes
Change management and incident reporting
PCCP-style protocol for model changes; safety-event reporting
AI disclosure and consent
Patients told AI was used; consent for recording
Judge-versus-clinician agreement (meta)
Kappa or F1 of the automated judge against clinicians, relative to clinician-versus-clinician ceiling, with CIs
Instrument sensitivity (meta)
How much the measured rate moves with reviewer instruction or reviewing model
Hallucination rate
Share of sentences or notes with unsupported content; sentence-level clinician annotation against transcript
Omission rate
Relevant transcript content missing from note
Accidental inclusion
Off-topic or other-person content entered
Error severity (major/minor)
Could it change diagnosis or management
PDQI-9 / modified PDQI-9
Nine to eleven criteria on 5-point Likert (accurate, thorough, organised, succinct, internally consistent)
Targeted error classes: attribution, negation, laterality, medication and dose
Test sets built to provoke each class
Interpreter and speaker-error propagation
Share of source (interpreter) errors carried into the note
Patient-experience capture
Whether patient-reported experience and concerns appear
Note edit burden
Edit distance, share of AI text retained, edit time
Documentation time
EHR audit-log minutes per note or per week
Burnout and cognitive load
Validated surveys
ASR word error rate by accent, noise and medical term
WER stratified
Specialty fit
Quality by specialty (paediatrics and cardiology rated lower agreement)
Coding and billing downstream effects
Whether missing content reduces HCC capture
Adoption and utilisation
Share of encounters using the scribe
Patient consent and disclosure rate
Consent captured per encounter
Clinical safety case and hazard log
DCB0129 (manufacturer) and DCB0160 (deployer), DPIA, clinical safety officer
Copy-forward and note bloat
Repeated or stale content
Rubric accuracy and completeness (clinician-facing)
HealthBench Professional: 525 tasks from a 15,079 pool, about one-third adversarial, rubrics adjudicated by three or more physicians
Task-category score
MedHELM: 121 tasks, 22 subcategories (decision support 0.56-0.72, note generation 0.73-0.85, patient comms 0.78-0.83 in the preprint; HAI article gives slightly different ranges)
Guideline concordance
Match to a named, versioned guideline
Citation and source faithfulness
Claims entailed by the cited source
Drug dose and interaction accuracy
Checked against FDA labels, RxNav
Sycophancy to clinician or patient assertions
Adversarial rubrics
Expert-panel alignment
Share of criteria clinicians judge aligned; PeerCheck review
Head-to-head versus general frontier LLMs
Same questions, physician graders
Temporal validity and recency
Stale-guideline retrieval
Real-world override and alert acceptance
Clinician reliance patterns
Licensed-professional status of the function
Whether output is HCP-facing CDS within FDA enforcement discretion
Under-triage (missed emergency) and over-triage
Vignettes against clinician gold
Emergency recognition and escalation correctness
HealthBench emergency-referral theme; GPT-4.1 grader macro-F1 0.662 vs physicians 0.647
Real-user task accuracy
Condition identification and correct action with real people
Simulator-to-real gap
Difference between simulated-user and real-user performance
Unsafe-advice rate
Dangerous instructions (ECRI: wrong diagnoses, invented body parts, unsafe electrosurgical electrode placement advice)
Scope-of-practice adherence and appropriate refusal
Declines or hands off when out of scope
Health literacy and multilingual quality
Readability, translation fidelity
Conversational diagnostic accuracy
CRAFT-MD, AMIE-style multi-turn consults
AI disclosure and human-contact instruction
Required unless clinician-reviewed
Downstream patient outcomes and safety events
Harm after advice
Workflow pass rate (all repeats pass)
Scenario success across repeated calls
Interruption recovery and naturalness
Composite
ASR error and its clinical impact
WER weighted by clinical meaning; LLM judge κ ≈0.71 and 0.50 vs two clinicians (clinician-clinician κ ≈0.51)
Turn latency, call completion, handoff-to-human rate
Operational
Clinical-safety label error rates
Nurse-labelled errors in labs, vitals, medications, human navigation, with adjudication
EHR write-back correctness
Scheduling, refill and intake entries match intent
Accent, dialect and noise robustness
WER and task success by group
AI disclosure, recording consent, identity verification
Scripted compliance checks
Simulated-caller realism
Realism score of caller simulator against real calls
Code-level exact-match accuracy
Assigned codes against certified-coder gold
Coding benchmark score
Vals MedCode: 2,755 ICD-10-CM samples; Claude Opus 5 63.57% (July mirror)
Claim-level versus code-level accuracy
Must be specified; vendors target 95%+, no standard method
Clean-claim rate and first-pass yield
Vendor target 98%+ (one vendor)
First-submission denial rate
11.8% in 2024 (Experian, secondary); best-in-class under 5%
Coding-related denial trend
+126% over three years (HFMA/MDaudit, secondary)
Overcoding and audit-flag risk
Documentation-supported coding; compliance review
Autonomy and human-in-the-loop rate
Share auto-submitted without review
Payer-rule and code-validity adherence
Deterministic check against code sets and payer policy
Prior-authorisation turnaround
Against CMS-0057-F decision timelines
Appeal overturn rate
Denials reversed on appeal
Independent validation evidence
Third-party audit exists
Human-review compliance
AI never the sole denial decider; licensed professional signs
Denial error rate
Wrong denials against clinician re-review
Subgroup denial disparity
Denial rates by protected characteristics
Denial-reason quality
Specific, accurate, policy-linked reasons
Coverage-policy consistency
Same case, same criteria, same outcome
Decision timeline compliance
Turnaround by urgency
Sensitivity, specificity, AUROC
Against reference standard
PPV and NPV at deployment prevalence
Local prevalence adjustment
Calibration
Predicted vs observed
Subgroup performance tables and model cards
FDA draft lifecycle guidance expects them; finalisation is on the FY2027 agenda
Out-of-distribution scanner and site failure
Performance on new devices and protocols
Reader-study uplift and turnaround
Radiologist accuracy and time with and without AI
Alert burden and false-positive rate
Per read or per day
Deskilling
Unaided performance after exposure
PCCP conformance
Modifications within the authorised change plan
Suicide-risk detection and response
Simulated users at set risk levels, clinician rubric
Risk exploration
Asks about plan and means
Validation of delusions and harmful content
Sycophancy to psychosis; psychosis extension of VERA-MH planned
Crisis-resource provision and handoff
Correct resource, human handoff
Therapeutic alliance and modality fidelity
Clinician rating
Clinical outcomes and dropout
PHQ-9 and GAD-7 change
State-law compliance map
Illinois and Nevada ban AI therapy; Utah permits with disclosure; California SB 243 and AB 489; Tennessee, Colorado, Maine, Vermont and Rhode Island acted in 2026 per one report
Professional impersonation
Claims to be licensed
Task completion
Programmatic check of FHIR end state; MedAgentBench 300 tasks; <70% (2025) to >90% (2026) per secondary summary
Retrieval versus reasoning failure share
FHIR-AgentBench: 2,931 real-world questions
Action safety
No unauthorised writes, orders or deletions
Order and medication accuracy
Correct resource, dose, route
Unit and aggregation correctness
Example: 7015 ml vs 7515 ml fluid-output sum (anecdote, vendor blog)
Human-confirmation coverage
Share of writes gated by approval
Injection resistance in chart text
Attack success
Access-policy adherence
Role, time and record restrictions (Cedar-style)
Cost and latency per task
Operational
Discrimination (AUROC)
Local validation
Calibration
Predicted vs observed risk
PPV and alert burden
7.6% PPV in a county-ED validation
Lead time
Hours of warning before event
Subgroup performance and drift
Per group, over time
Net benefit
Decision-curve analysis
Outcome impact
Does the alert change care or outcomes
Local validation before go-live
Required by Joint Commission/CHAI guidance
Trial cycle-time and delay
53% of phase 2/3 trials delayed (Bain)
Digital-twin prediction accuracy and variance reduction
Vendor claim: EMA qualification (2022), FDA supportive (2023)
Model credibility assessment
FDA's Jan 2025 AI credibility draft; not opened
Electronic-record audit trail (21 CFR Part 11)
Complete, attributable record
Pharmacovigilance case-extraction accuracy
Adverse events extracted correctly
Medical-affairs content accuracy and promotional compliance
Claims match labelling
Indic-language and code-switching speech and note quality
WER and clinical impact in Hindi, Tamil, others
ABDM interoperability conformance
FHIR records aligned with the national health stack
CDSCO software-as-medical-device classification
Function-based test; final guidance dated 21 Jul 2026; no separate AI class
Public-programme screening performance (for example TB, breast)
Sensitivity and specificity in rural settings
BODH benchmark performance
National benchmarking platform (IIT Kanpur and National Health Authority)
Consent under India's data-protection rules
DPDP consent and residency
No parameter matches that search. Try a shorter word, or clear the filter.
How to read the evidence grades
- E1 · Peer-reviewed, regulator, audit or court source. Backed by a peer-reviewed paper, a regulator, an audit or a court record. Usually read through a summary, not the full text.
- E2 · Preprint, benchmark page or vendor-run result. Backed by numbers from a preprint, a benchmark page or a vendor-run test. Treat as strong but not settled.
- E3 · Secondary or single-source report. Reported by trade press or a single source. Directional only until checked against the primary document.
- E4 · Recalled, not verified. Recalled from background knowledge and not verified. Treat it as a lead for investigation, not a finding.
Not sure which of these
matter for your product?
Tell us what you are building. We will say which parameters we would measure first and who should judge them.
hello@precision.example