Which open-source tools and benchmarks evaluate healthcare AI?

HealthBench, MedHELM, MedAgentBench and Epic's seismometer compared: what each measures, licence, GitHub traction on 15 November 2026 and where each falls short.

  • Global

1HealthBench has 5,000 multi-turn conversations, 262 physicians from 60 countries and 48,562 unique rubric criteria; its evaluation code lives in OpenAI's MIT-licensed simple-evals repository.

2MedHELM covers 121 tasks in five categories and was published in Nature Medicine in 2026; MedAgentBench is an MIT-licensed FHIR sandbox of 300 tasks over 100 patients.

3Epic's seismometer is BSD-3-Clause licensed with 272 GitHub stars and validates predictive models, not generative AI, as of its launch.

4HealthBench Professional, an April 2026 OpenAI benchmark, was rated 'Flawed' by Epoch AI on 10 September 2026 after finding clear errors in 25 of 50 sampled tasks.

Limits of this page. Star counts were read from the GitHub API on 15 November 2026 and change daily. We did not verify whether Epic's seismometer has since added generative-AI evaluation, or the MedHELM repository's licence and activity. The HealthBench data file's licence is unconfirmed.

What are the main open resources for evaluating healthcare AI?

Four are widely cited: HealthBench, MedHELM, MedAgentBench and Epic’s seismometer. They answer different questions, and none tests a deployed scribe, voice agent or chat product against the clinicians who use it.

Resource What it measures Licence GitHub (15 Nov 2026) Main limit
HealthBench (OpenAI) Chat answers against physician-written rubrics: 5,000 conversations, 48,562 unique criteria Code MIT in simple-evals; data licence unconfirmed simple-evals: 4,705 stars Fixed set; grader agrees with physicians 55 to 75%
MedHELM (Stanford) 121 tasks across five categories, scored by an LLM jury Not checked HELM repo: 2,970 stars Model-level, and score ranges differ between preprint and article
MedAgentBench 300 EHR tasks over 100 patients in a FHIR sandbox MIT Not recorded Sandbox, not your EHR
Epic seismometer Validation and fairness of predictive models BSD-3-Clause 272 stars Tabular models; generative AI not covered at launch

What does HealthBench measure?

OpenAI built it with 262 physicians from 60 countries. Its abstract reports 48,562 unique criteria, while the paper’s Table 3 counts 57,237 criterion instances in total because some consensus criteria repeat; the paper does not explain the gap. GPT-4.1 as grader reached a macro F1 of 0.709 against physicians.

What about HealthBench Professional?

It is a separate OpenAI benchmark of 525 clinician tasks, posted on 30 April 2026. Epoch AI rated it “Flawed” on 10 September 2026 because the vendor built and ran it, and found clear errors in 25 of 50 sampled tasks.

What are the data constraints?

PhysioNet’s terms bar sending MIMIC data to third-party LLM APIs. Public clinical sets rarely include clinician-labelled omission or hallucination spans, so teams often build their own gold set. See the parameter library for what to label.

What did we not verify?

MedHELM’s repository details, the HealthBench data licence, and whether the seismometer now covers generative AI.

Frequently asked questions

Is HealthBench enough to evaluate a healthcare AI product?

No. HealthBench tests a model on a fixed set of 5,000 conversations. It does not test your product, your workflows or your patients, and the grader's own agreement with physicians sits at 55 to 75 percent. It is a useful reference, not a validation of a deployed system.

Can I use MIMIC data with a commercial AI service for evaluation?

PhysioNet's data-use terms bar sending MIMIC data to third-party LLM APIs and recommend local models or approved cloud endpoints. That is one reason evaluation of clinical text often has to run inside the organisation's own environment.

Does Epic's seismometer evaluate AI scribes?

Not as far as we could verify. It was released as an open-source package for validating predictive models, and its launch material did not cover generative AI. We did not check whether that has changed since.

Sources

  1. HealthBench: evaluating large language models towards improved human healtharXiv (OpenAI) · 2025-05
  2. openai/simple-evalsGitHub · 2026-11-15
  3. MedHELM: holistic evaluation of large language models for medical tasksarXiv (Stanford) · 2025
  4. Measuring what medical AI can actually doStanford HAI
  5. epic-open-source/seismometerGitHub · 2026-11-15
  6. HealthBench Professional review: verdict FlawedEpoch AI · 2026-09-10
  7. Responsible use of LLMs with credentialed dataPhysioNet

Found something out of date? Write to hello@precision.example and we will fix it.

Written and reviewed byPrecision research teamHealthcare AI evaluation research