Replay and counterfactual diagnosis agent
Re-runs a failure with one part swapped to find the cause.
Checked against clinicians
Seeded-fault recovery; clinician review of top attributions.
Where a person decides
Reports causes; engineers approve changes.
Works on
Scribes, voice, Q&A, EHR agents.
In development. Early access is open. Reviewed .
The problem it solves
When a result is wrong, the cause could be audio, transcription, prompt, retrieval or the model. Reading logs rarely names the culprit, and published methods for it are right only 14% to 52% of the time.
How it works
Freeze the input
A recorded encounter is replayed through the same pipeline.
Swap one part
The agent changes one component at a time: speech model, prompt, retriever, model or guideline.
Measure the change
The effect on the error is recorded, and causes are ranked by effect size.
An example, illustrative
Allergy omitted. Cause unknown.
New prompt fixes it in 62% of cases. Speech model change: 24%.
Recorded inputs and the ability to re-run each pipeline step.
Works for text and FHIR today. Voice and Indian-language replay is still maturing.
What it helps measure
Quick answers
What does the replay diagnostician do?
Swap ASR, prompt, retriever, model or guideline version on fixed inputs; attribute failures.
How is the replay diagnostician checked against clinicians?
Seeded-fault recovery; clinician review of top attributions.
Why is this better than reading logs?
Swapping a part is an experiment on the same input, so the answer is a measured effect, not a guess.
Is it proven in healthcare?
The method is sound, but there's no published healthcare study yet. We say so on the how-it-works page.
What can the replay diagnostician not do?
Reports causes; engineers approve changes. No agent makes a clinical decision.