Retrieval versus reasoning failure share

FHIR-AgentBench: 2,931 real-world questions.

Who judgesAutomatable
EvidenceGrade E2
Product typeEHR agentsID A2

What retrieval versus reasoning failure share means

FHIR-AgentBench: 2,931 real-world questions.

Why it matters. Knowing whether failures come from fetching the wrong data or from reasoning tells you what to fix.

Example, illustrative

An agent reasons correctly from the wrong lab value.

How to measure it

01

Set tasks

Define tasks with a clear correct end state in the record.

02

Run in a sandbox

Use a FHIR test record, never a live chart.

03

Check the end state

Programmatically compare resulting records.

04

Review actions

Log every write and which needed approval.

Watch out for

Judging only the final answer misses harmful actions along the way.

What a good report shows

Task completion with resources touched, actions taken, and approval coverage.

Report task completion with the FHIR resources touched, the actions taken, and which actions needed human approval.

Reviewed by the Precision research team.

How strong is the evidence

E2 · Preprint, benchmark page or vendor-run result. Backed by numbers from a preprint, a benchmark page or a vendor-run test. Treat as strong but not settled.

What the sources say

(PMLR).

  • Automatable. Can be scored by software, including an AI grader that is spot-checked against clinicians.
  1. PMLRproceedings.mlr.press

Found something out of date? Write to hello@precision.example.

Quick answers

What is retrieval versus reasoning failure share in healthcare AI?

FHIR-AgentBench: 2,931 real-world questions. It sits in the agentic ehr workflows group.

Why does retrieval versus reasoning failure share matter?

Knowing whether failures come from fetching the wrong data or from reasoning tells you what to fix. An agent reasons correctly from the wrong lab value.

How is retrieval versus reasoning failure share measured?

Set tasks, then Run in a sandbox, then Check the end state, then Review actions. Report task completion with the FHIR resources touched, the actions taken, and which actions needed human approval.

What should I watch out for with retrieval versus reasoning failure share?

Judging only the final answer misses harmful actions along the way. A good report gives: task completion with resources touched, actions taken, and approval coverage.

Can retrieval versus reasoning failure share be automated?

Largely yes, including AI graders that are spot-checked against clinicians. Report the grader's error rate beside every score.

Measure retrieval versus reasoning failure share on your own data