Why chatbot benchmark scores miss real-user medical outcomes
A 2026 Oxford randomised trial of 1,298 people found chatbot users no better than controls at naming conditions or the right action. What it means for evaluation.
What did the Oxford trial test?
It tested 1,298 participants on 10 medical scenarios. Participants used GPT-4o, Llama 3 or Command R+, or a control method, to work out what condition they might have and what to do. The trial was published in Nature Medicine on 9 February 2026.
What did it find?
| Measure | Result |
|---|---|
| Relevant conditions identified | Under 34.5% of cases |
| Correct disposition (what to do) | Under 44.2% of cases |
| Compared with control | No better |
| Models tested alone | Scored well on benchmarks |
Real users gave incomplete information. Near-identical messages produced very different advice, including opposite advice in one subarachnoid-haemorrhage example. Simulated users scored 57 to 60%, much higher than real people.
What does it mean for evaluation?
Exam-style scores answer a different question from “does this help a worried patient?”. A test needs realistic users with varied wording, a deterministic red-flag escalation gate that does not depend on the model getting it right, and results reported separately for emergencies. ECRI ranked misuse of AI chatbots first on its 2026 health technology hazards list.
What are the limits?
The models were older. Newer systems may behave differently. A simulator is useful but should not be the only test, because of the gap between simulated and real users. See wording invariance and the patient simulator.
What did we not verify?
We read the trial through Oxford’s press summary and the abstract, not the full paper.
Frequently asked questions
Do high medical exam scores mean a chatbot is safe for patients?
No. In the Oxford trial, benchmark scores did not predict the failures real users experienced. Users identified relevant conditions in under 34.5% of cases and the correct action in under 44.2%, no better than controls.
Why did real users do worse than simulated users?
Real users supplied incomplete information, and near-identical messages produced very different advice, including opposite advice in one subarachnoid-haemorrhage example. Simulated users scored 57 to 60%, so a simulator alone is not a sign-off.
Does this study cover current chatbots?
Not fully. It tested GPT-4o, Llama 3 and Command R+, which are older models, not current frontier systems. The finding that benchmarks do not predict real-user outcomes is the transferable point.