Abridge, GAO and Ontario's auditor on evaluating AI scribes

Abridge's published method excludes omissions. GAO found few independent accuracy studies. Ontario's auditor found 11 of 20 scribe vendors had no third-party audit.

What does Abridge publish about its own evaluation?

In an August 2025 whitepaper, Abridge describes labelling each claim in a note as directly supported, circumstantially supported, unmentioned or contradicted, with severity rated major, moderate or minimal. It reports a detector trained on over 50,000 examples, benchmarked on over 10,000 encounters and validated by physicians over more than 1,000 hours. It says the detector caught 97% of unsupported claims against 82% for GPT-4o.

What does the Abridge method leave out?

Omissions are outside its scope. The method also rests on an internal benchmark with no independent check. The hard cases Abridge names are telling inference from extrapolation, speaker misattribution, unverbalised clinical judgement and subjective severity.

What did GAO find?

GAO-26-109116, from July 2026, found accuracy of AI note and coding tools “difficult to verify” and few independent studies. Its coverage traced a “>95%” accuracy figure to one vendor’s self-report. We read this through a secondary summary, not the GAO document.

What did Ontario’s Auditor General find?

The May 2026 report on AI in the Ontario government found that 11 of 20 approved scribe vendors submitted no third-party audit. Note accuracy carried about 4% of the procurement scoring, which is a weight, not an accuracy rate. We read the report through a search extract.

Source What it shows Caution
Abridge whitepaper Detailed vendor method, severity-graded Internal benchmark; omissions out of scope
GAO-26-109116 Few independent accuracy studies Read via summary
Ontario Auditor General 11 of 20 vendors without third-party audit “4%” is a procurement weight

What does this mean for buyers?

Ask vendors for omission as well as hallucination results, for the unit and sampling behind each number, and for independent or buyer-run testing. See why scribe error rates cannot be compared.

What did we not verify?

We did not read the GAO or Ontario documents in full, and Abridge’s 97% versus 82% result is the vendor’s own.

Frequently asked questions

How does Abridge evaluate hallucinations?

Abridge's August 2025 whitepaper labels each claim as directly supported, circumstantially supported, unmentioned or contradicted, and rates severity as major, moderate or minimal. It reports a detector trained on over 50,000 examples and benchmarked on over 10,000 encounters, which caught 97% of unsupported claims against 82% for GPT-4o. That is an internal benchmark.

Do AI scribe vendors get audited independently?

Often not. Ontario's Auditor General found that 11 of 20 approved scribe vendors had submitted no third-party audit, and GAO found few independent studies of AI note and coding tools in July 2026.

Does the Ontario auditor's 4 percent mean scribes are 96 percent accurate?

No. The 4 percent is a procurement weight, the share of Supply Ontario scoring given to accuracy of notes, not an error rate. The audit's error percentages circulating in coverage are unverified.

Written byPrecision research teamHealthcare AI evaluation research