Why do AI graders miss omissions in clinical notes?

Two 2026 preprints find LLM judges near chance at spotting what a clinical note leaves out, yet good at catching invented content. What the numbers say and do not.

What did the preprints find about LLM judges and omissions?

Two sets of authors tested whether LLM judges can tell a complete clinical note from an incomplete one. One team found AUC 0.49 to 0.66 across three rubric styles, three models and two clinician-annotated datasets, at or slightly above chance, with no triage benefit at 90% recall. The other found paired discrimination of 0.50 to 0.63 for omissions, against 0.79 to 0.94 for added or altered content.

Test Judges’ discrimination
Completeness (arXiv 2604.16383) AUC 0.49 to 0.66
Omissions (arXiv 2608.31016) 0.50 to 0.63
Added or altered content (same paper) 0.79 to 0.94

Is there a method that works better?

The second paper reports that a single call detects 36.9% of omissions at 6.2% false alarms, against 24.6% for a comparison method. A per-fact pipeline, which enumerates the transcript’s facts and checks each against the note, had 2.7% false alarms; the abstract does not give its recall. The 24.6% to 36.9% and 2.7% to 6.2% figures belong to different methods and should not be quoted as one range.

What are the limits?

The benchmark used 500 note pairs, 298 with one injected omission and 202 controls, validated by a physician on 70 items. Each note had a single injected error, the tuned threshold did not transfer to real vendor notes, and omissions restated elsewhere in the note defeated both methods.

How much should you trust the papers?

All three are non-peer-reviewed v1 preprints. The scribe-omission and census papers are by the same authors, so the second is not independent corroboration, and their code and data sit on a commercial AI-evaluation company’s accounts. We did not check affiliations in the PDFs.

What does this mean for evaluation?

A generic AI judge should not be the only grader for note completeness. Calibrate it against clinician labels, report its own miss rate, and measure omission with severity. See the omission rate and the omission auditor.

Frequently asked questions

Can an LLM judge find what an AI scribe left out of a note?

Only weakly. In the 2026 preprints we read, LLM judges discriminated omissions at 0.50 to 0.63, near chance, and completeness at AUC 0.49 to 0.66. They were much better at spotting content added or altered, at 0.79 to 0.94.

Why is omission harder than hallucination to detect?

A hallucination is something in the note you can check against the transcript. An omission is something absent, so a grader must first work out what should have been there. Enumerating the transcript's facts first, then checking each, helped in the one preprint that tried it.

Should I trust these numbers?

Treat them as a lead. All three papers are non-peer-reviewed v1 preprints. Two are companion papers by the same authors, and their code sits on accounts for a commercial AI-evaluation company, which suggests but does not state a commercial tie.

Written byPrecision research teamHealthcare AI evaluation research