Sycophancy to clinician or patient assertions
Adversarial rubrics.
What sycophancy to clinician or patient assertions means
Adversarial rubrics.
Why it matters. Tools that agree too readily with a wrong assertion can reinforce mistakes.
A user says a drug is safe in pregnancy and the tool agrees.
How to measure it
Write real questions
Specialists write questions from real decisions, including hard ones.
Name the reference
Fix the guideline version each answer will be judged against.
Grade with rubrics
Score answers on clinician-written criteria.
Compare
Run the same questions through a general frontier model.
Comparing a tool only with its earlier version flatters it.
Scores with named guideline versions, a general-model baseline and clinician agreement.
Report the question set, the specialty mix, who graded the answers and against which guideline version, and compare against a general frontier model, not only against an earlier version of the same tool.
Reviewed by the Precision research team.
How strong is the evidence
E2 · Preprint, benchmark page or vendor-run result. Backed by numbers from a preprint, a benchmark page or a vendor-run test. Treat as strong but not settled.
What the sources say
SycoEval-EM; judges under-label some behaviours (arXiv 2601.16529).
- Automatable. Can be scored by software, including an AI grader that is spot-checked against clinicians.
- Clinician judgement. Needs a clinician to judge or label, because the answer depends on clinical meaning.
- arXiv 2601.16529arxiv.org
Found something out of date? Write to hello@precision.example.
Agents that measure this
Quick answers
What is sycophancy to clinician or patient assertions in healthcare AI?
Adversarial rubrics. It sits in the clinical question answering and decision support group.
Why does sycophancy to clinician or patient assertions matter?
Tools that agree too readily with a wrong assertion can reinforce mistakes. A user says a drug is safe in pregnancy and the tool agrees.
How is sycophancy to clinician or patient assertions measured?
Write real questions, then Name the reference, then Grade with rubrics, then Compare. Report the question set, the specialty mix, who graded the answers and against which guideline version, and compare against a general frontier model, not only against an earlier version of the same tool.
What should I watch out for with sycophancy to clinician or patient assertions?
Comparing a tool only with its earlier version flatters it. A good report gives: scores with named guideline versions, a general-model baseline and clinician agreement.
Can sycophancy to clinician or patient assertions be automated?
Only in part. It needs automatable and clinician judgement. Software can screen, but a clinician must judge the cases that matter.