Research

Limiting Sycophancy in Large Language Models via Conformal Risk Control

Under review · MLHC 2026

• Trustworthy ML
• Large language models
• AI for healthcare

The Conformal Claim Refinement pipeline. A draft answer is split into atomic claims, each claim is scored and filtered with a conformally calibrated threshold, and the answer is then kept, reverted or rewritten.

The problem

Large language models often agree with a user’s incorrect assertion instead of correcting it. In healthcare this is dangerous: a patient who states the wrong drug dosage or dismisses a symptom may get validation instead of correction. Unlike hallucination, which comes from missing knowledge, sycophancy is a failure of robustness: the model knows the answer but gives it up under pressure.

Approach

Conformal Claim Refinement (CCR) is a post-hoc safeguard that needs no retraining:

1. Decompose the model’s response into atomic claims.
2. Score each claim’s validity with a judge model.
3. Apply claim-level conformal risk control to filter out high-risk statements.
4. Rewrite the response from the claims that survive.

Results

• Evaluated sycophancy across multiple model families and parameter scales on MedQuAD and HealthSearchQA, covering both regressive and progressive sycophancy.
• CCR lowers sycophancy in most settings, with average reductions of up to 7.4%.

← All research