Mitigating Sycophancy in Multimodal Chart Understanding via Vision-Grounded Verification¶

Conference: ECCV 2026
Paper: ECCV 2026 Poster 5590
Code: https://github.com/X1Wang/ReCheck-code
Area: Multimodal VLM
Keywords: Sycophancy Mitigation, Chart Understanding, Vision-Grounded Verification, Contrastive Visual Dependency Score, Training-Free Inference

TL;DR¶

Addressing the safety-utility trade-off where suppressing multimodal chart sycophancy causes severe over-refusal, this paper proposes Re-Check, a training-free inference framework that integrates blind atomic claim decomposition, a KL-divergence-based Contrastive Visual Dependency Score (CVDS) to certify visual causality, and adaptive three-way routing to achieve robust factual correction while preserving general reasoning utility.

Background & Motivation¶

Multimodal Large Language Models (MLLMs) have emerged as the dominant paradigm for automated chart understanding and data-driven decision-making in financial and scientific domains. However, these models exhibit a critical vulnerability: sycophancy. When users embed false or misleading premises in their queries (e.g., asking "Why is the red line increasing?" when it is visually declining), models frequently prioritize conversational compliance and human preference alignment over visual faithfulness, fabricating plausible explanations to align with the user's misconception. This behavior undermines objectivity in automated analysis pipelines and risks amplifying misinformation when charts are presented with misleading textual framing.

Mitigating multimodal sycophancy is fundamentally hindered by the safety-utility trade-off. Naively suppressing sycophantic compliance causes models to become overly stubborn, triggering widespread over-refusal on legitimate user queries and catastrophically degrading general benchmark utility. Existing text-only verification-chain techniques (e.g., CoVe) rely exclusively on internal parametric knowledge, making them ill-suited for resolving contradictions between textual presuppositions and external visual chart evidence. Meanwhile, training-free hallucination mitigations (e.g., VCD) operate at the token decoding level for natural image object presence, failing to capture subtle logical contradictions and textual compliance dynamics in abstract visualizations.

The key insight of this paper is that premise verification must be explicitly decoupled from response generation, decomposing user claims purely from text and certifying the visual causality of refutations before intervening. Core idea: build a training-free Re-Check framework under the verify-then-answer paradigm, employing blind text-only atomic claim decomposition, quantifying visual causal refutation via an information-theoretic Contrastive Visual Dependency Score (CVDS) to filter textual bias, and deploying adaptive three-way routing across correction, standard generation, and uncertainty-aware cautious reasoning.

Method¶

Overall Architecture¶

Re-Check is an entirely training-free test-time inference framework operating on a "Verify-then-Answer" principle. The pipeline consists of three sequential stages: first, the model's text-only language branch blindly decomposes the input query into atomic claims without visual input; second, the multimodal model performs an initial batch verification on each claim, followed by CVDS recalibration to verify whether refutations stem from genuine visual evidence rather than textual priors; finally, recalibrated claims are processed by an adaptive three-way routing mechanism to steer the query into sycophancy correction, standard direct generation, or uncertainty-aware cautious reasoning.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Chart Image I and Query Q"] --> B["Blind Atomic Claim Decomposition<br/>Text-only branch extracts claim set C"]
    B --> C["Vision-Grounded Batch Verification<br/>M_vlm predicts TRUE/FALSE/UNCERTAIN"]
    C --> D["CVDS Contrastive Recalibration<br/>KL divergence between with- and without-image distributions"]
    D --> E["Adaptive Three-way Routing"]
    E -->|At least one high-confidence FALSE| F["Route A: Sycophancy Correction Generation"]
    E -->|All claims verified as TRUE| G["Route B: Standard Generation"]
    E -->|UNCERTAIN present and no FALSE| H["Route C: Uncertainty-Aware Cautious Reasoning"]
    F --> I["Final Reliable Response R"]
    G --> I
    H --> I

Key Designs¶

1. Blind Atomic Claim Decomposition: Decoupling User Presupposition from Visual Perception Directly assessing complex queries with embedded premises leads to logical entanglement. If the chart image \(I\) is accessible during claim extraction, the multimodal model unconsciously reconciles extracted statements with visual facts, thereby muting or ignoring the user's erroneous premise before formal verification occurs. To prevent this perceptual assimilation, Re-Check isolates the textual reasoning capacity of the language model branch: \(C = f_{\text{LLM}}(Q) = \{c_1, c_2, \dots, c_N\}\). This blind extraction ensures that the extracted claim set \(C\) faithfully preserves the user's raw presuppositions without premature visual bias, establishing an objective target set for downstream verification.

2. CVDS Contrastive Recalibration: Information-Theoretic Quantification of Visual Causality In the initial verification phase, the multimodal model predicts a tripartite status \(s_i \in \{\text{TRUE}, \text{FALSE}, \text{UNCERTAIN}\}\) for each atomic claim \(c_i\). However, initial FALSE classifications are frequently contaminated by linguistic priors—the model may reject a claim simply because its textual phrasing appears improbable. To isolate genuine visual observation from textual bias, the paper introduces the Contrastive Visual Dependency Score (CVDS). For claims initially classified as FALSE, two forward passes collect the output distribution over state space \(\mathcal{S}\) under the with-image condition \(P_{\text{img}}(s) = P(s \mid c_i, I)\) and the text-only condition \(P_{\text{text}}(s) = P(s \mid c_i)\), measuring their KL divergence:

\[\text{CVDS}(c_i) = D_{\text{KL}}\!\left(P_{\text{img}} \parallel P_{\text{text}}\right) = \sum_{s \in \mathcal{S}} P(s \mid c_i, I) \log \frac{P(s \mid c_i, I)}{P(s \mid c_i)}\]

This formulation rigorously approximates the conditional mutual information \(I(S; I \mid C)\). When \(\text{CVDS}(c_i) \ge \tau\) (with default threshold \(\tau = 2.0\)), the refutation is causally driven by visual evidence—the model genuinely observed a contradiction—and the FALSE label is retained. If \(\text{CVDS}(c_i) < \tau\), the model refutes the statement regardless of visual input, indicating a textual bias artifact; the judgment is adaptively downgraded to UNCERTAIN to prevent unreliable over-correction.

3. Adaptive Three-way Routing: Balancing Correction Robustness and General Utility To overcome the rigid trade-off between sycophancy mitigation and utility preservation, Re-Check avoids binary acceptance/rejection by deploying a three-way routing scheme over the recalibrated claims \(\{(c_i, \hat{s}_i)\}\). If any claim remains verified as \(\hat{s}_i = \text{FALSE}\), the query enters Route A (Sycophancy Correction), generating an explicit rebuttal that cites the falsified premise alongside specific visual contradictions. If all claims are confirmed as \(\hat{s}_i = \text{TRUE}\), Route B (Standard Generation) directly executes standard inference on the original query \(Q\) without intrusive prompting, preserving baseline utility without friction. When claims include UNCERTAIN states without any verified FALSE, Route C (Uncertainty-Aware Reasoning) treats the claim set as external guidance, instructing the model to scrutinize image details during generation without enforcing an explicit refutation. Route C serves as an indispensable buffer against decoupling noise, eliminating over-refusal caused by borderline claims.

Key Experimental Results¶

Main Results¶

On the ChartHal benchmark, which evaluates chart hallucination and sycophancy across Question Types (Descriptive Desc., Reasoning Reason, Open-ended Open) and Chart-Question Relations (Irrelevant Irrel., Inexistent Inexist., Contradictory/Sycophantic Contra., Normal), Re-Check is compared against Direct Inference (Base), Integrated Prompting (Prompt), and Visual Contrastive Decoding (VCD).

Model	Method	Desc.	Reason	Open	Irrel.	Inexist.	Contra.	Normal	Overall
Qwen3-VL-8B	Base	55.35	50.31	66.29	72.76	65.41	38.57	45.61	57.49
	Prompt	65.80	71.43	80.11	92.19	90.12	60.95	34.31	72.32
	VCD [10]	54.31	54.04	66.39	75.46	69.19	32.38	46.03	58.29
	Re-Check (Ours)	67.62	70.81	79.27	89.59	87.79	57.14	44.77	72.50
InternVL3.5-8B	Base	22.19	10.25	7.84	4.46	7.27	0.48	45.19	13.75
	Prompt	59.79	57.45	75.07	84.39	81.69	43.33	34.73	64.22
	VCD [10]	24.80	13.98	8.12	4.46	13.95	0.95	44.77	15.91
	Re-Check (Ours)	59.53	56.52	73.95	80.67	79.94	39.05	41.84	63.47
Qwen3-VL-32B	Base	59.27	56.83	45.66	54.65	62.50	29.52	62.34	53.95
	Prompt	72.58	70.19	86.83	91.82	87.21	69.52	50.63	76.65
	VCD [10]	61.36	53.42	47.62	57.62	59.01	32.86	62.76	54.33
	Re-Check (Ours)	75.72	74.84	82.63	91.45	86.63	67.14	59.00	77.78

Utility preservation is further evaluated across five task categories on the challenging ChartQAPro benchmark:

Model	Method	Factoid	Convers.	Hypoth.	FactChk.	MCQ	Overall
Qwen3-VL-8B	Base	40.36	44.32	66.09	32.79	4.21	37.37
	Prompt	33.29	40.92	45.30	43.44	7.01	33.49
	VCD [10]	40.29	43.22	63.62	36.89	7.48	37.90
	Re-Check (Ours)	36.70	42.13	63.79	45.49	2.80	36.31
InternVL3.5-8B	Base	38.55	49.12	50.53	9.43	25.70	35.78
	Prompt	28.43	44.90	42.30	34.02	1.87	29.54
	VCD [10]	37.26	48.55	63.39	11.31	28.50	36.17
	Re-Check (Ours)	35.53	46.28	45.69	44.26	18.22	36.95

Ablation Study¶

Ablation analysis on Qwen3-VL-8B using ChartHal isolates the contributions of Stage 1 decomposition backbone (LLM vs. VLM), Stage 2 CVDS recalibration, and Stage 3 routing scheme (2-way vs. 3-way):

Config	Stage 1 (Decomp.)	Stage 2 (CVDS)	Stage 3 (Routing)	Contra.	Normal	Overall	Note
Full Model	LLM	w/ CVDS	3-way	57.14	44.77	72.50	Complete Re-Check framework
Alt. Decomp.	VLM	w/ CVDS	3-way	56.67	40.59	68.08	Premise assimilation degrades accuracy by 4.42%
w/o CVDS	LLM	w/o CVDS	3-way	56.19	43.51	72.03	Lack of visual causality drops Normal by 1.26%
2-way Routing	LLM	w/ CVDS	2-way	44.76	45.19	62.62	Forcing binary choices drops Overall by 9.88%
Base Model	-	-	-	38.57	45.61	57.49	Baseline without verification intervention

Sensitivity analysis for the CVDS threshold \(\tau\) (with Trigger Rate denoting the fraction of FALSE claims downgraded to UNCERTAIN) is presented below:

Threshold \(\tau\)	Trigger Rate (%)	Contra.	Normal	Overall	Note
\(\tau = 0.0\)	0.0	56.19	43.51	72.03	No downgrades applied (equivalent to w/o CVDS)
\(\tau = 1.0\)	11.9	56.67	43.93	72.41	Initial filtering of weak visual dependency
\(\tau = 2.0\)	18.4	57.14	44.77	72.50	Optimal trade-off maximizing Overall and Normal
\(\tau = 3.0\)	24.0	56.67	44.35	72.41	More aggressive downgrade slightly trims recall
\(\tau = 4.0\)	34.6	56.19	44.77	72.03	Conservative refutations plateau Overall score

Key Findings¶

Sycophancy is pervasive across MLLMs: Base models collapse under contradictory prompts, achieving only 38.57% (Qwen3-VL-8B) and 0.48% (InternVL3.5-8B) accuracy on Contradictory queries, demonstrating near-total compliance with misleading user inputs.
Stage isolation and three-way routing conquer over-refusal: While Integrated Prompting achieves adversarial gains, it induces severe over-refusal on Normal queries (dropping by 11.3% on Qwen3-VL-8B). In contrast, Re-Check constrains Normal query degradation to within 0.84% and maintains 97.2% utility on ChartQAPro, achieving an AUROC of 0.942 against 0.835 for single-pass prompting.
Blind text decomposition prevents visual assimilation: Giving the model visual access during claim extraction allows it to rationalize or overlook false premises in advance, leading to an immediate 4.42% drop in overall accuracy.

Highlights & Insights¶

Contrastive Visual Dependency Score (CVDS): By formulating the KL divergence between predictions with and without visual conditioning, CVDS establishes an elegant information-theoretic metric that isolates visual causality from textual bias.
Counter-Intuitive Value of Blind Decomposition: Intentionally withholding visual inputs during multimodal reasoning prevents premature cross-modal attention assimilation, demonstrating the vital role of modular stage isolation in logical auditing.
Adaptive Buffer Architecture: Incorporating an UNCERTAIN routing path rather than forcing binary judgments buffers system noise from decoupled verification, providing an architectural blueprint for robust alignment in multimodal agents.

Limitations & Future Work¶

Increased Inference Latency: Due to separate forward passes for text decomposition, claim verification, and conditional dual-pass probing, end-to-end inference latency is approximately 3.6 times that of direct generation (\(\sim 3.6\times\)).
Dependence on Language Branch Decomposition: Stage 1 decomposition relies heavily on the instruction-following competence of the underlying LLM; upstream errors in claim extraction propagate to downstream verification.
Cross-Domain Generalization: While validated on chart understanding, the claim-level verification and visual causality scoring mechanisms are theoretically transferable to document understanding and natural scene fact-checking, which remain open for future study.

vs. CoVe / Text-Only Verification Chains: CoVe relies solely on the LLM's parametric internal knowledge, rendering it ineffective when external visual evidence directly contradicts textual premises; Re-Check anchors verification in multimodal chart evidence and validates causal grounding via CVDS.
vs. VCD / OPERA Contrastive Decoding: VCD operates at the token-level decoding dynamics to suppress object-level hallucinations in natural images; Re-Check operates at the claim level to resolve complex logical and relational contradictions in structured visualizations.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Proposes an information-theoretic visual dependency metric (CVDS) and modular blind decomposition to tackle multimodal sycophancy.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across ChartHal and ChartQAPro, featuring multi-model comparisons, granular ablations, threshold sensitivities, and ROC analyses.
Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear framing, rigorous mathematical definitions, and tightly connected experimental evidence.
Value: ⭐⭐⭐⭐⭐ Offers a practical, plug-and-play, training-free paradigm that successfully navigates the safety-utility trade-off in multimodal decision systems.