Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs¶

Conference: ECCV 2026
arXiv: 2606.26387
Code: https://xixiaouab.github.io/VIGIL/ (Project Page)
Area: Multimodal VLM / Alignment RLHF
Keywords: Multimodal Hallucination, Visual Laziness, Counterfactual Alignment, DPO, Visual Information Gain

TL;DR¶

To address the "visual laziness" issue where MLLMs get misled by language priors despite perceiving correct visual evidence, this paper proposes VIGIL. It introduces a "counterfactual blind state" (where visual attention is masked) as a negative anchor into DPO preference optimization. By using geometric constraints to explicitly widen the log-likelihood gap between the "see" and "blind" states, VIGIL forces high-confidence model responses to be causally anchored to pixels. It outperforms state-of-the-art methods using only 25% of preference data and unexpectedly exhibits emergent spatial grounding capability.

Background & Motivation¶

Multimodal Large Language Models (MLLMs) have evolved from "perception tools" to "reasoning engines" capable of solving math problems and reading financial charts. However, hallucination remains a major hurdle for reliability. In multimodal contexts, hallucination is not simply "misseeing" but a deeper grounding failure: the generated text is not grounded in the actual pixels. Mechanistic interpretability research further points out that the root cause is a phenomenon known as "visual laziness"—modern MLLMs connect a visual encoder to a text-centric LLM pre-trained on massive corpora, which naturally carries strong language priors. When faced with complex multimodal inputs, the model default to relying on these priors rather than looking at the image. Several studies reveal an intriguing contrast: correct visual features actually exist within the model's latent space (which means the model "sees" the correct information internally), but it ultimately outputs incorrect responses due to language priors hijacking the decoding process.

Existing mitigation methods incur non-trivial costs. One class relies on architectural modifications, separating perception from reasoning via external tools or modular designs, which breaks the elegance of end-to-end learning. Another class employs online reinforcement learning to dynamically select visual tokens, introducing heavy computational overhead and training instability. Result-based preference alignment, such as DPO, has a more hidden issue: it minimizes preference loss on final textual outputs and penalizes incorrect tokens, but does not constrain how the model arrives at the answer. If the MLLM correctly guesses the answer based on language priors rather than visual evidence, standard DPO still rewards the output, unintentionally reinforcing shortcut learning and leaving visual laziness intact. This leads to a fundamental insight: correctness is not a sufficient proxy for grounding—a model can easily get the correct answer for the wrong reasons, and optimizing only the final output rewards can entrench causal attribution errors.

Thus, the key insight of this paper is straightforward: since the problem is the model's reluctance to rely on vision, it must confront the consequences of "blindness". Core Idea: Construct a "counterfactual blind state" \(x_{\text{v}}^{\emptyset}\) by masking visual attention, and incorporate the preference "seeing is better than being blind" as a preference term into DPO. This geometric constraint explicitly maximizes the mutual information between the visual input and the response (Visual Information Gain, or VIG), penalizing overconfident predictions made in the blind state and forcing high-confidence outputs to causally anchor to pixels.

Method¶

Overall Architecture¶

VIGIL is a purely offline post-training framework. Its inputs are standard multimodal preference data \((x_{\text{v}}, x_{\text{t}}, y_w, y_l)\) (visual input, textual instruction, winning/losing responses), and its output is an aligned strategy that avoids blind guessing. Based on the DPO skeleton, VIGIL inserts a geometric constraint via "counterfactual visual decoupling." The pipeline consists of three main steps: first, a dual-path forward pass is conducted for each sample—one path is normal ("seeing") and the other masks the text's attention to visual tokens, creating a "blind" state; second, a Visual Information Gain (VIG) geometric constraint compares the likelihood of the same winning response \(y_w\) under both states, penalizing high confidence under the blind state; finally, a dynamic gating factor \(\alpha\) adaptively fuses the CVD constraint and the standard DPO loss into a unified objective: \(\mathcal{L}_{\texttt{VIGIL}} = \mathcal{L}_{\text{DPO}} + \lambda\alpha\mathcal{L}_{\text{CVD}}\). Additionally, a self-adversarial hard negative mining module is employed to provide fine-grained visual discrimination gradients.

The key lies in the implementation of the "blind state": instead of physical destruction like blacking out or shuffling the image, the image, visual tokens, projector outputs, and position encodings remain completely unchanged. The attention mechanism simply cuts off the text queries' connections to the visual keys/values at each layer. Consequently, the only difference between the two paths is the readability of visual tokens. VIG measures the pure reduction in uncertainty brought by vision, avoiding distributional shifts introduced by blacking out the image.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Preference Sample<br/>(Image, Instruction, Win/Lose Response)"] --> B["Efficient Visual Counterfactual<br/>Cut text→vision attention for blind state"]
    B --> C["See Path<br/>logπ(y|xᵥ,xₜ)"]
    B --> D["Blind Path<br/>logπ(y|xᵥ∅,xₜ)"]
    C --> E["Counterfactual Visual Decoupling CVD<br/>Widen see/blind log-likelihood gap"]
    D --> E
    A --> F["Self-Adversarial Hard Negatives<br/>Generate plausible hallucinated negative answers"]
    E --> G["Dynamic Gating α<br/>Automatically weaken regularization for grounded samples"]
    F --> G
    G --> H["Unified Objective L_VIGIL<br/>= L_DPO + λα·L_CVD"]

Key Designs¶

1. Visual Information Gain (VIG): An Optimizable Metric for Visual Dependence

The limitation of standard DPO is straightforward: it only evaluates the correctness of final texts without measuring how much the answer depends on vision. Drawing on Maximum Mutual Information (MMI), VIG is defined as the pointwise mutual information (PMI) between visual input \(x_{\text{v}}\) and response \(y\) conditioned on instruction \(x_{\text{t}}\). Mathematically, it is the ratio of log-likelihoods of the identical response under the "see" and "blind" conditions:

\[\text{VIG}(y, x_{\text{v}}|x_{\text{t}}) = \log\frac{\pi_\theta(y|x_{\text{v}}, x_{\text{t}})}{\pi_\theta(y|x_{\text{v}}^{\emptyset}, x_{\text{t}})}\]

where \(x_{\text{v}}^{\emptyset}\) represents the attention-masked blind state, functionally equivalent to a do-intervention on the visual modality. The physical meaning of this quantity is "the reduction in uncertainty purely driven by visual evidence." A VIG near 0 indicates that the model heavily relies on language priors and treats vision as statistical redundancy, which is the core symptom of visual laziness. With this metric, multimodal alignment is reformulated as a constrained optimization problem: while maximizing preference rewards, the conditional mutual information of the winning response must exceed a threshold \(\mathbb{E}[\text{VIG}(y_w, x_{\text{v}}|x_{\text{t}})] \geq \delta\). This "bends" the multimodal manifold away from the text-only manifold, preventing optimization from collapsing into shell text-only shortcuts.

2. Counterfactual Visual Decoupling (CVD) Loss: Turning "See ≻ Blind" into a Preference Pair

Since the constrained optimization problem is difficult to solve directly, a Lagrange multiplier \(\lambda \geq 0\) is introduced to absorb the constraint into the objective. This is elegantly transformed by instantiating "for the same winning response \(y_w\), the see state \((x_{\text{v}}, x_{\text{t}})\) should be preferred over the blind state \((x_{\text{v}}^{\emptyset}, x_{\text{t}})\)" directly as a Bradley-Terry preference pair. Consequently, the Lagrange term becomes a DPO-isomorphic loss:

\[\mathcal{L}_{\text{CVD}}(\pi_\theta) = -\mathbb{E}_{\mathcal{D}}\left[\log\sigma\left(\beta\log\frac{\pi_\theta(y_w|x_{\text{v}}, x_{\text{t}})}{\pi_{\text{ref}}(y_w|x_{\text{v}}, x_{\text{t}})} - \beta\log\frac{\pi_\theta(y_w|x_{\text{v}}^{\emptyset}, x_{\text{t}})}{\pi_{\text{ref}}(y_w|x_{\text{v}}^{\emptyset}, x_{\text{t}})}\right)\right]\]

Its behavior is as follows: if the likelihood of a response does not drop after vision is removed, it implies that the output does not rely on visual cues, and CVD heavily penalizes such "blind overconfidence." A subtle but crucial implementation detail emphasizes that the frozen reference model \(\pi_{\text{ref}}\) must be evaluated under the identical attention masks as the policy model: the "see" term uses the see mask for both models, and the "blind" term uses the blind mask for both models. This compares "matched see/blind states" rather than normalizing the blind policy against a see reference, preventing spurious signals from misaligned reference models. Compared to implicit reweighting based on difficulty in DA-DPO, CVD performs physical intervention on the input geometry. This component contributes the most in ablation studies (w/o CVD leads to a 3.2% drop on POPE), serving as the lifeline of the method.

3. Dynamic Gating \(\alpha\) and Self-Adversarial Hard Negatives: Penalizing Only When Necessary

Applying CVD uniformly to all samples can over-regularize models that are already well-grounded, harming text generation fluency. Thus, a dynamic gating factor is designed to adaptively regulate the constraint intensity based on the "see/blind likelihood gap" (VIG gap):

\[\alpha = 1 - \tanh\left(\left|\log\pi_\theta(y_w|x_{\text{v}}, x_{\text{t}}) - \log\pi_\theta(y_w|x_{\text{v}}^{\emptyset}, x_{\text{t}})\right|\right)\]

When a sample has a large VIG gap (indicating adequate grounding), \(\alpha\) automatically approaches 0, relaxing the regularization. It focuses exclusively on lazy samples with small gaps. Ablations show that removing this gating factor causes a modest 1.1% performance drop, acting as a stabilizer for the gradient flow. Parallelly, self-adversarial hard negative mining utilizes the reference model to actively generate hallucinated negative samples \(y_{\text{hard}}\) containing plausible but non-existent details. Replacing the weaker standard \(y_l\) in DPO with \(y_{\text{hard}}\) provides sharper visual discrimination gradients. The unified objective is: \(\mathcal{L}_{\texttt{VIGIL}} = \mathcal{L}_{\text{DPO}}(y_w, y_{\text{hard}}) + \lambda\cdot\alpha\cdot\mathcal{L}_{\text{CVD}}\), so that the model learns not just "what to say" but "on what visual basis to say it."

Loss & Training¶

The pipeline is implemented using the OpenRLHF framework with full parameter fine-tuning. Training is limited to 1 epoch to prevent overfitting. The KL penalty coefficient \(\beta\) is fixed at 0.1, the weight coefficient is \(\lambda = 1.0\) (validated as a solid trade-off between hallucination mitigation and reasoning performance), and the learning rate is \(5\times10^{-7}\) with cosine decay. The global batch size is 2048 for 72B and 1024 for 7B. DeepSpeed ZeRO-3 with CPU offloading and FlashAttention-2 is used for 72B, while FSDP is employed for 7B. Because the counterfactual blind state is implemented via attention mask manipulation rather than physical input tensor edits, the blind-path forward pass incurs almost zero extra overhead—the additional TFLOPs from CVD constraints account for less than 1% of the total forward-backward passes.

Key Experimental Results¶

Main Results¶

Across Qwen2.5-VL models ranging from 7B to 72B, VIGIL achieves SOTA performance on hallucination and reasoning benchmarks. Notably, larger models yield larger gains—improving POPE-Adv by 4.1 points on 7B and 5.3 points on 72B, supporting the "post-training scaling law" where stronger base models leverage counterfactual visual signals more effectively. Conversely, text-centric alignment methods like SimPO exhibit diminishing returns on multimodal tasks.

Base Model / Method	POPE-Adv↑	AMBER-Gen↑	MMHal-Score↑	MathVista↑	MMBench↑
Qwen2.5-VL-7B + DPO	82.8	42.5	36.8	48.0	64.6
7B + DA-DPO (Prev. SOTA)	84.2	44.8	38.5	48.8	65.9
7B + VIGIL	86.9 (+4.1)	46.5 (+4.0)	40.2 (+3.4)	49.5	67.3
Qwen2.5-VL-72B + DPO	84.5	46.8	40.5	54.1	68.1
72B + DA-DPO (Prev. SOTA)	87.4	49.8	44.2	55.4	70.3
72B + VIGIL	89.8 (+5.3)	52.2 (+5.4)	46.0 (+5.5)	56.6	72.1

In cross-architecture validation, VIGIL steady improves POPE-Adv by ~4.0 points over DPO on both the projector-based LLaVA-OneVision-7B and the vision-centric InternVL2.5-26B (equipped with a 6B visual encoder). This demonstrates that even with extremely powerful visual encoders, models still exhibit visual laziness, which can be effectively rectified by counterfactual attention masking.

Ablation Study¶

Configuration	POPE-Rand↑	AMBER-Gen↑	Description
Full VIGIL	88.5	46.5	Full model
− Visual Anchor (\(x_{\text{v}}^{\emptyset}\))	85.3 (−3.2)	43.1 (−3.4)	Removed blind state anchor; largest drop, the lifeline
− Hard Negatives	86.5 (−2.0)	44.2 (−2.3)	Removed hard negatives
− Dynamic Gating (\(\alpha\))	87.4 (−1.1)	45.4 (−1.1)	Stabilized gradient flow
DPO baseline	86.2	42.5	—

A comparison of policy mechanisms highlights the difference: DPO (Equality) 82.8 → filtering simple samples 84.3 → DA-DPO implicit reweighting 85.7 → CVD physical masking 88.5 (+5.7). This confirms that "physical interventions on input geometry" provide significantly more effective supervisory signals than "numerical tuning of loss weights."

Key Findings¶

Visual Dependence Index (VDI) Bucketing: Standard DPO scores only 78.2 on high-VDI samples, whereas VIGIL boosts this to 87.5. It precisely rectifies "blind guessing" hallucinations, which are the primary pain points of the baseline.
Data & Compute Efficiency: VIGIL matches the performance of DA-DPO trained on full data using only 25% of the dataset, reducing total wall-clock training time by ~70% (finishing the 7B pipeline in 3.5 GPU hours on a single A100). Constructing the blind path via attention mask modification incurs negligible overhead.
Emergent Spatial Grounding: Despite receiving zero bounding box supervision, zero-shot RefCOCOg performance improves by 4.3 points (whereas DPO drops by 0.4). To satisfy the counterfactual grounding objective, the model spontaneously develops the capability to anchor text to specific image regions.
Information-Theoretic Evidence: On high-VDI samples, VIGIL's VIG jumps from DPO's 2.1 to 7.2, and its blind-state prediction entropy "explodes" from 0.65 to 1.88. This proves that the model genuinely experiences a loss of certainty without visual anchors, underscores that the constraint alters the dependency pathway rather than merely reweighting loss.
Preserved Text-only Capabilities: Performance on MMLU/GSM8K fluctuates by less than 0.2 points (well within the \(\pm 0.3\) random noise range), indicating no general capability tax.

Highlights & Insights¶

Constructing the "blind state" via attention masking rather than blacking out images is the most ingenious design: Blacking out or shuffling images introduces out-of-distribution (OOD) shifts that corrupt the likelihood ratios. In contrast, simply cutting off the "text query → visual key/value" attention connections while keeping the visual embeddings and positional masks intact allows the likelihood difference to be strictly attributed to cross-modal access. Ablation studies confirm that attention masking (86.9) is superior to blacking out (84.7), blurring (85.6), and shuffling (85.9).
Injecting causal counterfactuals into the DPO preference pair is nearly zero-cost: It bypasses the need for extra reference models or online sampling, directly formulating "see \(\succ\) blind" into a loss isomorphic to DPO. Fitting full parameter fine-tuning for 72B on this framework highlights a highly transferable approach of leveraging pre-existing preference frameworks to host new supervisory signals.
Design philosophy of the dynamic gating \(\alpha\): Instead of mindlessly applying global constraints, the philosophy is "relaxing constraints on grounded samples while penalizing lazy ones." It adaptively controls intensity via \(1-\tanh(\text{VIG gap})\), a clean one-line implementation that can be applied to any scenario requiring state-dependent regularization strength.
The emergent spatial grounding capability is highly inspiring: Preventing hallucination inherently obligates the model to understand the visual world at a deeper level. The visual grounding goal naturally pulls the model toward self-developing localization abilities.

Limitations & Future Work¶

The counterfactual blind state is constructed via a relatively straightforward "null mask." In highly cluttered scenes, it may not capture finer cross-modal conflicts. Future research could explore more granular counterfactual interventions like object-level spectral filtering or semantics-preserving transformations.
The performance hinges heavily on the quality and visual dependence distribution of the preference data. There is minimal benefit for low-VDI samples where answers can be inferred merely through text (Low bucket yields 89.2 vs DPO's 88.5); the gains are heavily concentrated in medium-to-high visual dependence scenarios.
The validity of VIG/CVD assumes that the blind-state likelihood truly reflects visual dependence. If the model possesses internal text-vision leakage channels that bypass the masked attention paths, the blind state might not be completely "blind." This assumption warrants further rigorous validation.
Thus far, validation is confined to the DPO family. While the authors discuss that GRPO might be computationally prohibitive due to the need to construct multiple blind variants per sample, whether geometric constraints remain effective under other RL frameworks remains an open question.

vs DA-DPO: Both aim to mitigate multimodal hallucination. DA-DPO relies on "implicit reweighting of preference pairs based on difficulty" to emphasize hard samples, whereas this paper performs "counterfactual physical masking on visual inputs." The key difference is that the former adjusts weights at the numerical level, while the latter cuts off access paths at the geometric level. Ablations show physical intervention (+5.7) is significantly superior to reweighting (+2.9).
vs VCD (Visual Contrastive Decoding): VCD contrastively decodes using logits from the original image and a distorted image to suppress hallucination. This almost doubles inference latency and is highly sensitive to hyperparameters. VIGIL internalizes this visual contrast into weights during post-training, producing a highly efficient inference policy. Experiments show that stacking VCD on top of VIGIL offers only marginal improvements (86.9 → 87.1) while doubling latency.
vs Architectural Methods (e.g., VGent): These approaches utilize external tools or decoupled reasoning modules to separate perception and reasoning, compromising end-to-end elegance and generalization. VIGIL introduces zero extra parameters, treating the counterfactual blind state purely as a negative anchor within an elegant end-to-end alignment framework.
vs Visual-GRPO: The GRPO baseline built by the authors uses POPE accuracy as a reward. Although competitive on MathVista (as GRPO excels at reasoning tasks), it lags behind on hallucination benchmarks. This confirms that "optimizing only final answer correctness" fails to distinguish true grounding from prior-driven guessing, which might even entrench visual laziness.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Instantiating the causal counterfactual "blind state" as a DPO preference pair and offering VIG as an optimizable grounding metric is a highly novel and clean approach.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Validated across axes of 7B-72B model scales, diverse architectures, and perspectives of ablation studies, mechanisms, efficiencies, and information theory, alongside unexpected findings of emergent localization.
Writing Quality: ⭐⭐⭐⭐ The logical flow is clear, and the mathematical derivations are thorough. However, metaphorical expressions like "sword of Damocles" are slightly overused, and the appendix is heavily packed.
Value: ⭐⭐⭐⭐⭐ Completely offline, incurs virtually zero extra cost, and outperforms SOTA using 25% of preference data—highly attractive for production-level multimodal alignment.