Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention

Conference: CVPR2026
arXiv: 2511.20032
Code: github.com/beta-nlp/VGA
Area: Hallucination Detection
Keywords: Multimodal hallucination, visual attention, visual semantic confidence, training-free, FlashAttention compatible

TL;DR

This paper proposes Vision-Guided Attention (VGA), a training-free method that constructs precise visual grounding from the semantic features of visual tokens to guide model attention toward relevant visual regions, effectively mitigating hallucinations in MLLMs while remaining compatible with FlashAttention.

Background & Motivation

Despite significant progress in visual understanding, MLLMs frequently produce hallucinated outputs that contradict actual visual content. Existing dehalucination methods fall into two main categories: training-based and training-free. Training-based methods construct datasets or design loss functions, but rapid model architecture iteration leads to diminishing returns. Training-free methods offer greater practical value, particularly those that optimize visual attention.

Core issues with current visual attention optimization methods: 1. Over-reliance on the quality of attention itself, whose localization capability is inherently limited (affected by the attention sink phenomenon) 2. Use of external tools or additional forward passes that introduce computational overhead 3. Methods relying on attention weights are incompatible with FlashAttention

Key finding: Models can accurately extract semantic features from visual tokens and convert them into conditional probabilities (visual logits), yet this advantage is not fully exploited during inference — suggesting that the visual understanding of MLLMs is underestimated.

Method

Overall Architecture

VGA proceeds in two steps: (1) constructing visual grounding via Visual Semantic Confidence (VSC), and (2) using the grounding to guide visual attention. Only one forward pass per token is required.

Key Designs

1. Visual Semantic Confidence (VSC):

   • Semantic confidence of visual token \(v_i\) for object O: \(c_{v_i}(O) = \text{softmax}[\text{logit}_{v_i}(O)]\)
   • Approximated using the first tokenized token \(o_0\) of object O
   • Confidence of object O over the entire image via max pooling: \(c(O) = \max c_{v_i}(o_0)\)
   • Visual grounding: \(G_O = \text{Norm}[\{c_{v_i}(o_0)\}_{i=1}^m]\)
   • Empirical validation shows VSC's localization capability significantly outperforms visual attention, especially for large objects (unaffected by attention sink)

2. Visual Semantic Salience (VSS) — target-free grounding for image captioning:

   • For tasks without specific targets (e.g., captioning), output uncertainty is used to measure the semantic salience of visual tokens
   • \(c_{v_i} = -\sum_k \log c_{v_i}(w_k) / \log K\) (entropy over Top-K tokens)
   • High-VSS tokens correspond to meaningful object regions; low-VSS tokens correspond to semantically insignificant background

3. Vision-Guided Attention (VGA):

   • Core formula: \(\hat{z} = z + \beta \cdot \gamma \cdot \Delta z\)
   • Where \(\Delta z\) is the guidance signal, \(\beta\) is the guidance strength, and \(\gamma\) is the attention head balancing coefficient
   • Key property: VGA does not require computing attention weights → fully compatible with FlashAttention
   • Exploits the associative law of addition: \(\hat{z} = (\alpha + \beta \cdot G)V = z + \beta \cdot \Delta z\)

4. Attention Heads Balancing:

   • Heads with stronger visual functionality receive weaker guidance; non-visual heads receive stronger guidance
   • The cosine similarity between \(z\) and \(\Delta z\) approximates the degree of visual functionality across heads
   • \(\gamma = \text{ReLU}(2 - H \cdot \gamma')\)

5. Programmed Visual Grounding (PVG) — dynamic guidance for captioning:

   • Dynamically updated during generation: \(G_{t+1} = (1+\lambda)G_t - \lambda G_w\)
   • Suppresses already-described regions and redirects attention to undescribed regions
   • Guidance strength automatically decays as generation progresses: \(\|G\|_0\) is used as the decay factor

Loss & Training

VGA is a fully training-free method applied only at inference time. Hyperparameters include guidance strength \(\beta\) and decay parameter \(\lambda\).

Key Experimental Results

Main Results

Dataset	Metric	VGA	Prev. SOTA	Gain
POPE (Acc, avg.)	Accuracy	SOTA	Multiple baselines	Consistent improvements over LLaVA-7B/13B/Next and Qwen2.5-VL
POPE (F1, avg.)	F1	SOTA	PAI/PAICD, etc.	Consistent gains across models

Ablation Study

Configuration	Key Metric	Notes
PSP only	Improvement	Validates position-timestep penalty effect
VGA on different MLLMs	Consistent improvement	Strong generalizability of the method
VSC grounding vs. attention grounding	Dice score significantly higher	Especially advantageous for large objects

Key Findings

• VSC prediction accuracy, while lower than the model's own responses, exhibits correct preference bias (significantly above 50%)
• A preference gap between VSC and model responses confirms that the model's visual understanding is not fully utilized
• VGA achieves state-of-the-art hallucination mitigation without introducing additional forward passes (one pass per token)

Highlights & Insights

• The core insight is particularly elegant: visual logits in MLLMs contain rich semantic grounding information that is underutilized during inference
• The method design is clean: exploiting the associative law of addition to bypass attention weight computation, achieving FlashAttention compatibility
• Attention Heads Balancing is a practical design choice that avoids disrupting the model's existing visual-functional heads
• PVG provides an effective paradigm for dynamic attention guidance in captioning scenarios

Limitations & Future Work

• Approximating object semantics with the first token may be insufficiently precise, especially for multi-syllable or multi-token objects
• The hyperparameter \(\beta\) requires manual tuning and may need adjustment across different models and tasks
• Integration with training-based methods has not been explored; complementary gains may be achievable
• The decay strategy in PVG is heuristic and may be unstable for long-form descriptions

Related Work & Insights

• Contrastive decoding methods (VCD, ICD, etc.) typically require additional forward passes to activate hallucination features
• Attention editing methods (PAI, OPERA, etc.) rely on attention weights and are incompatible with FlashAttention
• VGA successfully introduces visual semantic confidence as a novel visual prior for attention guidance; this paradigm is generalizable to other tasks requiring precise visual grounding

Rating

• Novelty: ⭐⭐⭐⭐⭐ VSC is an entirely new concept; the FlashAttention-compatible design is highly practical
• Experimental Thoroughness: ⭐⭐⭐⭐ Multi-model, multi-benchmark validation with both quantitative and qualitative analysis
• Writing Quality: ⭐⭐⭐⭐⭐ Clear logic with a natural progression from observations to methodology
• Value: ⭐⭐⭐⭐⭐ Training-free and FlashAttention-compatible — exceptionally high deployment value

Related Papers

• [ACL 2026] Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigation
• [CVPR 2026] Fighting Hallucinations with Counterfactuals: Diffusion-Guided Perturbations for LVLM Hallucination Suppression
• [CVPR 2026] Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidance
• [ICLR 2026] Copy-Paste to Mitigate Large Language Model Hallucinations
• [ICML 2026] Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating