Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigation

Conference: ACL 2026 arXiv: 2604.10071 Code: N/A Area: Multimodal VLM Keywords: Multimodal hallucination, contrastive decoding, layer-wise analysis, visual attention, training-free

TL;DR

This paper proposes DaID (Dual-Anchor Introspective Decoding), which mitigates hallucinations within a single forward pass by exploiting layer-wise differences in visual perception within MLLMs — Spotlight layers amplify visual signals while Shadow layers suppress language priors.

Background & Motivation

Background: Multimodal large language models (MLLMs) demonstrate strong performance on reasoning tasks but suffer from severe hallucination problems, where generated text is inconsistent with visual content.

Limitations of Prior Work: Existing contrastive decoding methods (VCD, ICD) exhibit two key drawbacks: (1) they require an additional forward pass per decoding step to obtain the negative distribution, increasing inference latency by 1.83×; (2) they rely on heuristic external perturbations (e.g., visual masking) to construct negative distributions, introducing stochastic noise that causes semantic drift.

Key Challenge: The uncertainty introduced by external perturbations may cause correct visual signals to be erroneously suppressed (e.g., VCD replacing the correct "yellow" with the incorrect "red").

Goal: To shift from the external perturbation paradigm to an internal introspection paradigm, leveraging perceptual differences across the model's own intermediate layers as the contrastive signal source.

Key Insight: Layer-wise diagnosis of MLLMs reveals that shallow layers exhibit strong hallucination tendencies (visual agnosia), middle layers achieve the strongest visual perception (peak fidelity), and deep layers have visual signals overridden by language priors (see-then-forget).

Core Idea: Visual Attention Scores (VAS) are used to dynamically locate, for each token, a Spotlight layer (peak visual perception) and a Shadow layer (language-noise-dominated), enabling hallucination mitigation through contrastive calibration within a single forward pass.

Method

Overall Architecture

During standard MLLM decoding, DaID selects two anchor layers in real time based on each layer's attention distribution over visual tokens: the Spotlight layer (highest visual attention → amplifies visual signals) and the Shadow layer (lowest visual attention prior to Spotlight → suppresses language priors). The final logits are calibrated via a dual-anchor contrastive formulation, requiring no additional forward passes.

Key Designs

1. Visual Attention Score (VAS) and Dynamic Anchoring:

   • Function: Identifies the optimal contrastive anchor layers for each token in a training-free manner.
   • Mechanism: \(\text{VAS}_t(l)\) = average attention weight of all heads over visual tokens. Spotlight = \(\text{argmax}\ \text{VAS}\) (peak visual perception); Shadow = \(\text{argmin}\ \text{VAS}\) (restricted to layers before Spotlight, representing pure language noise).
   • Design Motivation: Experiments verify that visual attention is highly synchronized with object recognition accuracy and hallucination rate, making it a reliable training-free proxy for the model's cognitive state.

2. Dual-Anchor Contrastive Decoding:

   • Function: Simultaneously enhances visual signals and suppresses language priors in the final logits.
   • Mechanism: \(L_{\text{DaID}} = [L_{\text{final}} + \alpha \cdot L_{\text{spotlight}}] \cdot (1+\beta) - \beta \cdot L_{\text{shadow}}\), where \(\alpha\) controls visual enhancement strength and \(\beta\) controls language suppression strength.
   • Design Motivation: Neither amplification nor suppression alone is sufficient — simultaneous "brightening and denoising" is required for maximum effectiveness.

3. Adaptive Plausibility Constraint:

   • Function: Prevents intermediate-layer logits from introducing grammatically implausible tokens.
   • Mechanism: Dual-anchor calibration is applied only to candidate tokens whose probability in the final-layer distribution satisfies \(p \geq \gamma \cdot \max\_\text{prob}\); all others are set to zero.
   • Design Motivation: Since Spotlight layers reside in intermediate layers, their logits may contain visually relevant but grammatically inappropriate tokens, necessitating a constrained candidate space.

Loss & Training

DaID is a training-free, inference-time method. Key hyperparameters: \(\alpha = 0.8\) (visual enhancement), \(\beta = 0.2\) (language suppression), \(\gamma = 0.9\) (POPE) / \(0.1\) (others).

Key Experimental Results

Main Results

Hallucination benchmarks on LLaVA-1.5-7B:

Method	POPE Acc	POPE F1	CHAIR_S↓	CHAIR_I↓	MME Total↑
Greedy	81.38	82.20	49.6	14.4	559.48
VCD	84.66	84.52	49.2	14.8	603.66
OPERA	84.88	85.21	45.4	12.7	549.00
SID	84.82	85.50	44.2	12.2	599.80
EAZY	84.97	85.78	38.8	11.4	596.16
DaID	85.08	85.92	35.9	11.3	633.68

Hallucination benchmarks on LLaVA-NeXT:

Method	POPE Acc	POPE F1	CHAIR_S↓	CHAIR_I↓	MME Total↑
Greedy	83.78	82.24	32.8	9.1	580.92
EAZY	84.91	85.40	26.8	8.3	611.14
DaID	85.32	85.76	24.2	8.2	644.40

Ablation Study

Hyperparameter analysis (LLaVA-1.5): - Increasing \(\alpha\) from 0.4 to 0.8 raises POPE Acc from 83.44% to 85.08%; \(\alpha > 0.8\) degrades performance as excessively strong visual signals disrupt grammaticality. - \(\beta = 0.2\) is optimal: compared to \(\beta = 0\) (no suppression), F1 improves by +0.93% and Acc by +0.51%; \(\beta > 0.2\) leads to over-suppression and performance degradation.

General reasoning capability: Across five benchmarks (GQA, VQAv2, MMB, SeedI, VizWiz) at both 7B and 13B scales, DaID consistently maintains or improves performance (e.g., +2.1% on SeedI).

Key Findings

• Layer-wise diagnosis reveals a "see-then-forget" phenomenon in MLLMs: object recognition accuracy peaks at intermediate layers but drops substantially in deeper layers (11.12% decline in LLaVA-NeXT).
• Visual attention is precisely synchronized with object recognition accuracy (both peaking at layer 25 in LLaVA-1.5), validating VAS as a reliable proxy for cognitive state.
• DaID consistently outperforms methods requiring additional forward passes (e.g., VCD, OPERA) while introducing no additional inference overhead (single forward pass).
• Generalization experiments across five MLLM architectures confirm the method's consistent effectiveness.

Highlights & Insights

• The paradigm shift from "external perturbation" to "internal introspection" is elegant — it avoids noise introduced by external perturbations while reducing computational overhead.
• The "Spotlight + Shadow" dual-anchor concept is highly intuitive: shallow layers = language noise → Shadow; middle layers = visual peak → Spotlight.
• The layer-wise diagnostic analysis has independent scientific value — the "see-then-forget" phenomenon and the attention-as-proxy mechanism can inspire further research.
• The method is entirely training-free and can be applied plug-and-play to any MLLM.

Limitations & Future Work

• \(\alpha\) and \(\beta\) require different settings across benchmarks (e.g., \(\gamma = 0.9\) on POPE vs. \(0.1\) on others), and hyperparameter selection lacks automation.
• The layer-wise analysis is based on the LLaVA series; optimal layers may differ for other architectures (e.g., Qwen2-VL).
• The advantage of a single forward pass may be partially offset by the overhead of attention extraction and layer-wise logits computation; specific latency is not reported.
• Extension to more complex multimodal settings such as video understanding remains to be explored.

Related Work & Insights

• The comparison with VCD is most illustrative: VCD constructs a negative distribution via external perturbations, whereas DaID treats shallow-layer language priors as a natural negative distribution — a more elegant and effective approach.
• DoLa similarly exploits layer-wise contrast but only compares early and late layers to extract factual knowledge; DaID's dual-anchor mechanism with dynamic VAS-based selection is considerably more fine-grained.
• Implications for MLLM architecture design: incorporating auxiliary objectives during training to preserve visual information in intermediate layers could address the "see-then-forget" phenomenon at its source.

Rating

• Novelty: ⭐⭐⭐⭐⭐ The internal introspection paradigm is novel, the dual-anchor dynamic selection mechanism is elegantly designed, and the "see-then-forget" finding is of significant scientific value.
• Experimental Thoroughness: ⭐⭐⭐⭐ Covers multiple hallucination and general-purpose benchmarks, multiple MLLMs, and complete hyperparameter analysis.
• Writing Quality: ⭐⭐⭐⭐⭐ The paper is well-structured, flowing coherently from motivation to observation to method.

Related Papers

• [CVPR 2026] Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention
• [AAAI 2026] InEx: Hallucination Mitigation via Introspection and Cross-Modal Multi-Agent Collaboration
• [CVPR 2026] Locate-then-Sparsify: Attribution Guided Sparse Strategy for Visual Hallucination Mitigation
• [ACL 2026] Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decoding
• [ACL 2026] HiPrune: Hierarchical Attention for Efficient Token Pruning in Vision-Language Models