See Only When Needed: Context-Aware Attention Intervention for Hallucination-Free LVLMs¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/Iris1946/CAI
Area: Multimodal VLM
Keywords: hallucination mitigation, vision-language models, attention intervention, uncertainty gating, training-free inference

TL;DR¶

Addressing the object hallucination vulnerability caused by language-prior dominance during autoregressive decoding, this paper introduces Context-Aware Attention Intervention (CAI), a training-free inference-time mechanism that derives token-specific spatial relevance from early layers and applies a conservative attention tilt only when deeper-layer predictive entropy spikes, strictly adhering to the "see only when needed" principle while preserving fluency.

Background & Motivation¶

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across image captioning, visual question answering, and multimodal reasoning tasks. Nevertheless, object hallucination remains a critical bottleneck impeding reliable deployment in safety-critical domains. Models frequently generate fluent, linguistically coherent statements describing entities that are entirely absent from the visual input. Detailed analyses reveal that such errors stem from both statistical co-occurrence biases in large-scale pretraining datasets and model-intrinsic over-reliance on strong language priors inherited from frozen language backbones. During autoregressive generation, minor early vision-text misalignments are cascadingly amplified across decoding steps, leading the model to hallucinate freely without visual grounding.

Existing remedies broadly divide into training-based methods and inference-time training-free approaches. Supervised fine-tuning on curated negative samples or reinforcement learning alignment (RLHF / DPO) achieves notable improvements, but retraining incurs immense computational costs and struggles to scale across heterogeneous backbones. In contrast, existing training-free approaches typically resort to uniform visual amplification—such as indiscriminately boosting attention weights on all visual patch tokens throughout decoding. However, uniform amplification introduces severe spurious correlations and amplifies irrelevant background noise, frequently degrading natural linguistic fluency and introducing new hallucination patterns.

An effective inference-time intervention must satisfy a strict non-interference principle: models should not "see more" everywhere, but rather see only when needed. Internal representation analysis uncovers two fundamental regularities. First, visual relevance is strictly token-specific: generating distinct words (e.g., "woman" versus "motorcycle") requires focusing on distinct spatial regions rather than global scenes. Second, hallucination risk strongly depends on layer depth and predictive uncertainty: in deeper decoding layers where visual representations tend to decay into lexical abstractions, hallucination-prone tokens exhibit sharp entropy spikes, whereas function words and already grounded tokens remain low-entropy and confident. Core Idea: Introduce a two-axis selective Context-aware Attention Intervention (CAI) that computes token-specific visual grounding from early-layer representations to identify where to look, and applies a conservative, KL-minimal attention tilt only on deeper-layer high-uncertainty tokens to dictate when to intervene, thereby eliminating hallucinations without damaging linguistic fluency.

Method¶

Overall Architecture¶

CAI operates entirely at inference time as a lightweight, plug-and-play intervention without modifying or retraining model weights. The overall workflow executes across three sequential stages during each autoregressive decoding step. First, at the bottom layer (layer 0), CAI computes the inner product between the current token's hidden state and all visual patch embeddings to produce a fine-grained, token-conditioned relevance map. Second, across intermediate-to-deep layers, the model monitors the predictive entropy derived via the language modeling head; an attention tilt toward the relevant visual regions is triggered only when the layer index exceeds $l_s$ and predictive entropy surpasses a threshold $\gamma$. Third, an optional contrastive decoding step can be appended to further penalize text-only prior hypotheses.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Inputs<br/>Image patch tokens v & text sequence x"] --> B["Token-Aware Visual Grounding<br/>Layer 0 hidden state h_t^0 dotted with patches yields w_t"]
    B --> C["Entropy- and Depth-Gated Intervention<br/>Check layer l ≥ l_s and predictive entropy H(h_t^l) > γ"]
    C -->|Gate Inactive / Low Risk| D["Preserve Baseline Attention<br/>Unperturbed forward propagation"]
    C -->|Gate Active / High Risk| E["Selective Attention Tilting<br/>Element-wise tilt on visual attention block A_t via w_t"]
    D --> F["Logits Aggregation"]
    E --> F
    F --> G["Optional Contrastive Decoding<br/>Penalize text-only hypotheses p(y_t|x)"]
    G --> H["Autoregressively Generated Grounded Token y_t"]

Key Designs¶

1. Token-Aware Visual Grounding: Resolving "where to look" from early representations Rather than uniformly amplifying all image patches, CAI identifies the exact image regions pertinent to the token under generation. Recent interpretability studies confirm that decoder-style LVLMs preserve localized, fine-grained visual representations primarily in their earliest layers (such as layer 0), whereas deeper layers progressively aggregate visual signals into global lexical abstractions. At decoding step $t$, CAI extracts the layer-0 hidden state $h_t^0 \in \mathbb{R}^{d_h}$, normalizes it via $L_2$ norm, and calculates its cosine similarity with the $N_v$ normalized visual patch embeddings $v \in \mathbb{R}^{N_v \times d_h}$, applying a Sigmoid activation: $$w_t^i = \sigma\left(\left(\frac{v_i}{\|v_i\|_2}\right)^\top \cdot \frac{h_t^0}{\|h_t^0\|_2}\right)$$ The resulting weight vector $w_t \in \mathbb{R}^{N_v}$ provides an accurate semantic footprint indicating exactly which visual regions align with the current token, preventing spurious attention drift toward irrelevant visual tokens.

2. Entropy- and Depth-Gated Intervention: Deciding "when to intervene" via uncertainty spikes Indiscriminately altering attention patterns across all tokens and layers impairs the grammatical structures and confident predictions established by the language backbone. To enforce conservative non-interference, CAI establishes a dual-gating condition: $$-\sum p(h_t^l) \log p(h_t^l) > \gamma, \quad \text{subject to} \quad l \ge l_s$$ where the predictive distribution $p(h_t^l) = \text{Softmax}(\text{LMHead}(\text{LayerNorm}(h_t^l)))$ is computed on the fly. Interventions remain inactive unless decoding advances into deeper layers ($l \ge l_s$, where visual grounding naturally decays) and the predictive entropy exceeds the risk threshold $\gamma$. Low-entropy syntax tokens (such as articles and prepositions) bypass the intervention completely, safeguarding fluent sentence flow and avoiding unnecessary computation.

3. Selective Attention Tilting: KL-minimal reweighting with provable safety bounds When both depth and entropy conditions are satisfied, CAI modulates the attention weights corresponding to image patch positions $A_{t, i_s:i_e} \in \mathbb{R}^{H \times N_v}$: $$\tilde{A}_{t, i_s:i_e} = A_{t, i_s:i_e} + |A_{t, i_s:i_e}| \odot w_t$$ Theoretical analysis demonstrates that this exponential attention tilt constitutes the unique KL-minimal reweighting that maximizes expected token-image similarity under a bounded relative entropy budget (Theorem 1). For high-entropy tokens whose induced visual shift aligns with the negative log-likelihood descent direction, small tilts guarantee monotonic loss reduction (Theorem 2). Furthermore, Pinsker's inequality establishes that when the gate is inactive or similarity signals are weak, the total variation distance $\|\tilde{a} - a\|_{\text{TV}}$ remains bounded and negligible (Theorem 4), guaranteeing non-interference.

Loss & Training¶

CAI is completely training-free and operates purely at test time without backward gradient computations. To further curb over-reliance on language priors, CAI can optionally be paired with contrastive decoding: $$\hat{p}(y_t \mid v, x) = \lambda p(y_t \mid v, x) - (\lambda - 1) p(y_t \mid x)$$ where $\lambda \ge 1$ governs the strength of prior penalization ($\lambda=1$ corresponds to standalone attention intervention $\text{CAI}^\dagger$). Hyperparameter sweeps on LLaVA-1.5 show optimal performance around $l_s = 26 \sim 27$, $\gamma = 0.1 \sim 0.2$, and $\lambda = 3.0$, while a label-free adaptive rule ($\gamma = \mu_H + 0.5\sigma_H$, $l_s = \lfloor 0.85L \rfloor$) enables effortless zero-shot transfer across distinct architectures.

Key Experimental Results¶

Main Results¶

Empirical evaluations are conducted across POPE (binary existence queries under Random, Popular, and Adversarial splits on MS-COCO, A-OKVQA, and GQA), CHAIR (sentence-level $CHAIR_S$ and instance-level $CHAIR_I$ on caption generation), and MME across multiple backbones (LLaVA-1.5, InstructBLIP, Qwen-VL, and Qwen3.5-4B).

Table 1 summarizes representative comparisons on LLaVA-1.5 evaluated on POPE (MS-COCO split) and CHAIR caption generation under maximum token limits of 64 and 128:

Benchmark / Dataset	Metric	Regular (Base)	VCD	PAI	ONLY	CAI (Ours)	Gain
POPE (COCO Random)	Accuracy (%) ↑	85.46	85.74	86.67	88.38	89.24	+3.78%
POPE (COCO Random)	F1-Score (%) ↑	86.07	86.70	87.26	88.41	89.14	+3.07%
POPE (COCO Popular)	Accuracy (%) ↑	81.20	81.93	82.77	85.00	86.03	+4.83%
POPE (COCO Adversarial)	Accuracy (%) ↑	75.87	76.90	76.83	79.93	82.77	+6.90%
CHAIR (Len=64)	$CHAIR_S$ (%) ↓	26.0	23.8	26.0	21.0	17.8	-8.2%
CHAIR (Len=64)	$CHAIR_I$ (%) ↓	8.8	8.2	8.7	7.3	6.9	-1.9%
CHAIR (Len=128)	$CHAIR_S$ (%) ↓	56.6	59.6	54.0	48.4	39.6	-17.0%
CHAIR (Len=128)	$CHAIR_I$ (%) ↓	16.8	16.6	15.2	14.7	12.4	-4.4%

Ablation Study¶

The ablation study validates the decoupled contribution of attention intervention versus contrastive decoding, together with inference latency, throughput, and memory consumption on LLaVA-1.5:

Config	Latency (ms/tok) ↓	Throughput (tok/ms) ↑	GPU Mem (MB) ↓	POPE Acc ↑	CHAIR_S (128) ↓	MME Total ↑	Note
Regular	89.62	9.41	14,241	74.81	56.6	588.33	Baseline unguided decoding
VCD	215.22 ($\times 2.40$)	2.22 ($\times 0.24$)	15,299	75.89	59.6	603.33	2.4× slower due to dual forward passes
PAI$^\dagger$ (Attn-only)	90.97 ($\times 1.02$)	9.27 ($\times 0.99$)	14,241	74.57	54.8	581.66	Uniform boosting harms POPE and MME
CAI$^\dagger$ (Attn-only, $\lambda=1$)	92.35 ($\times 1.03$)	9.15 ($\times 0.97$)	14,251	77.13	43.6	608.33	Robust gains without contrastive decoding
PAI (Full, w/ CD)	123.57 ($\times 1.38$)	7.09 ($\times 0.75$)	14,281	75.77	54.0	603.33	Relies almost entirely on contrastive decoding
CAI (Full, $\lambda=3.0$)	131.33 ($\times 1.47$)	6.80 ($\times 0.72$)	14,291	83.67	39.6	660.00	Full selective intervention + CD reaches SOTA

Key Findings¶

Two-axis selectivity is the essential driver of attention gains: Uniform amplification in PAI$^\dagger$ leads to degraded accuracy on POPE (74.57 vs. 74.81) and MME (581.66 vs. 588.33), verifying that indiscriminate visual enhancement introduces harmful background noise. In contrast, CAI$^\dagger$ alone improves POPE by +2.32% and drops $CHAIR_S$ by 13.0% with merely 3% latency overhead (92.35 vs. 89.62 ms/token), demonstrating that spatial targeting and entropy gating are fundamental.
Error accumulation in long generation is effectively arrested: In CHAIR captioning, expanding output length from 64 to 128 tokens causes baseline sentence hallucination to surge from 26.0% to 56.6%. CAI curtails this surge to 39.6% (a 17.0% absolute reduction), proving that gated intervention prevents autoregressive cascading of erroneous commitments.
Sparse activation ensures high inference throughput: Layer-wise tracking indicates that only 6.67% to 39.38% of tokens across deeper layers (layers 27-31) ever exceed the entropy threshold $\gamma$. By leaving the vast majority of tokens unperturbed, CAI runs at 131.33 ms/token—over 33× faster than costly external saliency-based baselines (e.g., LVLMs-Saliency at 4294.36 ms/token).

Highlights & Insights¶

Early-to-Late Grounding Transfer: Exploiting the structural disparity across network depths is an elegant insight: layer 0 preserves clean, unentangled spatial features before high-order semantic abstraction sets in, whereas deeper layers suffer from visual decay and language-prior domination. Transferring grounding coordinates from early layers to guide deep attention solves the visual degradation problem cleanly.
Entropy-Gated Non-Interference: Using internal predictive entropy as a risk proxy ensures interventions are strictly confined to uncertain tokens. Backed by KL-minimality and Pinsker's inequality, CAI guarantees bounded perturbation on confident tokens, preventing fluency degradation while mitigating hallucinations.
Broad Applicability and Architectural Flexibility: Beyond classic Transformer backbones, CAI readily generalizes to modern hybrid architectures like Qwen3.5-4B (GLA linear attention + Softmax attention), where applying CAI exclusively to deep Softmax layers achieves solid gains (MME score reaches 720.00 vs. 685.00 baseline) without task-specific tuning.

Limitations & Future Work¶

Vulnerability to Confident Hallucinations: When strong language priors produce hallucinated tokens with high model confidence (low entropy, e.g., outputting "plane" with high certainty when no aircraft exists), the entropy gate remains closed and CAI fails to intervene. Incorporating multi-modal feature consistency or external visual salience probes could address this gap.
Hyperparameter Sensitivity across Depths: While adaptive heuristic rules ($\gamma = \mu_H + 0.5\sigma_H$) exist, the optimal intervention starting layer $l_s$ and threshold $\gamma$ still show mild dependence on total model depth, requiring further study on fully parameter-free calibration methods for ultra-compact edge models.

vs PAI (Paying More Attention to Image): PAI applies uniform, global visual attention amplification and relies heavily on contrastive decoding to mask degradation; CAI introduces spatial targeting via layer-0 similarity and depth-entropy gating, demonstrating substantial gains with standalone attention intervention (CAI$^\dagger$).
vs VCD (Visual Contrastive Decoding): VCD requires creating corrupted visual inputs and executing paired forward passes, resulting in a 2.4× latency explosion; CAI operates in a single forward pass with under 1.5× overhead while achieving superior hallucination mitigation.
vs ONLY (One-Layer Intervention): ONLY restricts intervention to a single fixed layer; CAI establishes through Theorem 3 that visual decay is continuous across deeper layers, intervening adaptively across high-risk layers to provide smoother, more robust trajectory control during long generation.

Rating¶

Novelty: ⭐⭐⭐⭐ [Innovative early-to-late grounding transfer combined with principled entropy-gated KL-minimal attention reweighting]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated across 4 LVLM backbones, 3 major benchmarks, comprehensive ablations, efficiency analysis, and rigorous mathematical proofs]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, structured narrative around "see only when needed", tight alignment between theory, diagrams, and experiments]
Value: ⭐⭐⭐⭐⭐ [Plug-and-play, training-free, and computationally lightweight, offering high practical utility for deploying trustworthy LVLMs]