Skip to content

Visual Attention Faithfulness in Vision-Language Models is Heterogeneous

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Visual Attention Faithfulness, Causal Perturbation Analysis, Vision-Language Models, Comprehensiveness and Sufficiency, Attention Interpretability

TL;DR

This work develops the first token-level causal perturbation framework measuring comprehensiveness and sufficiency gap to evaluate visual attention faithfulness in Vision-Language Models (VLMs), revealing that visual attention faithfulness is heterogeneous across three distinct processing modes (Faithful-Sufficient, Faithful-Distributed, and Non-Focal) and that model visual dependencies systematically diverge from human-annotated ground-truth regions.

Background & Motivation

Whether attention weights provide a faithful explanation of neural model reasoning has long been one of the most vigorously debated topics in text-based Natural Language Processing (NLP). Seminal work by Jain and Wallace argued that attention distributions can be arbitrarily permuted or adversarial without altering final model outputs, leading to claims that attention is not an explanation. In counterpoint, Wiegreffe and Pinter demonstrated that enforcing strict constraints on attention during training yields measurably different inductive biases and output distributions. The consensus emerging from the NLP interpretability community is that faithfulness is neither an inherent binary property nor a uniform attribute of attention mechanisms; rather, it is a continuous, intervention-specific property that must be empirically audited on a per-sample basis through counterfactual causal perturbations (e.g., measuring comprehensiveness and sufficiency as formalized by the ERASER benchmark). However, this foundational dialogue has developed strictly within discrete, one-dimensional text-only domains.

Modern Vision-Language Models (VLMs) integrate a Vision Transformer (ViT) encoder with an autoregressive Large Language Model (LLM) backbone, encoding two-dimensional image patches into sequences of visual tokens that are concatenated directly with textual instruction tokens inside a unified transformer decoder. The 2D spatial arrangement, high spatial auto-correlation and semantic redundancy among adjacent visual patches, and asymmetric cross-modal information routing give visual attention characteristics fundamentally distinct from pure text. Despite this, downstream VLM research has largely bypassed validating the causal foundation of visual attention: practical inference acceleration techniques (such as token pruning and token merging) routinely assume that visual tokens receiving low attention weights are dispensable noise, while attention visualization pipelines (such as Grad-CAM and cross-modal attention heatmaps) depict attended regions as explanations without verifying whether the model's prediction causally relies upon them. Whether attention-ranked visual tokens truly correspond to the model's actual visual dependencies remains completely unverified.

The fundamental tension lies in the nature of visual representation within transformers: visual dependencies are simultaneously spatially diffuse and globally contextual, meaning attention heatmaps alone cannot prove causality, whereas pixel-level masking conflates the degradation of the visual encoder with the internal attribution of the language model backbone. To resolve this, the authors introduce a causal intervention framework operating directly at the hidden state entrance of the language model decoder, decoupling causal necessity (comprehensiveness) from standalone sufficiency (sufficiency gap). Core idea: by establishing the first token-level causal perturbation framework measuring normalized comprehensiveness and sufficiency gap, this paper reveals that visual attention faithfulness in VLMs is heterogeneous across three distinct processing modes (Faithful-Sufficient, Faithful-Distributed, Non-Focal) and demonstrates that human visual intuition systematically diverges from model causal dependencies.

Method

Overall Architecture

The framework formulates the assessment of visual attention faithfulness as a formal causal intervention problem. Given an input image \(I\) encoded into visual tokens \(V=\{v_1, \dots, v_N\}\) and a text query encoded into tokens \(T=\{t_1, \dots, t_M\}\), the VLM autoregressively generates an answer \(y=(y_1, \dots, y_{|y|})\). To eliminate non-deterministic sampling variance and isolate attribution strictly to token ablations, greedy decoding (temperature \(= 0\)) is maintained throughout all forward passes. The operational pipeline consists of four sequential stages: aggregating cross-layer attention weights to establish an unbiased visual token ranking, executing zero-ablation causal intervention at the first decoder layer, evaluating normalized comprehensiveness and sufficiency gap, and segmenting behavioral manifolds via two-stage Otsu thresholding into three processing modes.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Input Sequence<br/>Visual Tokens V + Text Tokens T"] --> B["Cross-Layer Attention Aggregation<br/>All-layer all-head mean attention ranking"]
    B --> C["Token-Level Causal Zero-Ablation<br/>First decoder layer hidden state suppression"]
    C --> D["Dual-Metric Faithfulness Evaluation<br/>Normalized Comprehensiveness & Sufficiency Gap"]
    D --> E["Two-Stage Otsu Clustering<br/>Mode A Faithful-Sufficient / Mode B Distributed / Mode C Non-Focal"]

Key Designs

1. Cross-Layer Attention Aggregation: Eliminating Routing Bias Across Network Depth

Single-layer attention weights or isolated attention heads are heavily skewed toward task-specific syntactic shortcuts or idiosyncratic token routing, failing to reflect the holistic computational flow of the language backbone. To derive an unbiased global importance score \(a_i\) for each visual token \(v_i\), the framework aggregates raw attention weights \(\alpha_{l,h}(i, y_s)\) across all \(L\) decoder layers and all \(H\) attention heads directed from each generated output token \(y_s \in y\) toward visual token \(v_i\): $\(a_i = \frac{1}{|y|} \sum_{s=1}^{|y|} \sum_{l=1}^L \frac{1}{H} \sum_{h=1}^H \alpha_{l,h}(i, y_s)\)$ Averaging across heads treats all attention streams without imposing unverified assumptions about head specialization, while summing linearly across layers captures native attention mass without requiring artificial information-flow assumptions. Sorting visual tokens in descending order of \(a_i\) identifies the top-\(k\) focus set \(S_k = \{v_{(1)}, v_{(2)}, \dots, v_{(k)}\}\).

2. First-Decoder-Layer Zero-Ablation: Causal Intervention Preserving 2D Spatial Structure

To establish true causality rather than correlational observation, traditional attribution methods either prune tokens entirely or corrupt input pixels. However, in modern VLMs, dropping visual tokens alters sequence length and shifts the 2D positional grid indices of all downstream tokens, hopelessly conflating the loss of token semantic content with the catastrophic collapse of spatial layout coordinates; conversely, pixel masking measures the vision encoder's robustness to noise rather than the language model's reliance on representations. The proposed method injects a zero-ablation intervention at the input of the first language decoder layer: $\(h_i^{(0)} \leftarrow \mathbf{0}, \quad \forall v_i \in S\)$ This zero replacement introduces zero foreign semantic information into the latent space while perfectly preserving sequence length and relative 2D coordinate embeddings, completely isolating the causal response of the language backbone.

3. Normalized Comprehensiveness and Sufficiency Gap: Orthogonal Quantifications of Visual Dependency

Because baseline prediction probabilities \(P(y \mid V, T)\) vary widely across different query-image instances, raw probability differentials induce severe statistical bias. Both metrics are therefore strictly normalized by baseline probability. Comprehensiveness (Comp) quantifies the causal necessity of the selected top-\(k\) tokens: $\(\text{Comp}(S_k) = \frac{P(y \mid V, T) - P(y \mid V \setminus S_k, T)}{P(y \mid V, T)}\)$ A high Comp value indicates that ablating the top-\(k\) visual tokens collapses the model's prediction probability, confirming their necessity. In contrast, the Sufficiency Gap (Sgap) measures the extent to which the model retains its prediction when restricted exclusively to \(S_k\) while zero-ablating all other visual tokens: $\(\text{Sgap}(S_k) = \frac{P(y \mid V, T) - P(y \mid S_k, T)}{P(y \mid V, T)}\)$ A low (or slightly negative) Sgap confirms that \(S_k\) alone carries sufficient evidence to fully reproduce the model's prediction, whereas a high Sgap reveals that broader context is required.

4. Two-Stage Otsu Thresholding and Three-Mode Heterogeneity Decomposition: Characterizing Architectural Processing Regimes

Projecting samples onto the two-dimensional (Comp, Sgap) manifold reveals distinct multimodal clustering rather than a uniform continuum. The framework applies a sequential two-stage Otsu thresholding pipeline: first applying Otsu's method along the Comp axis across the entire dataset to partition non-focal from focal samples (\(\tau_c=0.59\) on Qwen2.5-VL), and subsequently applying Otsu along the Sgap axis strictly within the high-Comp cluster (\(\tau_s=0.45\)). This objectively decomposes visual reasoning into three distinct processing modes: - Mode A: Faithful-Sufficient: High Comp (\(\approx 0.98\)) paired with near-zero or slightly negative Sgap (\(-0.026\)). The top-\(k\) attended tokens are both causally necessary and individually sufficient, while peripheral visual tokens act as neutral background or mild distractors. - Mode B: Faithful-Distributed: High Comp (\(\approx 0.95\)) paired with large Sgap (\(0.818\)). The attended top-\(k\) region is critical for prediction, but cannot produce the output in isolation; the model requires the broader visual layout and distributed contextual tokens. - Mode C: Non-Focal: Low Comp (\(0.213\)) and moderate Sgap (\(0.460\)). No localized attention region is causally essential, yet full-image ablation collapses prediction (Comp reaches \(0.973\)), demonstrating diffuse visual dependence acting as a global contextual trigger.

Key Experimental Results

Experiments are conducted across Qwen2.5-VL-7B (native-resolution 2D grid) and InternVL2.5-8B (dynamic-tiling ViT with thumbnail context), alongside scale replications on Qwen2.5-VL-3B and InternVL2.5-4B. The evaluation benchmarks cover 90 curated VQAv2 samples balanced across three reasoning types (30 yes/no, 30 number counting, 30 open-ended other) with dual-expert ground-truth (GT) spatial annotations, 60 correctly answered document information extraction samples from VRDU, and 100 chart visual reasoning samples from ChartQA.

Main Results: VQAv2 Perturbation Strategy Comparison & Mode Profiles

On valid VQAv2 samples (excluding 5 holistic full-image samples), the token ablation budget \(k\) is set exactly equal to the token count of the human-annotated GT region for each instance. Four perturbation strategies are benchmarked.

Perturbation Strategy Qwen2.5-VL-7B Mean Comp โ†‘ Qwen2.5-VL-7B Mean Sgap โ†“ InternVL2.5-8B Mean Comp โ†‘ InternVL2.5-8B Mean Sgap โ†“
Overall Top-k (Model Attention) 0.813 0.469 0.733 0.468
Non-GT Top-k (Model Focus outside GT) 0.659 0.653 0.474 0.659
GT Object (Human Ground-Truth Region) 0.505 0.846 0.609 0.555
Random (Uniform Baseline) 0.502 0.599 0.487 0.549

The empirical distribution of the three processing modes on Qwen2.5-VL-7B demonstrates consistent ground-truth task accuracy across all modes:

Processing Mode Sample Share (%) Mean Comp Mean Sgap Accuracy (%)
Mode A: Faithful-Sufficient 32.9% 0.978 -0.026 92.9%
Mode B: Faithful-Distributed 47.1% 0.952 0.818 100.0%
Mode C: Non-Focal 20.0% 0.213 0.460 94.1%

Document Understanding and Cross-Architecture Generalization

In visually-rich document understanding (VRDU) and chart reasoning (ChartQA), visual tokens are ablated across fixed proportions (1%, 3%, 5%, 10%, 30%).

Benchmark & Model Configuration Mask Ratio 5% (Comp / Sgap) Mask Ratio 10% (Comp / Sgap) Mask Ratio 30% (Comp / Sgap) Dominant Mode
ChartQA (Qwen2.5-VL-7B Top-k) 0.37 / 1.00 0.49 / 1.00 0.77 / 0.91 Mode B (Distributed)
ChartQA (Qwen2.5-VL-7B Random) 0.01 / 1.00 0.02 / 1.00 0.27 / 0.99 Random Baseline
ChartQA (InternVL2.5-8B Top-k) 0.65 / 0.42 0.71 / 0.26 0.79 / 0.08 Mode A (Sufficient)
ChartQA (InternVL2.5-8B Random) 0.01 / 0.82 0.01 / 0.74 0.06 / 0.39 Random Baseline
VRDU-IE (Qwen2.5-VL-7B Top-k) Comp: 0.68 / Sgap: 0.95 Comp: 0.826 / Sgap: 0.947 - Mode B (Layout Dependent)
VRDU-IE (InternVL2.5-8B Top-k) Comp: 0.65 / Sgap: 0.40 Comp: 0.650 / Sgap: 0.189 - Mode A (Self-Sufficient)

Model Scaling Behavior Across Parameter Capacities

Model Scale Overall Top-k Comp GT Object Comp Mode A Share (%) Mode B Share (%) Mode C Share (%)
Qwen2.5-VL-3B 0.739 0.432 31.8% 37.6% 30.6%
Qwen2.5-VL-7B 0.813 0.505 32.9% 47.1% 20.0%
InternVL2.5-4B 0.792 0.636 18.8% 54.1% 27.1%
InternVL2.5-8B 0.733 0.609 29.4% 45.9% 24.7%

Key Findings

  • Attention Rankings Outperform Human Intuition in Causal Faithfulness: Across both model architectures and all parameter scales, the model's own Top-\(k\) visual attention tokens achieve the highest comprehensiveness (0.813 on Qwen, 0.733 on InternVL) and the lowest sufficiency gap (0.469 and 0.468), significantly outperforming random selection and human-annotated GT regions. Visual attention reliably reflects causal reliance rather than serving as mere mathematical artifact.
  • Systematic Divergence Between Model Visual Reliance and Human Annotations: In Modes A and B, human-annotated GT regions satisfy comprehensiveness in only \(\sim\)60% of cases. In the misaligned subset, GT comprehensiveness plunges to a mean of \(-0.150\) while model Top-\(k\) comprehensiveness remains exceptionally high at \(0.945\). Qualitative inspection reveals that models often rely on correlated contextual background (e.g., chromatic ambient reflections around a sink to determine its color) rather than the object pixels themselves, proving that human bounding boxes cannot serve as an unquestioned proxy for VLM visual dependency.
  • Reasoning Task Geometry Governs Mode Distribution: Within VQAv2, binary yes/no questions predominantly populate Mode B (55.6%), requiring confirmation of localized targets combined with broader contextual scenes; counting queries split between Mode A (48.3% when instances are isolated) and Mode B (37.9% when scene clutter requires verification); open-ended queries exhibit the highest prevalence of Mode C (34.5%), where visual inputs trigger internalized parametric knowledge.
  • Model Architecture Dictates Processing Mode on Identical Tasks: On identical document IE and ChartQA tasks, Qwen2.5-VL operates under Mode B (masking 10% causes an 82.6% drop in confidence, but retaining 10% alone recovers only 5%), indicating extreme reliance on broader 2D layout context. In sharp contrast, InternVL2.5 operates under Mode A (retaining 10% yields an Sgap of 0.189 on IE and Sgap drops to 0.08 at 30% on ChartQA), because its dynamic-tiling preprocessor and global thumbnail structure package self-sufficient localized evidence blocks.

Highlights & Insights

  • Transitioning from Qualitative Polemics to Causal Measurement: The paper replaces speculative qualitative debates regarding attention interpretability with a rigorous interventionist framework adapted from ERASER, formulating hidden-state zero-ablation that respects 2D spatial tokenization.
  • Debunking the Human-Annotation Proxy Fallacy: Demonstrating that human ground-truth masks diverge from model causal evidence in 40% of focal cases provides a vital caution against naively penalizing or aligning model attention maps against human gaze or segmentation masks.
  • Actionable Constraints for Inference Token Pruning: The prominent discovery that Mode B accounts for 47.1% of general VQA and dominates document reasoning in single-grid architectures reveals that naively discarding low-attention peripheral tokens risks destroying essential layout and background dependencies.

Limitations & Future Work

  • Lack of Evaluation on Fixed-Resolution Architectures: The study is validated exclusively on dynamic-resolution models (Qwen2.5-VL and InternVL2.5) and does not examine classic fixed-resolution bicubic interpolation models (e.g., LLaVA-1.5).
  • Absence of Slow-Thinking Reasoning Models: Experiments focus on standard autoregressive generation without addressing test-time compute scaling or multi-step visual reasoning (e.g., visual CoT / slow-thinking models), where attention trajectories across extended reasoning steps may exhibit multi-phase temporal transitions.
  • Focused Sample Capacity: Because fine-grained per-token forward perturbation and dual-expert human region validation require extensive compute and annotation, the core VQA analysis utilizes 90 representative samples, pointing to the need for automated scaling pipelines in future benchmarks.
  • vs Jain & Wallace (2019) / Wiegreffe & Pinter (2019): While text-only debates treated attention faithfulness as a monolithic property, this work proves that in 2D multimodal transformers, faithfulness is inherently heterogeneous, decomposing into three distinct operational modes governed by question type and architecture.
  • vs DeYoung et al. (2020) ERASER: ERASER dropped text tokens outright; this paper adapts the comprehensiveness/sufficiency paradigm to visual tokens by identifying that literal token removal distorts 2D positional encodings, resolving this through first-layer zero-ablation.
  • vs Sparse-VLM (Zhang et al., 2025) / LLaVA-PruMerge (Shang et al., 2025): Modern token sparsification methods assume low-attention tokens are completely redundant; this paper's Mode B results caution that peripheral tokens frequently supply indispensable contextual support.

Rating

  • Novelty: โญโญโญโญโญ Pioneering token-level causal perturbation benchmark establishing heterogeneous visual attention modes in VLMs.
  • Experimental Thoroughness: โญโญโญโญโ˜† Rigorous evaluation spanning two leading VLM families, multiple parameter scales, natural VQA, document IE, and chart reasoning.
  • Writing Quality: โญโญโญโญโญ Exceptionally clear narrative progression, mathematically sound formulations, and dense, insightful empirical characterization.
  • Value: โญโญโญโญโญ Foundational reference for mechanistic interpretability, attribution fidelity, and token compression strategies in multimodal foundation models.