Skip to content

Tinted Frames: Question Framing Blinds Vision-Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: None
Area: LLM Reasoning
Keywords: Vision-Language Models, Question Framing, Cross-Framing Inconsistency, Attention Rollout, Attention Steering

TL;DR

This paper demonstrates that Vision-Language Models (VLMs) exhibit "selective blindness"—modulating visual attention allocation based on question framing even when tasks demand identical visual reasoning—and proves that constrained framings (Yes/No, MCQ) suppress visual energy and induce spatial misallocation away from target regions, which can be mitigated via attention steering and structured reasoning prompts.

Background & Motivation

Vision-Language Models (VLMs) have attained remarkable milestones on complex multimodal reasoning tasks, yet whether these architectures genuinely anchor their generated answers in visual evidence or primarily rely on shallow linguistic heuristics remains an unresolved foundational challenge. A growing body of literature documents symptoms of visual disengagement, often characterizing models as visually "blind" due to disproportionate allocation of attention to low-semantic visual sink tokens or spatial blind spots arising from rotary positional embeddings and causal masking. However, existing analyses have predominantly treated this visual disengagement as a static, intrinsic architectural flaw, aggregating empirical measurements holistically across heterogeneous benchmarks and formats.

This holistic diagnostic approach overlooks the fact that in both practical deployment and benchmark evaluations, visual queries are necessarily mediated through language. The same underlying visual concept is routinely queried via diverse formats, ranging from open-ended short-answer questions to constrained Yes/No affirmations or multiple-choice questions (MCQs). Standard multimodal evaluation protocols treat question framing as a neutral, interchangeable container, assuming that a model possessing true scene understanding should produce consistent predictions regardless of formatting. Prior prompt sensitivity studies have noted that accuracy fluctuates under prompt phrasing, but they largely attribute these variations to output-space prior shifts, answer label biases, or token-level surface formatting, leaving it entirely unexamined whether linguistic framing actively alters the internal visual processing of the model itself.

This disconnect establishes a crucial mechanistic dilemma: if a model can resolve an open-ended visual question correctly without candidate options to exploit, but fails when the identical query is restructured as a Yes/No or MCQ format, the failure cannot stem from an absolute deficiency in visual capability. Instead, the model is exhibiting dynamic, context-dependent disengagement—it "decides not to see" under constrained formulations. The authors reframe this phenomenon from a static limitation to an active failure mode, formalizing a latent causal chain connecting Question Framing (\(F\)), Visual Attention (\(A\)), and Final Prediction (\(Y\)). Core idea: using receptive-field normalized attention rollout as a mechanistic probe, the paper quantifies cross-framing inconsistency, demonstrates that constrained framings trigger substantial visual energy drops and spatial misallocation away from task-relevant regions, and proves via counterfactual attention steering and structured reasoning prompts that internal visual attention distortion directly drives downstream reasoning failures.

Method

Overall Architecture

The analytical pipeline systematically dissects the latent influence pathways between question framing and model behavior (\(F \to Y\), \(F \to A\), and \(A \to Y\)). In an ideal VLM reasoning workflow, visual input \(X\) and question semantics \(Q\) dictate the internal visual attention distribution \(A\), which subsequently generates the prediction \(Y\). However, empirical VLMs exhibit an intrusive framing pathway \(F \to A\), which silently corrupts the quality of visual attention and degrades accuracy via \(F \to A \to Y\).

To rigorously isolate framing without altering underlying semantic targets, the methodology establishes controlled benchmarks (GQAF and V*F) spanning open-ended, Yes/No, and MCQ variants. The internal information flow is quantified via receptive-field normalized attention rollout, tracking the probability mass transmitted from generated query tokens to input visual tokens across network layers. Finally, counterfactual intervention via multiplier-based attention steering and an attention realignment probe (optimizing soft prompt tokens under attention distillation objectives) are introduced to establish causality and mitigate framing-induced degradation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Input<br/>Image X + Core Semantic Query Q"] --> B["Controlled Framing Triplet<br/>Open-ended / Yes-No / MCQ"]
    B --> C["Causal Attention Rollout<br/>Receptive Field Normalization + Residual Tracking"]
    C --> D["Framing-Induced Blindness Analysis<br/>Visual Energy Collapse + Sink Token Divergence + Target Box Depletion"]
    D --> E["Dual-Axis Attention Steering<br/>Visual Energy (VE) Scaling + Target Bounding Box Boosting"]
    E --> F["Structured Reasoning Prompts & Learned Soft Tokens<br/>CoT / Grounding Prompts + Attention Distillation Realignment"]
    F --> G["Restored Visual Grounding & Consistent Predictions<br/>Reduced Cross-Framing Inconsistency Rate + Enhanced Downstream Accuracy"]

Key Designs

1. Cross-Framing Inconsistency & Annotation-Constrained Benchmark Curation: Isolating Framing Dynamics from Semantic Content

To establish whether question framing alters final predictions (\(F \to Y\)) without conflating visual competence with candidate-guessing heuristics, the authors anchor their evaluation on open-ended generation. Open-ended queries provide a strict, unguided reference point where models must synthesize answers freely, making accidental success via label priors improbable. Cross-framing inconsistency is formalized as the conditional probability that a model fails to preserve its correct open-ended answer under an equivalent Yes/No or MCQ framing: $\(I = P(C = 0 \mid O = 1) = 1 - P(C = 1 \mid O = 1)\)$ where \(O=1\) denotes open-ended correctness and \(C=1\) denotes correct predictions on both constrained variants. To address the fact that constrained formats can be guessed correctly by chance, the chance baseline under uniform random guessing is established as \(g = 0.5 \times 0.25 = 0.125\), yielding the chance-corrected metric: $\(\text{Acc}_{\text{chance-corrected}} = \frac{P(C = 1 \mid O = 1) - g}{1 - g}\)$ To eliminate LLM hallucination in data construction, GQA questions are transformed using GPT-5.1 conditioned on ground-truth scene graphs and bounding boxes to craft valid distractors, while V* reformulations follow deterministic rule-based option stripping. Label balance in binary Yes/No questions is controlled by sampling target labels before prompting, achieving a balanced 50:50 distribution across 30k samples in GQAF and 900 samples in V*F, validated by multi-national human evaluation achieving 96.9% spatial consistency and 95.7% option exclusivity.

2. Receptive-Field Normalized Attention Rollout: Tracing Autoregressive Information Propagation Across Layers

To overcome the inadequacy of naive attention-weight averaging across layers—which ignores residual bypasses and autoregressive causal masking artifacts—the paper extends Attention Rollout to multimodal generative transformers. Let \(W^{(\ell)}_{\text{att}} \in \mathbb{R}^{N \times N}\) be the raw attention matrix at layer \(\ell\). Accounting for residual stream connections yields \(A^{(\ell)} = 0.5 W^{(\ell)}_{\text{att}} + 0.5 I\). Because autoregressive causal masking disproportionately expands the effective receptive fields of later tokens, the key dimension is scaled by a diagonal receptive field matrix \(S\), followed by row-wise stochastic normalization: $\(R^{(\ell)} = \mathcal{N}\left( A^{(\ell)} \cdot \text{diag}(S) \right) \cdot R^{(\ell-1)}\)$ where \(R^{(0)} = I\) and \(\mathcal{N}(\cdot)\) enforces \(\sum_j R^{(\ell)}_{ij} = 1\). The cumulative stochastic matrix \(R^{(L)}\) tracks net information propagation from early input visual tokens to final output token representations. Visual reliance is quantified by slicing the sub-matrix mapping generated output tokens (queries) to input visual tokens (keys), defining Visual Energy (VE) as the sum of probability mass within this sub-matrix. Layer-wise trajectory analysis identifies that while early layers process inputs similarly, attention diverges starkly in middle cross-modal interaction layers (layers 12–22), where visual energy and target box attention drop by 40%–50% for Yes/No and MCQ framings compared to open-ended questions.

3. Dual-Axis Attention Steering: Causal Verification of Attention Misallocation Driving Errors

To resolve whether decreased visual attention is merely an efficient shortcut on easier constrained tasks or a malignant source of reasoning failure (\(A \to Y\)), the authors introduce parameter-free counterfactual Attention Steering during inference: - Visual Energy (VE) Steering: Computes the scalar ratio between open-ended and constrained total visual energy from attention rollout. During constrained inference, attention weights on all visual tokens are multiplied by this ratio for every post-image query token, while non-image token attention is scaled down proportionally to preserve a valid probability simplex, keeping spatial distribution within the image unchanged; - Box Attention Steering: Isolates spatial allocation by computing the ratio of open-ended to constrained bounding-box attention. Attention on tokens inside the ground-truth target bounding box is scaled up while clamping total visual energy constant. Steering experiments confirm that accuracy improves monotonically with Spearman rank correlation coefficients exceeding 0.94–0.98 as attention is shifted back toward open-ended levels. This demonstrates that spatial misallocation within the image is the primary driver of prediction degradation.

4. Attention Realignment Probe & Structured Prompting: Restoring Grounding via Inference-Time and Soft-Token Tuning

To translate these mechanistic findings into practical mitigations, the paper investigates both natural language prompting paradigms and learnable soft-token realignment probes. Evaluating prompting strategies on SeedBench and GQAF reveals that simple image-focused instructions (e.g., "examine the image carefully") yield negligible shifts in attention and fail to curb inconsistency. In contrast, structured reasoning prompts—such as Thinking, Chain-of-Thought (CoT), and Grounding prompts that mandate predicting bounding-box coordinates \([x_1, y_1, x_2, y_2]\) and object labels prior to the final answer—effectively restore target-region attention and slash inconsistency from 20.76% to 12.74%–16.41%. At the parameter level, an attention realignment probe optimizes \(K=8\) soft tokens prepended in an infix position using a combined cross-entropy and attention distillation loss: $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}} + 5 \cdot \left( \mathcal{L}_{\text{L2}}(\text{VE}) + \mathcal{D}_{\text{KL}}(\text{Spatial Distribution}) \right)\)$ Trained on 10k LLaVA-1.5 instruction samples with teacher-forcing confidence weighting, this lightweight probe (only ~60k parameters) successfully realigns attention under constrained framings, delivering consistent downstream gains across diverse benchmarks.

Key Experimental Results

Main Results

The primary quantitative evaluation of Attention Steering on Qwen2.5-VL-7B across 40 runs under varying random seeds is summarized below:

Benchmark & Format Metric Type Baseline Accuracy (%) VE Steering Only Box Steering Only Both Interventions Estimated Upper Bound (%)
V*F (Fine-grained) - Yes/No Accuracy (%) 65.57 ± 0.40 67.22 ± 0.28 66.73 ± 0.44 68.23 ± 0.52 (+2.66) 69.74
V*F (Fine-grained) - MCQ Accuracy (%) 70.77 ± 1.38 71.42 ± 1.29 73.35 ± 1.33 74.37 ± 1.43 (+3.60) 75.63
GQAF (General) - Yes/No Accuracy (%) 81.63 ± 0.18 81.81 ± 0.32 82.48 ± 0.42 82.48 ± 0.33 (+0.85) 83.46
GQAF (General) - MCQ Accuracy (%) 74.71 ± 0.15 74.70 ± 0.39 75.78 ± 0.51 75.68 ± 0.55 (+0.97) 76.24

Downstream benchmarking across 5 distinct VLM families evaluated with the learned soft-token attention realignment probe (\(K=8\) learned tokens) demonstrates consistent improvements across reasoning, alignment, and fine-grained visual grounding tasks:

Model Architecture Scale RealWorldQA (MCQ) MME (Yes/No) POPE (Hallucination) HRBench8k (Fine-grained) V* (Fine-grained)
Gemma3-12B Baseline 12B 61.87 2133.5 84.63 48.25 48.50
+ Soft Prompting 12B 61.87 2180.0 (+46.5) 84.98 (+0.35) 49.50 (+1.25) 49.79 (+1.29)
GLM-4.1V-9B-Base 9B 69.41 2208.1 88.52 51.62 58.80
+ Soft Prompting 9B 69.41 2237.8 (+29.7) 87.24 (-1.28) 52.25 (+0.63) 61.80 (+3.00)
Qwen2.5-VL-7B 7B 63.70 2199.5 87.68 54.13 66.95
+ Soft Prompting 7B 66.44 (+2.74) 2269.7 (+70.2) 88.65 (+0.97) 54.13 (=) 69.53 (+2.58)
Qwen3-VL-8B 8B 70.78 2390.8 89.26 61.38 72.96
+ Soft Prompting 8B 71.92 (+1.14) 2430.5 (+39.7) 89.41 (+0.15) 61.25 (-0.13) 75.11 (+2.15)

Ablation Study

Ablation of loss components and token count \(K\) in the attention realignment probe (evaluated on Qwen2.5-VL-7B):

Configuration RealWorldQA (MCQ) MMMU-Pro (Vision) POPE (Hallucination) V* (Fine-grained) Average Score (Avg)
Full Model (Ours, K=8, Infix) 66.44 31.50 88.65 69.53 64.03
w/o Attention Alignment Loss 64.84 32.89 87.05 65.67 62.61 (-1.42)
w/o Cross-Entropy Loss 64.61 31.27 83.40 63.95 60.81 (-3.22)
Learnable Tokens K=4 63.70 31.33 88.43 66.09 62.39 (-1.64)
Learnable Tokens K=12 63.93 31.50 88.36 67.81 62.90 (-1.13)
Learnable Tokens K=16 63.93 30.92 88.66 67.38 62.72 (-1.31)
Baseline (No tuning) 63.70 31.45 87.68 66.95 62.45

Token positioning ablations show that the Infix position (between question and instruction tokens) achieves 64.03 Avg, substantially outperforming Prefix (61.64 Avg) and Postfix (62.72 Avg), confirming that placing learnable adjustments directly adjacent to the question framing tokens delivers the most effective corrective signal.

Key Findings

  • Framing vs. Instruction Sensitivity: Coefficient of Quartile Variation (CQV) indicates that altering the core question formulation induces approximately \(3\times\) greater attention variation than merely swapping output instructions.
  • Monotonic Correlation with Scene Complexity: Grouping images by SAM2 segmentation count demonstrates that cross-framing inconsistency rises monotonically from 17.39% in simple scenes (10.1 segments) to 26.01% in cluttered scenes (65.8 segments), highlighting that visual disengagement is most damaging when visual clutter demands rigorous localization.
  • Asymmetric Reverse Inconsistency: The reverse transition rate (failing open-ended but succeeding on Yes/No or MCQ) is 44.64%, significantly higher than the 19.26% original inconsistency rate. However, chance-correction reveals this asymmetry is largely driven by guessing biases, proving that open-ended answering remains the only rigorous baseline for genuine grounding.

Highlights & Insights

  • Reconceptualizing Visual Blindness: Shatters the prevailing assumption that VLM blindness is a static architectural defect, proving instead that models actively modulate visual engagement dynamically according to surface question packaging.
  • Causal Attention Steering Methodology: Provides an elegant, parameter-free intervention tool that isolates total visual mass from spatial coordinate allocation, proving conclusively that spatial misallocation within the image is the primary culprit in reasoning breakdowns.
  • Prompting Mechanism Clarification: Reveals that superficial instructions like "look closely" do not change attention rollout, whereas explicit grounding constraints force models to traverse intermediate spatial representations, naturally restoring visual grounding.

Limitations & Future Work

  • Architectural Scope: The core empirical investigations center on dense autoregressive Transformer architectures (Qwen, LLaVA, Gemma, GLM); emerging paradigms such as Vision-Mamba state space models or Mixture-of-Experts (MoE) remain unstudied.
  • Framing Space Breadth: Controlled transformations focus on tripartite variants (Open, Y/N, MCQ); real-world human-VLM interactions exhibit richer linguistic nuances including presuppositions, conversational pragmatics, and open-vocabulary queries.
  • Pre-training Invariance vs. Test-Time Repair: Current mitigations (prompting and soft-token alignment) function as test-time or post-hoc adjustments; designing framing-invariant attention regularization during large-scale multimodal pre-training represents the ultimate solution.
  • vs. Static Visual Disengagement (Tong et al., 2024; Kang et al., 2025): While prior work identified visual sink tokens and spatial blind spots as fixed model characteristics, this paper demonstrates that visual allocation dynamically fluctuates under semantically identical linguistic shifts.
  • vs. Output-Space Prompt Sensitivity (Pezeshkpour & Hruschka, 2024; Song et al., 2025): Existing studies focus on option order biases and accuracy swings; this work uncovers the mechanistic \(F \to A \to Y\) causal chain by peering directly into internal attention distribution.
  • vs. Visual Search & Grounding Tools (Wu & Xie, 2024 - V*): Instead of introducing heavy external visual crop/search agents, this paper shows that native VLM visual competence can be largely reclaimed by steering internal attention back toward open-ended states.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ (Pioneering causal framing-to-attention paradigm that fundamentally alters our understanding of multimodal reasoning failures)
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Exhaustive rollout analysis across 5 model families, dual controlled datasets, attention steering, and human verification)
  • Writing Quality: ⭐⭐⭐⭐⭐ (Rigorous mathematical formulation, clear narrative structure, and compelling visual evidence)
  • Value: ⭐⭐⭐⭐⭐ (Essential reading for multimodal evaluation benchmark design, prompt engineering, and visual alignment)