Understanding the Effects of Distractors on Reasoning Vision-Language Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/effl-lab/Idis
Area: LLM Reasoning
Keywords: Vision-Language Models, Test-Time Scaling, Inverse Scaling, Distractor Robustness, Attribute Attribution
TL;DR¶
This paper constructs Idis, a controlled multimodal benchmark of over 277k samples, discovering that visual distractors degrade reasoning VLM accuracy by vertically shifting down the length-accuracy curve without increasing reasoning length, whereas textual distractors lengthen reasoning traces and amplify test-time inverse scaling; causal analysis further reveals that the proportion of distractor attributes allocated during thinking directly drives reasoning collapse.
Background & Motivation¶
Test-time computation scalingโenabling models to generate longer chain-of-thought traces during inferenceโhas emerged as a foundational strategy for advancing complex reasoning in language and vision-language models. However, producing lengthy reasoning outputs does not monotonically ensure superior prediction accuracy. Recent investigations into text-only reasoning models uncover pronounced failure modes such as overthinking and test-time inverse scaling, where irrelevant contextual information (distractors) systematically provokes models to spend more inference tokens while degrading final answer accuracy. These findings suggest that protracted thinking chains in language models often serve merely to amplify flawed heuristics seeded by distracting context.
In real-world multimodal scenarios, however, reasoning vision-language models (VLMs) encounter distracting evidence that spans both visual and linguistic modalities. Images inherently contain background clutter, incidental objects, and typographic overlays, while user prompts frequently introduce tangential textual statements. This raises a fundamental tension: do multimodal reasoning architectures fail under distractors through a unified, modality-agnostic mechanism, or do visual and textual distractors exert asymmetric influences on test-time scaling behavior due to intrinsic differences in how autoregressive attention accesses image tokens versus textual context?
To resolve this question, the authors establish a rigorous investigation spanning both perception-centric object recognition and reasoning-centric mathematical geometry. Core idea: construct the controlled multimodal benchmark Idis to decouple distractor modalities, quantities, and semantic relationships, demonstrating that visual distractors degrade accuracy via a vertical downward translation of the scaling curve without lengthening reasoning, and establishing that the verbalized distractor attribute ratio in thinking traces causally governs model collapse.
Method¶
Overall Architecture¶
The diagnostic and empirical framework comprises three interlocking components: (1) Controlled Benchmark Construction (Idis Suite): spanning perception-centric tasks (Idis-perception, 48,600 images built upon ImageNet-9) and visual mathematics tasks (Idis-math, 40,536 pairs derived from MathVerse), systematically orthogonalizing distractor modalities (visual objects, rendered typography, and textual prompt insertions), distractor density (\(N \in \{1, 2, 3, 4\}\)), and semantic relationships (Aligned, Conflicting, Irrelevant); (2) Modality-Dependent Sequential Scaling Analysis: evaluating open-weight and proprietary reasoning VLMs across repeated sampling trajectories to dissect how visual versus textual distractions alter the lengthโaccuracy Pareto frontier; (3) Attribute-Centric Causal Attribution & Mitigation: parsing fine-grained visual attributes in reasoning traces using an LLM extractor, causally verifying attribute influence via attention-blocking interventions, and deploying an attribute-guiding prompt strategy to restore reasoning robustness.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multimodal Base Benchmarks<br/>ImageNet-9 / MathVerse"] --> B["Controlled Distractor Benchmark Construction<br/>Decouple Modality / Count 1-4 / 3-Level Semantics"]
B --> C["Modality-Dependent Dual Scaling Dynamics<br/>Visual Vertical Shift vs Textual Length Expansion"]
C --> D["Attribute-Space Causal Attribution<br/>Distractor Attribute Ratio & Attention Blocking"]
D --> E["Target-Guiding Prompt Strategy<br/>Suppress Distractor Trace & Recover Accuracy"]
Key Designs¶
1. Controlled Multimodal Distractor Benchmark Construction: Decoupling Modality, Density, and Semantics
To eliminate problem hardness and baseline image complexity as confounding variables, the study constructs Idis (Images with distractors). For Idis-perception, the target object remains fixed while Gemini 2.5 Flash Image performs generative editing to seamlessly introduce distracting objects without boundary artifacts or target occlusion. For Idis-math, visual geometric distractors and typographic math formulas are deterministically composited onto an extended canvas to preserve the exact resolution and legibility of the core diagram. Distractors are systematically varied across three dimensions: modality (visual objects, typographic overlays, or textual prompt sentences), count (\(N \in \{1, 2, 3, 4\}\)), and semantics. Semantics are categorized into Aligned (spuriously co-occurring features, e.g., birdcage for bird), Conflicting (features indicative of an alternative competitive class, e.g., birdcage for vehicle), and Irrelevant (out-of-domain entities such as televisions or logic tables).
2. Modality-Dependent Dual Scaling Dynamics: Dissecting Vertical Degradation versus Length Expansion
Through extensive evaluation across leading reasoning VLMs (Qwen3-VL-8B-Thinking, Intern-S1-mini, GLM-4.1V-9B-Thinking, and R1-OneVision-7B-RL), the authors uncover a fundamental scaling asymmetry governed by distractor modality. Visual distractors and image-rendered typographic distractors consistently shift the entire lengthโaccuracy curve downward along the vertical axis without altering reasoning length: the per-step length delta is a negligible 11.8 tokens per distractor (vastly overshadowed by within-sample stochastic decoding variance of 122.4 tokens). Conversely, textual distractors inserted into the question prompt induce classic test-time inverse scaling: because textual distractors reside directly in the prompt's token address space, the model repeatedly attends to, quotes, and deliberates over them (appearing in 94.6%~97.3% of traces and occupying 22.8%~35.7% of word tokens), causing thinking lengths to explode and directly amplifying erroneous heuristics.
3. Attribute-Space Causal Attribution: The Decisive Role of Distractor Attribute Allocation
To explain why visual distractors degrade accuracy without inflating chain length, the authors extract visual attributes from generated reasoning traces using DeepSeek-V3.2-Exp, defining the distractor attribute ratio:
$\(r_{\text{attr}} = \frac{\#\text{Distractor Attributes}}{\#\text{Total Attributes}}\)$
While total trace attributes remain static across distractor counts (governed by intrinsic visual token density), \(r_{\text{attr}}\) scales monotonically with distractor quantity and spatial area ratio \(r_{\text{area}} = \frac{A_{\text{dist}}}{A_{\text{dist}} + A_{\text{target}}}\). When \(r_{\text{attr}} < 20\%\), model accuracy remains above 97%; once \(r_{\text{attr}} > 50\%\), accuracy collapses to near zero. A layer-wide Transformer attention-blocking experiment directly establishes causality: blocking attention from the final <answer> tokens to distractor attribute tokens in the trace improves accuracy (by up to +0.30%p), whereas blocking attention to target attribute tokens causes performance to drop drastically (-9.26%p), proving that trace-level attribute distribution causally determines the final prediction.
4. Target-Guiding Prompt Strategy: Restoring Robustness via Foreground Attribution
Leveraging the insight that final predictions are steered by attribute representation ratios rather than trace length, the authors introduce a training-free prompt mitigation strategy. The prompt explicitly instructs the reasoning engine: "First identify the main object. Base your reasoning only on that object's own visual attributes, and determine which category best matches the object. Do not assume that the most visually dominant object is the main object. Do not infer the category from surrounding objects or associated context." This mechanism actively suppresses distractor attribute verbalization in the chain of thought, successfully decreasing \(r_{\text{attr}}\) across all evaluated models and yielding consistent accuracy improvements across both Idis and the distribution-shift Waterbirds benchmark.
Benchmark & Evaluation Protocol¶
The evaluation leverages Idis-perception (4,050 ImageNet-9 base images expanded across four distractor counts and three semantic conditions), Idis-math (3,152 MathVerse problems augmented with geometric and handwritten distractions), and the Waterbirds spurious-correlation benchmark. Models evaluated include open-weight reasoning VLMs (Qwen3-VL-8B-Thinking, Intern-S1-mini, GLM-4.1V-9B-Thinking, R1-OneVision-7B-RL) and proprietary Gemini 3 Flash. Sequential test-time scaling curves are derived by sampling 5 responses per instance (temperature 0.7, top-p 0.95), ranking responses by generated reasoning length, and evaluating answer correctness using VLMEvalKit with Qwen3.5-27B as the objective judge model. Attention blocking is conducted on NVIDIA RTX A6000 Ada GPUs across all transformer heads and layers.
Key Experimental Results¶
Main Results¶
Accuracy of reasoning VLMs across distractor sizes (Small, Medium, Large) on the Idis-manual benchmark, and debiasing performance on the Waterbirds benchmark under the proposed prompt strategy:
| Benchmark / Model | Distractor Size: Small | Distractor Size: Medium | Distractor Size: Large | Accuracy Delta (S \(\to\) L) |
|---|---|---|---|---|
| Qwen3-VL-Thinking (Idis-manual) | 96.1% | 95.5% | 94.0% | -2.1%p |
| Intern-S1-mini (Idis-manual) | 94.4% | 93.1% | 90.9% | -3.5%p |
| GLM-4.1V-Thinking (Idis-manual) | 96.3% | 95.6% | 94.7% | -1.6%p |
| R1-OneVision (Idis-manual) | 92.86% | 91.68% | 88.25% | -4.61%p |
Performance on the Waterbirds spurious correlation benchmark across Aligned, Conflicting, and Overall subsets:
| Model Architecture & Setting | Aligned Acc. | Aligned Len. | Conflicting Acc. | Conflicting Len. | Overall Acc. | Overall Len. |
|---|---|---|---|---|---|---|
| Qwen3-VL-Thinking (Baseline) | 93.6% | 635.9 | 76.4% | 829.9 | 88.3% | 695.8 |
| + Prompt Strategy | 94.1% | 474.7 | 78.2% (+1.8) | 652.1 | 89.2% (+0.9) | 529.5 |
| Intern-S1-mini (Baseline) | 93.2% | 493.2 | 56.7% | 672.5 | 81.9% | 548.6 |
| + Prompt Strategy | 92.4% | 574.2 | 58.7% (+2.0) | 711.9 | 82.0% (+0.1) | 616.7 |
| GLM-4.1V-Thinking (Baseline) | 94.4% | 340.9 | 80.7% | 498.2 | 90.1% | 389.5 |
| + Prompt Strategy | 92.9% | 253.9 | 84.7% (+4.0) | 358.7 | 90.4% (+0.3) | 286.3 |
| R1-OneVision (Baseline) | 93.9% | 300.5 | 63.0% | 310.5 | 84.4% | 303.6 |
| + Prompt Strategy | 93.9% | 233.0 | 66.3% (+3.3) | 244.7 | 85.4% (+1.0) | 236.6 |
Ablation & Mechanism Study¶
Quantifying the verbalization and attention dynamics of visual versus textual distractors within autoregressive reasoning traces:
| Distractor Modality & Setting | Mentions per Trace | Traces with \(\ge 1\) Mention | Distractor Word Share | Trace Length Response |
|---|---|---|---|---|
| Visual Distractor (\(N=1\)) | 4.74 | 24.7% | 2.51% | Trace length flat (2558 words) |
| Visual Distractor (\(N=4\)) | 4.81 | 25.6% | 2.46% | No length inflation (downward shift) |
| Textual Distractor (\(N=1\)) | 48.6 | 94.6% | 22.8% | Substantial trace expansion |
| Textual Distractor (\(N=4\)) | 105.6 | 97.3% | 35.7% | Inflates to 3181 words (89% of growth on distractors) |
Key Findings¶
- Modality-Driven Scaling Divergence: Visual distractors do not provoke longer reasoning chains; instead, they depress the entire lengthโaccuracy curve downwards. Teacher-forced attention analysis demonstrates that image attention increases only upon initially extending canvas size (0.96% to 2.04%) and remains completely flat across \(N=1\sim 4\) (1.65%~1.76%). In contrast, textual distractors compete cumulatively in token space, with 89% of added tokens spent directly deliberating on distractors.
- Semantic Conflict Provokes Maximum Vulnerability: Across all benchmark configurations, Conflicting distractors induce catastrophic accuracy drops compared to Aligned or Irrelevant distractors. Models suffer from severe attribution illusions, classifying the distractor object itself when its physical area or attribute count exceeds that of the target.
- Invariance to Prompted Thinking Budgets: Prepending explicit reasoning budget constraints (e.g., capping thinking at 1024, 2048, or 4096 tokens) fails to alter the accuracy or trace length distributions in reasoning VLMs, indicating that current multimodal architectures lack the prompt-level reasoning budget controllability characteristic of text-only LLMs.
Highlights & Insights¶
- Asymmetric Multimodal Scaling Mechanism: Disproves the prevailing assumption that distractors inherently trigger overthinking in multimodal reasoning, proving that visual distractions operate via attribute allocation skew rather than reasoning sequence inflation.
- Causal Evidence through Attention Blocking: Moves beyond correlational observations by demonstrating via all-layer Transformer attention blocking that direct access to distractor attribute tokens in the generated chain causally governs the final classification.
- Generalizable Inference-Time Debiasing: Shows that simply guiding models to isolate foreground object attributes prior to step-by-step thinking decreases distractor attribute ratios and recovers up to 4.0%p in accuracy on spurious correlation tasks.
Limitations & Future Work¶
- Scope Restricted to Static VQA: The empirical analyses center on closed-set categorization and mathematical geometry; exploring open-ended agentic decision making, multi-image navigation, and embodied robotics represents an essential next step.
- Ceiling of Prompt-Level Mitigation: While prompt steering consistently dampens distractor attributes, it yields modest gains (+0.5%p to +4.0%p) and cannot fully insulate models against dominant, bias-conflicting visual objects, highlighting the need for architectural cross-attention filtering.
- Extraction Pipeline Granularity: Attribute attribution relies on semantic parsing via an external LLM, which captures explicit surface verbalizations but cannot directly observe implicit representation routing in intermediate latent states.
Related Work & Insights¶
- vs Gema et al. (2025) (Inverse Scaling in Test-Time Compute): While Gema et al. demonstrated that textual distractors trigger inverse scaling via expanded reasoning traces in text LMs, this work demonstrates that visual distractors degrade multimodal models without expanding trace length, establishing a crucial modality divergence.
- vs Deng et al. (2025) (Words or Vision in VLMs): Deng et al. identified a textual bias when VLMs encounter typographic overlays; this paper reveals that typographic distractors rendered as images behave mechanically like visual objects rather than textual prompt tokens in terms of test-time scaling dynamics.
Rating¶
- Novelty: โญโญโญโญโญ [Pioneering characterization of the modality-dependent test-time scaling asymmetry under multimodal distractors]
- Experimental Thoroughness: โญโญโญโญโญ [Extensive 277k-sample benchmark, spanning 4 open-weight frontier VLMs and Gemini 3 Flash, with rigorous attention-blocking causal proofs]
- Writing Quality: โญโญโญโญโญ [Exceptionally clear narrative structure, rigorous formalization of attribute metrics, and precise visual-linguistic decoupling]
- Value: โญโญโญโญโญ [Essential theoretical and empirical guidance for test-time computation, multimodal reasoning alignment, and hallucination mitigation]