Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning

Conference: ECCV2026
Paper: ECCV Paper
Area: Vision-Language Reasoning
Keywords: contrastive attention, visual question answering, attention entropy, pixel masking, training-free visual enhancement

TL;DR

CARVE extracts task-relevant regions through the ratio between the same model's attention to a specific question and a general image description, then masks, crops, magnifies, and answers again, raising LLaVA-1.5-7B's V* accuracy from 38.7% to 66.5% without training but requiring three generation passes.

Background & Motivation

A vision-language model (VLM) that understands an image does not necessarily see the details required to answer a question in a cluttered scene. Reading bottle brands, recognizing a shape through a cup handle, or locating text in a dense document can be hindered by competing colors and textures. ViCrop has shown that even an incorrect answer can coexist with useful target-location information in internal attention, which cropping can turn into a clearer input. However, a crop can still contain distractors, and a generic segmentation or detection tool may not know what the current question actually concerns.

The paper first investigates whether visual complexity corresponds to dispersed attention. Texture complexity is the fraction of Canny edge pixels, while color complexity is the normalized entropy of an HSV hue histogram with 180 bins. Attention dispersion is measured by Shannon entropy over visual tokens, averaged across selected layers at the final generation step. Analysis of Qwen2.5-VL-3B-Instruct on TextVQA finds that more complex images generally elicit more dispersed attention; as attention entropy rises from approximately 5.1 to 6.8, accuracy falls from approximately 76% to 65%. These correlations alone do not establish the full causal chain. Preliminary background-masking interventions additionally suggest that reducing distraction can increase the probability of a correct answer.

The difficulty is that selecting regions with high attention retains both locations important because of the question and locations important because they are visually salient. The authors therefore ask the same model to produce a task-agnostic image description and use its attention as a reference. Core Idea: emphasize regions receiving additional attention because of task semantics through the ratio of question attention to general-description attention, then denoise the input pixels instead of training a new module or directly modifying model weights.

Method

Overall Architecture

CARVE takes an image, a question, and a VLM that exposes internal attention, and returns an answer to the original question. The model first generates text for the specific question and the general description separately, yielding two sets of visual attention maps. Semantic Ratio Contrasting, Cross-Layer Temporal Fusion, and Pixel Selection and Re-observation then produce a refined image, which is passed to the same model with the original question.

These three generation passes are not teacher-student training: the first two provide localization signals, and the third produces the final answer. Model parameters remain unchanged, and no additional segmentation model is used. Even when the initial answer is wrong, its attended locations can guide re-observation, so the method does not require correct first-pass reasoning.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Image and question"] --> QUESTION["Question-conditioned generation<br/>Extract visual attention"]
    INPUT --> GENERAL["General-description generation<br/>Extract visual attention"]
    QUESTION --> CONTRAST["Semantic Ratio Contrasting"]
    GENERAL --> CONTRAST
    CONTRAST --> FUSION["Cross-Layer Temporal Fusion"]
    FUSION --> REFINE["Pixel Selection and Re-observation"]
    REFINE --> ANSWER["Generate again with the same model<br/>Return the final answer"]

Key Designs

1. Semantic Ratio Contrasting: separate image-inherent salience from task relevance

The authors assume that attention at a location approximately factorizes into the element-wise product of an image-inherent visual factor and a question-related semantic factor. Visual noise here does not mean sensor noise: it refers to image factors that attract the model without necessarily helping the current task. The general prompt is Write a general description of the image., intended to encourage global scanning rather than querying a particular object. Under the assumption that its task-specific semantic factor is approximately uniform, general-description attention estimates the image-inherent factor. This uniformity is an assumption, not an identity, and does not mean the general description produces ground-truth background segmentation.

For the two attention maps at the same layer and generation step, the central transformation is:

\[ \widehat{A}_{l,t,i}=\frac{A^{(Q)}_{l,t,i}}{A^{(G)}_{l,t,i}+\lambda}. \]

The numerator is question-conditioned attention to visual token \(i\), the denominator contains the corresponding general-description attention, and \(\lambda\) is a regularization parameter. Locations strongly attended to in both passes are relatively suppressed, whereas locations additionally emphasized by the question are more likely to survive. Unlike subtraction, the ratio corrects different amplification scales across locations. The derivation approximately recovers the semantic factor only when the image-inherent factor is substantially greater than \(\lambda\); otherwise, regularization also alters the signal. The paper reports KL divergences from uniform of 0.147 for general attention and 0.812 for question attention, supporting a more diffuse reference distribution without proving the multiplicative decomposition for every image.

2. Cross-Layer Temporal Fusion: construct spatial scores from more mature localization signals

Attention from one layer or generation step can be unstable. CARVE therefore maps contrasted visual-token attention back to image space and aggregates it across selected layers and generation steps. Layers are summed directly. When using the full generation trajectory, temporal weights increase linearly as \(w_t=t-t_{\mathrm{start}}+1\). Later tokens have longer prefix context and are expected to have localized the target more effectively. The aggregation operates on contrasted spatial maps, not on answer probabilities from the two generations, so this is not contrastive decoding.

The paper evaluates early-step, final-step, and full-trajectory weighted configurations. The main table uses layers \([20,25]\), with the final step usually slightly outperforming full-trajectory fusion. Using every time step is therefore not essential to the method. Cross-layer fusion and temporal selection are also distinct: selecting only the final step still permits fusion across multiple deep layers. The motivating observation is that shallow layers scan globally while deeper layers focus more strongly, so adding many shallow maps need not outperform using a smaller set of deep maps.

3. Pixel Selection and Re-observation: turn attention heatmaps into cleaner visual inputs

The fused spatial scores undergo top-\(p\) selection, retaining the highest-scoring \(p\) fraction of pixels, followed by connected-component analysis. Regions are ranked by their cumulative attention scores, and at most \(K\) are retained. This ranking is neither selection by a single peak nor simply selection of the largest box. The original image is then masked, cropped, and resized before being returned to the model. Masking removes unwanted pixels within the selected spatial extent, while cropping and magnification increase the effective scale of retained details. Together, they differ from extracting only a rectangular window.

Retaining too much leaves background distraction; retaining too little can cut away targets or necessary context. Too many regions reintroduce distractors, while too few may omit multiple objects needed for the answer. A parameter sweep on 1,000 TextVQA samples favors \(p\in[0.2,0.6]\) and \(K\in\{2,3\}\). The algorithm defines \(p\) as a retained proportion, whereas the threshold annotations and accompanying discussion in Figure 8 are ambiguous. This note follows the algorithm rather than treating all threshold numbers in that figure as the same percentile quantity.

A Worked Example

Figure 6 asks which shape is visible through a cup handle. General-description generation attends to the cup itself, while the initial question-conditioned answer incorrectly predicts a circle but still attends around the handle. CARVE uses the difference between the two attention patterns to distinguish the cup's overall salience from the relevance of the area inside the handle.

Cross-Layer Temporal Fusion then aggregates relevant responses into spatial scores, and pixel selection removes irrelevant background before cropping and magnifying the useful area. The third generation observes a clearer target and answers star. The example illustrates correction of visual evidence rather than direct rewriting of an incorrect answer string. The answer change comes from the paper's example and does not imply that every spatial-relation task benefits from background removal.

Loss & Training

The model is not trained. The objective in Section 4.1 is an analytical optimization problem used to derive the attention ratio: one term requires the estimated semantic signal multiplied by the visual factor to reconstruct question attention, while another applies a visual-factor-weighted quadratic regularizer to suppress excessively large estimates. First-order optimality yields the closed-form ratio above; this is not a new loss back-propagated over a training dataset.

Experiments use greedy decoding and \(\lambda=0.05\). The Qwen2.5-VL family receives \(448\times448\) inputs, while the LLaVA-1.5 family receives \(336\times336\) inputs. The results therefore concern this restricted-resolution protocol and should not be assumed to hold at the same magnitude under native high-resolution deployment. The general-description prompt is selected from five candidates, but the cached paper does not provide a complete candidate comparison table. It also does not explicitly disclose a common \(p\) and \(K\) for every main experiment, so the best sweep configuration cannot be assumed to apply to all tables.

Key Experimental Results

Main Results

The paper covers four models and seven benchmarks. The table below selects final-step results from original Table 1 and cross-dataset results from original Table 3. Scores are percentages, and gains are absolute percentage points over each model's Vanilla baseline, not the relative improvements shown as superscripts in the paper. TextVQA uses only images and questions, without external OCR. The cached paper does not explicitly identify the dataset splits and complete sample counts for these main tables.

Source and model	Dataset	Metric	Vanilla	CARVE	Gain (percentage points)
Table 1, Qwen2.5-VL-3B	A-OKVQA	Accuracy	73.0	79.2	+6.2
Table 1, Qwen2.5-VL-7B	V*	Accuracy	50.8	59.7	+8.9
Table 1, Qwen2.5-VL-7B	TextVQA	Accuracy	75.0	81.9	+6.9
Table 1, LLaVA-1.5-7B	V*	Accuracy	38.7	66.5	+27.8
Table 1, LLaVA-1.5-7B	TextVQA	Accuracy	47.8	58.2	+10.4
Table 3, Qwen2.5-VL-3B	DocVQA	Exact Match (EM)	49.70	61.70	+12.00
Table 3, Qwen2.5-VL-3B	ChartQA	Exact Match (EM)	67.64	70.80	+3.16
Table 3, Qwen2.5-VL-3B	ScienceQA	Multiple-choice accuracy	75.11	75.56	+0.45

EM requires exact answer matching, while ScienceQA reports the fraction of multiple-choice questions answered correctly. Tasks with stronger local-detail demands gain more, which does not imply an equivalent improvement in abstract reasoning. CARVE also does not lead every cross-method comparison: original Table 3 reports 82.50 on ScienceQA for Qwen2.5-VL-7B, below SD-RPN's 83.84.

Ablation Study

Original Table 4 uses Qwen2.5-VL-7B throughout. The ChartQA baseline of 79.12 matches Relaxed Accuracy in Table 3, so that metric is used here rather than conflating it with EM in the main results. Relaxed Accuracy allows the numerical tolerance specified by the benchmark.

Config	TextVQA accuracy	ChartQA Relaxed Accuracy	ScienceQA multiple-choice accuracy
Vanilla	75.0	79.12	81.21
Direct question-attention masking	78.6	79.92	82.30
Direct internal-attention modification	77.1	79.56	82.10
CARVE full model	82.0	79.96	82.50

Within this table, contrastive normalization adds 3.4 TextVQA percentage points over direct masking, and the full method exceeds internal-attention modification by 4.9 points. The corresponding score is 81.9 in original Table 1, 82.0 in original Table 4, and a best value of 82.1 in Figure 7's 1,000-sample subset. The subset sweep cannot be equated with the main table, and the 0.1 difference between the first two tables is not fully explained in the paper; the values are preserved as reported.

Key Findings

• Temporal selection matters: Qwen2.5-VL-7B achieves 80.7, 81.7, and 81.9 on TextVQA using early-step, full-trajectory, and final-step attention, respectively. Final-step localization is more effective, but its advantage is small.
• Deep-layer signals are more useful than shallow-layer signals: with full-trajectory fusion fixed, the same model scores 75.0 on TextVQA using layers \([10,15]\) and 81.7 using \([20,25]\), according to original Table 2.
• The cost is substantial: original Table 5 reports LLaVA-1.5-7B's TextVQA end-to-end latency increasing from 0.17 seconds to 1.34 seconds. Original Table 6, measured on 100 ChartQA samples, reports Qwen2.5-VL-7B GPU time increasing from 415 ms to 1,286 ms and peak GPU memory from 15.9 GB to 16.6 GB. These timing definitions and sample sets differ and should not be pooled; the cached paper also does not specify the GPU model.

Highlights & Insights

• The reference prompt supplies a control condition for the same image. Attention selection can therefore target the additional preference induced by the question rather than mistaking absolute salience for task relevance.
• An incorrect first answer does not make localization information useless. Converting attention into a new pixel input gives the same model a more targeted opportunity to observe the evidence.
• Improvements concentrate on perceptual bottlenecks. V* and document results more strongly support exposing details to existing capabilities than giving the model new knowledge-based reasoning abilities.

Limitations & Future Work

• The authors explicitly note that depth ordering and object relationships may depend on global context, which masking can remove. Inputs can approach the original as \(p\) approaches 1.0, but the paper does not provide a complete policy for automatically deciding when to fall back.
• The authors acknowledge the overhead of three generation passes and the memory cost of storing attention. Early termination and caching are proposed optimization directions, not demonstrated acceleration results.
• This note's assessment: multiplicative decomposition and approximately uniform general-prompt semantics are strong assumptions. General descriptions may have subject biases and cannot be guaranteed to provide a reliable denoising reference for every image.
• This note's assessment: incomplete reporting of main-experiment splits, hardware, common masking parameters, and some score differences limits exact reproduction. Further tests should use native high resolution, global-relation tasks, and matched compute budgets.

Related Work & Insights

• vs ViCrop: both exploit existing model attention for localization. CARVE adds a general-prompt reference and pixel-level masking beyond cropping. Its additional processing means deployment value cannot be judged from accuracy alone.
• vs SD-RPN: SD-RPN trains a region predictor. CARVE introduces no training process, instead shifting cost to multiple test-time generations and attention processing.
• vs contrastive learning and contrastive decoding: CARVE neither optimizes an embedding space nor subtracts final-token probabilities. It contrasts spatial attention under two prompt conditions and acts on the image input for the next pass.

Rating

• Novelty: 4/5. The general-prompt reference and ratio normalization form a distinct mechanism, although they build on existing attention-guided cropping.
• Experimental Thoroughness: 4/5. Seven benchmarks, four models, component ablations, and cost analysis provide breadth, but reproduction details and global-relation tests remain limited.
• Writing Quality: 3/5. The progression from observation to method is clear, while some threshold, metric, and cross-table score conventions require careful interpretation.
• Value: 4/5. Useful for fine-grained visual tasks when attention is accessible and extra inference cost is acceptable, rather than a universal reasoning enhancer.