Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models

Conference: ECCV2026
Paper: ECCV Paper
Project: Entropy-Gradient Grounding
Area: Multimodal VLM
Keywords: visual grounding, entropy gradients, multi-region evidence, iterative zooming, visual question answering

TL;DR

Backpropagating first-token predictive entropy to visual embeddings retrieves multiple image regions for adaptive zooming without updating model weights, raising Qwen2.5-VL 7B V* accuracy from 73.30 to 86.91 at the cost of additional forward and backward computation.

Background & Motivation

A vision-language model can answer incorrectly because it failed to perceive decisive evidence, not because it lacked reasoning ability. Compressing a full document into a limited visual-token budget can obscure small text, individual table rows, or distant sign numbers. Other questions require combining clues far apart in the image. Higher full-image resolution and dense tiling improve coverage, but also add substantial visual input unrelated to the question.

Training-free approaches such as ViCrop extract localization cues from model internals and append image crops. However, attention maps depend on layer and head selection, and strong attention does not necessarily identify answer-determining evidence. A single crop can also capture only the most salient clue. This paper consequently uses grounding not merely to explain an answer, but to determine what the model can inspect next. Localization errors, missing secondary clues, and excessive zooming must therefore be handled within the same inference process.

Core Idea: use the sensitivity of next-token entropy to visual features to retrieve candidate evidence, preserve multiple spatially separated regions, and control further zooming through the grounding mask's spatial entropy rather than a trained detector or one-step generation confidence.

Method

Overall Architecture

The input is an image and a question; the output remains a textual answer from the original VLM. The method extracts a query-conditioned heatmap, converts it into additional crops, and repeatedly grounds evidence in the original image and local views. The retained crops and global view are then supplied together for answer generation. The original image preserves context, while the local views concentrate visual resolution on potentially decisive content.

The three core designs are entropy-gradient grounding, multi-region selection, and spatial-entropy refinement. They respectively determine the localization signal, preserve disjoint evidence, and decide when zooming should stop. They are not separately trained modules.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image and question"] --> B["Entropy-gradient grounding"]
    B --> C["Multi-region selection"]
    C --> D["Spatial-entropy refinement"]
    D -->|Entropy decreases; update views| B
    D -->|Entropy increases; retain previous views| E["Original image and retained crops"]
    E --> F["Final answer"]

Key Designs

1. Entropy-gradient grounding: identify visual patches that affect predictive uncertainty

The model first processes the image and question without needing to generate a complete answer. By default, the method computes Shannon entropy over the full vocabulary at the first decoding position and backpropagates this scalar to image-token embeddings after the vision-language projector. Each token corresponds to a spatial patch; the gradient vector's L2 norm supplies its scalar saliency, which is reshaped into an image-aligned heatmap. The following combines the core relations from Equations (1)-(4), where \(p_1(y)\) is the first-token probability, \(\mathcal{Y}\) is the vocabulary, and \(\mathbf v_i\) is projected visual embedding \(i\):

\[ \mathcal L_{\mathrm{ent}}=-\sum_{y\in\mathcal Y}p_1(y)\log p_1(y),\qquad s_i=\left\|\frac{\partial\mathcal L_{\mathrm{ent}}}{\partial\mathbf v_i}\right\|_2. \]

Unlike observing an individual attention head, this signal starts from the final predictive distribution and measures the local influence of visual-feature changes on output uncertainty. It requires neither a correct-answer label nor a separately learned grounding head. Nevertheless, a large gradient indicates sensitivity, not proof that a region contains correct evidence. Taking its norm also discards direction: the method does not edit image features along a gradient to minimize entropy, but uses gradient magnitude to select crops from the real image.

The first-token choice matters because later predictions are already conditioned on generated text and may reinforce an existing interpretation. Using the first token reduces that dependence and avoids unrolling several decoding steps before grounding. In Table 6, first-token grounding obtains 67.96 on TextVQA, compared with 65.27 for the second token. This supports the default, but does not establish that the first token is best on every dataset.

2. Multi-region selection: convert saliency into a small set of readable evidence views

Gradient maps can contain isolated peaks, making the raw maximum an unreliable crop anchor. The method first applies Gaussian smoothing, sorts the smoothed responses, and selects a threshold with the elbow method: the point of maximum deviation from the chord joining the minimum and maximum values. Strong responses become a binary support mask. This removes the need to tune a retained percentile, but the pipeline still includes smoothing, binarization, and connected-component processing; it is not literally free of post-processing.

Connected components are extracted on the visual-token grid and ranked by the sum of original, unsmoothed saliency scores within each component, not merely by their maximum response. The top \(K\) regions are mapped to tight bounding boxes in the original image and supplied as separate crops alongside the global view. The default is \(K=2\). The goal is not to enclose everything in one large box, but to represent two potentially separated evidence regions more clearly.

This ranking favors components with substantial accumulated evidence. Gaussian smoothing supplies spatial coherence, the elbow threshold suppresses background, and original-score accumulation ranks candidates. In Table 4, increasing the number of regions from one to two raises LLaVA-1.6 DocVQA from 61.25 to 65.07, consistent with document questions needing both a row label and its associated value. These aggregate results do not, however, prove that every example uses complementary evidence.

3. Spatial-entropy refinement: check whether another zoom still concentrates evidence

Initial boxes may be too broad or overlook small and secondary clues. The algorithm retains the original image as global context and adds the selected crops to the current view set. Each iteration recomputes entropy gradients and binary masks on the views, then crops around activated support. Local views enable deeper zooming, while the global view provides an entry point for discovering other regions instead of committing exclusively to an initially incorrect crop.

The stopping signal is the area distribution of mask components, not language-output entropy or entropy over all pixel intensities. In Equations (7) and (8), \(C_i\) is a connected component of the binary mask and \(|C_i|\) counts its active locations:

\[ P(C_i)=\frac{|C_i|}{\sum_j|C_j|},\qquad H_{\mathrm{sp}}(\mathbf M)=-\sum_iP(C_i)\log P(C_i). \]

The implementation tracks spatial entropy associated with the most important view. Refinement continues while entropy decreases; when it increases, the method stops and uses the previous crops, as shown in Figure 2, to avoid discarding useful context. The main text does not specify implementation branches for equal entropy or an empty mask, so those cases should not be invented as part of the authors' algorithm.

This measure describes dispersion in component-area proportions rather than complete geometric compactness. If a mask contains one connected component, its spatial entropy is zero regardless of that component's size. It is therefore a stopping proxy, not a mathematical guarantee that enough evidence has been collected; its justification comes primarily from the stopping-rule ablation.

A Worked Example

Figure 3 shows a statistical table and asks how many patients were discharged from Neurology under private service. Relative-attention grounding selects an incorrect region and produces 6; the entropy-gradient map concentrates on the relevant table row, enabling the answer 696. The crop changes the evidence the model reads rather than repairing the generated answer afterward.

Within the complete pipeline, the model first identifies sensitive regions on the page, retains top-ranked components, and grounds again in enlarged views until spatial entropy no longer supports refinement. This walkthrough explains the mechanism; Figure 3 does not report the example's actual iteration count or region scores, so no per-iteration trajectory can be inferred from it.

Loss & Training

There is no additional training set, optimizer, or weight update. The entropy "loss" is only a scalar differentiation objective at test time; model weights remain frozen, and the final answer comes from ordinary generation. Training-free therefore does not mean forward-only: deployment requires access to visual embeddings and automatic differentiation.

Defaults are the first token, full-vocabulary entropy, two additional regions, and spatial-entropy stopping. Table 5 also compares transformer-layer settings for gradient computation and chooses the last layer as the shared default. However, layer 20 achieves V* 78.01 versus 73.30 for the last layer, so the default is not best on every metric. The cache does not include the referenced supplementary material; exact implementation hyperparameters and localization metrics are not guessed here.

Key Experimental Results

Main Results

Section 4.1 evaluates LLaVA-1.5 7B, LLaVA-1.6 Mistral 7B, InternVL-3.5 8B, and Qwen2.5-VL 7B on seven benchmarks. TextVQA uses standard VQA accuracy; DocVQA and InfoQA use ANLS from official test servers; V*, GQA, and RWQA use top-1 accuracy; POPE uses accuracy averaged across splits. The following excerpts Table 1 and preserves its percentage-scale presentation. Gains are score points, not relative percentages.

Backbone and dataset	Metric	Base model	Ours	Gain
LLaVA-1.5 7B / DocVQA	ANLS	22.32	33.70	+11.38
LLaVA-1.6 Mistral 7B / V*	Accuracy	57.59	73.30	+15.71
LLaVA-1.6 Mistral 7B / InfoQA	ANLS	24.66	33.93	+9.27
InternVL-3.5 8B / DocVQA	ANLS	58.73	79.54	+20.81
Qwen2.5-VL 7B / V*	Accuracy	73.30	86.91	+13.61
Qwen2.5-VL 7B / GQA	Accuracy	61.01	59.49	-1.52
Qwen2.5-VL 7B / RWQA	Accuracy	67.84	66.93	-0.91

The method does not win every same-backbone comparison. With LLaVA-1.5, ZoomEye achieves V* 72.25 versus 56.02 for this method; with LLaVA-1.6, ViCrop achieves TextVQA 68.65 versus 67.96. Training-based TEVA also differs in parameter count and training conditions, so cross-model score differences cannot be attributed solely to grounding.

Ablation Study

The following reproduces the region-count comparison from Table 4 using LLaVA-1.6 throughout. DocVQA reports ANLS; the remaining columns report accuracy, all on the paper's score scale.

Config	POPE	V*	DocVQA	RWQA
No additional regions, base model	87.80	57.59	64.94	58.30
1 additional region	88.79	69.63	61.25	60.39
2 additional regions	89.31	73.30	65.07	60.39
3 additional regions	89.06	72.25	64.87	59.87
4 additional regions	89.19	72.25	65.14	58.82

Two regions offer a strong overall compromise rather than a strict optimum in every column: four regions slightly improve DocVQA, while one and two regions tie on RWQA. Extra crops may add redundancy, so a larger visual-token count does not imply better performance.

The next table excerpts Table 2, again using LLaVA-1.6, and compares stopping rules. Timing values retain that table's reporting context and should not be merged with Table 7 as if they were one measurement series.

Stopping config	V*	DocVQA	RWQA	Table 2 inference time
Spatial entropy	73.30	65.07	60.39	4.80 s
Stop when first-token maximum probability decreases	70.16	64.26	58.69	3.84 s
Fixed 1 iteration	60.73	65.42	59.08	2.22 s
Fixed 2 iterations	65.96	63.34	60.39	2.70 s
Fixed 3 iterations	65.96	61.80	59.22	3.84 s

Key Findings

• Spatial-entropy stopping exceeds confidence stopping by 3.14 V* points but takes longer. One fixed iteration exceeds adaptive stopping on DocVQA, so no rule dominates every metric.
• In Table 3, full entropy and maximum-probability objectives achieve V* 73.30 and 73.29, respectively. This small difference supports the retrieval pipeline more strongly than a claim that entropy is the only effective objective.
• Table 7 separately reports average per-example runtime on RealWorldQA images of 726–1536 px: LLaVA-1.6 takes 2.04 s for a single retrieval pass and 3.37 s for full iteration. These differ from 2.22 s and 4.80 s in Table 2; the available main text does not reconcile the timing conditions or provide a verifiable hardware configuration.

Highlights & Insights

• Attribution determines the next input instead of serving only as a post-hoc explanation. The benefit comes from rereading enlarged evidence, creating a feedback loop between grounding and answering.
• Predictive uncertainty and localization dispersion have distinct roles. The former creates saliency, while the latter controls iteration, avoiding a direct equation between confidence in the next word and sufficient visual evidence.
• Multiple regions preserve separated clues, while the original image preserves scene relationships. This matches the information needs of tables, infographics, and small-text questions better than selecting only the strongest peak.

Limitations & Future Work

• The authors report latency from repeated forward and backward passes as a major cost. White-box gradient access is also a deployment constraint: the method cannot be directly applied to a closed VLM exposed only through a text-generation interface.
• Figure 6 shows that weaker backbones more often assign gradients to incorrect locations. When internal representations or uncertainty estimates are unreliable, cropping may reinforce an incorrect interpretation instead of correcting it.
• Spatial entropy measures component-area proportions, not semantic correctness, and does not distinguish different scales of a single connected component. This limitation follows from the formula; adding localization stability or context-retention constraints is a possible extension, not a result tested here.
• General understanding can regress, and some same-backbone competitors remain stronger. Small differences without error bars should not be treated as significant advantages. The supplementary material is absent from the cache, leaving localization accuracy, exact post-processing settings, and full reproducibility conditions to be checked against the original implementation.

Related Work & Insights

• vs ViCrop: both improve perception through additional crops. ViCrop uses attention-related cues and a single-crop strategy; this method uses entropy gradients, multiple components, and adaptive refinement, while still performing post-processing.
• vs ZoomEye: ZoomEye searches through tree-based exploration, whereas this method directs retrieval using internal gradients. The LLaVA-1.5 V* results in Table 1 show that targeted retrieval is not inherently superior to broader search.
• vs Grad-CAM / Integrated Gradients: the method shares gradient-attribution ideas, but differentiates entropy over the language-output distribution and uses attribution to change subsequent inputs. This does not make its heatmaps causally validated explanations.

Rating

• Novelty: 4/5. Combines entropy gradients, multi-region evidence, and stopping control into an actionable training-free retrieval pipeline.
• Experimental Thoroughness: 4/5. Four backbone families, seven benchmarks, and several ablations provide broad coverage, but timing conditions and statistical reliability need clarification.
• Writing Quality: 4/5. The mechanism and experimental narrative are clear, although claims about absent post-processing and timing conventions require careful interpretation.
• Value: 4/5. Useful for VLM applications with gradient access that can trade inference computation for better fine-detail perception.