SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs¶
Conference: ECCV 2026
Paper: ECCV Full Paper
Code: https://github.com/YinBo0927/SPOT-E
Area: Multimodal VLM
Keywords: Vision-language models, Test-time adaptation, Entropy shaping, Visual spotlights, Reinforcement learning
TL;DR¶
Addressing the failure of frozen vision-language models on evidence-intensive tasks caused by overlooking fine-grained visual cues, SPOT-E introduces an entropy-shaping principle combining answer-span entropy reduction with low-entropy anchor preservation, optimizing a question-conditioned visual spotlight via test-time GRPO without modifying backbone weights.
Background & Motivation¶
Vision-language models (VLMs) have made tremendous strides in multimodal understanding, yet they remain notably fragile on evidence-intensive tasks such as chart understanding, complex document parsing, and dense visual question answering. In these domains, the decisive evidence determining the final answer is often minute and highly localized—such as tiny numeric tick labels along an axis, specific table cells, or localized symbol legends. Models can frequently formulate a flawless high-level reasoning plan (e.g., explicitly stating that one should locate the y-axis tick and compare the relative heights of two bars), yet fail completely at the ground-level evidence readout step. This disconnect between sound high-level planning and flawed perceptual execution reveals what is known as the "evidence utilization gap." Closing this gap for already-deployed, frozen foundation VLMs without expensive end-to-end retraining or manual per-sample annotations is an urgent challenge.
Existing inference-time visual intervention methods, such as FGVP, Set-of-Mark (SoM), and ViCrop, attempt to alleviate this by overlaying visual cues, cropping candidates, or warping resolution towards salient image regions. Nevertheless, these approaches are largely open-loop: their interventions are fixed heuristics that lack any self-verifying feedback mechanism to ascertain whether the model genuinely utilizes the highlighted evidence, or whether the transformation accidentally degrades other critical context. When the heuristic intervention misidentifies the target or distorts decisive details, the failure remains undetected and irreversible. This underscores the necessity of establishing a closed-loop internal feedback signal accessible from frozen models at inference time.
The predictive uncertainty of the output distribution, specifically Shannon entropy over generated tokens, naturally tracks evidence usability: when visual evidence is clear and accessible, predictive entropy over the answer span remains low, whereas ambiguous or obscured evidence causes entropy to spike. However, naive entropy minimization suffers from an inherent ambiguity: low entropy can stem either from genuinely grounded confidence or from shortcut collapse, where the model erases difficult visual evidence and retreats into dominant language priors or salient distractors. Core idea: introduce low-entropy anchors to construct an entropy-shaping objective that drives down answer-span uncertainty while strictly penalizing disruptions to baseline deterministic tokens, steering lightweight test-time GRPO on question-conditioned visual spotlights for plug-and-play evidence amplification.
Method¶
Overall Architecture¶
SPOT-E operates as a modular, plug-and-play test-time adaptation pipeline. Given an arbitrary input image \(x\) and text instruction \(q\), the downstream vision-language model \(F_\phi\) is kept entirely frozen. The framework introduces a lightweight visual spotlight module (parameterized by LoRA adapters integrated into a CLIP vision encoder) that generates a continuous, question-conditioned soft mask \(m \in [0, 1]^{H \times W}\). A spotlight operator then blends the original image with a dimmed background to produce an intervened image \(\tilde{x} = \mathcal{S}(x; m)\).
The intervened image \(\tilde{x}\) is passed into the frozen VLM to extract the answer-span entropy and measure potential entropy inflation on pre-identified low-entropy anchors, synthesizing a composite entropy-shaping reward. At inference time, SPOT-E initiates a brief online optimization episode for each test sample: it perturbs spotlight parameters to sample a group of candidate masks, evaluates group-relative advantages via Group Relative Policy Optimization (GRPO), and updates solely the spotlight LoRA weights. Upon convergence, the best candidate mask is determined via Best-of-N selection for final response generation, followed by an immediate parameter reset to avoid cross-instance parameter drift.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Image & Instruction<br/>$x$ and $q$"] --> B["Multi-View Spotlight Generation<br/>CLIP Similarity + Local Crop Max Fusion"]
B --> C["Anchor-Preserving Entropy Shaping<br/>Dynamic Clarity Reward + Anchor Penalty"]
C --> D["Test-Time GRPO Optimization<br/>Group Relative Advantage LoRA Update + Reset"]
D --> E["Frozen VLM Final Inference<br/>Best-of-N Shortcut-Free Output"]
Key Designs¶
1. Multi-View Visual Spotlight Generation: Preserving minute cues via local-global cross-attention and max fusion
Static visual prompting often obscures surrounding context or fails to align with fine-grained query intents. To accurately spotlight minute visual cues conditioned on user queries, the spotlight module first distills a concise visual phrase \(\bar{q}\) from instruction \(q\) (retaining key entities and attributes) and encodes it via the frozen CLIP text encoder. On the visual side, the module ingests both the global view \(x^{(0)} = x\) and \(N_c\) overlapping local crops \(\{x^{(i)}\}_{i=1}^{N_c}\). A LoRA-adapted CLIP vision encoder extracts patch tokens across all views and computes patch-text cosine relevance against the query embedding:
To seamlessly reintegrate local crop relevance into the global scene coordinates, spatial warp operators \(\mathcal{W}_i(\cdot)\) map each crop grid back to full-image resolution. The system merges evidence across views using element-wise max fusion, \(u = \max\big(u^{(0)}, \max_i \mathcal{W}_i(u^{(i)})\big)\), preventing subtle localized signals from being averaged out. Bilinear upsampling followed by a temperature-scaled Sigmoid activation converts \(u\) into a smooth soft pixel mask \(m = \sigma(\frac{1}{\tau} u^{\uparrow}) \in [0, 1]^{H \times W}\). The final spotlighted image is composed via:
where \(\mathcal{B}(\cdot)\) applies a uniform background-dimming transformation. This spotlights candidate regions while suppressing surrounding visual clutter without introducing artificial edge artifacts.
2. Low-Entropy Anchors and Entropy-Shaping Reward: Disambiguating grounded confidence from shortcut collapse
Directly minimizing answer entropy is prone to pathological failure: if an intervention blacks out hard evidence, the model may confidently output majority-class answers driven purely by language priors. To mathematically separate evidence-grounded certainty from shortcut collapse, SPOT-E extracts low-entropy anchors \(\mathcal{I}_{\text{low}}(x, q)\)—the top-\(K\) token positions exhibiting the lowest predictive entropy during baseline greedy decoding on unmodified input \(x\). A faithful visual intervention should boost evidence clarity while strictly preserving the integrity of tokens the base model was already certain about.
The composite reward is formulated as \(R(\tilde{x}) = R_{\text{clarity}}(\tilde{x}) + R_{\text{preserve}}(\tilde{x})\). The dynamic clarity reward scales the entropy reduction along the structured final answer span \(\mathcal{T}_{\text{ans}}\) by an adaptive baseline uncertainty factor \(\gamma(x, q)\):
When baseline confidence is already high, \(\gamma(x, q)\) damps down exploration; when the model is confused, it amplifies the optimization gradient. Complementarily, the anchor-preservation reward penalizes any entropy increase across baseline anchor positions:
3. Test-Time Optimization with GRPO: Critic-free lightweight policy updates with instance reset
To ensure compatibility with massive foundation models and proprietary APIs providing token log-probabilities, the VLM backbone remains completely untuned. All parameter adaptations are confined to the compact LoRA parameters \(\theta\) inside the visual spotlight module. To achieve rapid per-sample convergence without training an auxiliary Critic value network, the framework leverages Group Relative Policy Optimization (GRPO).
During each test-time episode, Gaussian noise is injected into the spotlight policy to sample \(N\) candidate spotlight masks, yielding parallel candidates \(\{\tilde{x}^{(n)}\}_{n=1}^N\) evaluated by the frozen VLM. The group reward mean \(\mu_R\) and standard deviation \(\sigma_R\) normalize candidate payoffs into relative advantage estimates:
Standard clipped surrogate policy gradients with a KL divergence penalty against the initial spotlight state update LoRA parameters across very few steps (defaulting to 8 steps). Once optimization completes, a Best-of-N selection picks the candidate with the highest total reward for final token generation, and the spotlight parameters are immediately reinitialized (\(\theta \leftarrow \theta_0\)) to maintain zero-shot purity across test sequences.
Key Experimental Results¶
Main Results¶
SPOT-E was evaluated across nine diverse multimodal backbones spanning open-source models (Qwen-VL, InternVL, LLaVA families) and closed-source APIs (GPT-4o, GPT-4o-mini, Gemini-2.5-Flash). The table below summarizes benchmark performance before and after applying SPOT-E:
| Base Model / Method | TextVQA | DocVQA | ChartQA | MathVista | MMMU | POPE |
|---|---|---|---|---|---|---|
| GPT-4o (Frozen) | 77.4 | 91.1 | 86.7 | 63.5 | 69.2 | 86.9 |
| + SPOT-E (Ours) | 79.9 (+2.5) | 92.3 (+1.2) | 88.2 (+1.5) | 65.5 (+2.0) | 70.4 (+1.2) | 87.9 (+1.0) |
| GPT-4o-mini (Frozen) | 70.0 | 86.0 | 80.0 | 55.0 | 60.0 | 84.0 |
| + SPOT-E (Ours) | 73.5 (+3.5) | 88.0 (+2.0) | 82.5 (+2.5) | 58.0 (+3.0) | 62.0 (+2.0) | 85.2 (+1.2) |
| Qwen2.5-VL-7B (Frozen) | 84.9 | 85.7 | 87.3 | 67.8 | 55.0 | 86.4 |
| + SPOT-E (Ours) | 86.9 (+2.0) | 86.5 (+0.8) | 88.5 (+1.2) | 70.8 (+3.0) | 58.5 (+3.5) | 87.4 (+1.0) |
| LLaVA-NeXT-7B (Frozen) | 78.5 | 80.0 | 79.0 | 47.0 | 38.0 | 85.0 |
| + SPOT-E (Ours) | 84.7 (+6.2) | 82.0 (+2.0) | 81.5 (+2.5) | 50.5 (+3.5) | 41.0 (+3.0) | 86.5 (+1.5) |
| InternVL2.5-8B (Frozen) | 81.0 | 82.0 | 83.0 | 66.0 | 56.0 | 89.0 |
| + SPOT-E (Ours) | 84.5 (+3.5) | 83.5 (+1.5) | 85.0 (+2.0) | 69.5 (+3.5) | 59.5 (+3.5) | 90.0 (+1.0) |
Benchmarked against prior inference-time visual prompt engineering methods on frozen Qwen2.5-VL-7B:
| Method | TextVQA | GQA | MMMU | POPE | DocVQA |
|---|---|---|---|---|---|
| FGVP-Mask | 77.3 | 55.8 | 46.0 | 84.4 | 56.6 |
| FGVP-RBM | 72.3 | 55.8 | 46.5 | 81.3 | 38.6 |
| Set-of-Mark (SoM) | 61.5 | 47.8 | 45.1 | 75.8 | 57.4 |
| API | 81.6 | 61.1 | 47.4 | 85.8 | 68.4 |
| ViCrop | 83.8 | 60.6 | 47.1 | 86.7 | 82.5 |
| AttWarp | 84.7 | 64.0 | 50.4 | 87.4 | 84.1 |
| SPOT-E (Ours) | 86.9 | 65.0 | 58.5 | 87.4 | 86.5 |
Ablation Study¶
Ablations on Qwen2.5-VL-7B dissect the individual contributions of reward formulation and visual spotlight architectural choices:
| Study Aspect | Configuration / Variant | TextVQA | MathVista | POPE | Note |
|---|---|---|---|---|---|
| Reward Formulation | Clarity only (\(R_{\text{clarity}}\) only) | 85.8 | 69.4 | 86.9 | Suffers from shortcut collapse; MathVista drops 1.4% |
| Preservation only (\(R_{\text{preserve}}\) only) | 84.9 | 68.6 | 87.1 | Lacks drive to reduce answer uncertainty | |
| w/o Dynamic scaling | 86.4 | 70.1 | 87.2 | Fixed scaling ignores baseline confidence variance | |
| Full Reward (Ours) | 86.9 | 70.8 | 87.4 | Optimal balance of clarity and anchor preservation | |
| Spotlight Architecture | Global view only | 84.1 | 66.9 | 86.3 | Misses tiny text cues; TextVQA plummets by 2.8% |
| Mean fusion (MeanFuse) | 85.7 | 68.5 | 86.9 | Averaging dilutes peak localized activations | |
| No background degradation (NoBgDeg) | 85.2 | 67.8 | 86.7 | Background distractors remain unsuppressed | |
| Default Spotlight (Ours) | 86.9 | 70.8 | 87.4 | Multi-view max fusion + soft dimming yields best gain |
Key Findings¶
- Disproportionate gains on evidence-heavy benchmarks: SPOT-E yields its largest improvements (+2.0% to +6.2%) on text- and chart-intensive datasets (TextVQA, ChartQA, MathVista), proving that the bottleneck in base VLMs is fine-grained evidence readout rather than high-level reasoning capacity.
- Anchor preservation prevents catastrophic shortcut collapse: Omitting the anchor penalty (\(R_{\text{clarity}}\) only) leads to substantial performance degradation on complex reasoning benchmarks like MathVista, confirming that naive entropy minimization encourages models to erase hard evidence and guess based on visual priors.
- Efficient convergence envelope: Sweep studies over test-time update steps demonstrate that gains saturate near 8 adaptation steps, providing an ideal trade-off between inference compute and perceptual accuracy.
Highlights & Insights¶
- Turning internal predictive entropy into a closed-loop control signal: Rather than relying on external verifiers or hand-crafted scoring rules, SPOT-E closes the loop between evidence presentation and model readout using the model's own answer-span entropy.
- Resolving the ambiguity of unsupervised entropy minimization: By observing that destructive shortcuts inflate entropy on baseline high-confidence tokens, the paper establishes the low-entropy anchor preservation principle to separate true grounding from prior-driven collapse.
- Zero modification to downstream foundation backbones: The core VLM remains completely frozen, making the approach universally applicable across diverse open-source architectures as well as proprietary commercial APIs.
Limitations & Future Work¶
- Resolution ceiling on extreme micro-evidence: When decisive evidence is smaller than the spatial patch resolution of the CLIP visual encoder or corrupted by severe physical blur, spotlighting cannot recover lost visual information.
- Multi-pass inference computational overhead: Although the adaptation budget requires only 8 LoRA updates, sampling \(N\) parallel candidate masks across multiple steps incurs non-negligible inference latency and API cost.
- Promising future directions: Exploring knowledge distillation from test-time spotlight trajectories into a feed-forward visual spotlight generator, or integrating dynamic sub-patch super-resolution mechanisms.
Related Work & Insights¶
- vs Open-loop visual prompting (FGVP / Set-of-Mark / ViCrop): Heuristic region selection cannot assess whether highlighted evidence is actually ingested by the model. SPOT-E achieves significant performance leads by closing the feedback loop with model-internal entropy signals.
- vs Conventional test-time adaptation (TENT / MEMO): Traditional TTA updates the base classifier parameters via entropy minimization, risking severe representation collapse under domain shift. SPOT-E preserves all base weights and guards against shortcut traps via low-entropy anchors.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering the use of low-entropy anchors to shape answer-span entropy for unsupervised test-time visual adaptation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 9 foundation models, extensive visual prompt baselines, out-of-distribution corruption tests, and detailed ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear problem formulation, tight logical transitions, self-contained mathematical exposition, and high-quality visualizations.
- Value: ⭐⭐⭐⭐⭐ Provides an elegant, training-free paradigm for enhancing fine-grained perception and evidence grounding in deployed VLMs.