On Test-Time Scaling for Vision-Language Models¶

Conference: ECCV 2026
arXiv: 2606.28864
Code: None
Area: Multimodal VLM / LLM Inference / Test-Time Scaling
Keywords: Test-Time Scaling, Vision-Language Models, Chain-of-Thought, Attention Analysis, Systematic Evaluation

TL;DR¶

This is the first empirical work to systematically investigate whether test-time scaling methods developed for LLMs can be directly transferred to Large Vision-Language Models (LVLMs). Spanning 13 models, 9 scaling strategies, and 6 benchmarks, it overturns the conventional wisdom that "small models do not benefit." Instead, small but strong instruction-following models benefit the most (up to ~+30%, matching or even exceeding larger models). It also reveals two key phenomena: "excessive computation harms perception tasks" and "visual information is encoded into text tokens early in the reasoning chain."

Background & Motivation¶

Background & Limitations of Prior Work: Test-time scaling refers to allocating more compute during the inference phase (without modifying weights) to achieve stronger performance. It has emerged as a promising alternative to parameter scaling, often proven more cost-effective in LLMs than simply scaling up model size. These methods generally fall into two categories: those requiring additional training (SFT + RL to train so-called "thinking models"), and training-free approaches that "activate" inherent reasoning capabilities in instruction-following models via prompting, sampling, aggregation, or iterative refinement. This paper focuses on the training-free category. However, whether these methods work on LVLMs has remained largely unanswered. Worse, prior literature has established an almost consensus-like negative conclusion: multiple LLM studies report that small models (<10B) do not benefit from test-time scaling, and prompting them to reason can even degrade performance. Recent works on VLMs (e.g., evaluations on SmolVLM2 and CoT studies on the LLaVA series) similarly report that "Chain-of-Thought prompting fails on LVLMs." Consequently, researchers have assumed that LLM test-time scaling methods cannot transfer to VLMs, turning instead to VLM-specific designs (such as Compositional CoT, which extracts scene graphs from images before prompting).

Key Challenge & Goal: This paper directly challenges the assumption that "LLM methods cannot transfer to VLMs." The authors pose a simple yet fundamental question: can conventional test-time scaling methods developed for LLMs be directly applied to LVLMs? Rather than proposing yet another new method, their goal is to conduct a comprehensive control experiment covering multiple model scales, methods, and task types (perception vs. reasoning) to clarify "if, when, and to what extent" these methods are effective. They also delve into the inner workings of models (attention dynamics and informational sufficiency of reasoning chains) to explain the underlying mechanisms. Consequently, this paper represents a gap-filling and systematic study rather than a traditional method paper proposing a single pipeline.

The Core Idea of this paper is to conduct an exhaustive evaluation of 9 mainstream training-free test-time scaling strategies on 13 open-source LVLMs. By employing a unified "think-prompt + reliable answer extraction" protocol to eliminate confounding factors, the authors derive three actionable findings: (1) small-yet-strong models benefit the most (contrary to LLM literature), (2) excessive compute causes LVLMs to "lose focus" and degrade on perception tasks, and (3) visual information is "pre-encoded" into text tokens early in the reasoning chain, after which the model rarely refers back to the image. This paper attributes the contrary findings in prior work to the poor instruction-following capability of the models used (e.g., SmolVLM2, LLaVA-OneVision), which tend to answer directly while ignoring thinking prompts, behaving more like specialized VQA systems than conversational instruction models.

Method¶

Overall Architecture¶

This work does not propose a single pipeline, but rather a unified evaluation protocol + three-layered analysis: First, 9 test-time scaling methods (mostly from LLM literature, with a few VLM-specific ones) are applied to 13 models across 6 benchmarks using a unified prompt template to obtain the main results. Second, the mechanism is investigated from three perspectives: token budget (truncating to 300 tokens to observe performance changes), attention dynamics (evolving attention toward image tokens during reasoning chain generation, validated via KV-cache dropping causal intervention), and reasoning chain quality (evaluating informational sufficiency through an LLM-as-a-judge to test "whether reading the reasoning chain alone is sufficient to answer correctly," alongside sentence-level temporal analysis).

Two key engineering points in the unified protocol ensure the reliability of the conclusions: Think-prompt mechanism—all methods are implemented by inserting a think-prompt before the post-prompt, resulting in the final input format {question};{think-prompt};{post-prompt}. The model first outputs a reasoning chain $r$ followed by the final answer (e.g., the think-prompt for CoT is "think step-by-step"). Reliable answer extraction—if a standardized answer cannot be extracted on the first attempt, a second request is made using {question};{r};{answer-only};{post-prompt} to force the model to provide only the answer based on the generated $r$. This step is crucial: it enables successful answer extraction even when truncated at 300 tokens, and eliminates the confounding factor where performance drops simply because the answer format is not parser-friendly.

Key Designs¶

1. Unified Application of Nine Test-Time Scaling Methods: Migrating the LLM Arsenal to LVLMs

A major contribution of this work is not inventing new methods, but benchmarking nine diverse strategies under a unified protocol across three compute paradigms: "single-forward with extra tokens," "multi-sample voting," and "multi-turn refinement." Specifically: ① CoT (single-forward, consuming additional thinking tokens); ② S-CoT (Structured CoT, guiding question decomposition $\to$ information organization $\to$ logical reasoning $\to$ summary); ③ Plan-and-Solve (planning before execution); ④ Self-Consistency (sampling $b$ CoT chains and taking a majority vote, requiring $b$ forwards); ⑤ Self-Aggregation (concatenating all $b$ sample chains and prompt the model to aggregate them, requiring $b{+}1$ forwards); ⑥ Self-Refinement (generating a CoT chain and iteratively refining it for $k$ rounds, requiring $k{+}1$ forwards); ⑦ Describe-Answer (describing the image in detail first, then answering based on the description, requiring 2 forwards); ⑧ Compositional CoT / CCoT (generating a scene graph before answering, VLM-specific, requiring 2 forwards); ⑨ Prompt Repetition (repeating the question as the think-prompt to simulate a bidirectional-like attention effect, single-forward without generating reasoning tokens). Evaluating them side-by-side yields key insights, such as "vanilla CoT strikes the best efficiency-performance trade-off" and "conventional LLM methods significantly outperform VLM-specific CCoT."

2. Token Budget Analysis: Explaining Why Perception Tasks Are Harmed by Additional Compute

To understand why additional compute harms pure perception benchmarks, the authors reran all experiments with the maximum output length truncated from 1024 to 300 tokens, reporting $\Delta = \text{score}_{300} - \text{score}_{1024}$. Thanks to the reliable answer extraction protocol, final answers could still be retrieved even when the reasoning chain was truncated at 300 tokens, ensuring a clean comparison. The counter-intuitive finding is that, unlike in LLM literature where "models often know the answer before writing the reasoning chain," restricting the token budget leads to a drop in accuracy. This proves that reasoning chains actively drive the final answer, and spending tokens on intermediate steps is genuinely useful. However, this holds true only for reasoning benchmarks. On perception/hallucination tasks like RealWorldQA and HallusionBench, increasing the token budget is counterproductive or offers no benefits. This supports the "loss of focus" hypothesis: when given excessive compute to overthink, LVLMs tend to fabricate plausible narratives, establish incorrect assumptions, and accumulate chained errors, ultimately leading to hallucinations and incorrect answers. The authors suggest training a simple binary classifier to determine whether a task truly requires reasoning, thereby saving substantial compute.

3. Attention Dynamics & KV-Cache Causal Intervention: Revealing "Visual Pre-Encoding"

This is the most rigorous mechanistic investigation of the study. The authors decompose the attention of each generated token $g_y$ into three components: attention to image tokens $a^{img}=\sum a_{l}^{h}[1{:}N_m]$, attention to the text prompt $a^{pmt}=\sum a[N_m{+}1{:}N_m{+}T]$, and attention to previously generated tokens $a^{gen}=\sum a[N_m{+}T{+}1{:}N_m{+}T{+}(y{-}1)]$ ($N_m$ is the number of compressed visual tokens fed into the LLM, with $N_m = N/4$ for the Qwen series). Since these are normalized, $a^{pmt}=1-(\sum a^{img}+\sum a^{gen})$ is directly derived. Plotting the curves for $a^{img}$ and $a^{gen}$ across generation steps (averaged across layers, heads, and samples, supplemented by the upper bound of token-wise $\max[a^{img}]$) reveals that image attention peaks briefly at the beginning and rapidly decays, while attention to generated tokens rises continuously and eventually dominates. This suggests that visual grounding is "pre-encoded": there is a narrow window early in the reasoning chain where image information is absorbed into the hidden representations of text tokens. Beyond this window, the model mainly relies on these hidden states and prior textual context, rarely revisiting visual tokens. To establish causality beyond correlation, the authors perform KV-cache dropping interventions: selecting 300 samples per dataset, they completely purge the Key/Value cache of image tokens across all layers at generation steps $\{20, 50, 100, 200, 300\}$ and evaluate accuracy. The results show that dropping visual tokens early leads to significant performance drops (due to disrupting the critical "pre-encoding" phase), while dropping them after ~200 steps has almost no impact on accuracy. This validates the pre-encoding hypothesis and explains the "loss of focus" described in Design 2: the longer the chain, the more the model relies on its own generated text tokens to infer, propagating and amplifying errors.

4. LLM-as-a-Judge Dual Metrics for Reasoning Chain Informational Sufficiency

To evaluate whether the generated multimodal reasoning chains are high-quality and sufficiently informative, the authors employ an external LLM as a judge, validating its reliability through human evaluation. They introduce two complementary metrics: Rationale Sufficiency—the judge only receives the question $q$ and the reasoning chain $r$ (without access to the image, external knowledge, or ground truth) and predicts the answer $\hat{x}_r$ solely based on the text of $r$. If reading the chain alone suffices to answer correctly, $r$ is considered informative; this rationale accuracy is then compared against the LVLM's native accuracy. Rationale Dynamics—$r$ is decomposed into an ordered sequence of sentences $\{r_1,\dots,r_S\}$, and each sentence is judged to see if it supports the correct answer, the incorrect answer, or is undecided. These are aggregated to yield statistics such as the earliest decision point, stance-flipping/contradictions between reasoning steps, intermediate uncertainty patterns, and the point of final conclusion. This benchmark suite makes it measurable whether reasoning chains carry active decision-making signals and how information accumulates throughout the chain, moving beyond mere final accuracy metrics.

Case Study: How a 4B Model Matches a 32B Model¶

Take Qwen3-VL-4B as an example: its baseline on LogicVista is 40.85, which climbs to 57.59 with simple CoT priming. This outperforms the 8-times larger Qwen3-VL-32B baseline (44.42) by approximately +13%. Switching to Self-Consistency widens this margin further to 62.95. A similar "4B + CoT/S-CoT exceeding 32B baseline" trend is observed on HallusionBench and WeMath. The Pareto frontier indicates that under a ~9-second compute budget, the 4B model outright outperforms the 32B model on LogicVista and WeMath, achieves parity on HallusionBench, and comes close on MMStar. This illustrates the practical value of the first takeaway—deploying cheaper, faster, and more memory-efficient small VLMs with test-time scaling can match or exceed the accuracy of larger base models.

Key Experimental Results¶

The evaluation spans 13 models (Qwen2.5-VL 7B/32B/72B, Qwen3-VL 2B/4B/8B/32B, InternVL-3.5 2B/4B/8B/38B, Molmo2 4B/8B) and 6 benchmarks (MMStar, RealWorldQA, HallusionBench, WeMath, LogicVista, A-OKVQA). The maximum token length is set to 1024. The total experimental cost is approximately $6,000, and runtimes are measured on a single NVIDIA H200 (140GB) GPU (two for the 72B model).

Main Results: Selected Test-Time Scaling Results for the Qwen Series¶

The following table is selected from Table 1 and showcases that small models benefit the most, while methods easily degrade on perception tasks (higher is better, runtime is in seconds):

Model	Method	MMStar	RealWorldQA	HallusionBench	WeMath	LogicVista	Runtime
Qwen3-VL-2B	Baseline	51.69	65.10	63.72	35.46	32.59	0.1
Qwen3-VL-2B	Self-Consistency	63.83	64.31↓	71.08	64.25	46.88	40.2
Qwen3-VL-4B	Baseline	61.71	71.76	70.66	58.22	40.85	0.1
Qwen3-VL-4B	CoT	68.59	69.80↓	75.50	71.55	57.59	5.9
Qwen3-VL-4B	Self-Consistency	71.07	72.55	74.97	74.25	62.95	44.8
Qwen3-VL-32B	Baseline	72.66	77.25	74.76	66.03	44.42	0.2
Qwen2.5-VL-72B	Baseline	68.65	69.67	74.97	72.99	47.54	0.3

Key Findings: - Small models benefit the most, and can outperform larger models: Self-Consistency lifts Qwen3-VL-2B on WeMath from 35.46 to 64.25 (+29%). With simple CoT, Qwen3-VL-4B's scores on LogicVista (57.59), WeMath (71.55), and HallusionBench (75.50) completely surpass the baseline of the 8-times larger Qwen3-VL-32B (44.42 / 66.03 / 74.76). This directly contradicts findings in LLM literature stating that small models do not benefit. - Perception tasks are often harmed: On RealWorldQA (pure perception, space intensive), most methods degrade relative to the baseline (indicated by ↓ in the table), whereas they surge on reasoning-heavy benchmarks like WeMath/LogicVista. This shows that test-time scaling performance is highly dependent on whether the task genuinely requires reasoning. - Vanilla CoT is the most cost-effective: Generating a single CoT chain takes only ~6s of runtime but captures most of the gains of Self-Consistency (~45s, multiple samples), serving as the optimal efficiency-performance trade-off. Standard LLM methods consistently outperform VLM-specific CCoT. - VLM-specific methods are not necessarily better: CCoT and Describe-Answer underperform relative to vanilla CoT on many models and tasks, and occasionally drop below baseline.

Analysis: Token Budget Truncated to 300 ($\Delta = \text{score}_{300}-\text{score}_{1024}$, Selected)¶

The following table is selected from Table 2. Negative values indicate performance drops after truncation (i.e., long chains are useful), while positive values indicate truncation actually works better (i.e., long chains are harmful):

Model	Method	MMStar	RealWorldQA	HallusionBench	WeMath	LogicVista
Qwen3-VL-4B	CoT	-2.07	+0.79	-0.42	-11.32	-7.37
Qwen3-VL-4B	S-CoT	-2.80	+3.79	-0.11	-14.89	-13.61
Qwen3-VL-8B	CoT	-0.55	-0.79	-2.10	-6.04	-0.44
Qwen2.5-VL-72B	Self-Refinement	-0.15	+10.07	+1.68	+2.76	-0.23
Qwen2.5-VL-72B	Self-Aggregation	-1.52	+6.14	+1.58	-1.10	-0.67

Key Findings: - Reasoning tasks: Long chains drive answers. On WeMath and LogicVista, truncating the output to 300 tokens causes a sharp performance drop (e.g., 4B+CoT dropping by 11.32 on WeMath and 7.37 on LogicVista). This confirms that intermediate reasoning steps play an active role, rather than the model pre-computing the answer prior to outputting thoughts. - Perception tasks: Shorter answers are better. Truncating outputs on RealWorldQA often yields identical or improved scores (e.g., 4B+S-CoT gaining +3.79, and 72B+Self-Refinement gaining +10.07). This supports the hypothesis that perception tasks favor concise outputs, while excessive tokens cause models to lose focus and hallucinate. - KV-Cache dropping intervention: Purging image token KV-caches at early stages (steps 20–50) leads to massive degradation, while doing so after ~200 steps has minimal impact. This supports the "visual pre-encoding" mechanism. - Reasoning chain sufficiency: On most benchmarks, the judge's accuracy when reading "only the reasoning chain" closely mirrors the LVLM's native accuracy, indicating that reasoning chains embed relevant decision-making signals. The gap decreases with larger or more recent models, indicating better interpretability. Sentence-level temporal analysis on HallusionBench reveals that uncertainty manifests early, support for the correct answer begins around the midpoint ($\sim 0.5$) of the chain, and the conclusion is fully formed by the 0.75 mark. This indicates that reasoning chains are "progressively structured" rather than reaching conclusions immediately.

Highlights & Insights¶

A systematic "disproof-style" investigation of greater value than proposing a new method: It refutes two widely cited assumptions—that "LLM test-time scaling does not transfer to VLMs" and "small models do not benefit." It precisely attributes the failures in prior work to poor instruction-following capability in models like SmolVLM2 and LLaVA-OneVision, which directly output answers and bypass thinking instructions like specialized VQA models. This serves as a reminder to the community: negative conclusions are often due to poor base model choice rather than limitations of the method itself.
"Visual pre-encoding" is a highly transferable mechanistic insight: The early-peaking/late-decaying attention combined with KV-cache dropping causal validation demonstrates that models rarely look back at image tokens in the later parts of the reasoning chain. This inspires a highly practical engineering trick: safely dropping the KV-cache of image tokens after ~200 steps to save VRAM and compute (the paper reports GFLOPs savings in the supplementary material), which is highly beneficial for deploying long-chain VLM inference.
"On-demand scaling" deployment philosophy: Since compute harms perception tasks but benefits reasoning tasks, one can train a lightweight binary classifier to predict "whether the question requires reasoning" first. Test-time scaling can then be selectively enabled, transforming "blindly scaling compute" into "conditional scaling" to save massive resources.
Reliable answer extraction is an underappreciated experimental best practice: Relying on a secondary follow-up request to extract final answers ensures a clean comparison for the 300-token truncation baseline and avoids confusing formatting errors with genuine reasoning failure. Many previous negative findings may have suffered from this exact confounding factor.

Limitations & Future Work¶

Limited to training-free methods: This study focuses strictly on training-free (prompting/sampling) test-time scaling, leaving aside thinking models derived via SFT+RL. The relative performance of these two paradigms under equivalent compute budgets remains unexplored.
Limited evaluation on closed-source models: The drop in execution on perception tasks was only verified on GPT-5.2 (⚠️ the model name is likely a placeholder/future name; refer to the original paper) across MMStar and RealWorldQA, indicating that while "weaker, the phenomenon persists in advanced closed-source models," but the coverage remains narrow.
Fixed max tokens of 1024 and narrow package of benchmarks: The token budget is only contrasted at two coarse granularities (1024 vs. 300). While the 6 benchmarks are diverse, they are academically focused, leaving the gap to real-world product distributions unquantified.
"Loss of focus" remains a hypothesis: Although supported by attention analyses and interventions, the full causal chain ("overthinking $\to$ false assumptions $\to$ error propagation $\to$ hallucination") lacks fine-grained step-by-step ablation. Finer interventions (e.g., truncating specific reasoning clauses) represent crucial future work.
Future directions: Concretely train and end-to-end evaluate the proposed "reasoning necessity" binary classifier; make KV-cache dropping adaptive (dropping tokens dynamically based on attention decay thresholds) rather than utilizing fixed step intervals.

vs. CCoT / DCoT / DDCoT (VLM-specific test-time scaling): These methods draw scene graphs, extract bounding-box visual cues, or split identification and reasoning into sequential prompts, respectively. This work demonstrates that such specialized approaches underperform vanilla CoT on most models and tasks, suggesting that the necessity of "VLM tailoring" is somewhat overstated—getting general LLM methods right is often sufficient.
vs. kaya2026efficient (reporting failure on SmolVLM2): This work evaluated test-time scaling on small VLMs but arrived at a contrary (negative) conclusion. This study attributes their failure to poor instruction-following of the underlying model, emphasizing that "highly capable instruction models benefit the most," directly refuting and explaining the negative results reported in prior work.
vs. LLM Literature (e.g., Lanham et al., 2023 reporting that small LLMs do not benefit and models know the answer prior to writing the reasoning chain): This paper observes the opposite on LVLMs—small models benefit the most, and restricting tokens degrades performance (indicating that the reasoning chain genuinely drives the answer). This suggests that the visual modality may fundamentally change the functional mechanics of reasoning chains.
vs. Early VLM Reasoning (e.g., NLX-GPT / UniNLX): Early works co-generated answers with single reasoning phrases. In contrast, this study systematically analyzes the effects and intrinsic mechanisms of "scaling computation during reasoning" on modern large models, shifting the focus from "generating explanations" to "trading compute for accuracy and defining its boundaries."

Rating¶

Novelty: ⭐⭐⭐⭐ While not introducing a new method, the combination of "the first systematic evaluation + refuting two popular assumptions + providing insights into visual pre-encoding mechanisms" is highly novel and substantial.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extremely thorough, spanning 13 models $\times$ 9 methods $\times$ 6 benchmarks, complemented by token budget constraints, attention intervention, a three-layered LLM-as-a-judge analysis, and human verification.
Writing Quality: ⭐⭐⭐⭐ The three key takeaways are clear, and the mechanistic analysis is logical and progressive. The tables are information-dense, though some symbols (e.g., attention indices) require close reading.
Value: ⭐⭐⭐⭐⭐ Provides highly actionable deployment guidelines, such as "small models combined with simple CoT can substitute large models," "avoid adding compute on perception tasks," and "image tokens can be safely dropped after 200 steps to save compute," which are extremely useful to both academia and industry.