Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs

Conference: ECCV 2026
Paper: ECCV Official Page
Project: Project Page
Area: LLM Efficiency / Multimodal VLM / LLM Reasoning
Keywords: Multimodal LLMs, Adaptive Inference, Token Pruning, Dynamic Compute Allocation, Pareto Optimal

TL;DR

SmartVL introduces an end-to-end framework that jointly adapts visual token sequence length alongside LLM compute depth and width, utilizing shared budget encodings and a differentiable FLOPs latency estimator to dynamically co-schedule compute across stages, significantly expanding the accuracy-efficiency Pareto frontier across multimodal benchmarks and diverse compute budgets.

Background & Motivation

Multimodal Large Language Models (MLLMs) demonstrate remarkable comprehension in visual question answering, image captioning, and visual grounding. However, their exorbitant and static inference costs severely hamper real-world deployment. Conventional models allocate an identical compute expenditure regardless of whether an input is an isolated, salient object or a complex, cluttered visual scene. Furthermore, hardware serving environments experience continuous load fluctuations, while interactive edge applications and offline batch processing systems operate under drastically different latency budgets. Static, monolithic inference pipelines cannot strike an optimal trade-off across variable, resource-constrained operational regimes.

The prefill computation of MLLMs is governed by three tightly coupled dimensions: sequence length dictated by the number of visual tokens, model depth determined by Transformer layer depth, and model width controlled by active attention heads and FFN intermediate channels. Existing efficient inference methodologies typically optimize these dimensions in isolation—either executing static, heuristic visual token pruning (e.g., FastV, LLaVA-PruMerge) or independently skipping downstream Transformer layers and attention heads (e.g., AdaLLaVA). Such decoupled designs cause severe cross-stage mismatches: isolated token pruning lacks semantic guidance from downstream language decoders, risking irrevocable loss of critical visual evidence, while isolated layer/head dropping assumes unpruned, full-length visual sequences, failing to adapt architectural capacity once upstream tokens have already been pruned.

A fundamental cross-stage coupling exists between these dimensions: visual token redundancy is intrinsically linked to the requisite linguistic reasoning complexity, meaning the architectural capacity (depth and width) of the language model must be conditioned on the actual information density of the pruned visual input. Breaking this dimensional barrier requires unifying visual compression and language model compute pruning into a closed-loop framework. Core idea: develop SmartVL, a unified multimodal adaptive inference framework that couples vision encoders and LLMs through shared budget encodings and dynamic token survival embeddings, trained end-to-end via a differentiable latency estimator and asymmetric violation penalties to dynamically arbitrate between retaining richer visual context versus executing deeper reasoning layers.

Method

Overall Architecture

SmartVL frames MLLM adaptive inference as jointly learning an optimal execution policy \(\pi(I, T, b)\) over sequence length \(S\), effective depth \(L_{\mathrm{eff}}\), and per-layer effective width \(\alpha_l\), conditioned on multimodal inputs \((I, T)\) and a normalized compute budget \(b \in [b_{\min}, 1]\). The system comprises two interconnected, co-optimized controllers: at the visual stage, a Visual Token Controller receives ViT inputs injected with budget tokens to generate binary keep masks per token; at the language stage, an LLM Compute Controller processes the surviving visual tokens alongside text prompts, aggregates multimodal context across \(P\) mandatory prefix layers, and conditions on token survival embeddings to independently predict layer-skipping masks (SmartVL-L) or attention head group and FFN channel masks (SmartVL-LH) across the remaining \(L-P\) layers.

During training, continuous relaxations of decisions from both controllers pass through a hardware-agnostic differentiable FLOPs estimator, guided by an asymmetric budget violation loss to maximize predictive accuracy under global compute ceilings. At inference, continuous relaxations are replaced by deterministic binary projections to ensure bounded latency.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input: Image I + Text T + Compute Budget b"] --> TokCtrl["Visual Token Controller<br/>Sinusoidal budget fusion + Gumbel-Sigmoid pruning"]
    TokCtrl --> Svis["Filtered visual tokens + Prompt sequence"]
    Svis --> Prefix["Execute P mandatory LLM prefix layers<br/>Aggregate global multimodal context"]
    Prefix --> CompCtrl["LLM Compute Controller<br/>Token survival rate κ fusion + Gated routing"]
    CompCtrl --> DynLayers["Dynamic execution of L-P layers<br/>Adaptive layer-skipping (L) or head/FFN scaling (LH)"]
    DynLayers --> Out["Adaptive multimodal response"]

Key Designs

1. Visual Token Controller: In-encoder budget conditioning and Gumbel-Sigmoid discrete routing
To overcome the inflexibility of static pruning heuristics that remain agnostic to downstream compute targets, this module directly embeds the scalar compute budget \(b\) into the visual encoder (ViT). A sinusoidal positional encoding and a lightweight MLP project the budget into an embedding vector \(e_b^{\mathrm{ViT}} = \mathrm{MLP}_{\mathrm{ViT}}(\mathrm{PE}(b)) + \mathbf{p}_b\), appended as a learnable token to the patch sequence. This budget token participates in self-attention across all ViT layers, contextualizing visual semantics with global resource constraints into final state \(\mathbf{h}_b\). A linear projection predicts keep logits \(z \in \mathbb{R}^N\) for all visual tokens, sampled via the Gumbel-Sigmoid Straight-Through Estimator (STE):

\[y_i = \sigma\bigl((z_i + \xi_i)/\tau\bigr), \quad m_i = \mathbf{1}[y_i > 0.5] - \mathrm{sg}(y_i) + y_i\]

During training, discrete binary masks \(m_{\mathrm{token}} \in \{0, 1\}^N\) zero out discarded tokens while maintaining static tensor shapes, whereas backpropagation flows gradients through continuous relaxations \(y_i\). At inference, stochastic noise is omitted, and tokens with \(\sigma(z_i) \leq 0.5\) are physically discarded. Because RoPE natively accommodates arbitrary sequence lengths, no positional re-indexing or manual retention tuning across datasets is required.

2. LLM Compute Controller: Cross-stage survival awareness and unconstrained independent gating
Unlike prior compute adaptation methods that assume static input lengths and enforce rigid Top-K sorting, the LLM controller maps the scalar budget into language space via a separate MLP to produce \(e_b^{\mathrm{LLM}}\), appended to the prompt. The initial \(P\) Transformer layers (prefix layers) run unconditionally at full capacity. This establishes foundational multimodal representations while enabling the budget token to gather global cross-modal representations into state \(\mathbf{h}_b^{(P)}\).

Because subsequent LLM compute depends on the volume of surviving visual tokens, the upstream normalized token survival rate \(\kappa = \mathrm{sg}\bigl(\sum_{i=1}^N \sigma(z_i)/N\bigr)\) is encoded via sinusoidal embeddings and an MLP, then fused into the prefix state: \(\tilde{\mathbf{h}}_b^{(P)} = \mathbf{h}_b^{(P)} + \mathrm{MLP}_{\kappa}(\mathrm{PE}(\kappa))\). The stop-gradient operator \(\mathrm{sg}(\cdot)\) severs gradient flow from LLM architectural choices back to the visual token controller, preventing gradient magnitude disparities from destabilizing visual pruning early in training. Rather than imposing rigid Top-K constraints, the controller employs independent Gumbel-Sigmoid activations across subsequent \(L-P\) layers, allowing layer configurations to emerge organically. Two granularities are supported: Layer-level (SmartVL-L) generates scalar binary masks \(g_l \in \{0, 1\}\) for residual skipping; Layer-Head level (SmartVL-LH) partitions attention heads and FFN channels into \(G\) equal groups, generating group masks \(M_l \in \{0, 1\}^G\). The effective capacity ratio \(\alpha_l = \frac{1}{G}\sum_{j=1}^G M_{l,j}\) unifies depth and width scaling within a single routing structure.

3. Joint Optimization & Inference Scheduling: Differentiable latency estimation with asymmetric penalties
To enable end-to-end discrete architecture search, SmartVL approximates prefill computational complexity via a differentiable FLOPs formulation. With \(\tilde{S}_{\mathrm{vis}} = \sum_{i=1}^N \sigma(z_i)\) serving as a continuous surrogate for visual token count, total sequence length is \(S = S_{\text{text}} + \tilde{S}_{\mathrm{vis}}\). Prefill compute is approximated analytically:

\[\hat{\mathcal{C}} = \sum_{l=1}^L \alpha_l \Bigl[\bigl(8D^2 + 6D D_{\mathrm{int}}\bigr) S + 4D S^2\Bigr] + 2DV S\]

Normalizing by full-model FLOPs yields the differentiable cost ratio \(r = \hat{\mathcal{C}} / \mathcal{C}_{\mathrm{full}}\). To balance budget compliance and capacity utilization, the framework adopts an asymmetric violation penalty:

\[\mathcal{L}_{\mathrm{over}} = \max(0, r - b)^2, \quad \mathcal{L}_{\mathrm{under}} = \max(0, b - \mu - r)\]

Budget overruns \(\mathcal{L}_{\mathrm{over}}\) incur quadratic penalties to strictly safeguard latency deadlines, whereas underruns \(\mathcal{L}_{\mathrm{under}}\) use a linear penalty bounded by tolerance margin \(\mu\) to encourage compute utilization. The total loss \(\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{pred}} + \lambda(t) \cdot (w_{\mathrm{over}}\mathcal{L}_{\mathrm{over}} + w_{\mathrm{under}}\mathcal{L}_{\mathrm{under}})\) incorporates a linear warmup schedule \(\lambda(t) = \min(1, t/T_{\mathrm{warmup}})\) to prevent initial constraint gradients from disrupting representation learning. At inference, if discretization creates marginal budget violations, the system sheds visual tokens with the lowest keep probabilities first, resorting to additional layer/head skipping only if token reduction proves insufficient.

Loss & Training

During training, the ViT visual backbone is kept frozen to stabilize visual representations, while the LLM backbone, multimodal projector, and dual-stage controllers are fine-tuned end-to-end. At each optimization step, compute budget \(b\) is sampled uniformly from \([b_{\min}, b_{\max}]\) and broadcast across the mini-batch, preserving batch processing efficiency while covering continuous compute budgets. Gating linear biases are initialized positively to favor full-capacity execution initially, followed by smooth sparsification under the penalty warmup schedule.

Key Experimental Results

Main Results

Experiments are conducted on LLaVA-1.5-7B across seven multimodal benchmarks (VQAv2, GQA, TextVQA, ScienceQA, POPE, VizWiz, MMBench), comparing SmartVL against compute-adaptive baseline AdaLLaVA, static token pruning baselines (FastV, LLaVA-PruMerge+), and the cascaded combination AdaLLaVA-PruMerge.

Dataset	Compute Budget (FLOPs)	SmartVL (Ours)	AdaLLaVA	AdaLLaVA-PruMerge	Full Model (LLaVA-1.5)	Gain vs AdaLLaVA
VQAv2	50% Budget (~4.2T)	74.4%	67.9%	74.5%	76.5%	+6.5%
VQAv2	70% Budget (~5.9T)	75.7%	74.4%	-	76.5%	+1.3%
TextVQA	50% Budget (~4.3T)	54.4%	29.8%	51.3%	58.1%	+24.6%
TextVQA	70% Budget (~6.0T)	57.0%	50.8%	54.3%	58.1%	+6.2%
VizWiz	50% Budget (~4.2T)	55.1%	34.3%	-	-	+20.8%
GQA	50% Budget (~4.1T)	59.8%	56.8%	60.1%	62.0%	+3.0%
MMBench	50% Budget (~4.3T)	62.0%	63.3%	63.1%	64.3%	-1.3%
POPE	50% Budget (~4.1T)	86.4%	86.2%	86.5%	87.1%	+0.2%
7-Benchmark Average	~50% Budget	-	-	-	-	+7.8%

(Note: Data collected from Fig. 1, Fig. 3, and Section 4.2 of the paper. Across all seven benchmarks at 50% compute budget, SmartVL outperforms AdaLLaVA by an average of 7.8%.)

Ablation Study

Ablations evaluate controller granularity (SmartVL-L vs. SmartVL-LH) and architectural scalability to LLaVA-1.5-13B (40 layers total, \(P=20\) prefix layers):

Config / Backbone	Evaluation Setup / Budget	SmartVL (Ours)	Baseline	Note
L vs. LH Granularity (7B)	TextVQA @ 50% FLOPs	54.1% (L) / 53.6% (LH)	51.3% (Ada-PruMerge)	L marginally outperforms LH; full attention width is vital for OCR-centric reasoning
L vs. LH Granularity (7B)	TextVQA @ 100% FLOPs	58.1% (L) / 57.5% (LH)	58.1% (Full Model)	L matches full model capacity; LH exhibits minor degradation
13B Scale Extension	VQAv2 @ ~50% (~7.25T FLOPs)	75.45%	72.27% (AdaLLaVA 13B @ 8.27T)	Saves >1T FLOPs while retaining a +3.18% accuracy advantage
13B Scale Extension	VQAv2 @ Mid-High (~11.56T FLOPs)	77.66%	76.45% (AdaLLaVA 13B @ 11.57T)	Outperforms baseline by +1.21% under matched compute
13B Scale Extension	VQAv2 @ 100% (~16T FLOPs)	78.82%	77.85% (AdaLLaVA 13B)	Exceeds baseline full-model capacity by +0.97%

(Note: Data from Fig. 5, Fig. 8, and Section 4.4 of the original paper.)

Key Findings

• Superiority of Joint Cross-Dimensional Adaptation: Under aggressive 50% compute constraints, purely layer-skipping baselines like AdaLLaVA must shed more than half of the language layers because visual token sequences remain unpruned. This leads to catastrophic drops on reasoning-heavy benchmarks (TextVQA plunging to 29.8%, VizWiz falling to 34.3%). By contrast, SmartVL prunes visual redundancies first, preserving sufficient LLM depth for cross-modal integration and maintaining 54.4% on TextVQA and 55.1% on VizWiz.
• Content- and Task-Dependent Token-Compute Dynamics: Grid search analyses reveal distinct computational preferences across tasks. Fine-grained hallucination detection (POPE) demands high visual coverage, preferring 80% token retention with only 50% layer execution under tight budgets. Conversely, symbolic and text-heavy reasoning tasks (VQAv2, TextVQA) favor aggressive token pruning in order to sustain up to 90% layer depth. SmartVL dynamically discovers these optimal trade-offs without task-specific tuning.
• Depth vs. Width Pruning Trade-off: SmartVL-L (pure layer skipping) consistently matches or slightly outperforms SmartVL-LH (head and channel thinning). Preserving complete attention head and FFN widths appears critical for multimodal representation fidelity, with redundancy residing predominantly along sequence length and architectural depth.

Highlights & Insights

• Unified Cross-Stage Coordination: Overcoming isolated optimization silos, SmartVL establishes that visual spatial density and LLM structural capacity are inextricably coupled, delivering a unified formulation for joint multimodal adaptation.
• Gradient-Severed Cross-Stage Feedback: Incorporating token survival rate \(\kappa\) into the LLM controller alongside a stop-gradient operator eliminates gradient scale disparities, preventing premature collapse in upstream visual pruning.
• Smooth Learning via Differentiable Latency Formulations: Deriving an analytical, differentiable FLOPs surrogate coupled with asymmetric quadratic/linear losses recasts discrete combinatorial architecture search into smooth gradient-based optimization, circumventing complex reinforcement learning or heuristic hyperparameter searches.

Limitations & Future Work

• Compute Metric Restricted to Prefill FLOPs: The current latency estimator targets compute-bound prefill FLOPs, omitting memory-bound autoregressive decoding latency and dynamic KV-cache eviction.
• Hardware Wall-Clock Speedups Depend on Specialized Kernels: While layer dropping and token discarding yield immediate latency reductions, fine-grained head and channel pruning (LH variant) face operational overheads under standard GPU GEMM kernels without dedicated sparse execution runtimes.
• Orthogonal Integration with Post-Training Quantization: Joint token-compute adaptation operates orthogonally to weight quantization (e.g., AWQ, QLoRA). Combining these techniques could further depress memory and compute footprints on resource-constrained edge hardware.

Related Work & Insights

• vs AdaLLaVA (ICCV 2025): AdaLLaVA adapts Transformer layers, heads, and MLPs via Top-K Gumbel-Softmax but treats visual tokens as fixed inputs. Under low compute budgets, aggressive layer pruning degrades accuracy (TextVQA drops to 29.8% at 50% FLOPs). SmartVL incorporates visual token control and cross-stage coupling, boosting TextVQA to 54.4% and achieving a 7.8% average gain across seven benchmarks.
• vs LLaVA-PruMerge+ (ICCV 2025) / FastV (ECCV 2024): These approaches apply static visual token pruning using fixed, dataset-wide retention ratios, leaving the LLM structure untouched and limiting operational coverage to narrow compute bands (e.g., 3T–4.8T FLOPs). SmartVL dynamically adjusts both tokens and compute per instance across continuous budget intervals spanning 20% to 100% FLOPs.
• vs AdaLLaVA-PruMerge (Cascaded Baseline): Combining isolated pruning methods requires manual hyperparameter searches and lacks end-to-end coordination. SmartVL unifies budget routing and loss optimization, pushing past cascaded combinations to establish a superior Pareto frontier.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneers unified, end-to-end adaptive inference bridging visual token pruning and structural LLM compute allocation.]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 7 diverse benchmarks, spanning 20%–100% compute budgets with task-level trade-off analyses and 13B scale validation.]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical formulation, and consistent architectural narratives matching empirical findings.]
• Value: ⭐⭐⭐⭐⭐ [Delivers actionable design principles and operational strategies for deploying multimodal LLMs across heterogeneous and resource-varying serving environments.]

Related Papers

• [CVPR 2026] When to Think and When to Look: Uncertainty-Guided Lookback
• [ECCV 2026] Beyond Where to Look: Trajectory-Guided Reinforcement Learning for Multimodal RLVR
• [ECCV 2026] AdaThinking-E: One-Token Entropy Regulation for Adaptive Thinking
• [ECCV 2026] AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution
• [ECCV 2026] Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens