MVPruner: Dynamic Token Pruning for Accelerating Multi-view Vision-Language Models in Autonomous Driving¶
Conference: ECCV2026
Authors: Nan Yang, Zhanwen Liu, Linfeng Zhang, Shangyu Xie, Yang Wang, Wenzhuo Zhou, Xiangmo Zhao
Paper: Official ECCV page
PDF: Full paper
Area: Model Compression
Keywords: Token pruning, multi-view vision-language models, diversity, instruction relevance, adaptive budgets
TL;DR¶
MVPruner prunes multi-view vision-language models in two stages—first preserving rich scene information, then focusing on the current instruction—and, at 90% pruning with DriveMM on DriveLM, retains an aggregate score of 58.2 versus 59.1 while reducing prefilling latency from 1968 ms to 395.6 ms; this does not imply a 4.97-fold end-to-end speedup.
Background & Motivation¶
Vision-language models for driving typically process several camera views simultaneously. Each view produces many visual tokens, and the concatenated sequence slows down language-model prefilling and subsequent decoding. Applying the same retention ratio to every view assumes that all directions provide similar information value. Yet a question about a front-left object may require evidence outside the front view that commonly receives a large budget. Prune2Drive already accounts for differences between views, but its fixed allocations obtained through offline search can remain biased toward directions common in the calibration data.
The issue is not just which view matters, but at which layer its importance can be assessed reliably. Using questions with explicit view references, the authors examine whether instruction-to-visual attention identifies the correct view: recognition is weak in shallow layers and improves substantially in intermediate layers. Attention entropy and visualizations also suggest that the model first gathers broad scene context before concentrating on task-relevant regions. Aggressive instruction-attention pruning at the start can therefore discard useful content before cross-modal alignment has developed. Conversely, optimizing only visual diversity in deeper layers continues to retain content unrelated to the question.
The paper connects these two stages of processing. Shallow-layer view budgets primarily reflect information richness, while token selection already incorporates the instruction; deeper-layer allocation and selection then become instruction-driven. Core Idea: pruning criteria should follow the model's changing information requirements, first preserving nonredundant and potentially useful scene evidence, then concentrating computation on regions whose task relevance has become clear.
Method¶
Overall Architecture¶
The inputs are multi-view images and a textual instruction. A vision encoder and projector first produce visual tokens for each view. MVPruner prunes at two locations in the language model: shallow processing applies Diversity-aware Ratio Allocation followed by Cross-stage Contribution-aware Token Selection; deeper processing applies Instruction-aware Ratio Allocation followed by Instruction-guided Token Selection. The retained visual tokens and text continue through subsequent decoder layers. The output remains the original model's driving-scene answer, not that of an additionally trained action controller.
Both stages first determine how many tokens to keep from each view, then select the specific tokens. Dynamic allocation means that budgets change with the current images, question, and processing stage rather than assigning every camera a permanent ratio. The second stage can only select among tokens retained by the first; it cannot recover discarded evidence. The first stage must therefore also avoid premature loss of potentially relevant information.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-view visual tokens<br/>and textual instruction"] --> B["Diversity-aware Ratio Allocation"]
B --> C["Cross-stage Contribution-aware Token Selection"]
C --> D["Instruction-aware Ratio Allocation"]
D --> E["Instruction-guided Token Selection"]
E --> F["Subsequent decoder layers<br/>generate a scene answer"]
Key Designs¶
1. Diversity-aware Ratio Allocation: reserve capacity for views with nonredundant information
Because shallow-layer cross-modal attention is not yet sufficiently reliable, Diversity-aware Ratio Allocation (DRA) first estimates how much distinctive content a view contains from its visual features alone. It computes cosine distances—one minus cosine similarity—between distinct tokens within the same view. Each token's minimum distance to another token becomes its uniqueness score. The minimum matters: a token with a very similar neighbor should not appear distinctive merely because it is far from some unrelated region.
Averaging token uniqueness scores yields the view's diversity score. These scores are normalized across views to allocate first-stage retention budgets. Repetitive views with generally small nearest-neighbor distances receive less capacity, while views containing more mutually distinct content receive more. Allocation does not depend on brightness, object counts, or fixed camera directions. Nor does high diversity guarantee relevance to the current question: DRA establishes sufficiently rich early context, while subsequent selection adds the relevance constraint.
2. Cross-stage Contribution-aware Token Selection: constrain greedy coverage with task relevance
Once each view has a budget, Cross-stage Contribution-aware Token Selection (CCTS) does not simply rank tokens once and truncate the list. It first estimates task relevance through cosine similarity between visual tokens and the instruction representation, selecting the most relevant token as the starting point. Each subsequent addition should differ sufficiently from the selected set while remaining related to the task. This avoids retaining many nearly identical regions without spending a pure-diversity sampling budget on irrelevant background.
The paper uses the following product to score candidates:
Here, \(U_{p,i}\) is the candidate token's minimum semantic distance to the currently selected token set, and \(R_{p,i}\) is its semantic relevance to the instruction. Each iteration adds the candidate with the largest product and updates the selected set until the view budget is reached. Unlike DRA, which finds the nearest distance among all other tokens, this reference set changes during selection. Whether a token still adds information therefore depends on what has already been retained.
Cross-stage contribution does not mean executing future layers, reading future gradients, and then returning to prune. Current visual uniqueness represents immediate contribution, while shallow-feature relevance to the instruction serves as a proxy for potential future usefulness. This interpretation also limits the claim: the proxy can reduce premature loss of task evidence, but it does not guarantee preservation of every token that later becomes important. The cached algorithm expresses relevance as cosine similarity to an instruction representation without clearly expanding how multiple instruction tokens are aggregated; reproduction requires checking the original implementation.
3. Instruction-aware Ratio Allocation: use deeper task evidence to correct early budgets
In deeper layers, the model has established more reliable cross-modal correspondence. Instruction-aware Ratio Allocation (IRA) extracts attention from instruction tokens to retained visual tokens. It first averages over instruction tokens to obtain each visual token's relevance, then averages within each view to estimate that view's task relevance. Averaging rather than summing within a view avoids automatically assigning greater importance merely because the first stage retained more tokens there.
The second stage does not repeat DRA from scratch. It adjusts each view's continued retention ratio through a compensation based on the deviation of task relevance from its first-stage diversity score. A visually simple view that contains the object named in the question can thus receive a larger relative budget once deeper attention identifies its relevance. The adjustment concerns retention of existing tokens, not restoration of the original set: it compensates for allocation bias rather than providing lossless correction. Mathematical typesetting in the cached budget equations is damaged. This note therefore retains only the relative-deviation mechanism confirmed by the prose and does not invent definitive normalization, rounding, or clipping rules.
4. Instruction-guided Token Selection: retain highly relevant evidence within each view
Instruction-guided Token Selection (ITS) reuses IRA's token-level instruction attention scores. Within each view, it keeps the highest-scoring tokens up to the second-stage budget. Allocation and selection thus use a consistent relevance signal: IRA determines which views deserve more computation, while ITS determines where to spend it inside each view.
The difference from CCTS is not whether the instruction is used, since CCTS already incorporates it. Shallow selection still constrains coverage through uniqueness, whereas deeper selection focuses directly through attention after alignment. For the video experiments, the authors apply this procedure independently to every frame of six-view video sequences. They introduce neither cross-frame token memory nor an explicit temporal pruning module, so the video results should not be interpreted as a new temporal modeling method.
Loss & Training¶
MVPruner inserts token selection and budget allocation into an existing model's inference process. The main paper introduces no new training loss or pruning-specific fine-tuning stage. Experiments use DriveMM and DriveLMM-o1, already fine-tuned on driving data and based on LLaVA-OneVision-7B and InternVL2.5-8B, respectively. Their existing training should not be counted as additional MVPruner training.
The implementation places its stages before layers 0 and 16. The reported 75% and 90% pruning ratios measure average token reduction across language-model layers. Labels such as 72 retained tokens per image therefore describe the comparison budget; they do not establish that both stages retain exactly 72 tokens or that every view receives an identical allocation. The main text does not fully specify all budget boundaries and implementation details. Avoiding Prune2Drive-style offline view-ratio search does not mean the method has no settings such as pruning locations and total budgets.
Key Experimental Results¶
Main Results¶
Image evaluations cover the six-view DriveLMM-o1 and DriveLM benchmarks, together with MAPLM, which includes three-view images and synchronized point-cloud maps. STSnu evaluates six-view video question answering. The table consistently uses the paper's 90% pruning setting. Higher scores are better, but aggregate metrics from different datasets are not directly comparable.
| Dataset / Model | Metric ↑ | Unpruned | Prune2Drive | MVPruner | Score difference from Prune2Drive |
|---|---|---|---|---|---|
| DriveLMM-o1 / DriveLMM-o1 | Overall Reasoning | 74.37 | 70.31 | 72.91 | +2.60 |
| DriveLM / DriveMM | Average | 59.1 | 57.4 | 58.2 | +0.8 |
| MAPLM / DriveMM | Average | 72.24 | 64.81 | 68.30 | +3.49 |
| STSnu / DriveMM | Average | 44.21 | 43.17 | 43.00 | -0.17 |
DriveLM's Average is the benchmark's aggregate score, not a single classification accuracy; its table separately reports Accuracy, ChatGPT, language-generation metrics, and Match. MAPLM's FRM denotes frame-overall accuracy, while QNS denotes question-overall accuracy; Average summarizes the two. The reported 98.5% is the ratio of DriveLM's pruned aggregate score to its unpruned score, not a driving accuracy of 98.5%. The corresponding retained-performance ratios for DriveLMM-o1 and MAPLM are 98.0% and 94.5%.
The following efficiency results use DriveMM on DriveLM at 90% pruning. FLOPs are expressed as the remaining fraction of the unpruned model. Latencies follow the paper's measurement protocol and are not extrapolated to arbitrary vehicle hardware.
| Config | Prefilling latency ms ↓ | Decoding latency ms ↓ | Remaining FLOPs ↓ | GPU peak memory GB ↓ |
|---|---|---|---|---|
| Unpruned DriveMM | 1968 | 65.3 | 100% | 16.47 |
| Prune2Drive | 307.5 | 59.9 | 13.4% | 16.00 |
| MVPruner | 395.6 | 51.8 | 12.7% | 15.93 |
MVPruner achieves a 4.97-fold prefilling speedup over the unpruned model and reduces FLOPs by 87.3%. Prune2Drive nevertheless has faster prefilling, so MVPruner is not best on every speed measure. Peak GPU memory decreases only from 16.47 GB to 15.93 GB; the 87.3% computational reduction must not be presented as an equivalent memory reduction.
Ablation Study¶
The table extracts the CCTS ablation from Figure 4, using DriveMM at 90% pruning. It compares the full model against a variant without the future-contribution relevance factor, keeping the other modules.
| Config | DriveLM Average ↑ | MAPLM Average ↑ | Note |
|---|---|---|---|
| Full MVPruner | 58.2 | 68.30 | Selection combines uniqueness and task relevance |
| CCTS without future contribution | 57.7 | 67.10 | Uniqueness is no longer weighted by task relevance |
Removing future contribution reduces scores by 0.5 and 1.20 points, respectively. This supports the design choice that shallow selection should not depend on visual diversity alone, but does not prove that the future-relevance proxy is accurate for every input. The figure also compares single-stage pruning, reuse of the first-stage rule in both stages, and different diversity measures. Because the local text does not clearly preserve all multi-panel numerical alignments, the remaining bar-chart values are not reconstructed into a complete table here.
Key Findings¶
- The two stages matter because they switch criteria, not merely because they prune twice. Applying the second-stage attention rule in shallow layers performs worse, while reusing the first-stage rule deeper also loses performance, consistent with the layer-wise attention analysis.
- Video results are not uniformly superior. At 75% pruning, STSnu Average reaches 44.25 versus the unpruned 44.21; at 90% pruning, it trails Prune2Drive, showing task-dependent trade-offs under aggressive compression.
- CCTS is the principal added cost. On DriveLM with six views and 729 original tokens per view, it takes 26.48 ms per view, with total pruning overhead of 163.9 ms. Its quality-preservation benefit must be assessed together with this computation.
Highlights & Insights¶
- Both allocation between views and selection within views become dynamic. Improving only one can still assign the wrong total budget to useful tokens or waste an adequate budget on repetitive information.
- CCTS combines diversity coverage with task relevance without equating unreliable shallow instruction attention with a complete ban on using instructions early. It distinguishes instruction-assisted preservation from letting instructions dominate allocation decisions.
- Analysis and mechanism correspond closely: early global context motivates DRA and CCTS, while deeper local focus motivates IRA and ITS. This is an example of adapting compression criteria to layer-wise behavior rather than applying a fixed importance score throughout the network.
Limitations & Future Work¶
- The main experiments evaluate driving question answering and scene understanding, not closed-loop control or deployment-level guarantees. High aggregate-score retention cannot substitute for real-system validation.
- The second stage cannot recover tokens discarded early. If scene uniqueness and instruction-relevance proxies both miss small but important evidence, later budget compensation cannot restore it.
- Greedy selection requires within-view pairwise similarity computation and has non-negligible overhead; prefilling does not surpass Prune2Drive. Efficiency conclusions should jointly report scores, pruning overhead, and the specific inference stage.
- The local main-text cache contains damaged budget equations and table layouts and does not include the full supplementary material. Tables 2 and 5 also disagree on FastV's DriveLM score, so this note does not mix that baseline's results. Reproduction should first check the original PDF and implementation-level normalization and budget boundaries rather than rely on rules inferred from damaged equations.
Related Work & Insights¶
- vs Prune2Drive: Both address multi-view token budgets. Prune2Drive searches offline for fixed view allocations, whereas MVPruner adapts online to images and instructions. MVPruner preserves scores better on the three image benchmarks at high pruning, but Prune2Drive has faster prefilling and a higher aggregate score on STSnu at 90% pruning.
- vs FastV and SparseVLM: These attention-driven pruning methods show that language-model signals can guide visual token selection. MVPruner further emphasizes that a signal's reliability changes across layers and explicitly models budget allocation between views.
- vs DivPrune: Visual diversity reduces redundant information, but coverage alone does not guarantee preservation of the evidence required by the question. MVPruner's CCTS constrains diversity selection with task relevance before switching to instruction-attention selection.
Rating¶
- Novelty: 4/5 — Online multi-view allocation and stage-dependent importance criteria form a coherent combination, built on existing token-pruning and diversity-selection mechanisms.
- Experimental Thoroughness: 4/5 — Two models, four benchmarks, quality and efficiency measurements, and module ablations are covered, but closed-loop evaluation is absent and some reproduction details depend on supplementary material.
- Writing Quality: 4/5 — Layer-wise observations naturally explain the design, while some metric terminology and efficiency-table labels require careful interpretation.
- Value: 4/5 — Directly relevant to inference compression for multi-view vision-language models, provided pruning overhead and task sensitivity are assessed before adoption.