Skip to content

โšก VLM Efficiency

๐Ÿง  NeurIPS2026 ยท 11 paper notes

๐Ÿ“Œ Same area in other venues: ๐ŸŽž๏ธ ECCV2026 (57) ยท ๐Ÿ“ท CVPR2026 (63) ยท ๐Ÿ”ฌ ICLR2026 (18) ยท ๐Ÿ’ฌ ACL2026 (6) ยท ๐Ÿงช ICML2026 (4) ยท ๐Ÿค– AAAI2026 (5)

๐Ÿ”ฅ Top topics: Multimodal/VLM ร—5 ยท Model Compression ร—4 ยท Compression ร—2

Beyond Selection: Token Parameterization for Extreme Visual Token Compression

Rather than selecting a few original patches, Braco constructs a compact visual interface using a DCT low-frequency backbone, basis-coordinate embeddings, budget-dependent coordinate organization, and sparse-pooled spatial residuals; at 576โ†’16 tokens, it retains a Vanilla-normalized aggregate score of 94.0 while reducing full-pipeline prefill latency to 40.59 ms.

Breaking Modality Heterogeneity in Low-bit Quantization for Large Vision-Language Models

To break the modality heterogeneity barrier where vision and text activation outliers reside in disjoint channels and severely distort unified quantization, SplitQ proposes channel-splitting-driven decoupling alongside adaptive cross-modal calibration, maintaining near-lossless accuracy down to W4A4 and W3A3.

EventPrune: Cascaded Event-Assisted Token Pruning for Efficient First-Person Dynamic Scene Understanding

Proposes Event Cascade Pruning (ECP), the first training-free event-assisted pruning framework for first-person dynamic spatial reasoning, which combines causal temporal sampling, motion saliency filtering, and non-parametric rank fusion to outperform the full-token baseline (37.62% vs. 36.31%) at 80% token reduction while delivering a 1.89ร— inference speedup and 52% GFLOPs savings.

G2TR: Generation-Guided Visual Token Reduction for Separate-Encoder Unified Multimodal Models

G2TR uses VAE latent anchors to select and merge understanding-side visual tokens before LLM prefill in separate-encoder unified multimodal models; retaining 50% of the tokens yields 99.0% relative-average understanding performance and 98.0% editing performance on BAGEL, with approximately 1.94ร— lower prefill FLOPs, but not lossless performance on every task.

Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic Context Memory

Addressing the severe temporal retrieval bias and recall collapse when scaling frame token budgets in streaming video understanding, MemStream introduces training-free adaptive key selection (AKS) for sparse encoding and a reciprocal rank fusion (RRF) mixture-of-experts for frame retrieval, outperforming previous SOTA on CG-Bench and LVBench.

Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers

GoToHunt introduces a training-free hierarchical token selection framework that combines diversity-driven inter-frame selection via farthest point sampling with layer-adaptive intra-frame pruning guided by attention entropy, accelerating visual geometry transformers by over 85% on 500-frame sequences while maintaining or improving reconstruction fidelity.

OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models

Addressing the issues of audio-video token explosion and static modality-guided pruning failures in Omni-LLMs, OmniSelect employs a lightweight AudioCLIP to dynamically evaluate query relevance and switch compression regimes, combining fine-grained temporal group pruning to achieve 1.19ร—โ€“1.33ร— inference speedup while preserving 94%โ€“99% of full-token accuracy in a training-free manner.

SP2ec: Adaptive Self-Speculative Decoding for Vision-Language Models

Addressing the severe misalignment of small draft models and the suboptimality of fixed draft configurations in multimodal long contexts, this paper introduces SP2ec, a training-free bandit algorithm that exploits the single-peak throughput geometry of self-speculative decoding to adaptively select drafter configurations per query, achieving up to 1.62ร— lossless inference speedup.

SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass

SwiftVLM uncovers the non-monotonic layer-wise discriminability and cross-layer importance drift of visual tokens in VLMs, introducing a training-free cross-layer token bypass mechanism with residual offset alignment that temporarily shelves unselected tokens in early layers and re-evaluates them at deeper layers, eliminating localization degradation from premature hard pruning while achieving a ~2ร— prefill speedup at a 77.8% pruning ratio.

TwinPrune: Density-Aware Two-Phase Token Pruning for Vision-Language Models

To tackle both prefill compute and decode memory bottlenecks in vision-language models, TwinPrune introduces a unified training-free spatial-density signal derived from local cosine dissimilarity that drives density-tilted farthest-point bipartite merging during prefill and single-shot Top-\(k\) KV eviction during decode, establishing a superior compute-memory-accuracy Pareto frontier.

Vision Token Pruning via Query-Vision Interaction Decomposition

QViD introduces a training-free query-aware visual token pruning method that extracts the dominant singular modes from the pre-positional, pre-softmax query-vision interaction matrix at early Transformer layers. By scoring tokens with right-singular leverage scores, it retains critical visual cues without positional bias, outperforming prior baselines under aggressive compression budgets while reducing computation by over 70%.