โก VLM Efficiency¶
๐ง NeurIPS2026 ยท 11 paper notes
๐ Same area in other venues: ๐๏ธ ECCV2026 (57) ยท ๐ท CVPR2026 (63) ยท ๐ฌ ICLR2026 (18) ยท ๐ฌ ACL2026 (6) ยท ๐งช ICML2026 (4) ยท ๐ค AAAI2026 (5)
๐ฅ Top topics: Multimodal/VLM ร5 ยท Model Compression ร4 ยท Compression ร2
- Beyond Selection: Token Parameterization for Extreme Visual Token Compression
-
Rather than selecting a few original patches, Braco constructs a compact visual interface using a DCT low-frequency backbone, basis-coordinate embeddings, budget-dependent coordinate organization, and sparse-pooled spatial residuals; at 576โ16 tokens, it retains a Vanilla-normalized aggregate score of 94.0 while reducing full-pipeline prefill latency to 40.59 ms.
- Breaking Modality Heterogeneity in Low-bit Quantization for Large Vision-Language Models
-
To break the modality heterogeneity barrier where vision and text activation outliers reside in disjoint channels and severely distort unified quantization, SplitQ proposes channel-splitting-driven decoupling alongside adaptive cross-modal calibration, maintaining near-lossless accuracy down to W4A4 and W3A3.
- EventPrune: Cascaded Event-Assisted Token Pruning for Efficient First-Person Dynamic Scene Understanding
-
Proposes Event Cascade Pruning (ECP), the first training-free event-assisted pruning framework for first-person dynamic spatial reasoning, which combines causal temporal sampling, motion saliency filtering, and non-parametric rank fusion to outperform the full-token baseline (37.62% vs. 36.31%) at 80% token reduction while delivering a 1.89ร inference speedup and 52% GFLOPs savings.
- G2TR: Generation-Guided Visual Token Reduction for Separate-Encoder Unified Multimodal Models
-
G2TR uses VAE latent anchors to select and merge understanding-side visual tokens before LLM prefill in separate-encoder unified multimodal models; retaining 50% of the tokens yields 99.0% relative-average understanding performance and 98.0% editing performance on BAGEL, with approximately 1.94ร lower prefill FLOPs, but not lossless performance on every task.
- Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic Context Memory
-
Addressing the severe temporal retrieval bias and recall collapse when scaling frame token budgets in streaming video understanding, MemStream introduces training-free adaptive key selection (AKS) for sparse encoding and a reciprocal rank fusion (RRF) mixture-of-experts for frame retrieval, outperforming previous SOTA on CG-Bench and LVBench.
- Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers
-
GoToHunt introduces a training-free hierarchical token selection framework that combines diversity-driven inter-frame selection via farthest point sampling with layer-adaptive intra-frame pruning guided by attention entropy, accelerating visual geometry transformers by over 85% on 500-frame sequences while maintaining or improving reconstruction fidelity.
- OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models
-
Addressing the issues of audio-video token explosion and static modality-guided pruning failures in Omni-LLMs, OmniSelect employs a lightweight AudioCLIP to dynamically evaluate query relevance and switch compression regimes, combining fine-grained temporal group pruning to achieve 1.19รโ1.33ร inference speedup while preserving 94%โ99% of full-token accuracy in a training-free manner.
- SP2ec: Adaptive Self-Speculative Decoding for Vision-Language Models
-
Addressing the severe misalignment of small draft models and the suboptimality of fixed draft configurations in multimodal long contexts, this paper introduces SP2ec, a training-free bandit algorithm that exploits the single-peak throughput geometry of self-speculative decoding to adaptively select drafter configurations per query, achieving up to 1.62ร lossless inference speedup.
- SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass
-
SwiftVLM uncovers the non-monotonic layer-wise discriminability and cross-layer importance drift of visual tokens in VLMs, introducing a training-free cross-layer token bypass mechanism with residual offset alignment that temporarily shelves unselected tokens in early layers and re-evaluates them at deeper layers, eliminating localization degradation from premature hard pruning while achieving a ~2ร prefill speedup at a 77.8% pruning ratio.
- TwinPrune: Density-Aware Two-Phase Token Pruning for Vision-Language Models
-
To tackle both prefill compute and decode memory bottlenecks in vision-language models, TwinPrune introduces a unified training-free spatial-density signal derived from local cosine dissimilarity that drives density-tilted farthest-point bipartite merging during prefill and single-shot Top-\(k\) KV eviction during decode, establishing a superior compute-memory-accuracy Pareto frontier.
- Vision Token Pruning via Query-Vision Interaction Decomposition
-
QViD introduces a training-free query-aware visual token pruning method that extracts the dominant singular modes from the pre-positional, pre-softmax query-vision interaction matrix at early Transformer layers. By scoring tokens with right-singular leverage scores, it retains critical visual cues without positional bias, outperforming prior baselines under aggressive compression budgets while reducing computation by over 70%.