FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Models¶
Conference: ECCV2026
Paper: ECCV Paper
Area: VLM Efficiency
Keywords: visual token pruning, text guidance, cross-modal similarity, visual saliency, diversity preservation
TL;DR¶
At the vision encoder–language model interface, FlashVLM selects important regions using fused query relevance and visual saliency, then adds non-redundant background tokens; reducing LLaVA-1.5-7B from 576 to 128 visual tokens yields a reported 100.6% relative performance, but does not preserve every task's performance.
Background & Motivation¶
Vision-language models (VLMs) typically encode an image into dense patches and send hundreds of visual tokens to a large language model (LLM) alongside the question. A user may ask only about one object's color, yet the language model still processes the background and repeated textures across the image. Video and high-resolution inputs amplify this burden. Compressing the visual sequence can reduce prefill computation and shrink the KV cache, but the vision encoder has already run, so the token pruning ratio cannot be treated as the end-to-end speedup.
Deciding which tokens can be discarded is harder than shortening the sequence. Methods such as VisionZip and VisPruner rely on intrinsic visual saliency and cannot redirect selection when the question changes. FastV and SparseVLM use language-model attention, which can be affected by positional bias and attention-head sparsity. Retaining only patches most similar to the question words is also unreliable: counting needs multiple instances, relational questions need surrounding objects, and background regions outside the most salient object may contain essential evidence.
FlashVLM therefore separates query relevance from contextual coverage. Visual and textual signals first identify important regions, while a diversity budget preserves some of the remaining image content. Core Idea: fuse query relevance with visual saliency once before the language model, then retain non-redundant background tokens so that compressed inputs remain both question-relevant and contextually informative.
Method¶
Overall Architecture¶
The inputs are patch features from a frozen vision encoder and the user's question; the output is a shorter subset of visual tokens for the original language model. FlashVLM does not introduce another answer generator or repeatedly prune inside each Transformer layer. Selection occurs at the encoder–decoder boundary through two main steps: Query-Guided Relevance Fusion determines which regions deserve priority, and Diversity-Preserving Partitioning prevents similar background patches from consuming the remaining budget.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Image or video frames"] --> B["Vision encoder<br/>Features and saliency"]
C["User question<br/>Embeddings and early text attention"] --> D["Query-Guided Relevance Fusion"]
B --> D
D --> E["Diversity-Preserving Partitioning<br/>Important regions and diverse background"]
B --> E
E --> F["Compressed visual sequence + question<br/>Original language model generates an answer"]
Every edge represents inference-time data flow, with no additional training supervision. Here, attention-light means that deep text–image attention maps do not determine cross-modal relevance; it does not mean that attention is absent. Visual saliency comes from the vision encoder, and the query gating described in the methodology also uses early text-to-text attention from the language model. This distinction determines which intermediate states an implementation must expose and which overheads must be counted.
Key Designs¶
1. Query-Guided Relevance Fusion
The first requirement is a comparable representation space for vision and text. The model's existing multimodal projector maps visual patches into the language-model embedding space, followed by L2 normalization. Question tokens come from the language model's embedding layer. According to Section 3.1, the method sums the attention received by each text token in early text self-attention and normalizes it relative to the maximum sum, using the result as a contextual gate. The intention is to give semantically informative words greater influence than irrelevant words, rather than estimating importance from embedding magnitude alone.
The selector then computes dot-product similarity between every visual patch and every text token, producing an image-region-by-question-word matrix. This explicitly compares feature representations rather than reading text–image attention directly from the decoder. For each patch, similarities are divided by a temperature and passed through a softmax along the text dimension; the resulting weights form a weighted sum of that patch's similarities. The default temperature is 0.05. A low temperature lets the best-matching words dominate, preventing a relevant noun from being diluted by many ordinary words in the question.
The aggregated scores are filtered before use. The text describes computing their mean and standard deviation, clipping below-mean standardized values to zero, and applying min–max normalization to obtain extrinsic query relevance. Thus, the threshold adapts to the score distribution instead of using a fixed similarity cutoff. However, the paper calls this a "batch mean," and the cache does not fully clarify the exact axis for cross-sample versus within-sample statistics. Reproduction should verify the implementation rather than silently choose one interpretation.
In parallel, vision-encoder attention is averaged over heads and min–max normalized to obtain intrinsic visual saliency. Query relevance identifies what the question concerns, while visual saliency supplies structural cues beyond literal word matching. The authors implement a weighted geometric fusion of the two signals in the log domain, using a default weight of 0.5 and another normalization afterward. This favors regions supported by both signals, instead of allowing a large unimodal score to hide a low score from the other modality through linear addition. Without a question, the authors set extrinsic relevance to zero and describe the method as falling back to visual-attention-based pruning.
A reproducibility issue must remain explicit: the methodology says that each word vector is multiplied by a scalar gate and then L2-normalized. If normalization is per token, a positive scalar theoretically cancels out; normalizing the entire matrix would behave differently. The ablation and limitations sections also use "query-norm gating" or "norm-based gating," which is not fully consistent with the earlier attention-based description. The intended role of gating is understandable, but the exact operator order cannot be established from this cache alone.
Several extracted equations also lose brackets, minus signs, or indices. This note does not present repaired guesses as the authors' exact formulas. The similarity aggregation, standardization, and geometric fusion described above are supported by adjacent prose and explain the main algorithm; the gating implementation and normalization axes still require verification against original equations or code.
2. Diversity-Preserving Partitioning
Fused relevance does not lead directly to retaining the highest-scoring tokens for the entire budget. FlashVLM divides the budget equally by default: half goes to the highest-scoring important tokens, and half goes to diverse background tokens selected from the remainder. The first half protects explicitly queried objects; the second supplies additional evidence for relations, counting, and scene semantics. With global Top-K alone, similar patches around one salient object could repeatedly consume the budget and exclude other useful objects.
Background selection is not random sampling. The remaining candidate features are normalized and divided into two groups according to the parity of their flattened 2D spatial indices. Cross-group cosine similarities are computed. Each iteration finds the most similar pair, removes the member with the lower fused relevance score, and updates the candidate set until exactly the background budget remains. Important tokens have already been selected exclusively and are not subjected to this residual-pool removal. Both retained subsets are then passed to the original language model.
This procedure addresses both redundancy and which redundant item to retain: similarity identifies repeated information, while fused relevance selects the more useful instance. It does not average multiple patches into a new representation, making it subset selection rather than token merging. Its cost also needs qualification. Bipartitioning reduces the comparison set per iteration, but the cross-group matrix still grows quadratically with candidate count in the worst case, and removal is iterative. The low-overhead claim is primarily supported by runtime measurements, not by a demonstrated linear-complexity bound.
A Worked Example¶
Consider an image containing red tomatoes, yellow bananas, and background objects. The vision encoder produces 576 tokens, and the user asks, "What is the red fruit on the right?" This is a process illustration based on the paper's fruit visualization, not another quantitative experiment. Projected patches are compared with question words, allowing relevance aggregation to emphasize regions matching semantics such as "red," "fruit," and "right," while visual saliency supplies object-structure cues.
With a total budget of 128, the selector first keeps the 64 highest-scoring tokens. The remaining 512 enter the residual pool, are split by spatial parity, and undergo iterative near-duplicate removal until 64 background tokens remain. The language model receives the combined 128 visual tokens and the original question. The background allocation does not guarantee one token per object; it provides a mechanism that prevents the entire budget from concentrating on similar regions.
If the question changes to "What is the yellow fruit on the left?", the visual features can remain unchanged, but text relevance and the selected subset change, potentially shifting focus from tomatoes to bananas. This example also exposes a cost of query conditioning: a single compressed subset cannot be assumed reusable for every question about the same image.
Loss & Training¶
FlashVLM is an inference-time selection method. The paper introduces no new training loss or dedicated fine-tuning procedure; the vision encoder stays frozen, and the method relies on the existing projector and language-model representations. Defaults are 0.05 for aggregation temperature, 0.5 for the fusion weight, and equal budgets for important tokens and diverse background. Text Sharpening in the ablation refers to sparsity-enhancing sharpening, but the cache does not specify a standalone switch precisely enough for operator-level reproduction. Its name does not establish an additional learned module.
Key Experimental Results¶
Main Results¶
The image experiments use LLaVA-1.5-7B with 576 original visual tokens. Original Table 1 contains 10 image benchmarks. ACC is an aggregate relative-performance measure with the unpruned model set to 100%, not an absolute accuracy on one dataset; the cache does not fully specify its aggregation formula. The following table preserves the reported values. VQAv2, GQA, and MMVet contain native benchmark scores, and MMVet should not be treated as having the same accuracy definition as the first two.
| Method | Tokens kept | Pruning ratio | Relative ACC | VQAv2 | GQA | MMVet |
|---|---|---|---|---|---|---|
| Unpruned LLaVA-1.5-7B | 576 | 0% | 100.0% | 78.5 | 62.0 | 31.1 |
| VisPruner | 128 | 77.8% | 99.7% | 75.8 | 58.2 | 33.7 |
| FlashVLM | 128 | 77.8% | 100.6% | 76.4 | 58.9 | 34.1 |
| VisPruner | 64 | 88.9% | 96.6% | 72.7 | 55.4 | 32.3 |
| FlashVLM | 64 | 88.9% | 97.9% | 73.6 | 56.1 | 32.9 |
| VisPruner | 32 | 94.4% | 91.5% | 67.7 | 52.2 | 28.8 |
| FlashVLM | 32 | 94.4% | 92.8% | 69.3 | 52.8 | 29.6 |
The video experiments in original Table 2 use Video-LLaVA with 8 frames at resolution 224, totaling 2048 visual tokens. ChatGPT-Assistant evaluates TGIF-QA, MSVD-QA, and MSRVTT-QA. Keeping 455 tokens gives 48.7% average accuracy, still below the unpruned model's 49.3%, while its average score of 3.33 slightly exceeds the unpruned 3.32. These metrics must not be conflated into a claim of universally lossless video performance.
The authors state that they follow each benchmark's official protocol. Baseline values are taken directly from the VisPruner paper, whereas FlashVLM values average 5 independent runs. The current cache does not provide complete splits, all evaluator details, or variances, and it does not include the referenced supplementary material. These are therefore author-reported comparison conditions, not independently reproduced results in this note.
Ablation Study¶
The following rows come from the 64-token setting of original Table 5 using LLaVA-1.5-7B. All ablation rows are retained to show how component effects differ across tasks. Values are native scores, not relative ACC.
| Config | VQAv2 | POPE | MMB | MMVet | GQA |
|---|---|---|---|---|---|
| Full model | 73.6 | 81.7 | 61.8 | 32.9 | 56.1 |
| Without text guidance | 72.7 | 80.6 | 61.3 | 32.4 | 55.4 |
| Without visual saliency | 72.5 | 80.7 | 61.1 | 32.1 | 55.2 |
| Without diversity reserve | 72.4 | 81.3 | 61.5 | 32.2 | 55.6 |
| Log-domain fusion replaced by linear fusion | 73.2 | 80.2 | 61.2 | 32.5 | 55.8 |
| Without text sharpening | 73.0 | 80.9 | 61.2 | 32.3 | 55.6 |
| Without query gating | 72.8 | 81.0 | 61.4 | 32.3 | 55.3 |
Runtime efficiency comes from original Table 4 and Section 4.4, using LLaVA-NeXT-7B rather than the LLaVA-1.5-7B model above. Measurements use a single NVIDIA A100, batch size 1, and FP16. Latency is the authors' reported end-to-end CUDA latency.
| Method | Tokens kept | FLOPs (T) | KV cache (MB) | GPU memory (GB) | Latency (ms) |
|---|---|---|---|---|---|
| Unpruned LLaVA-NeXT-7B | 2880 | 43.6 | 1440 | 17.0 | 313 |
| FastV | 640 | 13.5 | 380 | 16.9 | 148 |
| VisPruner | 640 | 11.5 | 360 | 14.8 | 117 |
| FlashVLM | 640 | 11.3 | 356 | 14.6 | 115 |
| FastV | 160 | 6.3 | 95 | 16.9 | 112 |
| VisPruner | 160 | 3.8 | 80 | 14.7 | 78 |
| FlashVLM | 160 | 3.6 | 78 | 14.7 | 74 |
Key Findings¶
- "Beyond lossless" describes aggregate performance: 128-token FlashVLM reports 100.6% relative ACC, but VQAv2 falls from 78.5 to 76.4 and GQA from 62.0 to 58.9. It must not be presented as outperforming the original model on every task.
- Diversity and visual structure help, but there is no universally most important component. At 64 tokens, removing the diversity reserve reduces VQAv2 by 1.2 points; removing visual saliency reduces GQA by 0.9; replacing log fusion with linear fusion reduces POPE by 1.5.
- FlashVLM at 160 tokens takes 74 ms, approximately a 4.23-fold speedup over the unpruned 313 ms. Against VisPruner's 78 ms at the same budget, the improvement is only 4 ms. Large token reductions do not imply equally large speedups over a strong pruning baseline.
Highlights & Insights¶
- Query-conditioned selection at the language-model input helps retain dense internal computation and FlashAttention compatibility. The transferable insight is where to place the selector, not merely a different importance score.
- Important regions and background use different selection rules, matching visual question answering's evidence requirements better than global Top-K alone. A diversity budget preserves complementary information instead of treating every low-scoring token as useless.
Limitations & Future Work¶
- The authors acknowledge dependence on projected visual embedding quality, potentially larger budgets for fine-grained tasks, difficulty with pure logical negation, and the absence of multi-step refinement or temporal feedback.
- This note's reproducibility assessment: gating and normalization order, text-attention extraction, and residual-selection complexity require code-level clarification. Attention-light must not be expanded into completely attention-free.
- The available evidence has limited scope. The paper claims 14 benchmarks, but the visible main tables cover 10 image and 3 video benchmarks. Original Table 1's caption also says 12, inconsistent with its visible columns. Detailed ActivityNet-QA and additional-backbone results are referred to an absent appendix; this note does not invent their values or extend conclusions based on them.
- Aggregate improvements are small and variances are absent, so 100.6% alone cannot establish statistical significance. Useful follow-up evaluations would specify splits and evaluator versions, isolate selection overhead, and test long videos and negated questions.
Related Work & Insights¶
- vs VisPruner / VisionZip: These approaches emphasize intrinsic visual cues. FlashVLM adds explicit query relevance, but this also prevents assuming that one compressed representation can be reused for all questions.
- vs FastV / SparseVLM: FlashVLM avoids repeated selection using deep text–image attention inside the language model, reducing deployment intrusion. Its dependence on early text attention for gating nevertheless needs separate accounting.
- vs ToMe / PruMerge+: Merging methods reconstruct token representations, whereas FlashVLM keeps a subset of original tokens. Both address redundancy, but FlashVLM emphasizes query conditioning and a background allocation; its results do not establish that selection universally outperforms merging.
Rating¶
- Novelty: 3/5. Combining visual saliency, query relevance, and diversity is useful, but the contribution primarily integrates selection mechanisms and interface placement.
- Experimental Thoroughness: 3/5. Multiple budgets, ablations, and hardware measurements are available, while missing appendices, protocol details, and uncertainty estimates constrain the evidence.
- Writing Quality: 3/5. The overall idea is clear, but inconsistent gating terminology, benchmark counts, and equation-extraction issues impede exact reproduction.
- Value: 4/5. Relevant to researchers and engineers reducing existing VLM inference costs; its equal-budget quality and measured latency gains merit reproduction.