STVFocus: Query-guided Spatio-Temporal Visual Focusing for Video LLMs¶
Conference: ECCV 2026
Paper: ECCV 2026
Area: Interpretability
Keywords: Video LLMs, Spatio-Temporal Saliency, Hierarchical Temporal Focus, Spatial Focus, Training-Free
TL;DR¶
Addressing the intrinsically low task-relevant information density and substantial redundancy in video inputs, STVFocus introduces a training-free spatio-temporal visual focusing framework that combines coarse-to-fine hierarchical temporal frame sampling with query-guided intra-frame spatial saliency pruning, reinvesting the saved spatial budget into supplementary keyframes to significantly enhance Video LLM reasoning accuracy and interpretability.
Background & Motivation¶
Video Large Language Models (Video LLMs) have demonstrated impressive capability in reasoning over dynamic multimodal content. However, raw video streams inherently exhibit exceedingly low task-relevant information density. In the temporal domain, consecutive frames share substantial visual content with minimal incremental informativeness, while critical events answering a user query are frequently confined to fleeting moments. In the spatial domain, static background scenery, repetitive textures, and ambient clutter consume the vast majority of visual tokens within each individual frame, offering little to no value for answering specific questions. When Video LLMs are forced to process uncurated visual inputs under strict sequence length or memory budgets, task-relevant signals are inevitably diluted by massive background noise.
To alleviate spatio-temporal redundancy, recent studies have explored training-free visual context adaptation, yet temporal and spatial approaches remain fundamentally isolated. On the temporal front, adaptive sampling approaches such as BOLT and AKS leverage external multimodal encoders to extract query-relevant frames. However, these methods typically discard the heavy external model after a single frame-level scoring pass—leaving its fine-grained representational capacity underutilized—and suffer from severe query bias, which strips away global storyline context and harms short-video comprehension. On the spatial front, existing explorations rely primarily on vision token pruning or merging techniques originally designed for image models. These spatial methods are overwhelmingly driven by computational efficiency rather than semantic preservation; they lack fine-grained text-query guidance, frequently discarding small query-relevant objects or text regions alongside static backgrounds. Crucially, no existing approach bridges temporal and spatial budgets dynamically so that tokens saved across intra-frame regions can be reinvested to expand temporal context.
This paper tackles this gap by observing that human visual cognition naturally grasps global structure before local details, and that capacity pruned from uninformative spatial regions can be strategically traded for higher-density temporal coverage. Core idea: STVFocus establishes a training-free, model-agnostic unified spatio-temporal focusing framework that captures global video narrative and query-specific moments via coarse-to-fine hierarchical temporal selection, filters intra-frame tokens by fusing intrinsic feature activations with query-sensitive gradient saliency maps, and reallocates the saved spatial token budget to incorporate additional high-value temporal frames.
Method¶
Overall Architecture¶
STVFocus operates as a plug-and-play, training-free preprocessing pipeline. Taking as input a text query \(Q\) and a pool of video candidate frames \(\mathcal{F} = \{I_1, \dots, I_N\}\), it outputs a compact, information-dense sequence of visual tokens strictly adhering to a predetermined total token budget, ready for autoregressive decoding by any downstream Video LLM. The framework consists of four coordinated stages: spatio-temporal saliency extraction, hierarchical temporal focusing, query-informed spatial focusing, and temporal reallocation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Input: Text Query Q and Video Frame Pool F"] --> Saliency["Spatio-Temporal Saliency Extraction<br/>CLIP extracts temporal cosine similarity and spatial gradient saliency maps"]
Saliency --> HTF["Hierarchical Temporal Focus<br/>Coarse feature-diverse sampling + Fine query-relevant focusing"]
HTF --> QSF["Query-informed Spatial Focus<br/>Intrinsic AAM fused with text-guided spatial saliency map via Top-K retention"]
QSF --> TR["Temporal Reallocation<br/>Reinvest saved spatial token budget into additional candidate keyframes"]
TR --> Out["Output: High-Density Visual Token Sequence to Video LLM"]Key Designs¶
1. Hierarchical Temporal Focus: Balancing global narrative coverage with query-aligned moment selection
Relying solely on query-image similarity for temporal frame selection introduces severe tunnel vision, concentrating frames around narrow semantic keyword matches while destroying broader narrative continuity and causal context. Hierarchical Temporal Focus (HTF) addresses this by extending Farthest Point Sampling (FPS) into a coarse-to-fine two-phase selection procedure. First, an external multimodal embedding model (CLIP ViT-L/14) computes the cosine similarity \(s_i\) between text query embedding \(\mathbf{q} = f_T(Q)\) and frame visual embeddings \(\mathbf{v}_i = f_V(I_i)\), normalized and sharpened by a scaling power factor \(\alpha\): $\(\tilde{s}_i = \left( \frac{s_i - \min(s)}{\max(s) - \min(s)} \right)^\alpha\)$ where \(\alpha=2\). Frame selection proceeds iteratively from an initial seed frame set \(\mathcal{S}\), choosing the next frame to trade off visual diversity and query relevance: $\(I_{\text{next}} = \arg\max_{I_j \in \mathcal{F} \setminus \mathcal{S}} \Big[ \beta \min_{I_m \in \mathcal{S}} D(\mathbf{v}_j, \mathbf{v}_m) + (1-\beta) \tilde{s}_j \Big]\)$ where \(D(\mathbf{v}_j, \mathbf{v}_m) = 1 - \cos(\mathbf{v}_j, \mathbf{v}_m)\). Under a total frame budget of \(k\), HTF splits the budget equally between two phases. In the coarse phase, setting \(\beta=1\) isolates feature diversity, selecting a base frame set \(F_{\text{coarse}}\) that outlines global narrative progression. In the fine phase, \(\beta\) is decreased to \(0.5\), guiding selection toward query-relevant moments under diversity regularization to yield \(F_{\text{fine}}\). The union \(F_{\text{down}} = F_{\text{coarse}} \cup F_{\text{fine}}\) preserves overall contextual coherence while magnifying query-pertinent moments without excessive query bias.
2. Query-informed Spatial Focus: Fusing intrinsic activations with gradient saliency for intra-frame token selection
Even within keyframes selected by HTF, extensive spatial regions consist of irrelevant background. To eliminate intra-frame redundancy, Query-informed Spatial Focus (QSF) formulates a dual-stream importance metric. The first stream is an Activation-based Attention Map (AAM), which calculates the channel-wise \(L_1\) norm across patch token feature vectors \(\mathbf{X}_{t,p} \in \mathbb{R}^D\) and applies min-max normalization: $\(\mathcal{F}(\mathbf{X}_{t,p}) = \sum_{d=1}^D |\mathbf{X}_{t,p,d}|, \quad \text{AAM}(t,p) = \frac{\mathcal{F}(\mathbf{X}_{t,p}) - \min(\mathcal{F})}{\max(\mathcal{F}) - \min(\mathcal{F})}\)$ AAM captures the vision encoder's intrinsic perception of salient structures independent of language. The second stream derives a query-guided spatial saliency map \(\mathbf{M}_i \in \mathbb{R}^{H \times W}\) using a gradient-based visual explainability method (LeGrad) on the multimodal embedding model, smoothed by a Gaussian filter (\(\sigma=8\)) and downsampled via adaptive average pooling to match patch grid resolution \(\tilde{\mathbf{M}}_i \in \mathbb{R}^{H_p \times W_p}\). The final spatial focusing score linearly combines both streams: $\(\mathbf{s}_{\text{focus}} = \gamma \tilde{\mathbf{M}}_i + (1-\gamma) \text{AAM}_i\)$ With \(\gamma=0.5\), tokens are ranked and pruned via \(\text{TopK}_\tau(\mathbf{s}_{\text{focus}})\), retaining the top \(\tau=80\%\) most informative patch tokens. This dual formulation prevents pure AAM from discarding small query-targeted objects and avoids pure gradient saliency blurring high-frequency structural details.
3. Temporal Reallocation: Reinvesting saved spatial budget into expanded temporal coverage
Traditional vision token compression treats pruned tokens purely as FLOPs reduction. However, in fixed-budget inference scenarios, the pruned capacity represents unspent context budget. Temporal Reallocation (TR) institutes a closed-loop trade-off mechanism that converts spatial capacity savings into temporal coverage gains. Given a total token budget \(B_{\text{total}}\) and a spatial retention rate \(\tau\), suppressing \(1-\tau\) of the tokens leaves an available reserve \(B_{\text{remain}} = B_{\text{total}}(1 - \tau)\). Knowing the pruned token count per frame \(b_{\text{frame}}\), the number of supplemental frames that can be accommodated is calculated as: $\(k_{\text{add}} = \left\lfloor \frac{B_{\text{remain}}}{b_{\text{frame}}} \right\rfloor\)$ STVFocus directly samples the next \(k_{\text{add}}\) frames from the priority queue generated by HTF. In a 32-frame baseline setting with \(\tau=0.8\), the 20% spatial compression across 32 frames releases sufficient capacity to append exactly 8 additional processed frames (\(32 \times 0.2 / 0.8 = 8\)). As a result, 40 spatially focused frames enter the downstream Video LLM while strictly matching the original 100% token budget, bridging temporal gaps across long and complex action sequences.
Loss & Training¶
STVFocus is strictly training-free and model-agnostic, requiring zero weight updates or fine-tuning on vision encoders, multimodal projectors, or language decoders. A single universal hyperparameter configuration is used across all benchmarks and base models: temporal sharpening exponent \(\alpha=2\), Gaussian smoothing kernel \(\sigma=8\), an equal 50%/50% frame allocation between HTF coarse and fine stages, fine-stage trade-off parameter \(\beta=0.5\), spatial fusion weight \(\gamma=0.5\), and spatial retention threshold \(\tau=80\%\).
Key Experimental Results¶
Main Results¶
Evaluation was conducted across three comprehensive video benchmarks: VideoMME, LongVideoBench (LVB), and MLVU, spanning three leading open-source Video LLMs: LLaVA-OneVision-7B, VILA1.5-8B, and InternVL3-8B. STVFocus was compared against both state-of-the-art training-free sampling methods and training-based adaptation schemes.
| Base Model | Method Type | VideoMME Overall | VideoMME Short | VideoMME Medium | VideoMME Long | LVB | MLVU |
|---|---|---|---|---|---|---|---|
| LLaVA-OneVision-7B | Baseline (Uniform) | 58.5 | 70.3 | 56.6 | 48.8 | 56.4 | 63.2 |
| + AKS (CVPR 2025) | Training-Free Temporal | 59.6 | 71.1 | 57.4 | 50.1 | 59.3 | - |
| + BOLT (CVPR 2025) | Training-Free Temporal | 59.9 | 70.1 | 60.0 | 49.6 | 59.6 | 66.8 |
| + STVFocus (Ours) | Training-Free Spatio-Temporal | 61.1 | 71.9 | 61.0 | 50.4 | 61.2 | 68.7 |
| VILA1.5-8B | Baseline (Uniform) | 47.5 | 57.8 | 44.3 | 40.3 | 47.1 | 46.3 |
| + Frame-Voyager (ICLR 2025) | Training-Based Adaptation | 50.5 | 60.3 | 47.3 | 43.9 | - | 49.8 |
| + Q-Frame (2025) | Training-Free Multi-Res | 50.7 | 59.8 | 48.0 | 44.2 | 51.6 | 54.4 |
| + STVFocus (Ours) | Training-Free Spatio-Temporal | 52.6 | 61.8 | 50.8 | 45.2 | 52.5 | 50.9 |
| InternVL3-8B | Baseline (Uniform) | 64.5 | 75.7 | 64.1 | 53.8 | 58.5 | 67.8 |
| + STVFocus (Ours) | Training-Free Spatio-Temporal | 67.0 | 77.6 | 68.2 | 55.3 | 61.5 | 74.8 |
Ablation Study¶
A component-wise ablation study on VideoMME and LongVideoBench using LLaVA-OneVision-7B reveals the progressive contribution of coarse HTF, fine HTF, spatial pruning QSF, and temporal reallocation TR.
| HTF Coarse | HTF Fine | QSF Spatial | TR Realloc | Frames | Token Budget | VideoMME | LVB | Note |
|---|---|---|---|---|---|---|---|---|
| - | - | - | - | 32 | 100% | 58.5 | 56.4 | Uniform sampling baseline |
| ✓ | - | - | - | 32 | 100% | 59.2 | 56.8 | Visual diversity coarse selection (+0.7 / +0.4) |
| - | ✓ | - | - | 32 | 100% | 60.2 | 60.5 | Query-relevance fine selection (+1.7 / +4.1) |
| ✓ | ✓ | - | - | 32 | 100% | 60.8 | 60.6 | Full two-stage HTF (+2.3 / +4.2) |
| ✓ | ✓ | ✓ | - | 32 | 80% | 60.8 | 60.7 | QSF token pruning matches accuracy at 80% budget |
| ✓ | ✓ | ✓ | ✓ | 32 + 8 | 100% | 61.1 | 61.2 | Full STVFocus: 8 extra frames reallocated within 100% budget |
Sub-module ablation on QSF under a 50% spatial compression rate indicates that using AAM alone yields 59.1, using Spatial Saliency alone yields 58.4, and fusing both at \(\gamma=0.5\) reaches 59.4 (+0.9 over baseline), verifying strong complementarity between intrinsic activations and language-guided gradients. Across temporal frame scaling on InternVL3-8B, STVFocus delivers consistent +2.5 to +3.5 point improvements across 8 frames (59.8 → 63.3), 16 frames (61.6 → 64.5), and 32 frames (64.5 → 67.0).
Key Findings¶
- The coarse stage in HTF predominantly benefits short videos and maintains global narrative context, whereas the fine stage delivers major gains on long videos; combining both effectively prevents catastrophic context collapse.
- QSF achieves equivalent or superior accuracy while utilizing only 80% of spatial tokens, proving that static background tokens primarily contribute noise rather than useful signal.
- Reinvesting the saved 20% spatial token capacity into 8 additional keyframes via Temporal Reallocation maximizes spatio-temporal information density without exceeding overall compute bounds.
- Fixing the spatial retention rate \(\tau=80\%\) is empirically superior to dynamic thresholding; variable per-frame token counts disrupt attention alignment in long-context Video LLMs.
Highlights & Insights¶
- Dual-use of external multimodal foundation models: Rather than treating CLIP merely as a disposable frame-level scorer, STVFocus extracts spatial gradient explainability maps (LeGrad) from the identical model, maximizing feature reuse across temporal and spatial axes.
- Cross-dimensional resource reallocation: Reconceptualizing token pruning from a pure latency-reduction heuristic into an active context-enrichment mechanism enables models to trade uninformative background pixels for critical temporal event coverage.
- Perceptually grounded coarse-to-fine design: Aligning the frame selection process with the human cognitive principle of grasping global scene categories before inspecting task-specific details effectively resolves the severe query-bias dilemma in prior training-free selectors.
Limitations & Future Work¶
- Author-admitted limitations: Calculating gradient-based saliency maps (LeGrad) requires backpropagation passes through the vision encoder of the external CLIP model, incurring non-trivial pre-processing latency during the offline sampling stage.
- Independent observations: The framework employs static global hyperparameters (\(\tau=0.8\), equal split for coarse/fine stages). In extreme video categories—such as high-action sports versus static lecture slides—fixed retention ratios may not be optimal. Additionally, aggressive spatial pruning can occasionally clip small, discontinuous text strokes in dense OCR tasks.
- Improvement directions: Future investigations could explore parameter-free cross-attention maps to approximate spatial saliency without backward passes, and develop motion-adaptive budget allocation based on optical flow variance.
Related Work & Insights¶
- vs BOLT / AKS (Adaptive Temporal Sampling): Prior training-free temporal samplers operate solely at the frame level and suffer from severe query bias toward local clusters; STVFocus integrates coarse-to-fine temporal sampling to preserve global context, extends focusing to the intra-frame spatial token level, and introduces budget reallocation.
- vs Q-Frame (Multi-Resolution Frame Adaptation): Q-Frame dynamically adjusts entire frame resolutions based on temporal saliency, but retains full background clutter inside each active frame; STVFocus performs irregular, fine-grained token-level semantic selection within frames and closes the budget loop.
Rating¶
- Novelty: ⭐⭐⭐⭐ [First unified framework bridging training-free temporal sampling and spatial token pruning with budget reallocation]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated across 3 foundation models, 3 major benchmarks, with thorough ablations on modules, thresholds, and frame budgets]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally clear motivation, rigorous methodology, and consistent mapping between figures and formulations]
- Value: ⭐⭐⭐⭐ [Directly applicable to any open-source Video LLM without fine-tuning, offering substantial practical utility]