EventSTU: Event-Guided Efficient Spatio-Temporal Understanding for Video-LLMs

Conference: ECCV 2026
Paper: ECCV Official
Code: https://wh-xu1.github.io/EventSTU.github.io
Area: Video Understanding
Keywords: Event Cameras / Video-LLMs / Spatio-Temporal Understanding / Token Pruning / Keyframe Sampling

TL;DR

To tackle high computational costs and visual redundancy in long video understanding for Video-LLMs, EventSTU proposes a training-free, bio-inspired event-guided framework combining coarse-to-fine keyframe sampling with zero-cost physics-semantic token pruning, reducing FLOPs by 66.7% while consistently surpassing original baseline performance.

Background & Motivation

Video Large Language Models (Video-LLMs) have achieved remarkable breakthroughs in complex multimodal video understanding, temporal localization, and contextual reasoning. However, long videos encompass massive frame sequences that expand into tens of thousands of visual tokens. Because Transformer self-attention complexity scales quadratically with sequence length, processing such oversized token sequences incurs prohibitive computational overhead and memory footprints. In extreme situations, sequence lengths exceed the maximum context window of the language model, triggering system failures. Furthermore, long videos naturally contain static backgrounds, smooth panning, and long segments irrelevant to the user query, introducing substantial uninformative tokens that distract model attention and induce hallucinations.

Current spatio-temporal reduction methods confront severe efficiency-performance trade-offs. In the temporal domain, conventional uniform frame sampling frequently misses brief, high-dynamic critical visual moments, whereas advanced keyframe selection methods guided by auxiliary models or caption generation (e.g., VideoAgent, BOLT, AKS) require costly pre-processing that balloons overall computational FLOPs by several folds. In the spatial domain, inner-LLM token pruning (such as FastV and DyCoKe) only intervenes after several Transformer layers, leaving early-layer overhead intact; conversely, outer-LLM pruning approaches relying on pairwise token similarities or clustering (such as LLaVA-PruMerge and VisionZip) incur quadratic computational overhead, creating major prefilling latency bottlenecks.

Biological vision systems exhibit extreme computational efficiency by prioritizing motion and environmental changes, focusing neural resources on high-value dynamic cues. Bio-inspired event cameras mimic biological retinas by asynchronously recording only per-pixel brightness variations, yielding sparse, information-dense event streams that embody zero-cost physical priors of information increments. The angle of attack in this work is to exploit the change-triggered property of events to remove temporal frame redundancy, while capitalizing on their intrinsic visual saliency to guide spatial token pruning, extendable to standard videos via lightweight frame-difference simulation. The core idea is to establish a training-free, event-guided spatio-temporal understanding framework, EventSTU, which performs event-accumulation sampling with question-relevant binning in time, couples physical saliency with attention reuse in space, and adaptively allocates per-frame token budgets based on query relevance.

Method

Overall Architecture

EventSTU takes as input an RGB video sequence along with an associated event sequence (obtained directly from real event sensors or synthesized via lightweight consecutive frame differences), compressing excessive visual tokens into a compact, semantically dense visual representation for frozen Video-LLMs without any training or parameter updates.

The pipeline comprises two closely coordinated stages: first, temporal Coarse-to-Fine Sampling (C2FS) uses adaptive event-accumulation thresholds to filter static and slow redundancy, followed by text-image similarity ranking across uniform temporal bins to retrieve query-aligned keyframes; second, spatial Zero-Cost Adaptive Pruning (ZAP) partitions event frames into grids matching visual patches to eliminate uninformative background tokens (physics-aware), reuses the vision encoder's terminal self-attention matrix to preserve non-salient but semantically crucial tokens (semantic-aware), and adaptively modulates retention ratios per keyframe based on question relevance scores.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Long Video Frames + Event Stream<br/>(Real Event Stream or Lightweight Simulation)"] --> B["Temporal Coarse-to-Fine Sampling C2FS<br/>Event Accumulation Coarse Filter + Temporal Binning Fine Retrieval"]
    B --> C["Compact Keyframe Candidate Set<br/>(Capturing High Dynamics & Question Relevance)"]
    C --> D["Vision Encoder Feature Extraction<br/>(ViT Forward Pass & Terminal Attention Extraction)"]
    D --> E["Spatial Zero-Cost Adaptive Pruning ZAP<br/>Physics Saliency Pruning + Attention Reuse Selection"]
    B -.->|"Query Matching Score st"| F["Cross-Frame Adaptive Budget Allocation<br/>Dynamic Retention Ratios & Physics/Semantic Split"]
    F --> E
    E --> G["Compact High-Quality Visual Tokens<br/>+ Question Text Prompt Tokens"]
    G --> H["Large Language Model Inference<br/>Precise Question-Answer Generation"]

Key Designs

1. Temporal Coarse-to-Fine Sampling (C2FS): Event-Accumulation Triggering and Question-Relevant Binning

This design resolves the limitations of uniform sampling (which skips sudden dynamic cues) and existing model-guided keyframe selection (which introduces massive pre-processing FLOPs). C2FS decomposes keyframe extraction into redundancy removal followed by relevance retrieval: In the coarse stage, event density serves as an optimal proxy for visual information increment between frames. The system continuously accumulates event counts across frames until the accumulated value surpasses a dynamic threshold \(\tau\), triggering a keyframe sample and resetting the accumulator; intermediate redundant frames are discarded. To accommodate videos with diverse motion intensities, the threshold \(\tau\) is adaptively derived from the total sequence event count and a coarse sampling ratio \(S\): $\(\tau = \frac{\sum_{t=1}^{M-1} e_t}{\lceil (M-1)S \rceil}\)$ where \(M\) denotes total video frames and \(e_t\) represents the event count between frames \(F_{t-1}\) and \(F_t\), shrinking \(M\) frames down to a non-redundant subset \(M_c\). In the fine stage, to ensure the selected keyframes specifically answer the user query, candidate frames and the question are embedded via lightweight vision-language backbones (e.g., CLIP or BLIP) to evaluate cosine similarity. To prevent greedy Top-K selection from clustering all frames within a narrow temporal window and losing global context, C2FS introduces temporal bin sampling: the coarse candidate frames are partitioned into \(B\) uniform temporal bins, and the frame with the highest similarity score within each bin is selected, outputting \(M_f\) keyframes (\(M_f \ll M_c \ll M\)) that balance query alignment with temporal coverage.

2. Spatial Zero-Cost Adaptive Pruning (ZAP): Physical Saliency Prior and Semantic Attention Reuse

This design eliminates the quadratic overhead of pairwise token similarity calculations in outer-LLM pruning while overcoming the inability of inner-LLM pruning to save early-layer computations. ZAP emulates human visual cognition from low-level physical sensing to high-level semantic focus across two zero-parameter phases: The first phase is physics-aware selection. Event streams exhibit intrinsic spatial sparsity: static backgrounds like skies, roads, and walls trigger virtually no events, whereas moving objects and interactive foregrounds concentrate dense event bursts. ZAP partitions event frames into non-overlapping grids directly aligned with the spatial patches of the RGB vision encoder, counts event occurrences per patch as importance scores, and retains the top \(R_p\%\) scoring tokens. This physics-aware step operates via simple integer counting, incurring a negligible computational cost of only 0.00027% of total inference. The second phase is semantic-aware selection. Relying purely on physical motion can discard stationary but semantically vital visual entities (e.g., static road signs, text, or background reference objects). To preserve these without adding parameters or extra forward passes, ZAP reuses the attention matrix from the final layer of the vision encoder. Averaging the attention matrix across query tokens yields a global semantic score per visual token. ZAP selects the top \(R_s\%\) tokens among those not already selected by physics-aware pruning, marrying high-dynamic foregrounds with static semantic cues.

3. Cross-Frame Adaptive Budget Allocation and Lightweight Event Simulation: Unified Spatio-Temporal Synergy

Standard visual token pruning uniformly applies a fixed retention ratio to all frames, ignoring the varying informational importance of different keyframes. EventSTU bridges temporal and spatial stages by dynamically converting the temporal image-text similarity scores \(s^t\) into per-frame spatial token retention budgets. Given \(M_f\) keyframes, \(N\) tokens per frame, an expected overall retention ratio \(R\), and a baseline retention ratio \(B = 5\%\) to avoid over-pruning, the allocable token budget is \(N_{\text{alloc}} = M_f \cdot N \cdot (R - B)\). For keyframe \(F_t\), its adaptive retention ratio \(R^t\) is computed as: $\(R^t = \min\left( \frac{N_{\text{alloc}} s^t + BN}{N}, 1 \right)\)$ where \(s^t\) is normalized via Softmax. If \(R^t \ge 1\), all tokens in frame \(F_t\) are preserved, and surplus budget is redistributed. The balance between physics-aware retention \(R_p^t\) and semantic-aware retention \(R_s^t\) adapts dynamically: if \(R^t \le R_p\), the budget is allocated entirely to physics-aware selection; if \(R^t > R_p\), physics-aware selection saturates at \(R_p\), and the excess budget is allocated to semantic selection via: $\(R_s^t = \frac{R^t - R_p}{1 - R_p}\)$ For conventional videos lacking physical event cameras, EventSTU employs lightweight logarithmic frame differencing. Consecutive RGB frames \(F_t\) and \(F_{t-1}\) undergo inverse gamma correction \(\gamma^{-1}\) to calculate logarithmic intensity changes \(\Delta L_{t, t-1} = \log(\gamma^{-1}(F_t)) - \log(\gamma^{-1}(F_{t-1}))\), followed by polarity thresholding. This avoids computationally heavy microsecond interpolation, executing in just 1.52 ms and generalising event-guided efficiency to standard video benchmarks.

A Worked Example

Consider an aerial drone video query: "Did the black vehicle make an illegal lane change at the curve?" 1. Temporal Coarse Filtering: The original video contains 2,000 frames. While cruising along straight stretches, brightness variation is low, and accumulated events remain below threshold \(\tau\). When the drone turns or vehicles maneuver across lanes, event counts surge past \(\tau\). C2FS quickly prunes 2,000 frames down to 500 candidate frames. 2. Temporal Fine Selection: The 500 frames are divided into 32 temporal bins. Calculating text matching scores with "black vehicle illegal lane change" retrieves the best match from each bin, yielding 32 final keyframes spanning the entire maneuver. 3. Adaptive Budget Allocation: Three keyframes capturing the vehicle crossing lane markers receive high similarity scores \(s^t\), securing an 85% token retention ratio, whereas non-critical transit frames receive only a 15% ratio. 4. Spatial Dual Selection: Within the lane-change frames, rapid vehicle movement triggers dense events, and physics-aware selection keeps the car and lane boundaries (60% of tokens). Semantic attention reuse subsequently identifies a static overhead traffic sign (25% of tokens), while irrelevant sky and lawn regions are pruned. The LLM receives an ultra-compact visual token stream and outputs the correct answer instantaneously.

Loss & Training

EventSTU is a fully training-free framework that requires no fine-tuning or gradient updates on the vision encoder or language model. Experiments are conducted on 8 NVIDIA RTX 3090 GPUs with default coarse sampling ratio \(S = 25\%\) and fine sampling target \(M_f = 32\). LLaVA-OneVision-7B uses \(N=196\) tokens per frame, while Qwen3-VL adopts dynamic resolution (up to \(640 \times 28 \times 28\)). For static slide-presentation videos where motion is absent, a lightweight camera-shake perturbation trick is incorporated during event simulation to ensure continuous token differentiation.

Key Experimental Results

Main Results

EventSTU is benchmarked on the real-world physical event benchmark EventBench (EB, 500 expert-annotated multiple-choice questions across 8 tasks) and three widely adopted general video benchmarks using simulated events (Video-MME, LongVideoBench, and MLVU).

Performance and computational costs on LLaVA-OneVision-7B are summarized below:

Method	Retention Ratio	FLOPs (T) ↓	FLOPs Ratio ↓	EventBench ↑	Video-MME ↑	LongVideoBench ↑	MLVU ↑	Avg. ↑
LLaVA-OV-7B (Baseline)	100%	41.71	100%	52.50	58.48	56.40	64.81	58.05
Keyframe Sampling Only
BOLT	100%	170.6	409%	-	59.90	59.60	66.80	-
AKS	100%	170.6	409%	54.09	60.44	60.28	69.41	61.06
C2FS (Ours)	100%	79.34	190%	55.09	60.96	60.43	69.87	61.59
Token Pruning Only
FastV	50%	21.61	51.8%	50.90	57.74	56.10	63.98	57.18
LLaVA-PruMerge	50%	19.24	46.1%	51.90	58.04	55.95	64.63	57.63
DyCoKe	50%	24.45	58.6%	52.50	59.04	55.80	64.54	57.97
FastVID	50%	19.21	46.1%	52.69	58.33	56.55	64.49	58.02
HoliTom	50%	19.21	46.1%	52.69	58.52	56.62	65.87	58.43
ZAP (Ours)	50%	19.20	46.0%	54.09	59.15	57.37	65.69	59.07
Sampling + Pruning Combined
AKS + FastV	50%	150.5	361%	53.49	60.22	60.28	69.27	60.82
AKS + HoliTom	50%	148.1	355%	54.29	60.78	59.99	70.06	61.28
EventSTU (Ours)	50%	56.83	136%	55.09	60.70	61.11	70.98	61.97
AKS + FastV	30%	143.4	344%	51.10	58.89	58.56	69.41	59.49
AKS + HoliTom	30%	140.2	336%	51.90	60.78	59.01	70.84	60.63
EventSTU (Ours)	30%	48.93	117%	54.49	61.07	60.73	71.16	61.86

On Qwen3-VL-8B, where dynamic resolution generates substantially more tokens, EventSTU shows even stronger advantages: - Baseline Qwen3-VL: FLOPs of 169.1T (100%), average score 66.60. - EventSTU (50% retention): FLOPs reduced to 104.4T (62%), average score increases to 69.77 (+3.17). - EventSTU (30% retention): FLOPs drop to 73.64T (44% of baseline), average score remains at 69.29 (+2.69).

Ablation Study

1. Sampling Strategy Ablations (evaluated on EventBench and LongVideoBench):

Stage	Strategy	Description	EventBench ↑	LongVideoBench ↑
Coarse	UNI	Uniform interval sampling	53.09	59.54
Coarse	TOP	Select frames with highest instantaneous event density	54.09	58.94
Coarse	ACC (Ours)	Adaptive threshold triggering via event accumulation	55.09	60.43
Fine	TOP	Greedy global top-similarity frame retrieval	53.89	60.21
Fine	BIN (Ours)	Temporal binning with intra-bin best matching	55.09	60.43

2. ZAP Component Ablations:

Config	Semantic-Aware (Sem.)	Physics-Aware (Phy.)	Adaptive Ratio (Ada.)	EventBench ↑	LongVideoBench ↑	Note
Sem. only	✓	×	×	52.10	58.12	Evaluates token importance via attention only
Phy. only	×	✓	×	53.29	58.64	Selects tokens via event density only
Phy. + Sem. fixed	✓	✓	×	54.09	59.54	Fixed per-frame retention ratio
Full ZAP	✓	✓	✓	55.09	61.11	Adaptive allocation yields +1.28 gain

Key Findings

• Overcoming the Efficiency-Performance Trade-off: At a 30% retention ratio, EventSTU achieves an average accuracy improvement of 3.81 points over the original LLaVA-OneVision baseline. This confirms that uninformative background tokens and redundant frames serve as noise that hinders attention; pruning them directly improves the visual signal-to-noise ratio.
• Substantial Time-to-First-Token (TTFT) Speedup: Prior outer-LLM pruning methods (e.g., PruMerge, HoliTom) incur significant preprocessing latency for similarity computation and clustering. In contrast, ZAP introduces only 1.24 ms of pre-processing overhead, cutting TTFT by 66.2% relative to the original model.
• Broad Model Generalizability: Without any retraining, EventSTU delivers an average gain of 3.19 points across LLaVA-Video-7B (+2.79), Qwen2.5-VL-7B (+3.39), and InternVL3.5-8B (+3.39) on EventBench.

Highlights & Insights

• Bio-Inspired Physical Prior: By framing event camera triggers as a universal proxy for information increment, EventSTU replaces costly similarity matrices and neural ranking with near-zero-cost (0.00027%) integer event counting.
• Decoupled Redundancy Removal and Semantic Alignment: Temporal sampling separates content-agnostic motion triggering from query-conditioned binning; spatial pruning separates physical background stripping from attention-based semantic detail preservation.
• Physical-Simulated Dual Modality: By constructing the real-world benchmark EventBench while providing a lightweight frame-difference simulator, the framework bridges specialized neuromorphic hardware with mainstream general video models.

Limitations & Future Work

• Degradation in Extreme Environments: In near-total darkness or under violent, uncontrolled camera jitter, event streams can be dominated by sensor thermal noise or global event triggering, undermining physical saliency filtering.
• Simulation Fidelity for Sub-Pixel Dynamics: Because standard video feeds have bounded frame rates and compression artifacts, simulated frame differences cannot reproduce true microsecond-level neuromorphic responses.
• Future Directions: Exploring end-to-end adaptive neuromorphic-RGB neural fusion and deploying event-guided streaming models onto low-power edge robotics and autonomous platforms.

Related Work & Insights

• vs AKS / BOLT: AKS and BOLT improve frame quality at the cost of exceeding baseline FLOPs by over 300% due to dense multimodal scoring; EventSTU's C2FS filters redundant frames upfront, cutting sampling FLOPs by 53.5% relative to AKS while achieving higher accuracy.
• vs LLaVA-PruMerge / FastVID: These methods calculate token-to-token similarity matrices, causing quadratic pre-processing latency. ZAP leverages instant event counting and existing attention matrices, adding only 1.24 ms and reducing TTFT by 66.2%.
• vs EventCLIP / EventGPT: Prior event-based models were limited to short (<100ms) event recognition. EventSTU is the first to employ event streams as an efficiency accelerator for minutes-long Video-LLM reasoning.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneering the dual use of event camera principles for both temporal keyframe sampling and spatial token pruning]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Builds the 500-question EventBench benchmark, evaluates across 4 diverse benchmarks and 5 mainstream Video-LLM architectures]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear hierarchical structure, well-defined mathematical formulations, and compelling visualizations]
• Value: ⭐⭐⭐⭐⭐ [Completely training-free and plug-and-play, effectively resolving long-video computational bottlenecks for multimodal LLMs]