Dense Video Understanding with Inter-tokenization Acceleration

Conference: ECCV2026
Paper: Official page · PDF
Project: DIVE-Bench
Area: VLM Efficiency
Keywords: High-FPS video understanding, gated residual tokenization, motion compensation, scene merging, DIVE-Bench

TL;DR

The paper defines high-FPS video understanding through DIVE-Bench, where nearly every frame carries evidence, and proposes Gated Residual Tokenization (GRT), which encodes only changed patches before merging semantically redundant scenes, reducing tokenization latency by 53.6% at 1 FPS while improving answer quality on dense-text and high-motion tasks.

Background & Motivation

Current video large language models usually sample a small set of frames uniformly and then turn every selected frame into visual tokens. This engineering compromise works when scenes change slowly and questions concern global semantics, but it assumes that skipped frames contain no unique evidence. Brief subtitles in educational videos and fast hand motion in egocentric footage violate that assumption: halving the sampling rate can erase an entire subtitle or short action, and a fixed frame budget drives effective FPS even lower as videos get longer.

Simply increasing FPS creates two coupled bottlenecks. The vision encoder applies convolution and a Transformer to every patch of every frame even when most regions remain static, so tokenization time and token count grow approximately linearly with frame count. Once those visual tokens enter the language model, self-attention grows quadratically with sequence length. Worse, common video-QA benchmarks mostly ask about activity or object presence and can reward systems that inspect as few as six frames, hiding the cost of lost temporal detail.

The paper addresses both evaluation and representation. DIVE-Bench uses continuously changing subtitles and palm-grid trajectories to make dense evidence mandatory. GRT borrows the key-frame/predictive-frame structure of video codecs and handles change during tokenization rather than only after tokens have been produced. Core idea: instead of fully encoding every frame and removing redundant tokens afterward, let key frames carry static scenes, encode only motion residuals from subsequent frames, and then merge repeated scenes at the semantic level.

Method

Overall Architecture

The input is a high-FPS video plus a text query, and the output still comes from a standard video LLM. GRT does not train a new vision backbone. It inserts motion-compensated gating inside a pretrained ViT tokenizer and performs semantic scene merging after visual tokens are produced; the compressed visual sequence is linearly projected, concatenated with query tokens, and passed to a LLaVA-OneVision/Qwen2-based model.

The pipeline has three contribution stages. DIVE-Bench defines tasks that require frame-aligned evidence. Motion-compensated gated tokenization removes static patches before expensive visual encoding. Semantic scene merging then removes redundant key-frame representations while retaining motion tokens in temporal order.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["High-FPS video + query"] --> B["DIVE-Bench dense supervision<br/>subtitle sequence / palm trajectory"]
    B --> C["Motion-compensated gated tokenization<br/>key frame + changed patches"]
    C --> D["Semantic scene merging<br/>merge static, preserve motion"]
    D --> E["Linear projection + query tokens"]
    E --> F["Video LLM answer"]

Key Designs

1. DIVE-Bench dense supervision: make information loss from frame skipping measurable

DIVE-Bench treats high FPS as a distinct capability rather than another sampling setting for long-video QA. Its educational partition reuses lecture videos longer than 30 minutes and uses rapidly changing embedded subtitles as ground truth. Recovering the subtitle sequence requires dense observation, so the benchmark measures completeness with CER, WER, Token-F1, Exact Match, and a 0–5 MOS. This partition contains 317 videos and 317 QA pairs, averaging 44.26 source FPS and 72,068.5 frames per video.

The high-motion partition comes from egocentric hand videos. A \(3\times3\) grid is overlaid on each frame, turning the palm center from a continuous coordinate into a label such as r1c2; Grid ADE/FDE then measure trajectory-wide and final-frame location errors. It contains 277 videos and 3,243 clips, averaging 30 FPS and 254.8 frames per clip. The two partitions cover dense textual change and dense physical motion, ensuring that global semantics cannot conceal skipped evidence.

2. Motion-compensated gated tokenization: admit only changed patches before visual encoding

Conventional token pruning pays the cost of encoding a full frame before reducing its tokens. GRT first compares patches in adjacent frames. Each scene is represented by one complete key frame, while subsequent P-frames retain only patches whose structural similarity to the preceding frame falls below threshold \(\tau\); every key-frame gate is fixed to 1. The residual representation is:

\[ \Delta f_{s,k+j}=M_{s,k+j}\odot\left(f_{s,k+j}-f_{s,k+j-1}\right), \qquad f_{s,k+i}\approx f_{s,k}+\sum_{j=1}^{i}\Delta f_{s,k+j}. \]

To reuse a pretrained ViT, the authors flatten its non-overlapping patch convolution, whose stride equals its kernel size, into an equivalent MLP. Selected patches use the original convolution weights; masked positions receive zero placeholders so positional encodings remain correct, and placeholders are filtered before the sequence enters multi-head attention. This avoids retraining the tokenizer while preventing static regions from traversing the full vision encoder, enabling tokenization cost to grow sub-linearly with frame count.

3. Semantic scene merging: remove repeated static semantics without sacrificing motion residuals

Pixel differences alone cannot determine whether two scenes are semantically equivalent, and minor camera shake can trigger many residual patches. The second stage therefore uses semantic representations already produced by the pretrained tokenizer and compares normalized key-token distributions for adjacent scenes. The final formulation selects the most similar adjacent scene using Jensen-Shannon divergence:

\[ D_{\mathrm{JSD}}(s,t)=\mathrm{JSD}\!\left(P(T_{s,k}),P(T_{t,k})\right). \]

During a merge, the mean representations of the two key-token sets are averaged, while their P-token sequences are concatenated in temporal order. Compression therefore targets repeated static key frames, which dominate token count, rather than motion residuals that carry brief changes. At 1 FPS this stage reduces retention from 90% after gating to 14% of the original tokens, yet MOS rises only from 1.93 to 1.94, identifying it primarily as an efficiency component rather than the source of quality gains.

A Worked Example

Consider a lecture video. The first frame fully encodes the lecturer, slide background, and current subtitle; in the next frame only the subtitle ending and a hand gesture change. Gated tokenization admits those changed patches, keeps zero placeholders for the static background long enough to preserve position, and excludes them from the Transformer. A slide transition creates a new key frame. If later scenes still show the same lecturer and a similar layout, semantic scene merging retains one merged key representation but concatenates every subtitle and gesture P-token in order. The LLM receives neither sparse frames nor repeated full backgrounds, but a compressed temporal record of “static scene plus continuous change.”

Loss & Training

GRT is a zero-shot, inference-only method and introduces no training loss for gating or scene merging. Experiments use 0.5B and 7B variants based on LLaVA-OneVision and Qwen2, split \(224\times224\) frames into \(16\times16\) patches, and run FP16 inference on RTX 4090, RTX A6000 Ada, and H200 GPUs. Tokenization time covers raw frame extraction through completion of the token sequence and is averaged over 50 videos with LMMS-Eval; educational-video answers receive 0–5 MOS ratings from GPT-3.5 against ground truth.

Key Experimental Results

Main Results

Setting / Dataset	Method	Quality Metric	Efficiency / Motion Metric
DIVE-Bench educational	LLaVA-OneVision 0.5B	MOS 2.01	—
DIVE-Bench educational	GRT 0.5B	MOS 2.50	—
DIVE-Bench high-motion	LLaVA-OV baseline	—	4.49 FPS / ADE 1.3618 / FDE 1.6085
DIVE-Bench high-motion	GRT	—	8.69 FPS / ADE 1.1456 / FDE 1.1828
MLVU	LLaVA-OneVision 0.5B → GRT	33.002 → 34.066	+1.064
VideoMME	LLaVA-OneVision 0.5B → GRT	43.963 → 44.037	+0.074

The 0.5B GRT model's educational MOS also exceeds the reported LLaVA-Video 7B score of 1.47 and LLaVA-OneVision 7B score of 1.70. On high-motion video, effective FPS increases by 93.5% while ADE falls by about 15.9% and FDE by about 26.5%. This matters more than speed alone because it shows that retaining denser frames improves motion localization.

Ablation Study

Gated Tokenizer	Scene Merge	MOS	Change vs. Baseline	Interpretation
No	No	1.66	—	Full FPS without gating or merging
Yes	No	1.93	+0.27	Early static-patch filtering supplies the main quality gain
Yes	Yes	1.94	+0.28	Scene merging adds compression but only 0.01 MOS

Input FPS	LLaVA-OV Time	GRT Time	Speedup	Tokens after Gating	Tokens after Scene Merge
0.01	0.0170 s	0.0174 s	-2.4%	100%	100%
0.1	0.0186 s	0.0177 s	4.8%	96%	33%
1.0	0.0487 s	0.0226 s	53.6%	90%	14%

Key Findings

• Gated tokenization is the quality-preserving core: it contributes 0.27 MOS by itself, while scene merging adds only 0.01 MOS and primarily compresses tokens at high FPS.
• GRT is not universally faster. Its fixed gating overhead makes it 2.4% slower at 0.01 FPS, while the 53.6% gain emerges at 1 FPS, matching its intended dense-video regime.
• On conventional low-FPS benchmarks, MLVU improves by 1.064 and VideoMME by only 0.074. The method's advantage is concentrated in dense temporal evidence rather than general long-video semantic reasoning.
• Semantic and motion fidelity are supported indirectly: MLVU Ego-centric rises from 37.7 to 39.6 and Anomaly Recognition from 30.8 to 38.5, but the paper lacks a benchmark with pixel-level missed-motion recall.

Highlights & Insights

• Move pruning inside the tokenizer: Most compression methods first create visual tokens and only then prune them. GRT uses cheap patch differences to decide which locations deserve visual encoding, directly removing an earlier source of redundant computation; the idea transfers naturally to fixed cameras, screen recordings, and continuous robot observations.
• Compress static and dynamic information differently: Semantic distance deduplicates key frames, while P-frames retain temporal residuals. This mirrors the division of labor between I/P frames in conventional codecs but changes the objective from pixel reconstruction to VLM reasoning.
• Co-design the benchmark and method: DIVE-Bench deliberately creates tasks where missing a frame can remove the answer, preventing redundant global semantics from producing misleadingly strong results. Grid-based palm trajectories also convert coordinate regression into auditable discrete QA compatible with language models.

Limitations & Future Work

• DIVE-Bench currently covers subtitle/OCR streams and grid-based palm trajectories. These are dense signals but do not represent all high-FPS semantics; real-time interaction, games, surgery, and fine-grained causal events remain untested.
• The high-motion partition retains 244 invalid or empty grid parses (7.52%) and 171 repeated question patterns (5.27%). They are explicitly released as audit flags but may still affect metric stability.
• Gating relies on adjacent-patch SSIM and a fixed threshold. Camera shake, occlusion, or tiny but crucial changes can cause false retention or deletion. The paper acknowledges the absence of pixel-level motion ground truth for directly measuring missed-motion recall; MLVU subtasks are only proxy evidence.
• The paper claims sub-linear tokenization growth, but the main text reports short-video latency at only 0.01, 0.1, and 1 FPS. Complexity curves across video lengths, resolutions, and hardware are still needed, and GRT is slower at very low FPS.
• The implementation still consumes sampled decoded frames. Future systems could read codec key frames, motion vectors, and residual streams directly, avoiding repeated motion estimation and using content-adaptive thresholds to protect small but important changes.

Related Work & Insights

• vs. sparse or adaptive frame sampling: LLaVA-OneVision-style systems cap total frames, while Flexible-FPS allocates more frames to high-motion segments but still fully encodes every selected frame. GRT keeps a denser timeline and removes redundancy spatially and semantically, making it better suited to evidence that changes almost every frame.
• vs. post-tokenization pruning or merging: Post-processing shortens the language-model input but cannot recover vision-encoder compute already spent. Motion-compensated gating filters before patch embedding, and scene merging controls sequence length before the LLM; the two stages target different cost centers.
• vs. conventional video codecs: A codec optimizes visual reconstruction, whereas GRT borrows key/residual structure for question-answer semantics. A natural extension is to consume compressed bitstreams directly and use task-dependent semantic risk, rather than pixel error, to decide which residuals must reach the VLM.

Rating

• Novelty: ⭐⭐⭐⭐ The combination of a high-FPS QA task and codec-style gating inside the tokenizer is sharply targeted, although key/residual coding itself is established.
• Experimental Thoroughness: ⭐⭐⭐⭐ The paper covers quality, motion error, latency, token retention, ablations, and conventional benchmarks, but lacks direct missed-motion recall and broader complexity curves.
• Writing Quality: ⭐⭐⭐⭐ The problem, two-stage method, and efficiency gains are clear, though the methodology presents both cosine distance and JSD variants for scene similarity somewhat redundantly.
• Value: ⭐⭐⭐⭐ Reversing the default “fully encode, then compress” order is transferable to high-FPS VLMs, continuous robot perception, and long-video infrastructure.