Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?¶

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Area: Multimodal VLM
Keywords: Long Video Understanding, Deep Audio-Visual Coupling, Spatio-Temporal Evidence Grounding, Multimodal Causal Reasoning, Token Compression

TL;DR¶

To eliminate visual heuristics and ungrounded speculation in long video benchmarks, this paper introduces Video-HolmesV2 (784 long videos averaging 43 minutes, 4,000 QA pairs) requiring strict spatio-temporal audio-visual evidence grounding, accompanied by an audio-text guided token compression framework that outperforms full-token baselines.

Background & Motivation¶

Recent advancements in omni-modal foundation models—such as GPT-4o, Gemini 2.5 Pro, and Qwen2.5-Omni—have enabled native processing of interwoven audio and video streams across extended horizons. However, benchmarking paradigms have severely lagged behind these architectural breakthroughs. Dominant long-video benchmarks like Video-MME and LongVideoBench remain heavily visual-centric, treating the auditory stream as a negligible or secondary signal. Consequently, state-of-the-art models frequently act as "silent observers" that bypass genuine cross-modal causality, exploiting unimodal visual heuristics to correctly guess answers without understanding the underlying narrative.

Furthermore, traditional multiple-choice evaluation metrics fail to gauge authentic multimodal reasoning. Because models are only scored on their final selection, parametric memory shortcuts and hallucinated evidence are conflated with rigorous deduction. On the computational side, capturing fleeting evidence in hour-long films mandates high-density frame sampling, which inevitably causes catastrophic token explosion and attention dilution. Prevailing token pruning algorithms rely on naive top-K selection or unimodal signals, inadvertently severing cross-modal dependencies and stripping away indispensable macroscopic context.

This work directly resolves this dual dilemma by formulating an evaluation standard anchored in factual audio-visual evidence and developing an efficient token allocation strategy. Core idea: establish an extreme long-form benchmark Video-HolmesV2 requiring deep audio-visual interlock and spatio-temporal evidence chains, introduce an evidence-conditioned metric penalizing ungrounded guessing, and design an audio-text guided token compression framework with core deduplication and background scavenging.

Method¶

Overall Architecture¶

The Video-HolmesV2 ecosystem integrates three primary pillars: benchmark construction with adversarial filtering, spatio-temporal evidence-aware evaluation, and dual-stream guided token compression. During curation, high-fidelity narrative films are vetted via multi-dimensional analyzability scoring and unimodal trap ablation to guarantee multimodal interdependence. The evaluation protocol combines Gaussian soft temporal alignment and semantic scoring, strictly conditioning reasoning points on valid evidence retrieval. At test time, a lightweight student selector and audio self-attention anchors evaluate visual token saliency, followed by intra-set deduplication and background scavenging within a fixed sequence budget.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Long Video Input<br/>(Frames + Audio Stream + Query)"] --> B["Deep Audio-Visual Curation & Unimodal Trap Filtering"]
    B --> C["Spatio-Temporal Evidence-Aware Evaluation Protocol"]
    C --> D["Audio-Text Dual-Stream Saliency Scoring"]
    D --> E["Masked Differentiated Token Compression & Reassembly"]
    E --> F["Faithful Multimodal Long-Range Causal Reasoning"]

Key Designs¶

1. Deep Audio-Visual Curation & Unimodal Trap Filtering: Eliminating Unimodal Shortcuts To guarantee that answers cannot be deduced from isolated modalities, the data curation protocol implements an adversarial "Unimodal Trap" blind filter. Candidate QA pairs generated from cinematography narratives are evaluated under Visual-Only (silent frames) and Audio-Only (audio and subtitles) ablated conditions. Any question that can be answered with high confidence by a single modality is immediately discarded as a pseudo-multimodal shortcut. The surviving 4,000 QA pairs strictly necessitate synthesizing visual and acoustic evidence to form a coherent deductive chain.

2. Spatio-Temporal Evidence-Aware Evaluation Protocol: Penalizing Ungrounded Hallucination Traditional binary multiple-choice accuracy fails to verify whether a model actually observed the factual event. The evaluation protocol introduces a composite evidence score \(S_{\text{evidence}}\). To accommodate minor temporal drift without harsh cutoffs, length-similarity is multiplied by a Gaussian decay factor:

\[S_{\text{raw}}(p, g) = \frac{\min(L_p, L_g)}{\max(L_p, L_g)} \cdot \exp\left(-\frac{(C_p - C_g)^2}{2\sigma_{\text{drift}}^2}\right)\]

Coupled with an \(F_{0.5}\)-based semantic score \(S_{\text{sem}}\) that penalizes hallucinated claims, the overall benchmark performance strictly couples evidence quality with answer accuracy:

\[S_{\text{total}} = \lambda_1 \cdot \text{Acc}_{\text{MCQ}} + \lambda_2 \cdot (\text{Acc}_{\text{MCQ}} \cdot S_{\text{evidence}})\]

with \(\lambda_1 = 0.6, \lambda_2 = 0.4\). The conditional multiplier \((\text{Acc}_{\text{MCQ}} \cdot S_{\text{evidence}})\) ensures that evidence grounding is credited only when the final choice is correct, effectively exposing models that guess lucky answers while hallucinating evidence.

3. Audio-Text Dual-Stream Saliency Scoring: Unifying Subjective Query and Objective Events To prioritize visual tokens without incurring massive autoregressive overhead, the framework unifies top-down textual queries with bottom-up acoustic peaks. For textual guidance, a compact 0.5B student model is trained via Spearman rank correlation loss to mimic the cross-modal attention of an intermediate layer in a 72B teacher model, producing ranking scores \(S_{\text{text}}\). Concurrently, a training-free acoustic mechanism extracts salient audio anchors from the final self-attention layer of the audio encoder and computes maximum cosine similarity across visual patches to yield \(S_{\text{audio}}\). Both streams are normalized and linearly combined into a unified metric \(S_{\text{final}} = \alpha_{\text{fuse}} \tilde{S}_{\text{text}} + (1 - \alpha_{\text{fuse}}) \tilde{S}_{\text{audio}}\) (\(\alpha_{\text{fuse}} = 0.7\)).

4. Masked Differentiated Token Compression & Reassembly: Core Deduplication and Background Scavenging Rather than discarding all lower-ranked tokens, the framework splits visual features into a Core Set (\(M=1\)) and a Background Set (\(M=0\)) based on \(S_{\text{final}}\). Within the Core Set, spatially or temporally contiguous tokens whose visual cosine similarity exceeds threshold \(\tau_{\text{intra}}\) are merged via average feature pooling, liberating a dynamic quota \(R = B_{\text{vis}} - n\). This freed budget is scavenged to rescue the top \(M \times R\) tokens from the Background Set, which undergo aggressive \(M:1\) pooling (e.g., \(M=4\)) to fit the remaining budget. By preserving original spatio-temporal positional embeddings, the final sequence maintains sharp micro-evidence alongside essential global narrative context.

Loss & Training¶

The lightweight student selector (Qwen2.5 0.5B) is trained on 50,000 long-video sequences sampled from LLaVA-Video-178k using a differentiable Spearman rank correlation loss:

\[\mathcal{L}_{\text{rank}} = 1 - \frac{\sum_{i=1}^N (R(r_i^{\text{ref}}) - \bar{R}^{\text{ref}})(R(\hat{r}_i) - \bar{R})}{\sqrt{\sum_{i=1}^N (R(r_i^{\text{ref}}) - \bar{R}^{\text{ref}})^2 \sum_{i=1}^N (R(\hat{r}_i) - \bar{R})^2}}\]

Optimization is performed using AdamW across 4 \(\times\) H20 GPUs with batch size 8 and learning rate \(1 \times 10^{-5}\). The audio-guided scoring and differentiated compression stages are fully training-free at test time.

Key Experimental Results¶

Main Results¶

On Video-HolmesV2, closed-source foundation models and open-source omni-modal models were comprehensively evaluated (temperature set to 0.01, maximum generation length set to 8,192 tokens).

Category	Model	Perception Ave.	Reasoning Ave.	Overall Score
Proprietary	GPT-4o	56.1	46.4	46.9
Proprietary	Gemini-2.5-Flash	67.0	65.2	65.3
Proprietary	Gemini-2.5-Pro	75.8	73.1	73.2
Open-Source	Qwen3-VL-4B	47.3	27.0	28.0
Open-Source	Qwen2.5-VL-7B	41.9	37.9	38.1
Open-Source	MiniCPM-V-4.5	38.6	47.4	47.0
Open-Source	InternVL3-8B	50.6	48.0	48.1
Open-Source	Qwen2.5-VL-72B	52.2	50.1	50.2
Open-Source	Qwen3-VL-235B-A22B	58.5	55.1	55.3
Open-Source Omni	Qwen2.5-Omni-7B (Baseline)	42.9	40.6	40.7
Token Compression	Ours (Qwen2.5-Omni-7B + Compression)	46.7	44.4	44.5 (↑3.8)

When enforcing evidence-grounded conditional scoring, the gap between MCQ guessing and verified multimodal understanding becomes prominent:

Model	MCQ Reasoning Accuracy (Ave. %)	Evidence Score (Ave.)	Overall Conditional Score (\(S_{\text{total}}\))
Qwen3-VL-235B-A22B	55.1	8.5	34.9
Gemini-2.5-Flash	65.2	28.6	46.6
Gemini-2.5-Pro	73.1	35.2	54.2

Ablation Study¶

Ablation experiments on VideoMME and WorldScene validate the individual components of the proposed compression framework (using Qwen2.5-Omni-7B):

Text-Guided (TGC)	Audio-Guided (AGC)	Dual-Stream & Differentiated (DSFDC)	VideoMME Accuracy (%)	WorldScene Accuracy (%)
✗	✗	✗ (Full Tokens)	66.0	46.8
✓	✗	✗	67.0	46.5
✗	✓	✗	63.3	45.1
✓	✓	✗	67.4	46.9
✓	✓	✓ (Full Framework)	67.8	47.2

In token allocation studies, scaling visual/audio budgets from 4096/1024 (66.5 on VideoMME) to 8192/1024 (67.4) and 8192/2048 yielded the peak accuracy of 67.8; further scaling to 12288/4096 dropped accuracy to 67.6 due to attention dilution from redundant tokens.

Key Findings¶

Deploying audio-guided compression in isolation degrades accuracy significantly (VideoMME dropping from 66.0% to 63.3%) because it inadvertently strips silent narrative frames carrying critical visual causality. Fusing audio with text guidance resolves this and surpasses the uncompressed baseline.
Massive open-source models suffer an "illusion of reasoning": Qwen3-VL-235B-A22B achieves a respectable 55.1% MCQ accuracy, yet its evidence score drops to 8.5, indicating reliance on parametric memory and guesswork. Gemini-2.5-Pro maintains a balanced 35.2 evidence score.
Simply enlarging context windows and frame counts without filtering does not solve ultra-long video reasoning; attention dilution sets in at excessive token counts, demonstrating the necessity of fixed-budget differentiated compression.

Highlights & Insights¶

Adversarial Unimodal Trap Protocol: Employs double-blind unimodal ablation to automatically discard samples solvable by vision or audio alone, enforcing authentic multimodal reasoning.
Evidence-Conditioned Multiplicative Metric: Conditions evidence credit strictly on answer correctness, effectively penalizing lucky guesses and ungrounded hallucinations.
Quota-Recycling Differentiated Compression: Merges redundant patches within the core salient set and repurposes freed token quotas to salvage pooled global background tokens, beating uncompressed full-token baselines (101.5% relative performance).

Limitations & Future Work¶

Heuristic Sensitivity of Temporal Drift Tolerance: While Gaussian decay mitigates rigid IoU truncation, \(\tau_{\text{drift}}\) remains tied to average event durations and may penalize ultra-brief micro-actions.
Evidence Extraction Deficit in Smaller Models: Models under 30B parameters struggle to generate structured spatio-temporal timestamps and bounding boxes, showing near-zero grounding capability.
Absence of Binaural Spatial Acoustics: Current audio modeling relies primarily on temporal attention peaks rather than multi-channel spatialized sound localization to direct visual attention.

vs Video-MME / LongVideoBench: Conventional long video benchmarks treat audio as an optional bystander; Video-HolmesV2 enforces cross-modal coupling where neither modality alone is sufficient.
vs Video-Holmes (V1) / WorldSense: V1 was restricted to shorter clips and text-only reasoning chains; V2 scales to 600-7500s narratives with precise spatio-temporal audio-visual timestamps and bounding box annotations.
vs OmniZip / FastV: While previous compression algorithms apply rigid Top-K pruning that loses macroscopic narrative atmosphere, this framework dynamically scavenges freed quotas to retain global context.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ First benchmark enforcing deep audio-visual coupling with strict evidence grounding alongside a quota-recycling token compression framework.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensive evaluations across models from 4B to 235B parameters, proprietary baselines, and multi-benchmark ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Rigorous task formulation, clear mathematical definitions, and coherent narrative structure.
Value: ⭐⭐⭐⭐⭐ Offers crucial guidance for transitioning MLLMs from heuristic statistical guessing to grounded fact-based video reasoning.