LongVQUBench: Benchmarking Long-Term Video Quality Understanding of Vision-Language Models¶
Conference: ECCV 2026
Paper: ECCV Official
Project: https://longvqubench.github.io
Area: Multimodal VLM
Keywords: Video Quality Understanding, Long-Term Video Benchmark, Large Vision-Language Models, Needle Distortion QA, Temporal Quality Reasoning
TL;DR¶
Addressing the limitation of existing video quality benchmarks that focus on short clips and isolated artifacts while neglecting cumulative degradation and deep reasoning, this paper introduces LongVQUBench, the first comprehensive benchmark featuring 1,200+ diverse videos (spanning minutes up to two hours) and 1,500 QA pairs across a three-level hierarchy (LQU, CQR, GQU) with needle distortion QA, demonstrating significant performance drops in 14 state-of-the-art LVLMs as temporal horizon and reasoning depth increase.
Background & Motivation¶
With the rapid evolution of large vision-language models (LVLMs), architectures such as GPT-5, Gemini-3, and Qwen2.5-VL have demonstrated exceptional capabilities across fine-grained image recognition and long-context video question answering. However, multimodal research in video understanding has predominantly fixated on high-level semantic tasks—such as action categorization, event causality reasoning, and narrative tracking—while leaving low- and mid-level sensory perceptual quality understanding largely neglected. In real-world viewing conditions spanning minutes to hours, video quality is dictated not only by instantaneous spatial clarity, but also by temporal continuity, subjective viewing comfort, and cumulative perceptual degradation (e.g., progressive color drifts, intermittent stuttering, or repeated flicker that induces cumulative fatigue).
Traditional video quality assessment (VQA) approaches, ranging from classical metrics (PSNR, SSIM) to deep no-reference models (BRISQUE, VIDEVAL, DOVER), focus primarily on mapping short, isolated clips to scalar mean opinion scores (MOS), lacking conversational reasoning and explanatory causal attribution. Meanwhile, existing long-form LVLM benchmarks such as Video-MME, LongVideoBench, and MLVU evaluate semantic comprehension rather than visual quality degradation. Although Q-Bench-Video pioneered quality understanding in LVLMs, its evaluation remains strictly confined to micro-clips under 10 seconds, incapable of evaluating long-range quality evolution and cross-event degradation dynamics.
Closing the gap between long-video semantic comprehension and fine-grained perceptual quality assessment faces a core barrier: synthesizing long-term, controllable, and unbiased degradation data without narrative confounding. Core idea: build LongVQUBench, the first systematic benchmark for long-term video quality understanding, curating 1,200+ multi-domain videos up to two hours, structuring a three-level hierarchical evaluation spanning local event understanding (LQU), cross-event reasoning (CQR), and global quality understanding (GQU), combined with a needle distortion question-answering (NDQA) paradigm to probe the perceptual sensitivity, temporal attribution, and holistic synthesis of LVLMs.
Method¶
Overall Architecture¶
The construction of LongVQUBench follows an end-to-end pipeline: diverse long-video collection, controlled spatial-temporal distortion injection, three-tier temporal perception staging, and dual-format question-answer annotation with expert arbitration. The dataset encompasses over 1,200 videos drawn from six distinct genres—movies, documentaries, surveillance, egocentric footage, instructional clips, and animated sequences—with durations ranging from a few minutes to two hours (averaging 742.2 seconds). To establish clean and traceable ground truth, videos are divided into 15-second atomic units, where 14 spatial distortions and 4 temporal distortions are systematically applied at three controlled intensity levels to instantiate the Needle Distortion Question-Answering (NDQA) probe.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Long-Form Video Sources<br/>1200+ videos / 0-120 minutes"] --> B["Controlled Distortion Injection Pool<br/>14 spatial + 4 temporal distortions"]
B --> C["Hierarchical Evaluation Framework<br/>LQU / CQR / GQU three-level structure"]
C --> D["Needle Distortion QA Paradigm (NDQA)<br/>Multiple-choice (objective) + Open-ended (reasoning)"]
D --> E["Multimodal Benchmark Output<br/>1500 QA pairs with expert validation"]
Key Designs¶
1. Controlled Distortion Injection Pool: Decoupling degradation types and intensity levels from semantic content Spontaneous degradations in uncurated long videos are entangled with semantic content and lighting shifts, making objective temporal boundary labeling unreliable. LongVQUBench addresses this by collecting primarily high-quality, pristine videos (888 High, supplemented by 218 Medium and 94 Low samples) and applying controlled distortions exclusively to clean content. The distortion bank comprises 14 spatial types (e.g., Gaussian blur, sensor noise, blockiness, hue shift) and 4 temporal types (e.g., camera jitter, frame drops/stuttering, flicker, motion blur), each synthesized at three discrete intensity tiers. Videos are segmented into uniform 15-second clips, and distortions are sparsely injected into designated clips before recombining into the final full-length video, providing rigorous temporal boundaries and unbiased ground truth.
2. Hierarchical Evaluation Framework: Bottom-up cognitive progression from local perception to global attribution To systematically assess model capabilities across expanding temporal horizons, LongVQUBench introduces a three-tiered hierarchical evaluation protocol: - Local Event Quality Understanding (LQU): Evaluates the model's ability to identify isolated, transient quality events (lasting 5 to 20 seconds). It encompasses five dimensions: Detection (identifying artifact presence), Localization (pinpointing start/end intervals), Classification (specifying distortion category), Severity & Comfort Assessment (evaluating perceptual disturbance and viewer annoyance), and Open Reasoning (articulating visual causes). - Cross-Event Quality Reasoning (CQR): Probes 2 to 4 degradation events distributed across separate temporal segments, covering Comparison (ranking relative degradation severity), Cumulative Effect (gauging accumulated viewing fatigue), Integration (synthesizing evidence across disconnected clips), Temporal Relation (inferring temporal clustering or sequence), and Open Reasoning. - Global Quality Understanding (GQU): Evaluates holistic perceptual fidelity across the entire video (from minutes up to two hours). It targets Stability Evaluation (measuring viewing consistency and abrupt transitions), Dominant Factor Identification (isolating the primary source of perceptual decline), Trend Assessment (tracking quality evolution over time), Overall Evaluation (providing a unified global quality rating), and Open Reasoning.
3. Needle Distortion QA Paradigm (NDQA): Dual-track probe combining objective precision with interpretive reasoning Inspired by the needle-in-a-haystack paradigm in long-context language modeling, NDQA sparsely implants subtle spatial and temporal quality artifacts throughout the video timeline without disrupting the semantic narrative flow. The evaluation employs a dual-track strategy. The objective track leverages Multiple-Choice Questions (MCQ) categorized into Yes-or-No (presence), What (classification), Which (comparative severity), and How (intensity and temporal scope) to enable standardized, leakage-free accuracy benchmarking. The interpretive track incorporates open-ended questions requiring free-form natural language explanations of artifact dynamics, causal mechanisms, and perceptual consequences. Open-ended answers are scored for Relevance and Completeness using an automated LLM protocol validated by human expert consensus.
Key Experimental Results¶
Main Results¶
Following frame budget optimization on the 40% validation set, 14 state-of-the-art LVLMs—spanning 3 proprietary frontiers, 7 open-source video models, and 4 agentic architectures—were evaluated on the 60% held-out test split under zero-shot single-forward-pass inference. The table below presents accuracy across the MCQ dimensions (%):
| Model Class | Model | Frame Budget (#F) | LQU Mean | CQR Mean | GQU Mean | Overall Accuracy |
|---|---|---|---|---|---|---|
| Closed-Source LVLMs | GPT-5 | 256 | 65.4 | 82.2 | 60.9 | 69.5 |
| Gemini-3 | 128 | 61.6 | 79.3 | 59.4 | 66.8 | |
| Qwen-VL-Max | 64 | 57.6 | 77.3 | 57.1 | 64.0 | |
| Open-Source Video LVLMs | LLaVA-NeXT-Video | 8 | 44.7 | 70.0 | 54.7 | 56.4 |
| ShareGPT4Video | 16 | 30.4 | 31.7 | 25.3 | 29.1 | |
| Qwen3-VL | 64 | 50.6 | 72.6 | 57.8 | 60.3 | |
| MovieChat | 64 | 30.0 | 40.2 | 36.8 | 35.6 | |
| LLaVA-Video | 8 | 41.6 | 70.5 | 51.3 | 54.5 | |
| VQA2 (Dedicated VQA) | 8 | 51.4 | 64.1 | 47.2 | 54.2 | |
| Long-RL | 64 | 61.9 | 67.1 | 50.8 | 59.9 | |
| Agentic LVLMs | VideoAgent | Adaptive | 28.6 | 40.0 | 37.1 | 35.2 |
| VideoExplorer | Adaptive | 39.4 | 58.1 | 44.9 | 47.5 | |
| LongVT | Adaptive | 50.3 | 50.5 | 35.6 | 45.5 | |
| DeepVideoDiscovery | Adaptive | 69.2 | 72.1 | 56.8 | 66.0 |
Note: LQU, CQR, and GQU mean scores are computed from their respective constituent dimensions (e.g., LQU includes Detection, Localization, Classification, and Severity & Comfort Assessment).
Ablation Study: Impact of Sampled Frame Budgets on Quality Perception¶
On the LongVQUBench validation subset (40%), the effect of uniformly sampling different frame counts (capped at 1 FPS) was ablated across model families, uncovering distinct saturation and degradation patterns:
| Model | Sampled Frames (#frames) | LQU Acc | CQR Acc | GQU Acc | Total |
|---|---|---|---|---|---|
| GPT-5 | 8 | 72.4 | 77.8 | 61.2 | 70.5 |
| 32 | 71.5 | 78.5 | 63.0 | 71.0 | |
| 64 | 74.0 | 80.2 | 64.5 | 72.9 | |
| 128 | 75.5 | 81.6 | 65.0 | 74.0 | |
| 256 | 75.2 | 81.2 | 65.8 | 74.1 | |
| Gemini-3 | 8 | 71.2 | 73.6 | 59.6 | 68.1 |
| 32 | 67.2 | 75.0 | 56.2 | 66.1 | |
| 64 | 67.0 | 74.2 | 58.5 | 66.6 | |
| 128 | 71.0 | 76.5 | 59.2 | 68.9 | |
| 256 | 68.8 | 75.6 | 57.6 | 67.3 | |
| VQA2 (Domain-specific) | 8 | 63.2 | 65.5 | 49.5 | 59.4 |
| 32 | 59.0 | 63.5 | 47.5 | 56.7 | |
| 64 | 59.5 | 64.0 | 52.0 | 58.5 | |
| 128 | 58.0 | 63.0 | 50.5 | 57.2 | |
| 256 | 55.5 | 61.0 | 48.0 | 54.8 | |
| Qwen3-VL | 8 | 51.8 | 59.6 | 49.5 | 53.6 |
| 32 | 54.5 | 60.0 | 50.0 | 54.8 | |
| 64 | 54.0 | 75.0 | 60.4 | 63.1 | |
| 128 | 52.0 | 58.2 | 51.0 | 53.7 | |
| 256 | 49.5 | 56.5 | 49.5 | 51.8 |
Key Findings¶
- Diminishing and negative returns from scaling frame counts: Increasing input frames does not reliably enhance long-term video quality understanding. Proprietary models saturate at 128–256 frames, while domain-specialized models like VQA2 peak at 8 frames (59.4%) and degrade to 54.8% at 256 frames. Uniform dense sampling injects substantial visual redundancy that dilutes attention over localized artifact spikes.
- Consistent cognitive degradation from local to global reasoning: Across all models, accuracy drops sharply from CQR/LQU to GQU. In particular, Trend Assessment in GQU proves extremely challenging; even GPT-5 scores only 48.5%, indicating that tracking non-monotonic quality trends across hour-long videos remains beyond current multimodal capabilities.
- The exploration strategy of agentic LVLMs dictates performance: Agent architectures exhibit polarized results. DeepVideoDiscovery leverages sophisticated iterative search and tool use to reach 66.0% overall accuracy (achieving the leaderboard's top Localization score of 83.3%), whereas heuristic agents like VideoAgent achieve only 35.2%, highlighting the critical role of targeted temporal navigation in quality inspection.
- The "relevant yet incomplete" dilemma in open-ended reasoning: While open-ended answers achieve high Relevance scores across all models (GPT-5: 88.9%, open-source: 74%–81%), Completeness scores collapse universally (GPT-5 reaches only 45.8%, with open-source models below 40%), revealing that LVLMs struggle to provide exhaustive physical explanations and multi-event syntheses.
Highlights & Insights¶
- Extending the "needle in a haystack" test to continuous visual quality: Unlike textual needle retrieval based on explicit semantic keywords, NDQA tests perceptual resilience to subtle spatio-temporal corruptions, exposing representations that fail when low-level signal fidelity drifts.
- The diagnostic power of the three-tier hierarchy: Decoupling quality analysis into LQU (isolated perception), CQR (multi-event integration), and GQU (long-range holistic synthesis) offers a clear roadmap for diagnosing where temporal architectures fail.
- Implications for AI video generation and media streaming: As LVLMs are increasingly deployed to evaluate and reward generative video models (e.g., Sora, Gen-3), their blindness to cumulative flicker and frame drops signals an urgent need for quality-oriented temporal pretraining.
Limitations & Future Work¶
- Domain gap between synthetic and in-the-wild degradations: While 18 diverse distortion types were systematically introduced at three controlled intensities, real-world user-generated videos often feature compound, intertwined authentic degradations (e.g., sensor underexposure compounded with heavy non-linear codec compression).
- Subsampling bottlenecks under fixed context budgets: Standard evaluation protocols rely on sparse uniform sampling, which inevitably risks skipping brief, subtle distortion needles during temporal downsampling.
- Subjective perceptual modeling: The open-ended scoring leverages LLM-based evaluation; aligning these assessments more closely with psychophysical viewing fatigue and biological memory decay remains an open frontier.
Related Work & Insights¶
- vs Q-Bench-Video: Q-Bench-Video pioneered LVLM evaluation on video quality but was restricted to short clips (<10 seconds); LongVQUBench expands the temporal frontier to two hours and formalizes the LQU-CQR-GQU hierarchy.
- vs LongVideoBench & Video-MME: Prior long-video benchmarks emphasize high-level narrative comprehension; LongVQUBench demonstrates that high semantic proficiency does not translate to perceptual quality discrimination.
- vs Traditional VQA Models (DOVER, VQA2): Conventional deep VQA models act as black-box scalar regressors without reasoning; LongVQUBench establishes a benchmark for explainable, conversational, and localized video quality understanding.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes the first benchmark dedicated to long-term video quality understanding with an innovative three-level hierarchy and needle distortion QA.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously evaluates 14 state-of-the-art models across proprietary, open-source, and agentic paradigms, with detailed frame budget and open-ended completeness ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally structured, clear mathematical formulation of quality tiers, and comprehensive empirical validation.
- Value: ⭐⭐⭐⭐⭐ Provides a foundational testbed to guide the development of next-generation multimodal models with robust visual perceptual grounding.