TempCore: Are Video QA Benchmarks Temporally Grounded?¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Video Understanding
Keywords: Video Question Answering, Temporal Grounding, Frame Selection Sensitivity, Benchmark Evaluation, Vision-Language Models
TL;DR¶
The paper introduces Frame Selection Sensitivity (FSS) based on the contrast between most and least relevant frames and the Language Independence Score (LIS), auditing six popular Video QA benchmarks to reveal that merely 5.5%โ31% of samples genuinely demand temporal frame selection, and constructs the compact, temporally grounded benchmark subset TEMPCORE without manual re-annotation.
Background & Motivation¶
Vision-language models (VLMs) have revolutionized image understanding, spurring tremendous interest in extending their perceptual capabilities to long-form video understanding and procedural temporal reasoning. However, video processing faces a fundamental computational bottleneck: real-world videos span minutes to hours and comprise thousands or tens of thousands of frames, while contemporary state-of-the-art VLMs are constrained by finite context windows and GPU memory to process only roughly 16 to 64 frames at a time (with 32 frames serving as the standard working budget). This capacity constraint has spawned a thriving line of research on key-frame selection strategiesโencompassing combinatorial subset ranking, spatio-temporal saliency, hierarchical clustering, and generative index predictionโall anchored on the premise that identifying and feeding the most informative frames is paramount to downstream question answering.
Nevertheless, empirical findings paint an increasingly contradictory picture: on several popular benchmarks, naive uniform or random sampling performs on par with learned, compute-intensive selection modules; concurrently, an expanding body of diagnostic work highlights that VLMs are largely "temporally blind," exhibiting minimal sensitivity to chronological event ordering, action duration, and causal directionality, yet somehow manage to achieve high aggregate benchmark accuracy. This discrepancy points to an uncomfortable tension: if models that lack temporal comprehension still attain top-tier leaderboard scores, do existing Video QA benchmarks actually require temporal reasoning? Could most benchmark questions be solved using a single static snapshot or shortcut language priors alone? If downstream accuracy remains invariant to which frames are displayed, efforts devoted to engineering sophisticated frame selection algorithms risk solving an illusory problem.
To resolve this dilemma, this work develops a model-based diagnostic that monitors downstream VLM accuracy shifts under extreme frame interventions, disentangling the effectiveness of frame selection from benchmark temporal necessity. The core idea is to measure Frame Selection Sensitivity (FSS) via the performance contrast between the most relevant (MAXPROB) and least relevant (MINPROB) frames, coupled with a zero-vision Language Independence Score (LIS) to filter linguistic shortcuts, thereby automatically isolating the truly temporally grounded evaluation subsets TEMPCORE from existing benchmarks.
Method¶
Overall Architecture¶
The TEMPCORE diagnostic and curation pipeline operates in a two-stage sequential architecture: first, full-length video frames are scored against text queries using a vision-language encoder to construct contrastive frame subsets (MAXPROB vs. MINPROB), alongside uniform sampling, a sliding-window oracle ceiling, and a no-frame visual ablation; second, behavioral responses from an ensemble of 8 diverse open-source VLMs are measured to compute per-sample Frame Selection Sensitivity (FSS) and Language Independence Score (LIS), categorizing items into trivial shortcuts, frame-agnostic queries, and temporally grounded samples.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Video Frame Sequence + Question-Answer Pair"] --> B["Stage 1: Frame Scoring with Diversity Constraint<br/>SigLIP2 QA-mode relevance with integer-second grouping"]
B --> C["Stage 2: Frame Selection Sensitivity FSS Diagnostic<br/>Accuracy delta between MAXPROB and MINPROB partitions Key vs. Anti-key"]
C --> D["Stage 3: Language Independence LIS Filtering & 2D Classification<br/>Filter text-solvable trivial items to yield pure TEMPCORE subset"]
D --> E["Output: TEMPCORE Subsets & Temporal Purity Index TPI"]
Key Designs¶
1. Frame Scoring and Diversity-Constrained Sampling: Mitigating Interrogative Bias and Temporal Clustering To assign relevance scores to video frames \(f_i\) given query text \(t\), the framework leverages a pretrained SigLIP2-400M vision-language encoder computing sigmoid cosine similarity \(s(f_i, t) = \sigma(\cos(E_V(f_i), E_T(t)))\). The authors uncover a critical distributional flaw in query-only scoring (Q-mode): VQA questions inherently feature interrogative pronouns ("who", "what", "where") that lack concrete visual referents, causing the encoder's Grad-CAM attention maps to become diffuse across the entire frame. Appending the ground-truth answer (QA-mode) transforms the query into a declarative sentence matching the image-text pretraining distribution, focusing attention on task-relevant regions and yielding a benchmark-wide +2.3 pp accuracy gain; QA-mode is thus adopted as the analytical upper bound for frame relevance. Furthermore, because adjacent frames in a continuous shot share near-identical embedding scores, naive top-\(K\) selection heavily clusters into narrow temporal spans. To enforce broad coverage, the authors implement an integer-second round-robin temporal diversity constraint: each selection round draws the single highest-scoring (or lowest-scoring for MINPROB) frame from distinct seconds, ensuring the first \(\min(K, \lceil L/\text{fps} \rceil)\) frames span distinct temporal seconds and avoiding artificial performance collapses caused solely by temporal over-concentration.
2. Frame Selection Sensitivity (FSS) and Temporal Dependence Disentanglement: Isolating Key and Anti-key Behaviors To quantify whether downstream models genuinely depend on receiving the correct visual frames, Frame Selection Sensitivity (FSS) is defined over a model ensemble \(M\) (spanning 8 open-source VLMs from 256M to 8B parameters): $\(FSS_i = \frac{1}{|M|} \sum_{m \in M} (acc^{+}_{m,i} - acc^{-}_{m,i})\)$ where \(acc^{+}_{m,i}\) and \(acc^{-}_{m,i}\) represent the binary correctness (1 or 0) of model \(m\) on sample \(i\) under MAXPROB and MINPROB frames, respectively. With a decision threshold \(\tau = 0.15\) (requiring more than one net model flip across the 8-model panel), each item is assigned to one of three categories: - Key (\(FSS_i > \tau\)): downstream accuracy drops sharply when relevant frames are swapped for irrelevant ones, denoting genuine frame sensitivity; - Non-key (\(|FSS_i| \le \tau\)): performance is largely invariant to frame relevance, indicating that the question is solved through non-temporal cues or broad scene context; - Anti-key (\(FSS_i < -\tau\)): encoder-deemed "relevant" frames actually degrade performance, producing a counter-intuitive negative delta.
Crucially, the paper demonstrates that Anti-key samples are not annotation noise but genuine temporally dependent queries afflicted by static visual appearance bias. For instance, when querying object disappearance ("Did the item leave the table?"), SigLIP selects close-up shots where the item is most visually prominent, completely omitting the empty scene frames required to confirm the disappearance event. Similarly, in multi-step action sequencing, the encoder heavily prioritizes peak action endpoints, discarding the prior contextual frames necessary to decipher chronological order. Under window-oracle evaluation, Anti-key samples exhibit a striking U-shaped recovery with large headroom gains and receive high temporal necessity ratings from human annotators (mean 3.55 vs. 2.98 for Non-key), proving they are difficult, temporally localized items that must be retained in temporal evaluation.
3. Language Independence Score (LIS) and TEMPCORE Benchmark Extraction: Orthogonal Language Filtering High frame sensitivity alone does not guarantee visual necessity: a question with high FSS might still contain stylistic or common-sense language leakage that allows models to exploit text priors. To identify and exclude such items, a No-frame baseline replaces all video frames with uniform gray dummy images to compute the Language Independence Score: $\(LIS_i = 1 - \frac{1}{|M|} \sum_{m \in M} correct^{nf}_{m,i}\)$ where \(correct^{nf}_{m,i}\) is the binary accuracy on dummy inputs. \(LIS_i = 1\) denotes complete visual dependence (no model succeeds without visual input), while \(LIS_i = 0\) indicates a trivial text-solvable question. Across all benchmarks, the Spearman correlation between LIS and FSS satisfies \(|\rho| < 0.10\), proving mathematical and empirical orthogonality between language independence and frame selection sensitivity. TEMPCORE employs a two-tier curation protocol: Stage 1 removes Trivial items (\(LIS_i \le \lambda = 0.50\)); Stage 2 categorizes the remaining vision-dependent items along FSS into Frame-Agnostic (\(|FSS_i| \le \tau\)) and Temporally Sensitive (\(|FSS_i| > \tau\), combining Key and Anti-key). To summarize benchmark temporal grounding in a single metric, the paper introduces the Temporal Purity Index (TPI): $\(TPI = \frac{|\text{Temporally Sensitive}|}{N - |\text{Trivial}|} = \frac{|\text{Key} \cup \text{Anti-key}|_{LIS > \lambda}}{N - |LIS \le \lambda|}\)$ This automated diagnostic extracts focused, temporally grounded subsets from existing benchmarks without requiring expensive video re-recording or manual re-annotation.
Key Experimental Results¶
Main Results¶
The diagnostic framework is evaluated across 8 open-source VLMs (InternVL3.5-8B, LLaVA-OneVision-7B/0.5B, Qwen3-VL-8B, SmolVLM2-2.2B/500M/256M, VideoLLaMA3-2B) on six standard Video QA benchmarks: three short-video datasets (MVBench, NExTQA, EgoSchema) and three long-video datasets (MLVU, LongVideoBench, Video-MME). The table below summarizes the two-dimensional benchmark quality profiles (extracted from Table 4 of the paper):
| Benchmark | Total Samples \(N\) | Trivial (Text-solvable, %) | Frame-Agnostic (Fr.Agn, %) | Visual Bias (Vis.Bias, %) | Temporally Grounded (Temp.Grnd, %) | Temporally Sensitive (TempSens, %) | Temporal Purity Index (TPI) |
|---|---|---|---|---|---|---|---|
| MVBench (Short) | 3,786 | 33.4% | 61.0% | 1.2% | 4.3% | 5.5% | 0.083 |
| NExTQA (Short) | 8,564 | 37.0% | 55.7% | 1.5% | 5.8% | 7.3% | 0.117 |
| EgoSchema (Short) | 500 | 20.2% | 60.6% | 2.6% | 16.6% | 19.2% | 0.241 |
| MLVU (Long) | 2,173 | 35.6% | 33.4% | 5.8% | 25.2% | 31.0% | 0.481 |
| LongVideoBench (Long) | 1,337 | 36.6% | 43.9% | 4.2% | 15.3% | 19.5% | 0.308 |
| Video-MME (Long) | 2,700 | 31.3% | 43.0% | 3.9% | 21.8% | 25.6% | 0.373 |
Ablation Study and Diagnostic Validation¶
To substantiate that FSS isolates genuine temporal dependence, the table below reports the accuracy of uniform sampling alongside the sliding window-oracle upper bound decomposed by FSS category (extracted from Table 2 of the paper):
| Benchmark | FSS Category | Sample Count \(N\) | Uniform Accuracy (%) | Window Oracle Accuracy (%) | Oracle Headroom Gap (pp) |
|---|---|---|---|---|---|
| MVBench | Key / Non-key / Anti-key | 212 / 3,502 / 72 | 52.2 / 48.4 / 50.9 | 65.4 / 53.7 / 61.3 | +13.2 / +5.3 / +10.4 |
| NExTQA | Key / Non-key / Anti-key | 656 / 7,588 / 194 | 51.6 / 59.7 / 51.7 | 69.2 / 67.9 / 68.4 | +17.5 / +8.2 / +16.7 |
| EgoSchema | Key / Non-key / Anti-key | 94 / 390 / 16 | 41.4 / 37.1 / 43.0 | 61.7 / 50.2 / 64.1 | +20.3 / +13.1 / +21.1 |
| MLVU | Key / Non-key / Anti-key | 782 / 1,212 / 179 | 50.6 / 52.3 / 50.1 | 81.1 / 74.5 / 79.8 | +30.4 / +22.2 / +29.7 |
| LongVideoBench | Key / Non-key / Anti-key | 229 / 823 / 69 | 46.7 / 47.2 / 44.9 | 76.1 / 64.9 / 71.3 | +29.3 / +17.7 / +26.4 |
| Video-MME | Key / Non-key / Anti-key | 783 / 1,673 / 168 | 50.9 / 44.2 / 46.4 | 82.6 / 69.3 / 79.6 | +31.6 / +25.1 / +33.2 |
Key Findings¶
- Frame selection matters primarily for long-form video: Across short-video benchmarks, MAXPROB exceeds MINPROB by only +3.5 pp on average (+1.6 pp on MVBench), as a budget of \(K = 32\) frames already captures most event content across 1โ3 minute clips. In contrast, on long-video benchmarks spanning up to an hour, the MAXPROB advantage widens significantly to +10.2 pp (reaching +13.2 pp on MLVU), demonstrating that selection mechanisms become essential when relevant visual evidence is temporally sparse.
- Current benchmarks are overwhelmingly dominated by non-temporal samples: Even on MLVU, the benchmark with the highest temporal purity, only 31.0% of items are both vision-dependent and frame-sensitive (\(TPI = 0.481\)). On short-video benchmarks like MVBench and NExTQA, temporally sensitive items account for merely 5.5% and 7.3% of the data; more than 55% of questions are frame-agnostic, and over 33% are easily answered via language priors on blind dummy inputs.
- Anti-key samples reveal visual appearance distractions: Plotting the window-oracle gap across FSS bins reveals a distinct U-shaped curve: both Key and Anti-key tails achieve substantial oracle gains (+26 to +33 pp on long videos), confirming that Anti-key queries strongly require localized temporal evidence but suffer from static appearance traps introduced by off-the-shelf vision-language scorers.
Highlights & Insights¶
- Inverting frame selection into a diagnostic probe: Rather than competing to marginally improve frame retrieval precision, this work repurposes the performance spread between MAXPROB and MINPROB as a zero-cost, fully automated auditing lens to inspect benchmark health.
- Exposing the language prior shortcut in video benchmarks: Demonstrating that modern VLMs achieve 30%โ50% accuracy on standard benchmarks with completely blanked-out video inputs provides concrete evidence that leaderboards often rank language reasoning priors rather than multimodal video temporal comprehension.
- Identifying the interrogative gap in vision-language encoders: Attention map diagnostics reveal that question prompts generate diffuse, unanchored representations in contrastive vision encoders trained on descriptive captions; framing queries into declarative QA statements provides an effective, plug-and-play improvement for temporal frame scoring.
Limitations & Future Work¶
- Scope limited to frame selection sensitivity: The authors acknowledge that FSS captures only one dimension of temporal groundingโwhether answering is contingent on showing the right frames. Other intricate temporal faculties, such as directional causality, microscopic state transitions, and precise duration estimation, may not manifest as frame-budget sensitivity and remain outside FSS detection.
- Dependence on VLM panel behavior for LIS: While control experiments incorporating text-only LLMs (Qwen3-8B, Llama-3.1-8B) confirm that pretraining contamination shifts benchmark LIS by less than 2.9 pp, minor prompt compliance discrepancies across models could still slightly influence the exact boundary of Trivial samples.
- Future exploration: Integrating FSS with temporal permutation probes (such as sequence shuffling and reverse playback) could establish a comprehensive, multi-faceted benchmark suite that simultaneously audits frame content sensitivity and sequential relational reasoning.
Related Work & Insights¶
- vs. TempCompass / TemporalBench: Existing temporal diagnostics depend either on synthetic frame concatenation/reversal or costly, specialized human temporal boundary annotations; TEMPCORE operates non-invasively, carving out pure temporal subsets directly from widely deployed public benchmarks.
- vs. MVBench / Video-MME original splits: Prior benchmarks relied on coarse metadata categories (e.g., classifying a subtask as "action sequence") under the assumption that they test temporal dynamics; TEMPCORE introduces empirical verification grounded in actual VLM response shifts, demonstrating that nominal task categories frequently fail to impose true temporal constraints.
Rating¶
- Novelty: โญโญโญโญโญ Formulates an elegant, training-free diagnostic protocol combining extreme-frame sensitivity with text-prior filtering to audit benchmark validity.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation covering 6 major benchmarks, 8 open-source VLMs, 3 vision encoders, sliding-window oracles, and human verification studies.
- Writing Quality: โญโญโญโญโญ Exceptionally structured, systematically unravelling the core paradox of temporal blindness versus benchmark scores through rigorous empirical evidence.
- Value: โญโญโญโญโญ Provides a vital reality check for video multimodal evaluation; the open-source TEMPCORE subsets offer a reliable benchmark for genuine temporal reasoning.