Video-Oasis: Rethinking Evaluation of Video Understanding¶

Conference: ECCV 2026
Paper: ECCV page
Code: https://github.com/sejong-rcv/Video-Oasis
Area: Video Understanding
Keywords: video understanding evaluation, shortcut samples, visual dependency, temporal dependency, benchmark auditing

TL;DR¶

Rather than building yet another video benchmark, Video-Oasis proposes a sustainable diagnostic suite that audits 14 existing video understanding benchmarks along three axes — removing visual evidence, destroying temporal structure, and human-in-the-loop verification of annotation ambiguity — finding that 55% of samples are answerable without visual or temporal information; on the resulting 11,033 video-native challenge samples, state-of-the-art models sit only marginally above the 25.6% random-chance baseline.

Background & Motivation¶

Video understanding evaluation has spent recent years doing addition. Once narrow tasks like action recognition and temporal localization gave way, Video-LLMs were asked to handle fine-grained dynamics and long-form reasoning at the same time, and benchmarks proliferated accordingly: Video-MME, MVBench, LongVideoBench and MLVU for general long video, EgoSchema for very-long-form egocentric video, VSI-Bench and its relatives for spatial reasoning. As the volume grew, however, scores on these leaderboards became harder to interpret — when a model answers a given question correctly, is it because it actually understood the footage, because a linguistic prior already implies the option, or because the answer was written into the transcript or the audio track?

The reason this question has stayed open lies in how benchmarks are built. Video annotation is expensive, so mainstream pipelines take one of two automatic shortcuts: generate questions from selected keyframes, or have an LLM produce questions from the video transcript. Both routes can inadvertently bypass the video modality entirely, yielding samples that do not depend on temporal continuity, causal interaction, or multi-event narratives — precisely the properties that distinguish video from images and text. Auditing efforts exist, but each is narrow: EgoTempo examines only 2 benchmarks and only the temporal axis, Cambrian-S covers 9 benchmarks but with a single temporal test, and Apollo audits 6 benchmarks along a single visual axis. None of the three uses cross-model consensus to stabilize its conclusions, and manual verification is largely absent.

This paper's angle is to stop adding leaderboards and instead turn "does this video question actually require watching the video?" into a batch-executable, reproducible diagnostic problem. The notion of necessity is decomposed into three operational criteria — whether visual evidence is required, whether temporal context is required, and whether the annotation is reliable — each paired with a set of probe tests, with agreement across several diagnostic models deciding whether a sample counts as a "shortcut," and disputed cases handed to humans. Core idea: combine three groups of probes (visual ablation, temporal perturbation, ambiguity verification) with cross-model consensus and human verification to distill the samples in existing benchmarks that genuinely depend on spatio-temporal evidence, yielding a far stricter evaluation protocol that both exposes how much current models are overestimated and doubles as a testbed for algorithmic design.

Method¶

Overall Architecture¶

Video-Oasis is not a new dataset but a filtering layer that sits on top of existing benchmarks. The input is a benchmark's full set of QA pairs and their videos; the middle consists of nine checks under three criteria — visual dependency (Blind / Audio / Summary), temporal dependency (Center-Frame / Frame Shuffling / Bag-of-Frames), and annotation reliability (Consistency / Redundancy / Sensitivity), where the first six are answered by a panel of diagnostic models and adjudicated by cross-model consensus, and the last three feed human-in-the-loop review back into the pipeline; the output is a filtered "video-native challenge" set carrying five capability labels. Across 14 benchmarks, 24,416 QA pairs are reduced to 11,033 (from 4,938 unique videos), a 55% reduction in evaluation volume, while what survives is the part that leans on spatio-temporal evidence. The whole pipeline is reproducible and extensible: a new benchmark can simply be run through the same probes rather than requiring a redesigned protocol.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["14 video benchmarks<br/>24,416 QA pairs"] --> B["Visual dependency diagnosis<br/>Blind / Audio / Summary"]
    B --> C["Temporal dependency diagnosis<br/>Center-frame / Shuffling / BoF"]
    C --> D["Ambiguity verification<br/>Consistency / Redundancy / Sensitivity"]
    D --> E["Video-native challenge distillation<br/>five capabilities + ensemble labels"]
    E --> F["Video-native diagnostic set<br/>11,033 QA · 4,938 videos"]

Key Designs¶

1. Visual dependency diagnosis: three no-vision probes that force out samples solvable from linguistic priors or spoken text

The most direct way to prove that a question must look at the video is to strip the visual evidence away and ask again: if the model still answers correctly, the question was not testing visual perception. The three probes remove visual information at different levels. Blind gives only the question and the answer options, catching samples solvable from linguistic bias or world knowledge alone — a question about the debut year of a typewriter, for instance. Audio transcribes the video's audio track and hands the text to the model in place of the video, catching samples whose answers live in the narration. Summary replaces the raw video with a concatenation of captions extracted at fixed intervals, catching samples whose key information is already covered by a textual summary. The paper explicitly states that the latter two probes are deliberately simple and are not meant to be optimal text-only video reasoners; they serve as diagnostic probes, and if a sample can be answered from a transcript or a caption sequence, its dependence on raw visual perception does not hold.

The crux is who counts as answering correctly. A single model is too brittle — one model failing may just mean it is weak, not that the question requires vision. Hence the cross-model consensus threshold \(c\): a sample counts as a shortcut under a given test only if at least \(c\) diagnostic models answer it correctly. The shortcut ratio of benchmark \(X\) is defined as

\[R_{\text{shortcut}}(X)=\frac{\bigl|\{x\in X:\ \exists\,t\in\mathcal{T},\ \#\{m\in\mathcal{M}: m\ \text{answers}\ x\ \text{correctly under}\ t\}\ge c\}\bigr|}{|X|}\]

where \(\mathcal{T}\) is the set of six diagnostic tests and \(\mathcal{M}\) the set of diagnostic models. Relaxing \(c\) to 1 pushes the ratio up to 92.7% on average; tightening it to 3 brings it down to 44.6%–63.0%. Reporting three thresholds turns "how prevalent are shortcuts" into a statement with a tunable criterion rather than a number resting on one experimental run.

2. Temporal dependency diagnosis: from a single frame through shuffled order to order-free matching

The visual probes govern whether the model looks at the footage at all, but half of video understanding is whether it looks in order. The three temporal probes escalate in how much structure they destroy. Center-Frame supplies only the middle frame, catching tasks that are called video but are in substance static recognition or spatial relations. Frame Shuffling randomly permutes the frame order, cutting temporal causality outright: if the model still answers correctly under shuffling, it is insensitive to chronological order. Bag-of-Frames goes furthest, using a frozen CLIP-family encoder (CLIP / Long-CLIP / EVA-CLIP) that models no temporal order to perform top-k frame matching against the query — a retrieval-style solution that sees patterns but not sequence. If it succeeds, the task needs only visual pattern recognition, not temporal reasoning. The three-level design separates two qualitatively different defects: not looking at the footage at all, versus looking at it but ignoring order — and the latter is the more insidious failure of systems that advertise themselves as video models.

3. Ambiguity verification: human-machine collaboration that separates wrong annotations from false shortcuts

All six probes rest on the inference that "the model answered correctly, therefore the sample contains a shortcut," but video data is long and dense and the annotations themselves can be broken: imprecise temporal grounding (the question asks when something happens while the answer is "throughout the video"), non-unique subject reference (several men with brown hair in the clip), insufficient evidence, or a non-unique answer. Such samples surface as "the model answered correctly" and get misclassified as shortcuts. Three checks handle this, and their role is not to filter but to flag samples for manual inspection. Consistency exposes intrinsic ambiguity or non-unique answers by checking whether multiple models fail to agree. Redundancy investigates whether a sample can be answered from any arbitrary video segment, revealing flawed question designs that fail to anchor the answer to a specific temporal span. Sensitivity targets the false positives of Frame Shuffling: humans review the samples that models still solve under shuffling and restore the ones that genuinely require temporal ordering. The paper's worked examples are telling — a question asking when the action "person fixes their hair" takes place, annotated as "throughout the entire video," is flagged by the redundancy check and judged a wrong annotation; a question asking what the baseball umpire did after the second ball passes the shuffling test, yet manual review confirms it does depend on the ordering implied by "after the second ball," so it is restored.

4. Video-native challenge distillation: letting the surviving samples grow their own five capability labels

Once filtering is done, what the survivors actually are needs a bottom-up answer rather than a predefined taxonomy imposed from above. The pipeline reuses existing annotations: it aggregates the source-benchmark metadata (original task categories and sub-task labels) of the surviving QA pairs, prompts Gemini-2.5-Pro to abstract candidate challenge clusters from it, and consolidates those into five capabilities — Fine-Grained Perception (recognition grounded in a spatio-temporal context, where what is identified depends on how details evolve across space and time), Spatial World Understanding (synthesizing fragmented multi-view evidence across frames to infer relative positions, geometry and trajectories), Temporal Dynamics & Tracking (object tracking, action sequencing, state transitions, all requiring ordering), Causality & Logical Reasoning (deducing latent cause-effect relations, physical laws and unobserved intentions), and Global Narrative (integrating events across the full timeline to infer long-term semantics while filtering irrelevant context). Categorization is not left to a single model: five proprietary LLMs label each sample independently and a label is accepted only on a three-way agreement, with just 122 samples failing to reach consensus and going to manual inspection. The ordering — filter first, induce categories afterwards — is the key point: the categories grow out of the surviving data rather than forming a pre-set checklist, so traditional image tasks cannot be repackaged and slipped back into video evaluation.

A Worked Example¶

Walking the two examples from Figure 3 through the pipeline shows how the three groups of checks cooperate.

The first asks when the action "person fixes their hair" takes place in the video. It passes Blind, Audio and Summary cleanly (neither linguistic priors nor the captions can produce the answer), and also passes the center-frame and shuffling tests. But in the Redundancy check the models find that any segment of the video suffices to answer it — the action spans the entire clip. The sample is flagged for manual review, which confirms that the annotation "throughout the entire video" does not match the temporal grounding the question demands, and it is removed as a wrong annotation.

The second asks what the baseball umpire did after the second ball. Under Frame Shuffling the model still answers correctly, so it is initially marked a temporal shortcut. But during Sensitivity review a human reads the phrase "after the second ball" as itself demanding event ordering, and concludes that success under shuffling more likely reflects memorized answers than an order-free question. The sample is restored to the final diagnostic set.

Both questions "pass" some probes, yet they end up on opposite sides — which is exactly why three axes are needed rather than one: any single probe produces both false positives and false negatives at once, and cross-model consensus plus human review is the mechanism that suppresses that bias.

Key Experimental Results¶

Main Results¶

The audit first reports the aggregate diagnostic-test results over all 14 benchmarks. Random chance is 25.6%:

Diagnostic test	Eagle2.5	Qwen2.5-VL	Qwen3-VL / others	Note
Blind	35.6	33.5	36.2	question and options only, 8–11 points above chance
Audio	47.6	46.8	45.9	transcript replaces the video
Summary	44.0	42.6	45.0	sampled captions replace the video
Center-Frame	42.2	—	40.2 (Qwen3-VL) / 43.0 (VideoAuto-R1)	middle frame only
Frame Shuffling	52.2	—	50.7 (Qwen3-VL) / 52.4 (VideoAuto-R1)	random frame permutation
Bag-of-Frames	—	—	31.4 (CLIP) / 32.8 (Long-CLIP) / 32.7 (EVA-CLIP)	frozen CLIP top-k frame matching

Splitting the benchmarks into four task groups, the prevalence of shortcuts varies with the consensus threshold as follows:

Consensus threshold	Spatial [21,33,43]	Temporal [9,27,42]	Reasoning [7,22,52]	General [12,16,35,39,50]
c ≥ 1	95.6	95.7	85.8	94.0
c ≥ 2	86.1	85.2	69.2	83.9
c = 3	58.8	54.4	44.6	63.0

Filtering yields 11,033 video-native challenge samples, on which 13 models are ranked (accuracy %, random baseline 25.6):

Model	Fine Percep.	Spatial World	Temporal Dynamics	Causal Reason.	Global Narrat.	Overall
GPT-4o	25.6	33.2	26.3	27.3	26.5	27.5
Gemini-2.5-Pro	40.2	49.8	50.9	45.4	43.0	46.7
Qwen2.5-VL (7B)	23.3	28.7	32.3	28.6	21.2	29.2
Qwen3-VL-Instruct (8B)	27.0	42.4	36.5	28.0	21.5	33.8
Qwen3-VL-Think (8B)	29.0	41.6	37.7	27.7	23.2	34.6
Eagle2.5 (8B)	26.9	31.0	39.7	33.2	22.7	34.5
InternVL-3 (8B)	27.0	31.3	34.1	30.6	24.5	31.6
InternVL-3.5 (8B)	29.5	41.9	35.1	29.8	23.3	33.6
Video-R1 (7B)	24.0	24.0	29.1	27.3	18.4	26.3
LongViLA-R1 (7B)	28.4	25.4	31.5	27.9	20.6	28.6
VideoAuto-R1 (8B)	27.5	44.3	39.5	31.1	28.9	36.8
VideoTree (GPT-5-mini)	28.6	34.0	32.3	24.6	20.7	30.1
STAR (GPT-5-mini)	31.6	44.4	42.2	34.0	32.9	39.5

Ablation Study¶

The role of temporal grounding. Using AKS to retrieve the 16 most relevant frames as a representative grounding method, ablated on the video-native challenges:

Method	Fine Percep.	Spatial World	Temporal Dynamics	Causal Reason.	Global Narrat.	Overall
Eagle2.5 (no grounding)	25.0	29.4	34.9	30.5	25.7	31.5
Eagle2.5 + temporal grounding	28.4	31.9	35.7	30.4	27.5	32.9
Qwen3-VL-Instruct (no grounding)	22.9	37.1	28.8	24.8	16.9	27.8
Qwen3-VL-Instruct + temporal grounding	25.2	37.9	32.3	23.3	19.9	30.1

To separate "incorrect grounding" from "weak reasoning," the paper runs an oracle grounding experiment on 2,945 QA pairs from ImplicitQA and KFS-Bench (1,060 video-native and 1,885 shortcut samples, each paired with ground-truth temporal regions):

Method	Fine Percep.	Spatial World	Temporal Dynamics	Causal Reason.	Global Narrat.	Video-native (overall)	Shortcut (overall)
Eagle2.5	37.2	22.9	40.5	38.5	16.0	35.0	78.0
Eagle2.5 + oracle grounding	50.4	27.5	61.3	48.1	48.0	50.8	80.8

Reasoning-depth modulation. Using Qwen3-VL (8B) as the base, three modes are compared, plus an oracle voting baseline that counts a response correct if either mode succeeds:

Method	Fine Percep.	Spatial World	Temporal Dynamics	Causal Reason.	Global Narrat.	Overall
Qwen3-VL-Instruct	27.0	42.4	36.5	28.0	21.5	33.8
Qwen3-VL-Think	29.0	41.6	37.7	27.7	23.2	34.6
Qwen3-VL-AutoR1 (adaptive thinking)	27.5	44.3	39.5	31.1	28.9	36.8
Qwen3-VL-Voting (oracle either-or)	38.4	57.7	49.4	37.8	30.1	46.2
Gemini-2.5-Pro	40.2	49.8	50.9	45.4	43.0	46.7

Training paradigms. With Qwen2.5-VL as the unified base, long-context SFT is compared against two RLVR reward designs:

Model	QA reward	Grounding reward	Fine Percep.	Spatial World	Temporal Dynamics	Causal Reason.	Global Narrat.	Overall
Qwen2.5-VL	—	—	23.3	28.7	32.3	28.6	21.2	29.2
Eagle2.5 (long-context SFT)	—	—	26.9	31.0	39.7	33.2	22.7	34.5
Video-R1	✓	—	24.0	24.0	29.1	27.3	18.4	26.3
VideoAuto-R1(Qwen2.5)	✓	✓	25.4	29.7	35.9	33.7	28.6	32.7

Key Findings¶

The ranking genuinely reshuffles, and RL-based video models are the most overestimated. Video-R1 (26.3) and LongViLA-R1 (28.6), presented in the literature as progress in video reasoning, sit essentially on the chance line (25.6) under the new protocol; Video-R1 even falls nearly 3 points below its own base, Qwen2.5-VL (29.2). What RLVR learned is plausibly a pattern over shortcut samples rather than spatio-temporal reasoning. By contrast VideoAuto-R1(Qwen2.5), with a grounding reward on the same base, rises from 29.2 to 32.7 — the reward design, not the presence of RL, is the watershed.
GPT-4o is substantially overestimated as well. Its overall score under the new protocol is only 27.5, level with 7B open-source models near the chance line, while Gemini-2.5-Pro reaches 46.7 and is the only system pushing every dimension above 40 — yet even that is far from solving the task, which shows how demanding the video-native challenges really are.
Global Narrative is the shared bottleneck. For most of the 13 models this column sits at or below chance (Qwen2.5-VL 21.2, Eagle2.5 22.7, GPT-4o 26.5), with the best being STAR at 32.9 and Gemini at 43.0. Integrating events across an entire timeline while filtering irrelevant context is what current architectures handle worst.
The payoff of temporal grounding depends on whether the sample truly needs it. The oracle experiment makes the split stark: +15.8 on video-native samples (35.0 → 50.8) versus +2.8 on shortcut samples (78.0 → 80.8). The implication is that shortcut samples let models score without precise grounding, which actively rewards ignoring temporal structure; grounding only becomes decisive in environments with strong spatio-temporal dependencies.
Agentic gains cannot be attributed to the underlying reasoning model alone. VideoTree and STAR share GPT-5-mini as the reasoning model yet score 30.1 and 39.5 overall — a 9.4-point gap — showing that the orchestration of reasoning steps, not the reasoning model itself, is the key variable.
"When to think" is worth nearly a frontier model. Always-on thinking (34.6) beats instruction-following (33.8) by only 0.8; adaptive thinking (36.8) adds more; and the oracle either-or voting baseline (46.2) nearly matches Gemini-2.5-Pro (46.7). In other words Qwen3-VL (8B) can already solve most of what Gemini solves — what it lacks is the ability to decide whether a given question warrants deeper thinking.
Video-Oasis is not merely difficulty-based filtering. Against a difficulty baseline of 6,609 questions that Eagle2.5, Qwen3-VL and VideoAuto-R1 all answer incorrectly, the overlap is only 44.6%, confirming that samples are selected by the visual / temporal / ambiguity criteria rather than by whether a model got them wrong.

Note on comparability: the main video-native leaderboard (13 models) and the ablation table disagree on baselines for the same models — Eagle2.5 scores 34.5 on the main board but 31.5 in the "no grounding" row, and Qwen3-VL-Instruct 33.8 versus 27.8. The original paper does not state whether the two evaluations differ in sample scope; the numbers are transcribed as published, ⚠️ refer to the original paper. Also, VideoAuto-R1 (8B) on the main board and Qwen3-VL-AutoR1 in the reasoning-depth table share the value 36.8 and are presumably the Qwen3-VL-based variant, whereas VideoAuto-R1(Qwen2.5) in the training-paradigm table is 32.7 on a different base — the two are not directly comparable.

Highlights & Insights¶

Turning "can we trust this benchmark?" into a reproducible batch experiment rather than an opinion piece. The combination of three probe groups with a consensus threshold \(c\) gives the question of shortcut prevalence a tunable criterion and a transferable toolchain. Any new benchmark can be run through the same probes, which makes auditing continuous rather than one-off.
The anti-circularity check (correlation rate) is the most methodologically careful part of the paper. Using different models from those that identified the shortcuts (InternVL-3.5, LongViLA-R1, STAR) to verify that flagged samples are still solved under standard evaluation yields an average of 76%, with no single test below 65.7%. This rules out self-validation and simultaneously delivers a strong conclusion: shortcut samples are already well handled by current models, so future evaluation value should shift to what remains.
The oracle grounding experiment disentangles "inaccurate grounding" from "weak reasoning." It is the most convincing experimental design in the paper: with ground-truth grounding, video-native samples gain 15.8 points while shortcut samples gain only 2.8. That single contrast explains why grounding research yields diluted returns on shortcut-laden benchmarks, and the pattern transfers to any attribution setting contrasting the error of module A against module B.
The voting upper bound points to a path that does not require scaling the model. An 8B model with an oracle selector over its instruction and thinking modes reaches 46.2, close to Gemini's 46.7. The number is not deployable (it uses ground truth), but it quantifies how much headroom reasoning-depth control offers — follow-up work should move toward a learnable selector, not a larger model.
Transferable practices: add a modality-blind probe as an admission check for any multimodal benchmark; replace single-model adjudication with multi-model consensus thresholds; and test any "hard subset" filter against a difficulty baseline to make sure it is not simply a difficulty picker in disguise.

Limitations & Future Work¶

Acknowledged by the authors: caption substitution and frame shuffling can still preserve partial temporal cues, producing both false positives (genuinely temporal questions misjudged as shortcuts) and false negatives. The countermeasure is stacking complementary diagnostic axes with cross-model consensus and human review, but no quantitative estimate of the residual bias is given.
The shortcut ratio is sensitive to the strength of the diagnostic models. Whether a sample counts as a shortcut depends on that panel of models; a stronger panel would only flag more samples, so 55% is closer in nature to "the fraction bypassable at current model capability." Reporting three consensus thresholds mitigates but does not remove this, and no sensitivity curve over diagnostic-model capability is provided.
The text probes may over- and under-estimate at the same time. Audio and Summary turn the task into pure text QA: if the caption model misses information that is present in the frames, the sample is judged to need vision and the shortcut count is underestimated; conversely, samples solvable from world knowledge merely duplicate what Blind already caught. The paper states these probes are deliberately not optimal, which also means their error rates are uncharacterized.
Key details of the Bag-of-Frames decision rule are missing from the main text. The top-k frame matching uses a frozen CLIP-family encoder, but the value of \(k\), the similarity aggregation (max or mean), and the definition of "answering correctly" are all deferred to the supplementary material, which directly affects the reproducibility and false-positive rate of that test.
The five capability labels are not validated for boundary consistency. The categories come from a five-LLM vote with human input on only 122 samples, yet "Causality & Logical Reasoning" and "Global Narrative" are semantically adjacent; no confusion matrix or inter-annotator agreement metric is reported.
The frame budget itself is never ablated. All evaluations cap input at 128 frames uniformly sampled at 1 fps. For long-video models this cap may by itself suppress performance, and no sensitivity analysis over frame count or sampling rate is provided — so the leaderboard conclusion that "long-video models are not strong either" need not come purely from architecture.
Concrete improvement directions: report the human-audited false-positive rate of each diagnostic test; express the shortcut ratio as a function of diagnostic-model capability rather than a fixed model panel; and run a systematic frame-budget ablation to separate architectural shortcomings from input-budget limits.

vs EgoTempo [26]: they audit 2 egocentric benchmarks, cover only the temporal axis, and use a single test with no cross-model consensus or human verification. Video-Oasis widens the scope to 14 benchmarks and nine checks across three axes, and replaces single-model adjudication with multi-model agreement.
vs Cambrian-S [44]: it covers 9 benchmarks with multiple visual tests, but the temporal axis remains a single test and conclusions still lack a consensus mechanism and human validation. Video-Oasis's main increment is splitting the temporal axis into three levels of severity (single frame / shuffled order / order-free matching) and adding the annotation-ambiguity dimension.
vs Apollo [53]: it audits 6 benchmarks along a single visual axis with partial human verification. The two are complementary — Apollo focuses on redundancy across benchmarks, this paper on whether the samples within one benchmark hold up.
vs conventional benchmark construction (Video-MME, MVBench, LongVideoBench, MLVU, etc.): those works add data, tasks and duration, whereas Video-Oasis subtracts — trading 55% of the samples for much higher discriminative power. One immediately actionable implication: a benchmark shipped with its own "shortcut report" would cost its users far less trust to interpret.
vs video RLVR work (Video-R1, VideoAuto-R1): Video-R1 stands for "RL helps video reasoning" on conventional benchmarks but scores only 26.3 under this protocol, below its own base; VideoAuto-R1 reaches 32.7 through QA plus grounding rewards. The gap suggests that what the reward is composed of — whether temporal grounding is explicitly rewarded — matters more than whether RL is used at all, which the paper leaves as an explicit opening for follow-up research.

Rating¶

Novelty: ⭐⭐⭐⭐ Individual probes (blind testing, frame shuffling) are not new, but assembling benchmark auditing into a reproducible three-axis diagnostic protocol with cross-model consensus, human verification, and reuse as an algorithmic testbed is new for video, and it explicitly resists the temptation to build another leaderboard.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ 14 benchmarks, 24,416 samples, a 13-model comparison, plus four ablations (temporal grounding, oracle upper bound, reasoning depth, SFT vs RLVR) and two validity checks covering shortcut-identification soundness and the distinction from difficulty-based filtering.
Writing Quality: ⭐⭐⭐⭐ The chain of criteria → probes → consensus → validation is clear, and the Figure 3 cases of wrong annotation and restoration are persuasive; points off because the main board and the ablation table disagree on baselines without stating the sample scope, and the key BoF hyper-parameters are deferred to the supplementary material.
Value: ⭐⭐⭐⭐⭐ The full audit pipeline is open-sourced and any new benchmark can plug in; the 55% figure together with the finding that SFT and RLVR have complementary strengths and that grounding rewards are the more promising investment places direct constraints on future benchmark construction and video model training.