Benchmarking Scientific Understanding and Reasoning for Video Generation using VideoScience-Bench¶

Conference: ECCV2026
Paper: Official Page · PDF
Code: https://github.com/hao-ai-lab/VideoScience
Area: Video Generation
Keywords: Scientific reasoning, composite scientific concepts, video evaluation, evidence grounding, VLM-as-a-Judge

TL;DR¶

VideoScience-Bench places video generators in experimental scenarios requiring combinations of scientific concepts, evaluates their outputs with expert ratings and an evidence-grounded VideoScience-Judge, and shows that visual coherence is not scientific correctness, while checklist-based judging reaches a Spearman correlation of 0.96 with expert rankings.

Background & Motivation¶

Video generators can render convincing objects, camera movements, and smooth transitions without correctly predicting an experiment's outcome. A visually stable clip can still depict motion in the wrong direction or show an apparatus without the expected phenomenon. Distributional metrics such as FVD do not readily locate these mistakes. Physical-commonsense benchmarks centered on everyday falling objects, reflections, and contact also provide limited evidence about outcomes that depend on interacting scientific principles.

This paper selects undergraduate-level physics and chemistry scenarios and requires every case to involve at least two scientific concepts. A laser bending through a sugar solution with a concentration gradient, for example, involves both refraction and diffusion rather than merely recognizing a laser and liquid. Evaluation then becomes a second challenge: a VLM may infer the expected answer from a familiar setup without checking whether the generated video actually depicts it. The benchmark therefore needs controlled experimental conditions and judgments supported by identifiable visual evidence.

Core idea: test observable outcome correctness with composite scientific cases, then ground automated scores in itemized checklists, causally salient frames, and computer-vision evidence so that looking like an experiment is not mistaken for depicting the right phenomenon.

Method¶

Overall Architecture¶

The paper does not train a new video generator; it develops a test suite and an automated evaluation pipeline. Inputs are experiment prompts, with initial setup images for I2V items. After a target model generates a video, experts or VideoScience-Judge apply the same five-dimensional rubric. The automated judge builds a checklist from the reference phenomenon, identifies frames supporting or contradicting its requirements, and combines detection, tracking, and motion evidence to produce dimension scores and a weighted aggregate.

The diagram represents the multi-stage benchmark and judging workflow, not the internal architecture of the video generator.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Source["Scientific concepts<br/>and teaching examples"] --> Cases["Composite Case Construction"]
    Cases --> Generate["Prompt and optional first frame<br/>Target model generates video"]
    Generate --> Rubric["Five-Dimensional Scoring"]
    Rubric --> Checklist["Checklists and Key Frames"]
    Checklist --> Evidence["CV Evidence Augmentation"]
    Evidence --> Output["Dimension scores and aggregate<br/>Comparison with expert labels"]

Key Designs¶

1. Composite Case Construction: make outcomes depend on interacting concepts rather than isolated recall

Eight domain-specialist graduate students each author 20 prompts, yielding 160 T2V cases. Every case combines at least two scientific concepts, draws on public educational experiments, and targets a phenomenon representable within a video lasting no more than 10 seconds. Coverage includes 14 topics and 103 concepts: physics contributes 9 topics, 79 concepts, and 135 cases; chemistry contributes 5 topics, 24 concepts, and 25 cases. The point is not longer prompts but outcomes that depend on jointly satisfied conditions, making familiar visual patterns less sufficient.

Of these cases, 40 receive paraphrased prompts and initial frames to form I2V items, including 35 physics cases and 5 chemistry cases. The total is therefore 200 evaluation items, not 200 independent scientific scenarios; T2V and I2V are not disjoint case collections. I2V reference videos come from public datasets and online resources, with a key frame from the initial segment showing the apparatus. Experts cross-review prompt specificity, procedural correctness, and expected outcomes. Sora-2 and Wan-2.5-T2V-Preview are then used to check whether opening frames can depict the essential setup. This checks representability; it does not make generated outcomes scientific ground truth.

2. Five-Dimensional Scoring: separate experimental conditions, target outcomes, and visual stability

Prompt Consistency (PCS) checks whether the apparatus and procedure follow the prompt. Phenomenon Congruency (PCG) checks the target scientific outcome, while Correct Dynamism (CDN) checks other basic physical constraints on motion and interactions. Immutability (IMB) requires objects to remain stable when no transformation is expected, without penalizing transformations the experiment actually requires. Spatio-Temporal Coherence (STC) measures natural frame transitions and continuity over space and time. Separating PCG from CDN distinguishes an absent target outcome from an outcome depicted through physically invalid dynamics.

Each dimension uses an ordinal scale from 1 to 4: absent or contradictory, weak or partially incorrect, mostly correct, and clearly correct. The aggregate gives scientific outcomes the largest weight. The following equation only expresses the weights explicitly stated in the text; it does not reconstruct a missing equation:

\[ S = 0.30\,\mathrm{PCG} + 0.20\,\mathrm{PCS} + 0.20\,\mathrm{STC} + 0.15\,\mathrm{CDN} + 0.15\,\mathrm{IMB}. \]

Applied to raw dimension scores, this produces a score between 1 and 4. Normalized results in the experimental tables undergo separate processing and must not be read as accuracy. The authors describe PCS and STC as checks on whether a valid setup and video have been produced. However, the available main text does not specify a complete computational gating rule, so it would be unjustified to add a threshold that automatically sets the aggregate to zero.

3. Checklists and Key Frames: connect every deduction to observable evidence

Asking a VLM for a holistic scientific-video score can produce unjustifiably generous ratings. VideoScience-Judge first uses an LLM agent to generate prompt-specific checklists organized by evaluation dimension. It then maps the fraction of satisfied items in each dimension to a four-point rating. The checklist follows an itemized deduction scheme in which deductions require concrete evidence. Key-frame identification uses the reference phenomenon and, for I2V, the available reference video to find moments that demonstrate violations. The judge must therefore answer whether a state change appears in the video, rather than whether it knows the scientific answer in language.

Human validation reports 95.6% checklist accuracy against ground-truth phenomena. This measures checklist quality, not video-generation success or final judging accuracy. The cached main text does not contain the supplementary checklist templates, exact fraction-to-rating thresholds, or key-frame sampling parameters. This note consequently does not assume uniform bins or invent sampling intervals. Reference information supplied to the judge also does not imply that the generator receives the expected answer.

4. CV Evidence Augmentation: constrain language-based judgments with location, identity, and motion signals

Grounding DINO checks entity presence and attributes, ByteTrack associates identities across frames, RAFT optical flow measures motion direction and magnitude, and CLIP4Clip measures text-video alignment. When a reference video exists, the system additionally reports LPIPS similarity using a VGG backbone. Each module emits compact JSON records containing frame_second, type, region, score, and attributes. An aggregator organizes these records into an evidence table and can attach the checklist, the evidence table, both, or neither to the judging prompt.

These tools do not directly prove that scientific laws hold. They help verify local observations such as object disappearance, trajectory continuity, or motion direction. For a ball changing position during sustained rotation, a still frame only establishes that the ball appeared at a location; tracking and flow provide evidence about its continuous movement. The paper also acknowledges that severe deformation and identity loss make off-the-shelf CV tools brittle. They provide inspectable evidence, not an independently reliable scientific simulator.

Loss & Training¶

There is no new training loss or generator fine-tuning. The central experiment compares expert and automated ratings on the same generated videos. The main text states that each model generates 3 video samples, but the cache does not expose all run parameters. Table 2 uses GPT-5 mini for the unified automated-judge comparison, whereas the checklist and CV ablation in Table 3 uses GPT-5 pro. Their numbers must not be combined as experiments using the same backbone.

Key Experimental Results¶

Main Results¶

The following values are extracted from the original Table 2. Base denotes basic judging, CL adds a checklist, and CL+CV adds visual evidence. These are reported normalized aggregate scores, not success rates. Model names follow the experimental table, retaining Veo-3 rather than substituting the introduction's Veo-3.1.

Video model	VSci-Judge Base	CL	CL+CV
Sora-2	0.76	0.75	0.76
Veo-3	0.67	0.65	0.65
Kling-v2.5-Turbo-Pro	0.63	0.59	0.59
Wan-2.5-T2V-Preview	0.68	0.60	0.59
Seedance 1.0 Pro	0.63	0.56	0.54
Hailuo 2.3	0.58	0.52	0.50
Ray2	0.54	0.39	0.34

The same source table reports correlations between evaluation rankings and expert rankings; higher is better. LMArena-T2V uses ELO derived from human votes, not the unified VLM judge.

Evaluation method	Kendall \(\tau\)	Spearman \(\rho\)
VSci-Judge Base	0.78	0.86
VSci-Judge CL	0.90	0.96
VSci-Judge CL+CV	0.90	0.96
T2V-CompBench	0.62	0.71
VideoScore2	0.45	0.53
PhyGenEval	0.62	0.82
LMArena-T2V	0.52	0.71

Ablation Study¶

The original Table 3 uses GPT-5 pro and reports signed gaps \(\Delta\) relative to expert scores as mean ± standard deviation. Positive values indicate overestimation and negative values underestimation. Distance from zero matters; a lower signed value is not automatically better.

Scoring dimension	Without CL	With CL	CL+CV
Prompt Consistency	-0.10 ± 0.10	-0.06 ± 0.08	-0.05 ± 0.09
Phenomenon Congruency	-0.22 ± 0.10	-0.20 ± 0.11	-0.20 ± 0.11
Correct Dynamism	+0.32 ± 0.13	+0.27 ± 0.10	+0.29 ± 0.13
Spatio-Temporal Coherence	+0.28 ± 0.08	+0.26 ± 0.09	+0.21 ± 0.09
Immutability	+0.27 ± 0.12	+0.24 ± 0.14	+0.18 ± 0.17

Key Findings¶

The checklist provides the clearest ranking improvement: \(\rho\) rises from 0.86 to 0.96. CL+CV has the same correlation as CL, so these results do not show further ranking gains from CV.
CV helps calibrate certain dimensions: relative to no CL, the STC gap falls from +0.28 to +0.21 and the IMB gap from +0.27 to +0.18. However, CDN rises from +0.27 with CL to +0.29 with CL+CV; benefits are not uniform.
The main text reports Sora-2 expert aggregates of 3.37 in chemistry and 3.12 in physics. In Energy, its PCS is 3.53 but PCG is 2.67, illustrating that a correct-looking apparatus can precede an incorrect outcome. These subgroup values come from the main text's account of supplementary analysis, not complete category tables in the cache.
The main text credits LPIPS with IMB calibration and RAFT with motion-related dimensions, but numerical per-tool ablations are only referenced in unavailable supplementary material. No per-tool ablation numbers are invented here.

Highlights & Insights¶

Separating target-outcome correctness from other physical constraints makes errors diagnosable. A single realism score conceals distinct failures in apparatus construction, outcome generation, and trajectories.
Ranking correlation and dimension-level calibration answer different questions. CV can leave ranking correlations unchanged while reducing biases in specific dimensions; both should be reported separately.
Checklists turn domain knowledge into verifiable questions, while CV adds temporal and spatial grounding. The transferable idea is evidence-constrained evaluation, not relabeling generic visual metrics as scientific correctness.

Limitations & Future Work¶

The authors explicitly limit the benchmark to externally observable scientific correctness, not recovery of latent reasoning. Visual priors can partially imitate phenomena, and combining concepts does not by itself demonstrate learned causal reasoning.
The 160 cases contain 135 physics and 25 chemistry scenarios, with I2V reusing 40 of them. Aggregate results should not be interpreted as balanced coverage of all scientific topics. Paired condition interventions and new concept combinations would strengthen future evaluation.
Generic detection and tracking tools may fail on severely distorted videos. The text says structural breakdown is penalized before phenomenon credit is assigned, but the cache lacks complete implementation rules for verifying how erroneous credit is prevented.
Ranking evaluation covers 7 generators, and the cache provides no confidence intervals for the correlations. High ranking agreement does not automatically extend to new models, individual videos, or other domains.
The main text repeatedly references unavailable supplementary material and contains unresolved appendix references. Table 2 specifies GPT-5 mini as a unified judge, but later baseline descriptions mention GPT-4o and Llava-1.5-7B. This note follows Table 2's explicit protocol for that table; reproductions still need to resolve those descriptions.

Compared with VBench / VBench++: these frameworks systematically evaluate multiple dimensions of video quality, whereas this work emphasizes outcomes under composite scientific conditions. They are complementary; stronger scientific-benchmark performance does not imply better visual quality in general.
Compared with PhyGenBench / PhyGenEval: physical-commonsense evaluation is a direct predecessor, with this paper placing greater emphasis on outcomes jointly determined by multiple scientific concepts. Correlation differences hold for this test suite and the reported protocol, not as a universal dismissal of other benchmarks.
Compared with T2V-CompBench and VideoScore2: compositional relationships and multidimensional quality already have established evaluation frameworks. The more specific contribution here is scientific-phenomenon judging through domain checklists and local visual evidence. A useful next test is whether the judge can construct reliable checklists for unfamiliar scientific concepts.

Rating¶

Novelty: 4/5. Combining composite scientific scenarios with evidence-grounded judging is well targeted, although checklists and VLM judging are not new paradigms themselves.
Experimental Thoroughness: 4/5. The study includes 7 generators, expert comparisons, and configuration ablations; scale, missing confidence intervals, and unavailable supplementary details limit the strength of its conclusions.
Writing Quality: 3/5. The rubric and central argument are clear, but inconsistent baseline-backbone descriptions and unresolved appendix references hinder reproduction.
Value: 4/5. The benchmark exposes scientific errors hidden by visual coherence and provides more interpretable evaluation than a generic overall score.