Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generation

Conference: ECCV 2026
arXiv: 2606.25306
Code: https://github.com/atinpothiraj/pqsg
Area: Video Generation
Keywords: Physical Plausibility Evaluation, Scene Graph, Fine-grained Evaluation, Text-to-Video Generation, VLM Evaluation

TL;DR

PQSG decomposes the evaluation of physical plausibility in videos into a three-layer hierarchical question graph (Object \(\rightarrow\) Action \(\rightarrow\) Physics). VLMs automatically generate questions with logical dependencies and answer them node-by-node. PQSG achieves a significantly higher correlation with human judgment than existing metrics on the FinePhyEval dataset, while precisely localizing which dimension (Object/Action/Physics) violates physical laws.

Background & Motivation

Video generation models (Sora 2, Veo 3, Wan 2.1, etc.) have made rapid progress in visual realism. However, generating physically plausible videos remains a core bottleneck—objects might dissolve instead of absorbing liquids, Newton's cradle collision trajectories could be disordered, or specular reflections might miss crucial objects. This lack of physical plausibility severely limits the utility of video generation models in downstream applications such as robotics simulation, embodied AI training, and synthetic data generation.

Nevertheless, why did reliable, fine-grained physical evaluation methods not exist previously? The answer lies in twofold technical barriers: first, early evaluation metrics (FID, FVD, CLIPScore) only measure visual quality or semantic alignment and ignore physical laws entirely; second, even though recent efforts like VideoPhy-2 and PhyGenEval specialize in evaluating physical plausibility, they only output a single aggregated score. They fail to localize "whether the object itself was generated incorrectly, the action was wrong, or the physical interaction was implausible"—which is precisely the diagnostic feedback most needed for model improvement and video editing.

Why is this task feasible now? The key enabling factor is the maturity of Vision-Language Models (VLMs, e.g., Gemini-2.5-Pro, GPT-5.5). VLMs can not only understand video content but also generate and answer atomic verification questions. This turns the task of "automatically generating a set of hierarchical, dependency-aware physical validation questions" from impossible to feasible. PQSG seizes this opportunity: it leverages VLMs to automatically generate a question scene graph from a text prompt, queries VLMs to answer the questions node-by-node against the generated video, and finally prunes invalid questions based on logical dependencies to produce independent scores across three dimensions: Object, Action, and Physics.

Core Idea: To model physical evaluation as a Directed Acyclic Graph (DAG) where nodes represent atomic validation questions and edges encode logical dependencies (e.g., if an object does not exist, do not query its actions; if an action does not occur, do not query its physical dynamics), thereby surpassing existing aggregated metrics in both evaluation granularity and reliability.

Method

Overall Architecture

The core problem PQSG solves is: given a text prompt and a generated video, how to provide an overall physical plausibility score while precisely localizing which dimension (which object, action, or physical interaction) went wrong. The overall design is a two-stage VLM pipeline: the first stage automatically generates a hierarchical Question Scene Graph (QG) from the prompt, and the second stage answers each question in the scene graph against the video (QA). Finally, scores are aggregated after pruning based on the graph's dependency structure.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Text Prompt"] --> B["QG Stage<br/>VLM Generates Hierarchical Question Graph"]
    B --> C["PQSG Scene Graph<br/>Object / Action / Physics Nodes + Dependency Edges"]
    C --> D["QA Stage<br/>VLM Answers Node-by-Node<br/>Open-ended Reasoning -> Yes/No Classification"]
    D --> E["Dependency Pruning<br/>If Parent Node is No,<br/>Child Nodes Automatically Marked No"]
    E --> F["Score Aggregation<br/>Object / Action / Physics<br/>Three Independent Scores + Total Score"]

The input to the QG stage is a text prompt and an in-context example (including an example prompt, example PQSG nodes, and edges), and the output is a complete question scene graph. The input to the QA stage is the generated video and a single question node. It utilizes a two-step answering mechanism: first letting the VLM generate open-ended reasoning, and then classifying the reasoning output into "Yes" or "No". This avoids the lack of Chain-of-Thought (CoT) caused by direct yes/no prediction. Each question is processed independently to ensure fine-grained results. After answering all questions, pruning is performed according to the dependency structure: if a parent node is answered "No", all of its child nodes are automatically marked "No" and are not queried. Finally, the ratio of "Yes" answers is calculated for each node category to serve as the score for that dimension, and the average across all nodes serves as the total score.

Key Designs

1. Three-layer Hierarchical Question Classification: Decoupling Video Evaluation into Object / Action / Physics

The biggest flaw of existing evaluation methods is that they mix qualities of all dimensions into a single aggregated score, making it impossible to determine at which stage the model failed. PQSG strictly categorizes validation questions into three levels: Object nodes verify whether the key objects from the prompt appear in the video (e.g., "O2: Are there two pillows?"); Action nodes verify whether objects execute the correct actions (e.g., "A1: Does one grabber tool release the brown tennis ball?"); Physics nodes evaluate the physical plausibility of the actions (e.g., "P8: Do the pillows visibly deform or compress upon impact?"). The distinction between Action and Physics is that Actions are explicit behaviors mentioned directly in the prompt, whereas Physics refers to the implicit physical common sense (such as deformation, absorption, gravitational acceleration, etc.) implied by the prompt. This separation allows the evaluation to accurately distinguish three failure modes: incorrect object generation, incorrect action, and implausible physical interactions. Experiments demonstrate that this hierarchical separation is effective—across all video generation models, the Object score (average 0.93) is significantly higher than Action (0.66) and Physics (0.57), indicating that "generating objects but failing on physics" is the true bottleneck of current models.

2. Dependency Graph Structure and Pruning Inference: Ensuring Only Meaningful Questions are Asked

This is the core design that distinguishes PQSG from a simple list of questions. PQSG organizes questions into a Directed Acyclic Graph (DAG) where directed edges encode logical dependencies: most Action nodes have Object nodes as parents, and most Physics nodes have Action nodes as parents. Intra-category dependency edges (e.g., Object \(\rightarrow\) Object, Action \(\rightarrow\) Action) are also allowed to model sequential dependencies (e.g., subsequent actions preconditioned on prior actions). During evaluation, this dependency structure is strictly enforced: if a parent node is answered "No", its entire sub-path of child nodes will not be queried and is automatically marked as "No". The key benefit of this approach is avoiding VLM hallucinations on invalid questions—for example, if there is no tennis ball in the video, directly asking "is the gravitational acceleration of the tennis ball realistic?" will force the VLM to invent an answer. Ablation studies (Table 8) validate this design: removing the dependency graph structure drops the Pearson correlation with human judgment from 0.48 to 0.44; when QA is executed by humans, the drop is even more pronounced (\(0.80 \rightarrow 0.75\)), demonstrating that the dependency structure is essential to evaluation reliability.

3. Two-Stage VLM Evaluation Pipeline: QG Question Graph Generation + QA Two-Step Node Answering

PQSG decouples the evaluation into two independent stages, QG (Question Generation) and QA (Question Answering), which can be handled by different VLMs, making it model-agnostic. The QG stage employs in-context learning: by providing an example prompt containing complete PQSG nodes and edges, the VLM learns to generate structurally consistent question graphs for new prompts. The QA stage incorporates a critical implementation detail: directly prompting a VLM for a yes/no answer suppresses Chain-of-Thought (CoT) reasoning, leading to hasty judgments that miss crucial visual details. PQSG's solution is a two-step QA: first, let the VLM output open-ended reasoning (describing phenomena observed in the video), and then classify its own reasoning into yes/no. This introduces negligible latency but significantly boosts QA quality. As shown in Table 5, even for the strongest GPT-5.5, the QA accuracy in the Physics category is only 64.6%, far below the 88.4% of the Object category, demonstrating that physical reasoning remains a bottleneck for VLMs and constitutes PQSG's current performance ceiling.

4. FinePhyEval Dataset: Four-Dimensional Human Annotation over 195 Videos

To validate the efficacy of PQSG and establish a robust evaluation benchmark, the authors constructed the FinePhyEval dataset. Key design choices include: (a) selecting all 65 prompts from Physics-IQ dedicated to testing physical understanding, covering 5 physical categories: solid mechanics (38), fluid dynamics (15), optics (8), thermodynamics (3), and electromagnetism/magnetism (2); (b) generating videos using three of the strongest video generation models: Sora 2, Veo 3, and Wan 2.1, resulting in 195 prompt-video pairs; (c) rating each video by 8 non-author human annotators on a 5-point Likert scale across four dimensions: Object, Action, Physics, and Overall, gathering 780 ratings in total; (d) performing additional manual annotations of question generation (QG) for 20 prompts and question answering (QA) for 30 prompt-video pairs (444 QA pairs) to establish ground truth for sub-task evaluations. Annotation reliability is verified by a high consistency of ICC = 0.84. This dataset allows PQSG to not only study correlation with human judgment but also independently evaluate VLM performance on the QG and QA sub-tasks, making every step of the evaluation framework diagnostic.

A Complete Example: Tennis Ball Drop Scene

Using the opening example of the paper to illustrate PQSG's complete workflow: given the prompt "Two pillows on a table and two grabber tools hanging above them from which a brown tennis ball and an orange block are suspended. The grabber tools let go of the ball and block.", the VLM in the QG stage generates a question scene graph:

• Object nodes: O1 "Is there a table?", O2 "Are there two pillows?", O9 "Is there a brown tennis ball?"
• Action nodes (with Object as parent): A1 "Does one grabber tool release the brown tennis ball?" (parent O9), A2 "Does the other grabber tool release the orange block?"
• Physics nodes (with Action as parent): P8 "Do the pillows visibly deform or compress upon impact?" (parent A1/A2), P5 "Does the tennis ball exhibit realistic gravitational acceleration?"

During the QA stage, nodes are queried sequentially: the VLM answers O2 = Yes (pillows exist), O9 = Yes (tennis ball exists), A1 = Yes (grabber released the ball), P8 = No (pillows do not visibly deform on impact). Dependency pruning is not triggered on a large scale here since most preconditions are met. Final scores: high Object score (objects are correctly generated), low Physics score (pillows do not deform, gravity acceleration is incorrect), which precisely outputs the diagnostic conclusion that "the model generated correct objects and actions, but the physical interaction is implausible". Without hierarchical decomposition, a single aggregated score would merely say "the video quality is average", offering no developer insights on what to fix.

Loss & Training

PQSG itself does not involve model training and is a zero-shot evaluation framework. Both QG and QA utilize off-the-shelf VLM API calls without fine-tuning. In the iterative optimization experiments (Sec 5.5), the authors use the fine-grained feedback of PQSG to guide prompt refinement in conjunction with Wan 2.2 TI2V-5B for iterative generation, while PQSG itself remains a fixed evaluator without any training.

Key Experimental Results

Main Results

Evaluation Method	Pearson's r	Kendall's tau	Spearman's rho
VideoScore	0.289	0.262	0.378
VideoPhy-2-Autoeval	0.346	0.277	0.349
PhyGenEval	0.272	0.220	0.264
DSG (image-based)	0.302	0.195	0.265
Direct VQA	0.382	0.290	0.360
PQSG w/ Gemini-2.5-Pro	0.467	0.306	0.406
PQSG w/ GPT-5.5	0.478	0.336	0.456

PQSG outperforms existing methods across all three correlation coefficients. Notably, even though DSG is also a question-graph-based evaluation method, because it is designed for images rather than videos and lacks modeling of temporal attributes (actions, physical interactions), its correlation (Pearson 0.302) is far lower than PQSG's. Furthermore, PQSG's score upper bound is extremely high: when the QA is conducted by humans (Table 8), the Pearson correlation reaches 0.80, indicating that as VLM capabilities improve, there is significant room for growth in PQSG's automated evaluation quality.

Multi-Model Comparison

Video Gen Model	Object	Action	Physics	Overall
Sora 2	0.95	0.75	0.69	0.78
Veo 3	0.98	0.78	0.68	0.80
Wan 2.1	0.86	0.53	0.46	0.59
Cosmos 2.5	0.93	0.56	0.46	0.62

Three findings: (1) All models score highest on Object and lowest on Physics, suggesting that "drawing objects correctly" is no longer the bottleneck, but "simulating physics correctly" is; (2) Closed-source models (Sora 2, Veo 3) comprehensively outperform open-source models (Wan 2.1, Cosmos), with particularly substantial gaps in the Action and Physics dimensions (Veo 3 Physics 0.68 vs. Wan 2.1 Physics 0.46); (3) Although Cosmos is designed specifically for physical world simulation, its Object score is close to closed-source models while its Physics score is on par with Wan 2.1, indicating that its physical simulation architecture advantages have not yet fully translated into better video generation quality.

Ablation Study

Configuration	Pearson's r (GPT-5 QA)	Pearson's r (Human QA)
PQSG (Full)	0.48	0.80
w/o Dependency Graph	0.44	0.75
w/o Fine-grained Questions (Direct VQA over 3 Dimensions)	0.40	0.68

Two clear ablation conclusions: (1) Fine-grained question decomposition yields the largest contribution—removing it causes a correlation drop of 0.08 (auto) and 0.12 (human), since directly prompting the VLM to output integrated scores for the three dimensions abandons intermediate reasoning; (2) Although the contribution of the dependency graph structure is smaller than that of fine-grained decomposition, its contribution is more pronounced under the human QA setting (0.05), suggesting that while logical dependencies are intrinsically implicit in human judgment, explicit modeling benefits VLM reliability much more.

Key Findings

• Physics is the primary evaluation bottleneck: VLM QA accuracy is 88.4% on Object but only 64.6% on Physics, and VLMs exhibit a noticeable "yes-bias"—tending to default to answering "yes" when unsure about details. For instance, in judging smoke direction, the VLM claimed "smoke drifts upwards," while the actual video showed the smoke shooting out sideways like a flare.
• The upper bound of human QA is as high as Pearson 0.80 (Table 8), proving that PQSG's evaluation framework itself is highly robust; the current limitations on automated evaluation quality stem primarily from the VLM's physical reasoning capability. As VLMs advance, PQSG's automated evaluation will inherently improve.
• Model Iterative Optimization Experiment: Guided by PQSG's fine-grained feedback for prompt refinement, the mean PQSG score of Wan 2.2-generated videos improved from ~70% to ~82% (+15% in the first round), stabilizing at 81.9% in the second round, proving that PQSG serves not only as an evaluation tool but also as a diagnostic feedback signal in generation-optimization loops.
• Generalizability: On the external VideoPhy-2 dataset, PQSG improves the Pearson correlation of Semantic Alignment (SA) from 0.450 to 0.550, and Physical Commonsense (PC) from 0.420 to 0.498, demonstrating that the method is independent of specific data distributions.

Highlights & Insights

• Dependency pruning is the most ingenious engineering design: While many evaluation frameworks generate a flat list of questions, PQSG organizes them in a DAG and enforces dependency pruning. This acts as a logical shield for the VLM, preventing it from answering meaningless questions when preconditions are unmet, thereby reducing hallucinations that pollute final scores. This design philosophy can easily migrate to other VLM evaluation scenarios requiring hierarchical validation (e.g., long-form fact-checking, multi-step reasoning verification).
• Explicit separation of Action and Physics is a conceptual contribution: Distinguishing "actions explicitly specified in the prompt" from "implicit physical commonsense" allows the evaluation to segregate "the model failed to execute the action" from "the model executed the action but in an unrealistic manner." Although intuitive, no evaluation framework prior to PQSG explicitly modeled this distinction, leading to mixed failure modes.
• Flexibility from decoupling QG and QA: Since the two stages are independent, they can be driven by different VLMs or substituted independently. When VLMs upgrade, developers only need to swap out the QA model to obtain superior evaluation quality without redesigning the architecture. This modular and model-agnostic nature grants PQSG long-term viability.
• Multi-layered annotation design of FinePhyEval is highly worth adopting: Not only does it label overall Likert scores, but it also labels ground-truth for QG and QA, enabling independent verification of each sub-component of the evaluation pipeline. When building benchmark publications, such a design significantly boosts the convincingness and reusable value of the dataset.

Limitations & Future Work

• VLM physical reasoning capability serves as a hard upper bound: The authors candidly acknowledge that PQSG's evaluation quality is entirely capped by the QA VLM's physical understanding (Physics QA accuracy is only 64.6%). Although human QA's upper bound of Pearson 0.80 proves the framework's intrinsic reliability, automated evaluation for the Physics dimension is still lacking under current VLM limits. An undiscussed question is whether PQSG's QG stage (which uses only a single in-context example) could benefit from more examples covering diverse physical classes to improve QG quality.
• Only evaluating prompt-specified content: PQSG only generates questions for objects and actions explicitly mentioned in the prompt. If a severe physical violation occurs in areas not mentioned by the prompt (e.g., water flowing upward in the background), PQSG will not capture it. This is an inherent limitation of the "prompt-based evaluation" paradigm, whereas humans can judge physical plausibility independent of prompts. The paper notes this as a direction for future work.
• Dependence on closed-source VLMs affects reproducibility: The main experiments utilize Gemini-2.5-Pro and GPT-5.5, which are closed-source models. Although the authors promise to release all code, prompts, and annotation data, and have validated open-source VLM utility on VideoPhy-2, subtle changes in closed-source API behavior may still impact long-term reproducibility.
• Insufficient diversity in Physical categories: Among the 65 prompts in FinePhyEval, solid mechanics accounts for 38 (58%), while magnetism accounts for only 2 (3%), resulting in unbalanced coverage of physical phenomena. Evaluation conclusions for certain sparser physical categories (e.g., electromagnetism) may lack reliability due to too few samples.

Related Work & Insights

• vs. VideoPhy-2: VideoPhy-2 outputs physical plausibility scores by fine-tuning VLMs on physical video datasets, but still yields a single aggregated score and cannot provide fine-grained diagnostic feedback. PQSG's hierarchical question graph is inherently interpretable and localizable while requiring zero-shot execution (no fine-tuning), offering lower deployment overhead.
• vs. DSG (Davidsonian Scene Graph): DSG is the most direct predecessor of PQSG, likewise employing a question graph to evaluate generated content. However, DSG is tailored for images, lacks the Object/Action/Physics hierarchical division, and does not model temporal dependencies. PQSG can be viewed as a natural extension of DSG to the video and physical domains. Experimental data confirms the value of this extension: DSG's Pearson correlation on FinePhyEval is only 0.302, significantly below PQSG's 0.478.
• vs. PhyGenEval: PhyGenEval also evaluates physical commonsense using VLMs, but operates at a coarse rating granularity and lacks a dependency graph. Its Pearson correlation (0.272) is the lowest among the baselines, demonstrating that direct "VLM scoring" strategies are unreliable for complex physical scenarios.

Rating

• Novelty: Four Stars — Upgrades video physical evaluation from "giving an overall score" to a "hierarchical question graph + dependency pruning," bridging a clear gap with a simple yet elegant concept; the explicit separation of Action/Physics, although intuitive, has not been done before.
• Experimental Thoroughness: Four Stars — Comprehensively covers six sets of experiments, including correlation with human judgment, multi-model evaluation, ablation studies, generalization on external datasets, independent QG/QA sub-task evaluation, and iterative optimization experiments; one star deducted due to the unbalanced distribution of physical categories and lack of systematic comparisons across more open-source VLMs.
• Writing Quality: Four Stars — Features a clear chain of motivation (why fine-grained is needed \(\rightarrow\) why it did not exist before \(\rightarrow\) why it is possible now), well-structured method descriptions, and ablation verifications for each design choice; the running example in Figure 2 is highly intuitive.
• Value: Four Stars — Holds direct guidance for evaluation practices in the video generation domain, with the dependency graph + hierarchical decomposition design being highly transferrable to other generative tasks (e.g., image editing, 3D generation, audio synthesis); FinePhyEval itself also holds substantial independent value as a dataset.

Related Papers

• [ICCV 2025] ETVA: Evaluation of Text-to-Video Alignment via Fine-Grained Question Generation and Answering
• [ECCV 2026] PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generation
• [ICLR 2026] \(PhyWorldBench\): A Comprehensive Evaluation of Physical Realism in Text-to-Video Models
• [ECCV 2026] Your Data Manifold is Secretly a Reward Model: Shell-LCC for Text-to-Video Generation
• [ECCV 2026] PhysRAG: Enhancing Physics-Awareness in Video Generation via Retrieval-Augmented Generation