Predictive Structure Improves Video Diffusion Dynamics¶
Conference: ECCV 2026
Paper: ECCV Official
Code: None
Area: Video Generation
Keywords: video generation, diffusion models, world models, intuitive physics, information bottleneck
TL;DR¶
Addressing the frequent violations of gravity, support, and object permanence in video diffusion generation, this paper proposes Latent Dynamics Optimization (LDO), which uses a frozen V-JEPA 2 predictive world model as a dynamics prior through Gram-matrix relational distillation and rollout-level GRPO policy optimization to significantly improve physical commonsense.
Background & Motivation¶
Video diffusion models have demonstrated extraordinary photorealism and single-frame synthesis quality, capable of rendering rich textures and complex natural scenes. However, as synthesis unfolds across extended temporal horizons, even the most capable video foundation models regularly produce severe physical inconsistencies. Generated objects hover without physical support, pass through boundaries, spontaneously vanish behind occluders, or glide unnaturally across contact surfaces. Although individual frames appear crisp and aesthetically pleasing, the underlying physical trajectory remains fragile and implausible, severely restricting the utility of video models in embodied robotics, interactive world simulation, and reliable content editing.
Analyzing this failure mode through the Information Bottleneck (IB) principle reveals that physical inconsistency is not simply a data-scale deficit, but an objective-formulation pathology. Standard latent diffusion and flow-matching objectives seek to minimize conditional cross-entropy over high-dimensional VAE latents. In doing so, high-entropy stochastic nuisance factors—such as minute texture shifts, illumination flickers, and background clutter—inflate the conditional entropy, compelling the generative network to spend excessive representation capacity on non-causal appearance noise rather than on the underlying low-dimensional physical state dynamics. In contrast, self-supervised predictive latent world models like V-JEPA 2 learn to forecast masked target latents from causal context purely in feature space, establishing an implicit information bottleneck that naturally filters out unpredictable high-frequency texture variations while isolating smooth, coherent physical state trajectories.
Building on this insight, the authors argue that physically grounded video generation can be achieved without modifying the core diffusion sampling pipeline during inference. Core idea: propose Latent Dynamics Optimization (LDO), a post-training framework that employs a frozen V-JEPA 2 world model as a dynamics teacher and physics critic, combining intermediate Gram-matrix relational distillation (Predictive Dynamics Alignment, PDA) with rollout-level Group-Relative Policy Optimization (GRPO) to instill causal physical dynamics into pretrained video diffusion generators.
Method¶
Overall Architecture¶
Latent Dynamics Optimization (LDO) is a two-level post-training framework that guides pretrained diffusion transformers toward physically grounded dynamics without interfering with the inference sampling process. At the representation level, intermediate features from the diffusion transformer (DiT) are aligned with the causal future predictions of a frozen V-JEPA 2 model across discrete temporal bins via relational Gram-matrix geometry. At the trajectory level, the diffusion model acts as a parameterized policy generating multi-candidate rollouts, while the frozen V-JEPA 2 serves as a physics critic that scores each rollout by its causal predictability over sliding temporal windows, optimizing the diffusion policy through Group-Relative Policy Optimization (GRPO).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Input Video / Prompt C"] --> DiTBranch["Diffusion Backbone Pathway<br/>VAE Encoding + DiT Features"]
In --> JEPA上演["Frozen V-JEPA 2 Pathway<br/>Binned Temporal Latents + Predictor P"]
DiTBranch & JEPA上演 --> PDA["Predictive Dynamics Alignment (PDA)<br/>Gram-Matrix Manifold Distillation"]
DiTBranch --> Rollout["Stochastic Multi-Sample Rollout<br/>stochastic DDIM Trajectory Generation"]
Rollout --> Critic["V-JEPA 2 Physics Critic<br/>Sliding-Window Causal Prediction Reward"]
Critic --> GRPO["Structure-Aware GRPO Optimization<br/>Group-Relative Advantage Policy Update"]
Key Designs¶
1. Predictive Dynamics Alignment (PDA): Coordinate-Invariant Manifold Geometry Distillation
Directly forcing diffusion intermediate tokens to match the absolute coordinate values of world model latents causes coordinate-space mismatch and risks compromising the generative model's fine-grained appearance fidelity. PDA addresses this by aligning the internal relational geometry rather than absolute coordinates. The temporal axis is partitioned into \(K\) uniform bins, each spanning \(\Delta_t\) frames. For each target bin \(k\), the frozen V-JEPA 2 predictor \(\mathcal{P}\) takes context tokens from preceding bins \([0, k-1]\) and causally forecasts the target features \(\hat{\mathbf{h}}^{(k)}\). Concurrently, intermediate features \(\tilde{\mathbf{z}}\) from layer \(l\) of the diffusion transformer and the predicted features \(\tilde{\hat{\mathbf{h}}}\) are L2-normalized, from which spatial token-to-token Gram matrices (cosine similarity relational matrices) are constructed: $$ \mathbf{G}^{\text{diff}}t = \tilde{\mathbf{z}}_t \cdot \tilde{\mathbf{Z}}^\top, \quad \mathbf{G}^{\text{pred}}_t = \tilde{\hat{\mathbf{h}}}_t \cdot \tilde{\hat{\mathbf{H}}}^\top $$ The PDA loss enforces structural parity between the diffusion feature geometry and the world model's causal prediction manifold within a relaxation margin \(m\): $$ \mathcal{L}_t\right| - m\right)\right) $$ This relational matching provides structural invariance, isolating the causal dynamics manifold while leaving diffusion feature representations unconstrained to handle photorealistic visual textures.}} = \frac{1}{|T_{\text{valid}}|} \sum_{t \in T_{\text{valid}}} \text{mean}\left(\text{ReLU}\left(\left|\mathbf{G}^{\text{diff}}_t - \mathbf{G}^{\text{pred}
2. Structure-Aware Reward GRPO: Trajectory-Level Dynamics Reinforcement via a Physics Critic
Because feature alignment alone cannot directly penalize compounding kinematic errors across multi-step iterative denoising loops, LDO introduces an end-to-end reinforcement learning feedback mechanism. The diffusion model is formulated as a stochastic policy \(\pi_\theta\) that generates a group of \(G\) candidate rollouts per conditioning prompt via stochastic DDIM. A frozen V-JEPA 2 physics critic processes each rollout using a sliding-window scheme \(\mathcal{W}\). Within each window, the first \(C\) frames provide causal context to predict future token representations, and the trajectory reward is defined as the mean cosine similarity between predicted future tokens and actual encoded tokens: $$ r(Y) = \frac{1}{|\mathcal{W}|} \sum_{w \in \mathcal{W}} \frac{1}{|w_{\text{future}}|} \sum_{i} \cos\left((\mathbf{h}^w_{\text{pred}})i, (\mathbf{h}^w)_i\right) $$ Group-relative advantages }\(A_i = (r_i - \mu_G) / (\sigma_G + \epsilon)\) are computed and clipped, guiding policy gradient updates across a randomly sampled subset of denoising timesteps to directly penalize physically erratic trajectories.
3. Interleaved Dynamic Objective: Unified Optimization under Memory Constraints
Simultaneously executing dense feature distillation graph computation and full generative rollouts on high-resolution video models introduces prohibitive GPU memory bottlenecks. LDO addresses this via an interleaved step-wise scheduling regime where the overall loss is defined dynamically: $$ \mathcal{L}{\text{total}}^{(n)} = \mathcal{L}}} + \lambda \mathcal{L{\text{PDA}} + \gamma(n) \mathcal{L} $$ During standard training iterations (}\(\gamma(n) = 0\)), the model jointly backpropagates the standard diffusion denoising loss and the PDA relational distillation loss on video data. Every \(m\) steps, \(\gamma(n) > 0\) activates to prioritize policy-gradient updates (\(\mathcal{L}_{\text{GRPO}}\)). Combined with gradient checkpointing and random temporal subsampling (computing policy gradients over only a fraction \(\rho=0.6\) of denoising steps), this design enables stable end-to-end training within standard accelerator memory limits.
Loss & Training¶
- Backbone & Settings: T2V experiments build upon CogVideoX-5B (with Wan2.2-T2V-A14B verified qualitatively), and I2V gravity evaluations build upon Open-Sora. PDA is applied at layer 18 of the DiT with LoRA rank 128; the world model uses V-JEPA 2-ViT-G.
- Hyper-parameters: Temporal bins \(K=4\); GRPO group size \(G=16\), sliding window length 16 frames (8 causal context frames, temporal stride 8); DDIM noise scale \(\eta > 0\), classifier-free guidance \(w=6.0\); trained for 1 epoch on 8 NVIDIA GH200 (120GB) GPUs.
Key Experimental Results¶
Main Results¶
On the VideoPhy benchmark (evaluating solid-solid, solid-fluid, and fluid-fluid interactions) and the PisaBench gravity benchmark, LDO demonstrates substantial quantitative gains over base models and physics-specialized baselines.
| Benchmark / Category | Metric | CogVideoX-5B | VideoREPA-5B | LDO-5B (Ours) | Relative Gain (vs Base) |
|---|---|---|---|---|---|
| VideoPhy Overall Physics | PC (%) ↑ | 32.0 | 37.8 | 45.6 | +13.6% |
| VideoPhy Overall Text-Align | SA (%) ↑ | 69.2 | 70.3 | 71.8 | +2.6% |
| VideoPhy Solid–Solid | PC (%) ↑ | 19.6 | 26.6 | 34.3 | +14.7% |
| VideoPhy Solid–Fluid | PC (%) ↑ | 33.6 | 37.7 | 43.8 | +10.2% |
| VideoPhy Fluid–Fluid | PC (%) ↑ | 60.0 | 67.3 | 80.0 | +20.0% |
On the PisaBench falling-object benchmark, LDO achieves consistent trajectory and shape consistency gains against PISA's specialized supervised baseline:
| Evaluation Set | Method | Trajectory Error L2 ↓ | Shape Error CD ↓ | Object Permanence IoU ↑ |
|---|---|---|---|---|
| Real Videos | Sora (OpenAI) | 0.174 | 0.488 | 0.065 |
| Real Videos | PISA-psft (Strong Baseline) | 0.079 | 0.194 | 0.138 |
| Real Videos | LDO (Ours) | 0.070 (-11.39%) | 0.170 (-12.37%) | 0.145 (+5.07%) |
| Simulated Videos | Open-Sora (Base) | 0.135 | 0.389 | 0.035 |
| Simulated Videos | PISA-psft (Strong Baseline) | 0.035 | 0.076 | 0.155 |
| Simulated Videos | LDO (Ours) | 0.030 (-14.29%) | 0.064 (-15.79%) | 0.154 (-0.65%) |
Ablation Study¶
Ablations on VideoPhy dissect the individual contributions of representation alignment (PDA) and trajectory reinforcement (GRPO):
| Config | Semantic Adherence SA ↑ | Physical Commonsense PC ↑ | Note |
|---|---|---|---|
| Vanilla diffusion (CogVideoX-5B) | 69.2 | 32.0 | Baseline denoising model |
| Only PDA (Feature alignment) | 72.4 | 39.2 | Highest semantic score (+3.2), strong physics boost (+7.2) |
| Only GRPO (Rollout reward) | 70.6 | 40.4 | Stronger physics prior (+8.4), slight semantic drop |
| All losses (Full LDO) | 71.8 | 45.6 | Complementary synergy yields peak physical accuracy (+13.6) |
Key Findings¶
- Complementary Synergy: PDA excels at retaining and even refining text alignment while injecting internal dynamics structure, whereas GRPO enforces global trajectory-level consistency. Combining both achieves a peak Physical Commonsense score of 45.6% without sacrificing semantic fidelity.
- Elimination of Unphysical Floating and Melting: Baselines frequently show books floating above table surfaces or falling objects melting into terrain upon collision. LDO maintains continuous boundary support and gravity-consistent parabolic freefall.
- Dramatic Gains on Complex Fluid Mechanics: In fluid-fluid interactions—where pixel entropy is exceptionally high—LDO boosts physical commonsense by +20.0%, proving that predictive latent spaces successfully isolate continuous state transformations from chaotic appearance fluctuations.
Highlights & Insights¶
- Information Bottleneck Diagnostic: Framed the root cause of physical hallucination in diffusion models as an objective misalignment where likelihood-based objectives waste capacity on high-entropy appearance factors rather than causal state transitions.
- Sampling-Agnostic Post-Training: Distills physical priors into weights during post-training, requiring zero modifications to the inference sampling schedule or additional compute overhead at test time.
- Relational Gram Distillation: By matching cosine similarity topology rather than forcing exact feature coordinate mimicry, LDO effectively injects dynamical structure while safeguarding generative diversity.
Limitations & Future Work¶
- Conservatism under Implicit Causality: When a prompt describes an outcome without an explicit causal force (e.g., "the bottle topples off the table"), LDO's strict physical prior may over-stabilize the object into remaining stationary to satisfy physical inertia, causing it to disregard the prompt.
- Teacher Capacity Ceiling: The fidelity and scope of physical knowledge transferred are strictly bounded by V-JEPA 2's pre-trained world model representation. Phenomena unseen by the teacher cannot be reliably transferred.
- Future Directions: Integrating explicit causal reasoning mechanisms in prompt processing to infer implicit physical agents, resolving the trade-off between strict physical realism and creative prompt execution.
Related Work & Insights¶
- vs VideoREPA: VideoREPA aligns diffusion features with static self-supervised visual encoders (e.g., DINOv2) using frame-level relational matching. LDO leverages a temporal predictive world model with causal autoregressive forecasting, achieving an additional +7.8% gain on Physical Commonsense.
- vs PISA: PISA relies on external object segmentation and bounding-box tracking heuristics to formulate physics rewards. LDO learns purely through latent world model predictability without needing specialized segmentation or tracking annotations.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates physical grounding through the Information Bottleneck lens and introduces V-JEPA 2 as both a feature teacher and an RL physics critic.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluations on VideoPhy and PisaBench, spanning T2V and I2V across multiple backbones with detailed ablations and failure case analysis.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptional clarity, mathematically self-consistent explanations, and seamless narrative progression.
- Value: ⭐⭐⭐⭐⭐ Provides a practical, sampling-free blueprint for advancing generative video models into reliable physical simulators.