PhysPO: Physics-Aware Local Preference Optimization for Physically Consistent Video Diffusion

Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://shanpoyang654.github.io/PhysPO/page.html
Code: https://shanpoyang654.github.io/PhysPO/page.html
Area: Video Generation (video_generation)
Keywords: Video Generation, Physical Plausibility, Local Preference Optimization, Counterfactual Data, Diffusion Model

TL;DR

PhysPO resolves physical commonsense violations in video diffusion by creating a counterfactual pair generation pipeline (CounterPhyPipe) that preserves global scene identity while introducing local physical violations, coupled with a local DPO framework that isolates physically evolving regions via spatiotemporal high-frequency filtering and enforces background neutrality regularization.

Background & Motivation

Text-to-video (T2V) diffusion models have made remarkable progress in synthesizing high-resolution, long-duration, and visually compelling videos. However, ensuring physical plausibility remains a fundamental roadblock. Leading foundation models, including Wan2.1, CogVideoX, and HunyuanVideo, frequently generate videos that blatantly violate real-world physical laws, such as metal pendulums swinging against gravity in dynamics, candle flames showing unnatural abrupt upward motion in thermodynamics, or drinking straws exhibiting misplaced refraction angles at water boundaries in optics. Previous attempts to inject physical priors either resort to physics simulation engines (which fail to scale beyond simple parameterized environments) or rely on iterative LLM-prompt enrichment (which incurs heavy inference latency without fundamentally improving the generator's intrinsic physical dynamics).

Supervised fine-tuning (SFT) on curated video datasets also fails to reliably ensure physical consistency because it lacks contrastive negative signals. Without explicit penalties on physically implausible patterns, models easily collapse into exploiting shortcut visual correlations. While Direct Preference Optimization (DPO) offers a promising alternative, standard video DPO encounters three severe bottlenecks: first, generating multiple candidates per prompt to sample preference pairs requires exorbitant sampling time; second, pairs generated from independent noise initializations exhibit drastically different camera motions, compositions, and scene layouts, causing optimization signals to become entangled with global aesthetic variations rather than isolated physical dynamics; third, conventional DPO optimizes videos monolithically, spreading gradients uniformly over the whole video frame and thereby diluting supervisory focus on localized physical failures while corrupting non-physical background areas.

This paper tackles these challenges by shifting from global, uncontrolled video comparisons to strictly controlled counterfactual supervision with localized gradient routing. The core idea is to construct a counterfactual preference dataset (CounterPhy) preserving global scene identity with isolated local physical violations, and localize dynamic transition regions via spatiotemporal high-frequency decomposition to apply adaptive local preference optimization while enforcing neutrality on background tie regions.

Method

Overall Architecture

PhysPO consists of two coordinated pipelines: the CounterPhyPipe counterfactual data construction pipeline and the PhysPO local preference optimization framework. CounterPhyPipe takes high-quality real physical videos as preferred samples, uses a multimodal LLM and an image editor to generate counterfactual end frames, and synthesizes dispreferred negative videos using a first-last-frame-to-video generator under physics-aware VLM filtering. Subsequently, during post-training, frame-wise 2D and temporal 1D Fourier high-pass filtering extract a unified spatiotemporal physical mask identifying structural deformation and velocity shifts. Finally, optimization is localized: preference loss is concentrated within the physical mask weighted by physical implausibility, while a tie-region neutrality regularization suppresses background gradients.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Real Video & Caption Input"] --> B["Counterfactual Video Generation<br/>CounterPhyPipe constructs paired samples"]
    B --> C["Spatiotemporal High-Frequency Mask<br/>2D & 1D FFT isolates entities & state transitions"]
    C --> D["Local Preference Optimization LPA-DPO<br/>Adaptive penalty modulated by 1-PC"]
    C --> E["Tie-Region Neutrality Tie-DPO<br/>Zero preference difference guided by SA"]
    D --> F["Hybrid Training Objective<br/>Physically aligned video generator"]
    E --> F

Key Designs

1. Counterfactual Video Generation Pipeline: Surgical Local Violations with Identical Scene Layout

To overcome the heavy computational cost of multi-candidate generation and the lack of apples-to-apples correspondence in independent-noise pairs, CounterPhyPipe constructs preference pairs directly from high-quality real videos \(x^w\). A representative start frame \(i_{start}\) with maximal text-frame alignment is first identified via a cross-modal embedding model. A multimodal LLM (Qwen3-VL) inspects \(i_{start}\) along with the caption \(c\) and proposes an alternative, physically implausible proposal \(c^l\) (e.g., "the liquid spills upward defying gravity"). Next, an image editing model (Qwen-Image-Edit) modifies \(i_{start}\) according to \(c^l\) to create an erroneous ending keyframe \(i_{end}\). A first-last-frame-to-video model (Wan2.1-FLF2V) then synthesizes the dispreferred counterpart \(x^l\) conditioned on \((i_{start}, i_{end}, c^l)\). To eliminate semantic drifting during interpolation, a physics VLM (VideoCon-Physics) filters the generated samples by retaining only those with high semantic adherence (\(SA\)) and low physical commonsense (\(PC\)). Thanks to the strong boundary conditioning of FLF2V, the filtering rejection rate is only ~8%, yielding over 30K tightly controlled pairs in the CounterPhy dataset without multi-candidate sampling overhead.

2. Spatiotemporal High-Frequency Physical Mask: Unsupervised Isolation of Dynamics and Deformations

Physical violations inherently manifest as abnormal variations in object geometry or temporal transitions, which are predominantly concentrated in high-frequency spectral components. Rather than relying on fragile manual annotations or noisy open-vocabulary segmentors, PhysPO isolates these regions directly in latent space using frequency-domain decomposition. Spatially, a frame-wise 2D Fourier transform and radial high-pass filter isolate high-frequency spatial boundaries:

\[ H_s(u, v) = \mathbb{I}\left(\sqrt{u^2 + v^2} > \tau_s\right), \quad \tau_s = \beta_s f_{s}^{\max} \]

Inverting back to the spatial domain with an adaptive frame-wise threshold \(\delta_s = \mu_s + \alpha_s \sigma_s\) produces the spatial mask \(M_{sp}\), which captures physical object contours. Temporally, a 1D Fourier transform along the temporal dimension at each spatial coordinate combined with high-pass filtering \(H_t(f) = \mathbb{I}(|f| > \tau_t)\) and threshold \(\delta_t = \mu_t + \alpha_t \sigma_t\) generates the temporal mask \(M_{tp}\), isolating abrupt velocity shifts and state discontinuities. Applying this decomposition symmetrically across \(x^w\) and \(x^l\) and taking their union yields the unified candidate physical mask:

\[ M = M_{sp}^w \cup M_{tp}^w \cup M_{sp}^l \cup M_{tp}^l \]

3. Local Preference Optimization with Tie-Aware Neutrality: Dual-Zone Decoupled Gradient Guidance

Standard DPO applies uniform loss over the entire spatio-temporal volume, allowing diffusion denoisers to exploit background artifacts rather than rectifying physical errors. PhysPO partitions the latent space into the physical mask \(M\) and the non-physical tie region \(1 - M\). Inside \(M\), an implicit reward difference \(s'(x^*, c, t, \theta, M)\) is defined, and local preference loss \(\mathcal{L}_{\text{PA-DPO}}\) is computed:

\[ \mathcal{L}_{\text{PA-DPO}} = -\mathbb{E}_{d \sim \hat{\mathcal{D}}}\left[\log \sigma\left(\beta \cdot (1 - PC) \cdot \mathbb{E}_t \left[ s'(x^w, t, \theta, M) - s'(x^l, t, \theta, M) \right]\right)\right] \]

Here, \((1 - PC)\) acts as an adaptive penalty scaler reflecting violation severity. Outside \(M\), the preferred and dispreferred videos share identical scene semantics and should be treated as neutral ties. PhysPO introduces a tie-aware neutrality regularization \(\mathcal{L}_{\text{Tie-DPO}}\) modulated by semantic adherence \(SA\):

\[ \mathcal{L}_{\text{Tie-DPO}} = \mathbb{E}_{d \sim \hat{\mathcal{D}}}\left[ SA \cdot \left| \mathbb{E}_t \left[ s'(x^w, t, \theta, 1 - M) - s'(x^l, t, \theta, 1 - M) \right] \right| \right] \]

Combined with full-frame standard diffusion DPO \(\mathcal{L}_{\text{DPO}}\) in a hybrid objective \(\mathcal{L}_{\text{total}} = \lambda_{\text{PA-DPO}}\mathcal{L}_{\text{PA-DPO}} + \lambda_{\text{Tie-DPO}}\mathcal{L}_{\text{Tie-DPO}} + \lambda_{\text{DPO}}\mathcal{L}_{\text{DPO}}\), PhysPO concentrates learning on physical laws while completely eliminating background shortcut exploitation.

Key Experimental Results

Main Results

PhysPO is evaluated on VideoPhy and PhyGenBench using Wan2.1-T2V-1.3B and CogVideoX-5B backbones, assessing both Semantic Adherence (SA) and Physical Commonsense (PC).

Table 1: Benchmark Results on VideoPhy

Methods	Solid-Solid SA/PC	Solid-Fluid SA/PC	Fluid-Fluid SA/PC	Overall SA↑	Overall PC↑	Inference Time (s)
VideoCrafter2	50.4 / 32.2	50.7 / 27.4	48.1 / 29.1	50.3	29.7	-
DreamMachine	55.1 / 21.7	59.6 / 23.3	58.2 / 18.2	57.5	21.8	-
LaVie	40.8 / 18.3	48.6 / 37.0	69.1 / 50.9	48.7	31.5	-
HunyuanVideo	55.2 / 16.1	67.1 / 30.1	54.5 / 54.5	60.2	28.2	1080
Cosmos-Diffusion-7B	- / -	- / -	- / -	57.0	18.0	600
PhyT2V	- / -	- / -	- / -	61.0	37.0	1800
WISA	- / -	- / -	- / -	67.0	38.0	220
PhysMaster	- / -	- / -	- / -	67.0	40.0	26
Wan2.1-T2V-1.3B Baseline	34.9 / 10.0	64.3 / 25.4	43.5 / 48.1	49.0	24.0	180
PhysPO-Wan (Ours)	55.4 / 26.0	80.1 / 44.7	62.0 / 81.5	67.0 (+36.7%)	42.6 (+77.5%)	180
CogVideoX-5B Baseline	53.1 / 18.2	75.3 / 32.9	56.4 / 61.8	63.1	31.4	170
VideoREPA-CogVideoX	58.0 / 28.0	82.9 / 39.0	80.0 / 74.5	72.1 (+14.3%)	40.1 (+27.7%)	170
PhysPO-CogVideoX (Ours)	62.3 / 31.8	82.5 / 52.2	87.0 / 88.9	74.7 (+18.4%)	49.4 (+57.3%)	170

Table 2: Benchmark Results on PhyGenBench

Methods	Semantic Adherence SA↑	Physical Commonsense PC↑
VideoCrafter	27.0	20.0
OpenSora	23.0	21.0
Cosmos	43.0	14.0
PhyT2V	38.0	42.0
WISA	40.0	43.0
Wan2.1-T2V-1.3B Baseline	30.0	22.0
PhysPO-Wan (Ours)	43.0 (+43.4%)	46.0 (+109.1%)
CogVideoX-5B Baseline	39.0	41.0
PhysPO-CogVideoX (Ours)	52.0 (+33.3%)	64.0 (+56.1%)

Ablation Study

Ablations on Wan2.1-T2V-1.3B demonstrate the individual contributions of dataset quality, mask-guided preference loss, and neutrality regularization.

Table 3: Ablation Results on Wan2.1-T2V-1.3B (PhyGenBench)

Settings	SA↑	PC↑	Note
PhysPO (Full Model)	43.0	46.0	Full proposed model
w/o CounterPhy	35.0 (-18.6%)	38.4 (-16.5%)	Replacing with randomly sampled noise negatives
w/o PA-DPO + Tie-DPO (VanillaDPO)	33.2 (-22.8%)	26.0 (-43.5%)	Standard full-frame DPO without masking
- w/o PA-DPO	39.1 (-9.1%)	29.4 (-36.1%)	Mask removed; unweighted global optimization
- w/o Tie-DPO	34.6 (-19.5%)	40.2 (-12.6%)	Background neutrality regularization removed

Key Findings

• Local physical masking is critical for physical reasoning: Removing PA-DPO results in a 36.1% relative drop in PC score (from 46.0 to 29.4). Reverting entirely to VanillaDPO triggers an acute 43.5% collapse in PC. Without spatial and temporal masking, gradients disperse across static areas, failing to provide sufficient learning signals to correct dynamics.
• Tie-region neutrality prevents semantic degradation: Discarding Tie-DPO yields a substantial 19.5% drop in SA (from 43.0 to 34.6). Qualitatively, models without Tie-DPO tend to alter scene elements inappropriately (e.g., pouring liquid outside a cup) to fulfill overall preference loss differences.
• Counterfactual pairs outperform independent noise sampling: Training with randomly generated dispreferred videos rather than CounterPhy pairs leads to noticeable drops in both SA (-18.6%) and PC (-16.5%), confirming that apples-to-apples scene alignment is essential for learning genuine causal physical transitions.

Highlights & Insights

• Scalable Counterfactual Negative Synthesis: By pivoting from independent random sampling to start-end frame conditioned generation guided by MLLM physical inversions, CounterPhyPipe synthesizes controlled physical counterfactuals with an ~8% filter rate, drastically reducing annotation and generation costs.
• Unsupervised Physical State Localization via FFT: Utilizing 2D and 1D Fourier transforms to build spatiotemporal masks leverages the fundamental insight that physical violations produce high-frequency edge tearing and temporal velocity steps, avoiding expensive object detectors.
• Dual Decoupled Optimization Dynamics: Formulating adaptive reward scaling (\((1 - PC)\) for the local violation mask and \(SA\) for the tie region) creates a clean separation of concerns, eliminating background shortcut learning while stabilizing global video semantics.

Limitations & Future Work

• Dependence on Upstream Generative Components: The fidelity of CounterPhy relies on Qwen-Image-Edit and Wan2.1-FLF2V. In highly complex multi-object collisions or chaotic fluid behaviors, upstream generation failures can inject noisy counterfactual samples.
• Sensitivity of High-Pass Filtering to Complex Camera Motions: Aggressive dynamic camera pans or high-frequency environmental textures (e.g., dense foliage, turbulent water surfaces) may trigger false-positive activations in the high-pass mask, expanding the supervised mask beyond the genuine physical event.
• Future Directions: Integrating explicit 3D geometry or monocular depth/motion flow could sharpen mask extraction under rapid camera movement; expanding this localized preference paradigm to embodied AI world models offers substantial promise.

Related Work & Insights

• vs Vanilla Video DPO (e.g., VideoDPO, Diffusion-DPO): Standard video DPO optimizes over the entire latent volume using pairs with disparate backgrounds, leading to gradient noise and background hacking; PhysPO enforces controlled counterfactual pairs and restricts preference updates to high-frequency motion regions.
• vs PhyGDPO & PhysMaster: PhyGDPO still suffers from scene discrepancy by pairing real videos with independently generated samples; PhysMaster demands complex multi-step physics priors; PhysPO achieves a 350.8% boost on VideoPhy2's Hard track over baseline, outperforming PhyGDPO (0.0532 vs 0.0444).
• vs SFT Approaches (WISA, VideoREPA): SFT lacks explicit contrastive negative signals and frequently overfits to surface correlations; PhysPO provides explicit negative penalties, outperforming VideoREPA-CogVideoX on VideoPhy PC (49.4 vs 40.1).

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant combination of counterfactual first-last-frame data generation and spatiotemporal high-frequency masking for local preference optimization.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Thoroughly benchmarked on VideoPhy, PhyGenBench, and VideoPhy2 with detailed ablations and qualitative interaction studies.
• Writing Quality: ⭐⭐⭐⭐⭐ Lucid problem formulation, rigorous mathematical definitions, and intuitive visual illustrations.
• Value: ⭐⭐⭐⭐⭐ Solves the fundamental efficiency and shortcut bottlenecks in video physical alignment, offering high practical value for video generation and world simulator research.

Related Papers

• [ECCV 2026] PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generation
• [ECCV 2026] Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints
• [ECCV 2026] SIFT: Self-Imagination Fine-Tuning for Physically Plausible Motion in Video Diffusion Models
• [ECCV 2026] Salt: Self-Consistent Distribution Matching with Cache-Aware Training for Fast Video Generation
• [ECCV 2026] DualCamCtrl: Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generation