ECCV2026 Video Generation physics-controllable generation part-aware semantic grounding physical property prediction temporal physics editing trajectory-conditioned generation

PhysChoreo: Physics-Controllable Video Generation with Part-Aware Semantic Grounding¶

Conference: ECCV2026
Paper: ECCV Paper
Area: Video Generation
Keywords: physics-controllable generation, part-aware semantic grounding, physical property prediction, temporal physics editing, trajectory-conditioned generation

TL;DR¶

PhysChoreo reconstructs objects in a single image as language-editable, part-level physical fields and drives a pretrained video model with temporally editable simulation, improving physical plausibility and temporal instruction alignment on the paper's 10 video evaluation cases without guaranteeing that the final pixels strictly follow the simulation.

Background & Motivation¶

Video models can produce convincing appearances without necessarily representing why an object moves in a particular way. A metal frame and soft padding should respond differently to the same impact; assigning a single "soft" or "hard" label to the whole object discards the structure that determines local deformation. Approaches such as PhysGaussian and PhysGen delegate dynamics to physical simulation, but the simulator still needs credible materials and parameters. Manually assigning one global material, or repeatedly optimizing an individual object using video diffusion priors, makes it difficult to obtain both fine-grained semantic editing and efficient initialization.

A second obstacle concerns time: initial velocity and gravity specify how motion begins, but do not naturally express a sequence such as "fall first, soften after impact, then change the applied force." Asking a video model to imitate the prompt does not expose which property or force causes the change; abruptly modifying stiffness inside a simulator can instead create numerical instability. The goal is therefore not merely material prediction, but connecting "which part, when, and how to change it" to an executable dynamical state.

Core Idea: predict an initial physical field through explicit correspondences between part descriptions and geometric points, translate temporal instructions into smooth simulation interventions constrained for stability, and let the video model turn the controlled trajectories into images. Physical plausibility here means a reasonable response under specified materials and forces, including deliberately counterfactual material changes, rather than a claim that these changes would occur spontaneously in the real world.

Method¶

Overall Architecture¶

The inputs are an image and text describing object attributes, local materials, and action timing; the output is an image-to-video result. Grounded-SAM first separates object instances and InstantMesh reconstructs their meshes. The system then predicts material classes and continuous physical properties for sampled points, fills the object volumes with particles, generates trajectories in an editable simulator, and passes them to Wan2.2-Fun-5B-Control for video generation.

Three connected components organize this process: part-aware physics reconstruction determines which material occupies each location, temporally editable physics simulation determines its response over successive stages, and trajectory-conditioned video generation determines how that motion appears. The predictor is trained, whereas the video generator is pretrained. Part annotations and physical regularizers supervise property learning; they are not additional ground-truth inputs available at inference time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Single image and text instructions"] --> Geometry["Instance segmentation and mesh reconstruction"]
    Geometry --> Reconstruction["Part-aware physics reconstruction"]
    Supervision["Part annotations and physical regularizers"] -.->|Training only| Reconstruction
    Reconstruction --> Simulation["Temporally editable physics simulation"]
    Input -->|Time-indexed actions| Simulation
    Simulation --> Generation["Trajectory-conditioned video generation"]
    Generation --> Output["Controlled video"]

Key Designs¶

1. Part-aware physics reconstruction: bind local descriptions to points instead of labeling the whole object

After uniform mesh-surface sampling, each point carries both geometric position features and part-semantic features from PartField. CLIP encodes a global description and optional part phrases. The central operation is soft assignment: points and part descriptions are projected into a shared space, normalized weights are computed over the phrases for each point, and the weighted text information is injected into point features. "Metal frame" and "soft cushion" therefore do not act equally on the whole object; they produce an inspectable point-to-phrase distribution. Algebraically, this is equivalent to single-head cross-attention. The paper emphasizes keeping the operation simple so that its assignments can be directly supervised, rather than presenting it as an entirely new attention operator.

After soft assignment, hierarchical cross-attention lets the points read global text before part text. The former supplies coarse object-wide conditions, and the latter refines local differences, reducing competition between prompts at different granularities. A 6-layer Transformer then aggregates non-local context and predicts 6 simulation material classes together with Young's modulus \(E\), Poisson's ratio \(\nu\), and density \(\rho\) for each point. These quantities describe stiffness, lateral deformation relationships, and mass distribution; a label such as "rubber" alone cannot replace them. Training also explicitly constrains within-part physical continuity, between-part response differences, and text assignments, as detailed below. The output is thus an editable spatial physical field rather than an image-level material classification.

2. Temporally editable physics simulation: control the entire process rather than only its initial conditions

The paper maintains physical properties, external forces, and motion states for objects or parts in material point method (MPM) and rigid-body solvers. GPT-5 decomposes natural language into global descriptions, part prompts, and solver instructions. The latter execute as predefined, time-indexed action sequences while simulation runs. For example, the system can first change a wind field and later modify a part's material or stiffness without reinitializing the scene. Physical properties, gravity or wind, and velocity constitute different control channels, turning pixel-level indications of movement into explicit interventions on its causes.

Numerical stability is the cost of this flexibility. For continuous physical parameters, an exponentially determined mixing weight gradually moves the current value toward the user's target instead of applying an instantaneous jump; solver step size and an object-specific time constant determine the transition rate. The system also limits stiffness through material wave speed: when the predicted maximum wave speed exceeds a threshold, it rescales Young's modulus with a shared coefficient, preserving relative stiffness while bounding extreme states. This can soften editing-induced discontinuities, but does not imply that arbitrary material changes obey real-world energy conservation. Equations (13)–(14) and several elasticity formulas are damaged in the extracted cache, so this note describes only the smoothing and bounding mechanisms explicitly supported by the prose rather than reconstructing exact author equations.

3. Trajectory-conditioned video generation: assign motion to simulation and appearance to the generative model

Property prediction operates on surface points, whereas MPM requires particles inside solid objects. The system uses Dust3r to estimate object positions and scales, then fills the volume through surface-to-interior propagation. Each interior particle inherits the properties of its nearest surface particle found by a nearest-neighbor search. This turns a semantically conditioned surface field into a simulatable volumetric representation, but the interior material is extrapolated rather than genuinely measured from the single image. This interface explains both how local property prediction enters the dynamics solver and why internal-state estimation remains a limitation.

The simulator produces point-cloud motion trajectories, which condition the pretrained Wan2.2-Fun-5B-Control model. The source image and text retain visual content and semantic requirements, trajectories constrain motion, and the generator supplies texture, lighting, and detail. The paper does not describe this as end-to-end retraining of the video backbone. Nor should stable simulated trajectories be equated with physically correct final video: the generator can still produce shadows or local appearances inconsistent with geometric changes. Its reported failure cases occur at this final stage, demonstrating that property prediction is not the source of every error.

A Worked Example¶

Consider an illustrative input showing a chair with a rigid frame and a soft cushion, accompanied by "fall first, then make the cushion softer." This example explains the method; it is not an additional experimental result reported by the paper.

Reconstruction first isolates the chair and obtains a mesh. Part features help distinguish frame from cushion, soft assignment injects the two descriptions into the corresponding points, and the predictor produces different local properties. With only one global material, the subsequent simulation could make both frame and cushion equally soft, losing local control before the motion even begins.

Simulation fills the interior with particles and first models falling under the initial properties. At the scheduled time, it smoothly modifies the cushion's parameters while applying stability bounds. The video model then receives the trajectory and turns the distinct part motions into continuous images. The paper demonstrates effects such as liquefaction after collision, but the interface explicitly described in the main text is time-indexed control; this does not establish a general collision-event detector and natural-language trigger system.

Loss & Training¶

Training data comprise 9,580 PartNet objects spanning 24 semantic categories. The authors clean part labels, use GPT-5 with renderings and category information to generate global and part descriptions and infer properties, and then calibrate these annotations with manually specified material constraints and GLM-4.5. They manually check 9% of samples and assign counterfactual material labels to another 5% of examples. The dataset maps real-material descriptions to simulation material tags, but "real material" here names an annotation category; it does not imply that every physical property was experimentally measured.

The point-wise task loss combines smooth regression on normalized continuous targets with cross-entropy for material classification. An additional wave-continuity loss connects only points from the same semantic part in a three-dimensional neighborhood graph and penalizes differences in longitudinal and shear wave speeds. Not imposing the same smoothing across parts avoids erasing material interfaces. Contrastive regularization uses a log-domain, normalized embedding of shear and bulk moduli, forms triplets with same-part positives and different-part negatives, and keeps elastic responses separable across parts. Prompt-assignment supervision aligns the point-to-phrase distribution with the corresponding part index, directly calibrating where a language edit takes effect.

The full objective is a weighted sum of these terms. Regression, classification, assignment, smoothness, and contrastive weights are respectively 1, 0.3, 0.1, 0.02, and \(5\times10^{-4}\). Because cached equations (7)–(12) contain missing symbols, this note does not reconstruct the exact wave-speed or triplet formulas. The prose is sufficient to distinguish the three supervisory roles: continuity within parts, separability between parts, and alignment between text and parts.

The implementation uses 96-dimensional PartField features and 96-dimensional positional MLP features, with frozen 256-dimensional CLIP text features. Soft-assignment temperature is 0.07, fused features are mapped to 512 dimensions, and attention uses 8 heads. The predictor is trained on 2 A6000 GPUs with AdamW, a learning rate of \(3\times10^{-4}\), batch size 32, and cosine decay. Validation-based early stopping uses patience 10 and an improvement threshold of \(10^{-4}\); the main text reports stopping at epoch 41.

Deployment is possible on one RTX 5090. Physics reconstruction and pose estimation take approximately 120 seconds, and simulation takes approximately 30 seconds. These timings do not specify the complete cost of final video generation and must not be reported as "150 seconds per finished video." Complex instructions also undergo multiple GPT-5 optimization rounds, so the final results include the contribution of prompt preparation.

Key Experimental Results¶

Main Results¶

Physical-property comparison randomly selects 100 samples from a held-out test set and maps all predictions to 6 simulation material classes. Mat.Acc. is material-class accuracy, where higher is better; continuous-property errors are evaluated on \([\log E,\nu,\log\rho]\), where lower is better. Because Pixie lacks a plastic class, the authors use a permissive mapping that also accepts elastic, metal, or rigid predictions for plastic ground truth. This is therefore not a direct comparison in each model's native label space.

Method	Mat.Acc. ↑	log E error ↓	Poisson's ratio error ↓	log rho error ↓
NeRF2Physics	0.628	2.033	0.064	0.521
PUGS	0.283	2.778	0.076	0.627
Pixie	0.349	4.129	0.103	0.848
PhysChoreo	0.789	0.661	0.061	0.249

The table reproduces the selected results from original Table 1. Relative to NeRF2Physics, material accuracy improves by 16.1 percentage points, while log E error falls from 2.033 to 0.661. The improvement in Poisson's ratio error is much smaller; changes in differently scaled quantities should not be described as uniform gains.

Video evaluation covers 10 generated cases. Gemini-2.5-Pro assigns scores on a 5-point scale for three dimensions: SA measures alignment of content and motion with text, especially temporal instructions; PC measures physically plausible deformation and dynamics; VQ measures visual detail quality. The following results come from original Table 3, not from objective conservation-error measurements.

Method	SA ↑	PC ↑	VQ ↑	AVG ↑
PhysGen3D	2.30	2.10	3.50	2.63
PhysCtrl	3.75	4.25	4.30	4.1
Wan2.2-5B	1.75	1.70	4.20	2.55
CogVideoX-3	2.40	2.55	4.15	3.04
Veo 3.1	4.10	4.20	4.90	4.40
PhysChoreo	4.70	4.55	4.75	4.67

The authors also collect 642 valid selections from 31 users; original Table 2 reports an overall preference share of 50.1% for PhysChoreo. It achieves the highest average VLM score but lower VQ than Veo 3.1. Better physical control must not be restated as superiority on every visual metric.

Ablation Study¶

Original Table 5 removes individual physical-supervision terms. Iter. denotes the early-stopping epoch, not the number of solver steps. Total Err. is retained as reported because the provided main text does not explicitly specify its aggregation details. It is not video error, and accuracy in this table should not be conflated with the 100-sample comparison in Table 1.

Config	Mat.Acc. ↑	Total Err. ↓	Iter.
Without prompt-assignment supervision	0.8534	0.3753	50
Without wave-continuity supervision	0.8578	0.3579	37
Without contrastive regularization	0.8310	0.3451	38
Full model	0.8605	0.3318	41

Key Findings¶

Removing prompt-assignment supervision changes early stopping from 41 to 50 epochs, suggesting that explicit geometry-language correspondence affects learning efficiency rather than merely enabling visualization.
Removing contrastive regularization reduces accuracy to 0.8310, the largest classification degradation in Table 5. Its role is consistent with maintaining separability between material responses.
In original Table 4, adding soft assignment to dual-stage attention reduces Total Err. from 0.5801 to 0.5046 and changes early stopping from 36 to 26. Adding the part prior further reduces error to 0.3318 but increases early stopping to 41; not every component accelerates training.

Highlights & Insights¶

An interpretable control interface matters more than simply adding text attention. Direct supervision of point-to-part-phrase assignments gives "which part to edit" a geometric meaning before video generation begins.
Physical supervision distinguishes continuous regions from material interfaces. Within-part wave-speed smoothing and between-part elastic contrast serve different purposes, avoiding excessive homogenization of the object.
Temporal control acts on state evolution. Smooth updates and stiffness bounds allow language edits to enter a running solver, a principle that could transfer to parameter scheduling in interactive animation.

Limitations & Future Work¶

Author-acknowledged: the method focuses on independent objects and remains insufficient for large scenes; surface sampling and interior filling cannot precisely recover internal physical states.
Author-demonstrated: video rendering can produce incorrect lighting, shadows, and detail artifacts. Stable simulation trajectories do not automatically yield physically correct pixels.
This note's assessment: labels are largely inferred by models and calibrated with rules, with 9% manually checked. Prediction errors therefore measure agreement with that annotation system, not necessarily accuracy against measured physical materials.
This note's assessment: 10 video cases, VLM scores, and preference votes support preliminary effectiveness, but broader and conservation-based validation is missing. Multiple instruction-optimization rounds also make one-shot natural-language control difficult to isolate.
A possible next step is to represent uncertainty about invisible internal structure and measure generated-video deviations from simulated trajectories, separately quantifying property-estimation errors and rendering departures.

vs NeRF2Physics / PUGS: these baselines infer physical properties through multiple views and vision-language models. This work trains a three-dimensional predictor explicitly conditioned on part text, making local properties editable conditional outputs.
vs Pixie: Pixie builds physical fields from multi-view features; this work emphasizes directly combining point geometry, part semantics, and language at multiple granularities. Comparison must retain the caveat that six-class material mappings were adapted.
vs PhysGen3D / PhysCtrl: all connect physical motion with video generation. This paper emphasizes part-level initial properties and temporal orchestration during simulation rather than improvements to the video backbone alone.
Implication for video generation: interpretable state variables offer a useful control interface for counterfactual effects, but the final images still require an independent check of whether they faithfully realize those states.

Rating¶

Novelty: 4/5. Combining part-semantic physical fields with temporally editable simulation is well motivated, although the underlying attention and simulation operators are not entirely new.
Experimental Thoroughness: 3/5. Prediction, video evaluation, and component ablations complement each other, but video coverage and validation against measured physics or conservation laws remain limited.
Writing Quality: 4/5. Module motivations and interfaces are clear, but the main text leaves aggregation of errors and some evaluation protocols insufficiently specified.
Value: 4/5. Provides interpretable physical orchestration for controllable video and interactive-animation research, but should not be treated directly as a high-precision physical digital twin.