Learning to Generate Rigid Body Interactions with Video Diffusion Models¶

Conference: ECCV2026
Paper: ECCV Paper
Project: KineMask
Authors: David Romero, Ariana Bermudez, Viacheslav Iablochnikov, Hao Li, Fabio Pizzati, Ivan Laptev
Area: Video Generation
Keywords: velocity masks, rigid body interactions, two-stage training, synthetic data, text conditioning

TL;DR¶

KineMask combines velocity-mask control, two-stage training that removes future motion conditions, and event descriptions to generate subsequent rigid-body interactions from an image and initial velocity, improving motion and interaction quality in synthetic tests and perceptual evaluation of real-world scenes.

Background & Motivation¶

Video generation can maintain convincing appearance and temporal coherence, yet collisions can still produce interpenetration, disappearing objects, or movement without contact. These may be local defects in creative videos, but they make the relationship between actions and consequences unreliable in a world model. Some existing methods reconstruct a scene before computing motion with a physics simulator, although geometry, materials, and occlusion are already difficult to recover from one image. Drag-based control can instead move an object along a specified path, but requires the user to supply its future trajectory and may implicitly prescribe its post-collision destination. Consequently, following a path and predicting consequences from initial motion are different tasks.

This paper addresses the latter: a user draws a velocity arrow on an object, and the model generates its subsequent motion and effects on other objects. The difficulty is that first-frame conditioning is sparse, making direct control training difficult; providing full trajectories throughout training instead creates dependence on information unavailable at inference. The authors therefore establish control with complete velocity masks before removing future conditions to require dynamics generation from initial states. Training uses interactions between boxes and cylinders in Blender, where motion changes following contact have explicit support in the data. However, these simple shapes do not cover effects such as shattering or splashing, so the system also needs to retain the visual and event knowledge of its pretrained video backbone.

Text is not a replacement for velocity here: it describes events that interactions may cause, allowing low-level motion signals to activate richer generative priors. Training descriptions come from complete synthetic videos, whereas GPT-5 predicts inference descriptions from the input image and motion direction; these information sources must be distinguished. This setup combines synthetic motion supervision with pretrained real-world knowledge, without introducing a verifiable physics solver. Core Idea: learn control from dense velocity masks, remove future conditions to learn continuation, and use event descriptions to retain the ability to generate complex interaction effects.

Method¶

Overall Architecture¶

The inputs are a scene image, a target object, and its user-specified initial velocity; the output is a video of future motion and interactions. The base model is CogVideoX-I2V-5B: the video backbone remains frozen while an additional ControlNet branch learns to receive velocity masks. The system has three key designs: velocity-mask encoding, two-stage mask-dropout training, and high-level text conditioning. Training videos supply actual motion and retrospective event descriptions; inference supplies only initial masks and predicted descriptions, not ground-truth future trajectories. Solid arrows below indicate data or conditioning, while dashed arrows indicate learned parameters and training-only text.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Data["Blender videos and velocities"] --> Mask["Velocity-mask encoding"]
    Mask --> Train["Two-stage mask-dropout<br/>training"]
    Data --> Text["High-level text conditioning"]
    Input["Input image and velocity arrow"] --> Text
    Input --> Initial["SAM2 and initial velocity mask"]
    Text -.->|Training video descriptions| Train
    Train -.->|ControlNet parameters| Model["Frozen video backbone<br/>and trained ControlNet"]
    Initial --> Model
    Text -->|Predicted event description| Model
    Input -->|Reference image| Model
    Noise["Random noise"] --> Model
    Model --> Output["Future interaction video"]

Key Designs¶

1. Velocity-mask encoding: specify both the moving object and its motion

A binary mask describes object location but cannot distinguish leftward from rightward motion or encode initial speed. KineMask fills the object segmentation region with a three-channel instantaneous velocity, using red, green, and blue for the three axes of the velocity vector. Stacking these masks over frames produces a spatiotemporal control tensor containing object shape, location, and motion state. The colors encode control values rather than modify the object's actual appearance; the image-to-video backbone still receives the reference image. Blender supplies boxes, cylinders, textured backgrounds, and random initial velocity directions and magnitudes, making these supervision signals directly available. Interactions includes collisions, while Simple Motion contains isolated moving objects, separating the ability to move objects from the ability to model interactions.

A crucial restriction is that only objects already moving in the first frame receive this encoding; objects set in motion by collisions never receive control masks. Thus, even during dense conditioning, the response of a struck object cannot be prescribed pixel by pixel and must be learned from the training videos. At inference, SAM2 extracts the target region from the image, and the user arrow is converted into its initial velocity encoding. Initial masks can control several active objects simultaneously; the paper shows collisions between them and with a third object. The main text does not fully specify how numerical velocities are calibrated to RGB ranges, so arrow length should not be interpreted as a calibrated real-world speed. This representation provides object-level motion conditioning rather than a complete state describing mass, friction, or joint actuation.

2. Two-stage mask-dropout training: transition from following complete motion conditions to continuation from initial states

The first stage supplies per-frame velocity masks for actively moving objects, allowing ControlNet to learn how dense control maps to shape and motion in videos. Training only with the first frame from the outset would require the control branch to learn mask interpretation and future-state prediction simultaneously. The second stage continues from the first-stage parameters and randomly removes some later velocity conditions, filling the removed entries with zeros. The main text describes sampling biased toward earlier frames because interactions usually occur early in the synthetic sequences; its exact distribution is deferred to the unavailable appendix. The model consequently encounters different amounts of missing conditioning and must complete motion without all future masks. What is removed is the control input, not the target video frames: complete videos still supervise generation.

At final inference, only the initial mask remains and all subsequent control slots are zero:

\[ \widetilde{m}=[m_0,0,\ldots,0]. \]

This directly follows the inference setting in Section 3.2; zero denotes absent conditioning, not a command for the object to stop. The struck object's velocity changes, the active object's deceleration, and its eventual location must be generated rather than supplied by the user. Table 1 tests whether dense pretraining is necessary by comparing against second-stage-only training, supporting the use of both stages. A separate comparator receives ground-truth masks for every test frame and is a privileged upper bound with future information, not a deployable method under the same input budget.

3. High-level text conditioning: activate generative priors beyond the geometric training set

Simple collision data can teach when one object should push another, but does not directly contain complex phenomena such as overflowing liquid or shattering ceramics. The authors use Tarsier to describe complete training videos, emphasizing interactions and their consequences instead of merely stating that an object moves on a surface. These captions remain tied to synthetic training elements and do not add shattering or fluid-simulation videos to the training set. Their purpose is to keep the text channel active during control training, preventing exclusive reliance on velocity signals from weakening responses to complex event prompts. At inference, GPT-5 observes the new image and motion direction, predicts an outcome description, and supplies it alongside the initial velocity mask. Complex effects in the final video therefore reflect motion conditioning, predicted text, and the pretrained backbone's knowledge together.

The paper contrasts a fixed short prompt with rich event descriptions to test the importance of retaining text semantics during training. In qualitative real-world examples, training with rich descriptions better supports shattering pots and water disturbances, even though these effects are absent from the simple geometric training data. However, a language description can itself predict an incorrect outcome, and the generator has no separate physics module that verifies it. Generating semantically plausible interactions should therefore not be taken to mean that the visual control branch alone recovers true dynamics. Velocity specifies how the user wants motion to begin, while text constrains possible events; the two are complementary but have no hard consistency guarantee. This also explains why text can improve perceived effects without improving every motion-distance metric.

A Worked Example¶

Consider a left cup moving toward a right cup, an explanation of the Figure 1 pipeline rather than an additional experiment. The user draws an arrow only on the left cup, SAM2 extracts its shape, and initial velocity values fill that region while later control frames remain zero. The right cup is initially stationary, so the user need not specify its post-collision position or trajectory. GPT-5 describes a possible collision and subsequent effects from the scene and direction, providing text conditioning to the video backbone. The trained control branch injects the left cup's motion condition during denoising, while the backbone generates contact, the right cup's response, and possible liquid effects. Removing the right cup means the same initial arrow no longer implies the same endpoint, because the entire scene still determines the generated continuation. Figure 7 shows related response differences when velocity or the struck object changes, but reports neither physical mass-estimation error nor conservation error.

Loss & Training¶

Training retains the video diffusion generation objective and optimizes the control branch, without introducing an additional collision-conservation or Newtonian dynamics loss. The main text presents a KL objective over reverse diffusion conditionals and states that it is implemented through noise prediction. Because the cached equations contain corrupted symbols, this note does not reconstruct them as an exact reproducible objective. The stages share the video generation objective; their key difference is changing the control tensor from dense to randomly truncated, not introducing a new supervision task. The CogVideoX implementation attaches ControlNet to the first 8 layers with a control weight of 0.5 and generates 49 frames at inference. Each synthetic dataset has 10,000 training samples and 100 test samples, with test colors and textures disjoint from training. Real World contains 50 images collected online or generated with ChatGPT and has no ground-truth future trajectories for framewise motion evaluation. The main text does not fully specify training steps, learning rates, or the mask-dropout distribution; the referenced appendix is absent from this cache, so reproduction needs supplementary material.

Key Experimental Results¶

Main Results¶

Table A excerpts original Table 2 on page 11, evaluating training data and model choices on the synthetic Interactions test set. This experiment isolates low-level motion control with a fixed short prompt; it is not an error table for the rich-text full system on real-world scenes. MSE measures pixel error against ground-truth videos, FVD measures video distribution differences, and FVMD measures motion-feature distribution differences; lower is better for all three. IoU measures intersection over union between SAM2-extracted generated object masks and ground-truth masks, with higher being better; the original tables spell FVMD as FMVD, normalized here to the metric name in the prose.

Method	Training Data	MSE ↓	FVD ↓	FVMD ↓	IoU ↑
CogVideoX	No task fine-tuning	344.6	807.3	3514.9	0.192
CogVideoX	Interactions	201.9	368.2	162.9	0.243
KineMask	Simple Motion	166.2	301.1	160.5	0.334
KineMask	Interactions	158.7	250.7	143.8	0.355

With the same KineMask architecture, interaction data reduces FVD from 301.1 to 250.7, an absolute decrease of 50.4, showing that learning translation alone does not replace learning contact consequences. CogVideoX fine-tuned on interaction data still trails KineMask, supporting the role of object-level conditioning rather than merely exposure to more synthetic videos.

Ablation Study¶

Table B excerpts original Table 1 on page 10, evaluating training stages on Simple Motion to isolate control from complex text and collisions. KineMask uses a fixed short prompt for low-level control training and inference; the text separately says the CogVideoX comparator uses video descriptions, so that comparison does not purely change the number of stages. The key controlled comparison is second-stage-only versus two-stage training, while the final row additionally receives true control masks for all future frames.

Config	Test Control	MSE ↓	FVD ↓	FVMD ↓	IoU ↑
CogVideoX	No velocity masks	158.3	601.1	1504.6	0.051
Second stage only	Initial velocity mask	86.3	288.8	201.0	0.237
Two stages	Initial velocity mask	47.2	160.3	199.8	0.367
First stage + full masks	Per-frame true velocity masks, privileged upper bound	24.9	89.7	165.6	0.684

Compared with second-stage-only training, two stages reduce MSE from 86.3 to 47.2 and increase IoU from 0.237 to 0.367, whereas FVMD only decreases from 201.0 to 199.8. The gains are therefore particularly apparent in spatial and shape consistency, rather than equally large improvements across all motion metrics. The full-mask upper bound still achieves an IoU of 0.684, indicating a substantial gap between free continuation and access to the true future.

Key Findings¶

The discussion of Table 3 on page 13 reports IoU of 0.376 with rich-text training versus 0.356 for the second-best configuration; the corresponding FVD values discussed are 231.3 versus 238.8.
The same passage reports a motion metric of 174.4 for the rich-text model versus the table's best value of 143.8, explicitly showing a trade-off between text-supported effects and motion consistency.
Rows and columns in the cached Table 3 are merged, so the full table is not reconstructed; only numbers explicitly supported by the prose are quoted, without inferring MSE or row assignments from damaged layout.
The user study on page 9 involves 30 participants, comparing motion fidelity, interaction realism, and overall physical consistency while allowing equal-quality answers.
Preference percentages in Figures 5 and 12 cannot be read from the current text, so only the authors' reported improvement direction is retained; no percentages are invented for Wan2.2-5B or Cosmos2.5-2B extensions either.

Highlights & Insights¶

Learning dense control before removing conditions provides a practical curriculum from supervised trajectory following to initial-state generation. It simplifies learning the control signal while keeping deployment free of future-trajectory requirements.
Struck objects never receive direct velocity masks, making their responses something the model must learn. Otherwise, dense conditions could encode the entire apparent physical effect in advance.
The data ablation separates appearance transfer from interaction transfer. Preserving convincing textures on real objects does not imply that translation-only training suffices for collisions.
The text channel helps preserve complex event capabilities of the pretrained model. This conditioning-preservation idea is reusable, but behavior under conflicts between text and low-level motion needs separate evaluation.

Limitations & Future Work¶

The authors explicitly limit control to velocity, whereas real motion also depends on mass, friction, shape, and air resistance; the method neither explicitly inputs nor estimates this complete parameter set.
The method focuses on center-of-mass motion rather than part-level or joint actuation; training cameras are static, camera motion is not explicitly controlled, and soft-body interactions remain future work.
Reader assessment: real-world scenes lack future-trajectory ground truth, so preferences and selected examples support greater plausibility rather than accurate mass estimation, causal identification, or physical conservation.
Reader assessment: predicted text can introduce useful event priors or incorrect outcomes; incorrect descriptions, occlusion, and repeated long-horizon generation require independent failure testing.
Stronger validation should report velocity, collision timing, and conservation errors under controlled physical parameters, separating real photographs from generated images; these are suggestions, not completed experiments.

Versus PhysGen, PhysGen3D, and WonderPlay: these approaches combine scene recovery with physical simulation, whereas KineMask confines the simulator to training-data creation and lets the video model predict outcomes at inference.
Versus TORA and MotionI2V: trajectory control requires future paths; initial-velocity conditioning makes the path scene-dependent, at the cost of lacking an explicit physical guarantee for subsequent motion.
Versus InterDyn: it relies on per-frame masks of controlling elements, whereas KineMask reduces deployment conditioning to the initial frame through two-stage training, directly motivating its training design.
Versus Force Prompting: both use physically inspired controls and synthetic data, but this paper emphasizes object-level velocity encoding and interaction data; input velocity and input force are not equivalent physical quantities.
A useful research direction is to evaluate whether an action condition is followed separately from whether its consequences are realistic. Improving the former does not establish the latter, as the different roles of Tables 1 and 2 illustrate.

Rating¶

Novelty: 4/5. Velocity masks and dense-to-sparse training form a clear combination, although control branches and synthetic training have precedents.
Experimental Thoroughness: 4/5. Training stages, data, text, and multiple backbones are evaluated, but quantitative physical ground truth for real-world scenes is missing.
Writing Quality: 4/5. The central argument and ablations are clear, while reproduction depends on the appendix; corrupted cached equations and Table 3 also limit current verification.
Value: 4/5. A reusable approach to initial-state-controlled video world models, with remaining validation gaps before reliable robotic dynamics prediction.