DeforM: Reasoning-Guided Physics-Aware Video Generation via Spatial-Temporal Masking

Conference: ECCV2026
Paper: ECCV official page · PDF
Authors: Yunyi Li, Yu Qiao, Yaohui Wang, Xinyuan Chen
Area: Video Generation
Keywords: Physics-aware generation, deformation dynamics, vision-language reasoning, spatial-temporal masks, attention modulation

TL;DR

DeforM converts vision-language reasoning about which objects should deform into spatial-temporal masks, then guides Wan2.2 through either training-free attention modulation or learned dual-attention injection, improving the same-backbone physical score from 3.49 to 4.40 on a deformation test set; this measures evaluator-assessed plausibility, not guaranteed compliance with physical laws.

Background & Motivation

Video generators can produce detailed textures without making objects change in physically plausible ways. Melting, squeezing, fracturing, stretching, and slicing require more than deciding where an object moves: its shape, apparent volume, contact relationships, and fragments must evolve coherently. A rigid trajectory or a few bounding boxes can describe translation, but cannot fully specify melting wax or a compressed sponge. Explicit simulators require scene assets and reconstruction, while data-driven training and reward optimization depend on adequate data coverage and well-designed rewards.

The paper identifies a more local bottleneck between semantics and spatial generation: a prompt says "melting," but the generator does not concentrate that change on the appropriate object. The authors observe relevant cross-attention spreading across the frame; in examples, wax fails to melt or liquid appears in unrelated locations. They therefore ask whether explicitly identifying where change should happen can activate dynamic knowledge already present in the backbone, before trying to relearn all the necessary physics. The training-free branch tests this mechanism, while the trained branch addresses deformation capabilities that attention redistribution alone cannot supply.

Core idea: first use a VLM to identify the object that should deform and plan its spatial-temporal extent, then use a mask to alter both textual semantic retrieval and information propagation within the video, concentrating learning and generation on physics-critical regions.

Method

Overall Architecture

The inputs are an initial image and a text description of a physical action; the output is a subsequent video. The shared entry point, DeforM-Reason, uses Qwen2.5-VL-7B to identify the target object and the open-vocabulary detector LLMDet to ground it, producing a spatial-temporal mask aligned with the video latents. This mask is a regional prior indicating where generation should concentrate, not a deformation solver or a specification of material stress and velocity fields.

The generator then follows one of two alternative branches. DeforM-Free modulates cross-attention queries during inference without updating pretrained parameters. DeforM-Injection trains a Dense Mask Modulator and LoRA weights, injecting features into both cross-attention and self-attention. These branches are not a sequential Free-then-Injection pipeline, and Injection does not require a video previously generated by Free.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Initial image + action text"] --> Reason["DeforM-Reason<br/>Object reasoning and localization"]
    Reason --> Mask["Latent-aligned<br/>spatial-temporal mask"]
    Mask -->|Training-free branch| Free["DeforM-Free<br/>Cross-attention modulation"]
    Mask -->|Trained branch| Injection["DeforM-Injection<br/>Dual-attention injection"]
    Free --> Output["Flow-matching sampling<br/>and video decoding"]
    Injection --> Output

Key Designs

1. DeforM-Reason: ground the region that should deform instead of merely rewriting the prompt

The VLM reads the first frame and text together to identify the object most relevant to the deformation, such as the wax rather than the spoon holding it. The resulting object description is passed to LLMDet, which grounds the name in image bounding boxes. Reasoning and detection have different responsibilities: the former selects what should change, while the latter determines where it is. Detecting every visible object would not establish which region matters for the requested physical action.

Training has access to real subsequent frames, so the detector can obtain a bounding-box sequence that follows the actual deformation. At inference, only the first frame can be detected; the VLM extrapolates subsequent boxes from the initial box and text. This distinction matters: inference does not use real future frames, but training provides more reliable region conditions than inference, introducing a potential distribution gap caused by planning errors. The paper does not provide a complete planning prompt or sufficient motion-constraint details to reproduce this extrapolation, so it should not be described as a known, exact physical trajectory.

Filling each frame's bounding box with 1 and its exterior with 0 produces a binary mask of size \(T\times H\times W\), which is downsampled to the latent dimensions \(T'\times H'\times W'\). This is a filled bounding-box region, not pixel-accurate object segmentation; it may include background. Temporal variation lets the mask cover the object's expansion, compression, or changing location. However, a box does not determine internal texture, cracks, or contact behavior: the generator still has to synthesize them.

2. DeforM-Free: test whether spatial grounding helps through cross-attention modulation

In cross-attention, video latents supply queries, while text embeddings supply keys and values. The mask is flattened and interpolated to match the video token sequence, and a focus scale greater than 1 modulates queries in target regions; the experiments use \(\alpha=1.4\). This changes how video locations retrieve text information rather than hard-coding a trajectory or melting process into pixels. All pretrained weights remain unchanged, making this branch useful for testing whether the backbone already contains deformation knowledge that better grounding can activate.

Operators and trailing terms in Equation (5) are incompletely extracted in the cache. The full gating expression and exact handling of unmasked regions cannot be reliably recovered, so this note does not supply a guessed formula. The surrounding prose establishes target-region query scaling and attention focusing. More precisely, query scaling does not mathematically guarantee increased probability for a particular physical keyword: the result depends on existing similarities between queries and text keys. The attention maps and video examples provide empirical evidence rather than a guarantee for arbitrary inputs.

This branch has a clear ceiling. If the backbone lacks a representation of a complex fracture or flow pattern, stronger attention cannot create that knowledge and may even reinforce an incorrect response. Its advantage over ordinary SFT in the ablation establishes the independent value of localization, but does not make training data or foundation-model capability irrelevant.

3. DeforM-Injection: let masks influence semantic retrieval and internal video relationships together

The trained branch encodes the mask into high-dimensional features instead of using it only as a scalar control. A Dense Mask Modulator uses 3D convolutions and scale alignment to produce three decoupled offset streams for self-attention keys, self-attention values, and cross-attention queries. Zero-initialized final linear layers avoid immediately disrupting the backbone's attention behavior at the start of training, allowing useful region conditions to emerge gradually. The cached description does not specify layer-by-layer tensor sizes for mask downsampling, so convolution counts, strides, and channel widths cannot be inferred.

The injection relationships below are organized from the explicit prose surrounding Equation (6). They restate the relationships that can be confirmed, rather than claiming a character-for-character recovery of the damaged equation:

\[ \begin{aligned} K'_{\mathrm{self}} &= K_{\mathrm{self}} + \Delta k_{\mathrm{self}},\\ V'_{\mathrm{self}} &= V_{\mathrm{self}} + \Delta v_{\mathrm{self}},\\ Q'_{\mathrm{cross}} &= Q_{\mathrm{cross}} + \Delta q_{\mathrm{cross}}. \end{aligned} \]

Cross-attention query offsets help target regions retrieve action semantics such as squeezing or melting from the text. Self-attention key and value offsets alter matching and information propagation among video locations, supporting spatial-temporal relationships within local deformation. Semantic control alone may not produce coherent internal changes, while internal coordination alone may not follow the requested action; this motivates synchronized injection. The ablation supports complementarity, but does not isolate the three offset streams individually, so benefits cannot be attributed separately to a particular key or value branch.

A Worked Example

Consider the scene in Figure 3: two small blue wax cubes sit inside a copper ladle, and the prompt asks them to melt. DeforM-Reason selects the wax as the target, detects its initial region, and plans where subsequent changes should occur during inference. It does not use a real melting video to construct the test-time mask.

With DeforM-Free, the mask directly modulates textual retrieval at the corresponding latent locations. The authors show a baseline video without melting and a modulated result in which blue liquid appears around the wax. With DeforM-Injection, the same kind of region condition is encoded into learned 3D features that influence semantic retrieval and internal video propagation, allowing the trained model to synthesize the dynamics. This is the paper's qualitative example; the displayed frames do not establish a melting rate, conservation error, or framewise bounding-box coordinates.

Loss & Training

The backbone is Wan2.2-TI2V-5B, trained through flow matching rather than noise prediction. Real videos are encoded into latents, and velocity prediction is trained along an interpolation path from data to Gaussian noise; the target velocity is the noise latent minus the data latent. Inference integrates backward from the noise end to the data end and decodes the result. Equations (2) and (3) are also damaged in the cache, so this note explains the confirmed training target in prose without inventing a complete expectation or norm expression.

DeforM-Injection jointly optimizes the mask modulator and LoRA parameters in the Transformer. LoRA rank is 256, batch size is 64, the LoRA learning rate is \(1\times10^{-6}\), and the modulator learning rate is \(2\times10^{-6}\). Training uses approximately 6k high-quality real deformation videos and their captions selected from WISA-80K.

Optimization combines flow-matching loss over the full latent sequence with a region-sensitive loss emphasizing errors inside the mask, using a balancing parameter \(\lambda_{\mathrm{mask}}=0.3\). Equation (7) is damaged in the cache, and the normalization of the regional term is insufficiently specified. It would therefore be unjustified to rewrite the objective as "global loss plus 0.3 times local loss." The design concentrates learning on deformation regions without discarding background training altogether. Training epochs, hardware runtime, and the complete sampling configuration are not reported in the readable main text.

Key Experimental Results

Main Results

Visual evaluation uses an independent set of 320 test samples; 160 are randomly selected for physical evaluation. Test samples are excluded from training. Visual metrics follow VBench but exclude subject consistency and background consistency because deformation itself changes these attributes. Quality Score is the reported visual aggregate; the available cache does not specify its aggregation weights, so it should not be assumed to be a simple average.

Qwen3-VL-32B evaluates generated videos together with their captions on a scale from 1 to 5. PC measures global Physical Commonsense, SA measures Semantic Adherence, and LDF measures Local Deformation Fidelity. Physical Score is explicitly their arithmetic mean, not a physical error measured in a human experiment:

\[ \mathrm{Physical\ Score}=\frac{\mathrm{PC}+\mathrm{SA}+\mathrm{LDF}}{3}. \]

The following selection from Table 1 uses higher-is-better metrics throughout. DeforM is trained on a 5B backbone, whereas the Wan2.2 comparator here is 14B. Training conditions differ across models, so this is not a parameter-count-controlled comparison.

Method	Dynamic Degree	Quality Score	PC	SA	LDF	Physical Score
CogVideoX1.5-I2V	48.32	71.40	4.07	3.62	3.24	3.64
Hunyuan-I2V	27.74	67.50	3.86	3.34	3.01	3.40
Wan2.2-I2V-14B	46.07	71.18	4.28	3.91	3.65	3.95
MAGI-1	28.03	67.26	3.93	3.39	3.01	3.44
DeforM-Injection	54.74	73.17	4.59	4.46	4.16	4.40

Relative to Wan2.2-I2V-14B, LDF increases by 0.51, approximately a 14.0% relative improvement; Physical Score increases by 0.45 and Quality Score by 1.99. These differences are calculated from the table. The method does not lead on every visual submetric: the original table reports temporal flickering and motion smoothness scores of 99.30 and 99.54 for Hunyuan-I2V, above DeforM's 97.24 and 98.40, although Hunyuan-I2V also has much lower Dynamic Degree.

Ablation Study

The following selection from Table 2 uses the Wan2.2 5B backbone family. W/O DeforM-Reason replaces the reasoned masks with random spatial-temporal masks; it does not remove mask conditioning entirely.

Configuration	Dynamic Degree	Quality Score	LDF	Physical Score
Vanilla Wan2.2	42.61	70.65	3.19	3.49
SFT	44.77	70.93	3.41	3.76
W/O DeforM-Reason	47.58	71.40	3.45	3.82
DeforM-Free	51.57	71.99	3.53	3.89
DeforM-Injection	54.74	73.17	4.16	4.40

The full method exceeds random-mask conditioning by 0.58 in Physical Score. The training-free branch exceeds SFT by 0.13, and the fully trained branch exceeds the training-free branch by another 0.51. This supports the interpretation that localization helps independently and learned region conditioning adds further value, not that VLM reasoning alone is sufficient.

The following selection from Table 3 tests the complementarity of cross-attention and self-attention.

Injection Strategy	Quality Score	PC	SA	LDF	Physical Score
DeforM-Cross	72.64	4.42	4.01	3.83	4.09
DeforM-Self	72.83	4.22	3.73	3.33	3.76
DeforM-Injection	73.17	4.59	4.46	4.16	4.40

Key Findings

• Random-mask conditioning reduces LDF by 0.71 relative to the full method, indicating that mask semantics and location matter. However, this ablation does not separate errors in object selection, detection, and temporal extrapolation.
• Self-attention-only injection has a slightly higher Quality Score than cross-attention-only injection but lower SA and LDF. Better visual quality does not substitute for correct actions; joint injection improves these dimensions together.
• The paper provides no sensitivity curves for \(\alpha\), the regional loss weight, or LoRA rank, and reports no repeated-run variance. The ablations establish the usefulness of component combinations, but not hyperparameter robustness or statistical significance.

Highlights & Insights

• The paper separates an apparent failure to understand physics into failure to apply an action at the correct location and an actual lack of deformation knowledge. Its training-free and trained branches probe these bottlenecks separately, offering more explanation than simply scaling up training.
• The mask is a low-bandwidth interface between language reasoning and video latents rather than a complete material-dynamics description. It reduces the need for explicit scene modeling while leaving texture and fine detail to the generator.
• Separating semantic retrieval from internal spatial-temporal coordination suggests a transferable control mechanism. Local editing and interaction-video generation could use dual-attention injection, but preservation of non-target regions would still require dedicated evaluation.

Limitations & Future Work

• The authors explicitly acknowledge inaccurate spatial-temporal localization in complex scenes and the dependence of LoRA-based generation quality and scene diversity on the foundation model.
• Training uses detections from real subsequent frames, while inference relies on the first frame and planning, creating a condition-quality gap. Comparisons among static masks, planned masks, and an upper bound using real future boxes could evaluate occlusions, interacting objects, and unseen materials more precisely than random masks alone.
• Physical scores depend on a single VLM evaluator. The cache provides no human blind evaluation, inter-evaluator agreement, or conservation measurements. Higher scores do not establish mass, energy, or contact-constraint compliance.
• Testing covers a specialized deformation subset, and visual evaluation omits subject and background consistency. Non-target preservation and out-of-distribution tests are needed to rule out more obvious deformation coming at the cost of deterioration elsewhere in the scene.
• Several equations are incompletely preserved in the readable cache, and mask planning, regional-loss normalization, and training costs lack sufficient detail. Reproduction requires checking the original PDF or implementation; this note does not fill those gaps with assumptions.

Related Work & Insights

• vs PhysGen / PhysMotion / WonderPlay: these methods rely on physical simulation or scene structure, providing more explicit constraints. DeforM instead activates generative priors through masks, reducing asset-modeling requirements without offering numerical physics guarantees.
• vs VLIPP / PhyRPR: the paper characterizes these as VLM-guided trajectory-planning approaches mainly aimed at rigid-body dynamics. DeforM treats box sequences as region conditions, not complete deformation specifications, and leaves local shape evolution to the learned generator.
• vs WISA / ordinary SFT: DeforM uses a subset of WISA-80K while adding localization, dual-attention modulation, and regional training emphasis. The same-backbone SFT ablation shows that domain fine-tuning alone does not explain the gains.
• Research direction: explicitly pass mask uncertainty to the modulator and constrain generation with contact events or local conservation checks. These are opportunities suggested by the limitations, not capabilities already validated by the paper.

Rating

• Novelty: 4/5. It turns semantic grounding for deformation into analyzable attention control and extends it with dual-attention training, although the underlying components are established.
• Experimental Thoroughness: 3/5. Cross-model comparisons, random-mask controls, and injection-location ablations are useful, but human evaluation, statistical uncertainty, sensitivity analysis, and out-of-distribution tests are missing.
• Writing Quality: 4/5. The problem, mechanism analysis, and trained extension connect clearly; exact reproduction still needs planning and loss implementation details, and cache extraction damage should not be attributed to the original manuscript's typesetting.
• Value: 4/5. Regional control offers practical insights for local deformation videos, but the method is best viewed as a plausibility enhancement rather than a physical world model that can replace a simulator.