SIFT: Self-Imagination Fine-Tuning for Physically Plausible Motion in Video Diffusion Models¶

Conference: ECCV 2026
arXiv: 2606.27741
Code: None
Area: Video Generation
Keywords: Video Diffusion Models, Motion Decoupling, Self-Imagination Fine-Tuning, Physical Plausibility, Motion Entanglement

TL;DR¶

SIFT proposes a self-imagination fine-tuning paradigm that discards real video inputs and starts entirely from pure Gaussian noise, forcing the diffusion model to "imagine" and generate videos solely based on LLM-generated text prompts. By combining decoupled motion feedback supervised by dual heterogeneous motion classifiers with a progressive hard case replay strategy, SIFT significantly improves the physical plausibility and motion decoupling capabilities of two backbone models, Wan and CogVideoX, without relying on decoupled motion annotation data. VLM and human evaluations consistently show that SIFT outperforms baseline methods such as SFT and VideoREPA.

Background & Motivation¶

While video diffusion models (VDMs) have made significant breakthroughs in visual fidelity and semantic consistency, generating physically plausible motion remains a fundamental challenge. Existing works (PhysGen, PhysMotion, etc.) mostly focus on the dynamics level (gravity, collision, fluid). In contrast, this paper identifies a complementary and more intuitive failure mode from a kinematic perspective: Motion Entanglement. This occurs when the model fails to independently control and decouple motion from different sources, typically manifested as the unexpected coupling of camera motion and object motion. For instance, when a prompt asks for a camera orbiting around a stationary object, the generated video often shows the object drifting as well. Conversely, when the object moves and the camera should remain static, the camera unexpectedly follows the object. This phenomenon is prevalent across commercial models (Veo3, Kling AI), open-source backbones (Wan, CogVideoX), and even models with explicit camera control (CameraCtrl) or trajectory control (Wan-Move). Fundamentally, motion entanglement reflects the model's inability to maintain independent reference frames and accurately model relative motion. Beyond perceptual realism, such kinematic entanglement severely limits the utility of VDMs as world simulators in downstream applications, such as autonomous driving, which require strict decoupling of self-motion and environmental dynamics.

The authors attribute motion entanglement to two core factors. First, data-induced bias: in real-world videos, camera motion and object motion often co-occur. Large-scale video datasets lack explicit annotations to distinguish their individual contributions, leading the model to learn spurious statistical correlations and treat independent kinematic variables as inherently coupled. Second, inherent limitations of the diffusion training paradigm: diffusion models are trained to reconstruct clean videos from noisy ones using pixel-level MSE objectives. However, even highly noisy videos retain significant residual structures and temporal cues (such as coarse object trajectories and frame-to-frame coherence). This creates a "reconstruction shortcut" where the model learns to copy the residual motion patterns from the noisy input rather than reasoning kinematically correct dynamics from the text prompt. Furthermore, the pixel-level MSE objective only constrains RGB matching accuracy and does not distinguish whether pixel displacement stems from camera motion or the object itself, further solidifying the model's bias towards "appearance fidelity" over "understanding of motion structure."

Supervised fine-tuning (SFT)—fine-tuning on additional motion-decoupled video data—appears to be a straightforward solution but is practically ineffective. This is due to two reasons: motion-decoupled video data is extremely scarce, making the data engineering cost to cover full motion diversity prohibitively high; more fundamentally, SFT inherits the exact same reconstruction shortcut and pixel-level reconstruction bias as pre-training, meaning motion patterns are still inherited from the input data rather than reasoned from the prompt semantics. Diagnostic experiments in Fig. 2 validate this observation. When testing four input configurations on Wan2.1-1.3B (original video + correct prompt / original video + mismatched prompt / shuffled frames + correct prompt / shuffled frames + correct prompt targeting the original temporal sequence), the loss curves of the first three configurations almost completely overlap. This indicates that the model heavily bypasses the text prompts or temporal order, and its reconstruction behavior is dominated by the residual information in the noisy inputs.

The core idea is to break the reconstruction shortcut and transition the training paradigm from "reconstruction-based copying" to "imagination-based reasoning". By discarding real video inputs and starting from pure Gaussian noise, the model is forced to imagine and synthesize videos utilizing only text prompts, while guided by motion-aware discriminative supervision instead of pixel-level reconstruction objectives.

Method¶

Overall Architecture¶

The core mechanism of SIFT is to invert the diffusion model training from "given noisy real video $\to$ denoising reconstruction" to "given pure noise + text prompt $\to$ imaginary generation $\to$ motion discriminative feedback". The overall framework consists of three key components: self-imagination generation (to break the reconstruction shortcut), decoupled motion feedback (to provide motion-aware supervision), and progressive hard case replay (to stabilize training).

The workflow is as follows: An LLM (such as GPT-4o) freely generates diverse text prompts covering various motion combinations alongside their corresponding motion category labels (camera-only / object-only / both-in-motion / static). The process initializes from pure Gaussian noise $x_T \sim \mathcal{N}(0, I)$ and performs a few denoising steps ($t = 1000 \to 980 \to 960$, 3 steps) only within the high-noise interval to generate video $\hat{x}_0$. This video is sent to two heterogeneous motion classifiers (R3D and SlowFast, alternatingly applied) to predict the motion category. A cross-entropy loss $\mathcal{L}_{\text{motion}}$ is computed against the ground-truth motion label. Misclassified samples are routed to a hard case buffer and re-introduced to training with a progressive probability $p_s = \min(1, s/S_{\text{warmup}})$. Concurrently, a minor real-world video branch is maintained using standard MSE loss $\mathcal{L}_{\text{MSE}}$ to preserve visual quality. The total loss is defined as $\mathcal{L}_{\text{total}} = \lambda \mathcal{L}_{\text{motion}} + \mathcal{L}_{\text{MSE}}$.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["LLM Generates Text Prompts"] --> B["Self-Imagination Generation<br/>Pure Noise + Few-step Denoising"]
    B --> C["Generated Video x̂₀"]
    C --> D["Decoupled Motion Feedback<br/>R3D / SlowFast Alternating Classification"]
    D --> E["L_motion Cross-Entropy"]
    C --> F{"Classification<br/>Correct?"}
    F -->|"No"| G["Progressive Hard Case Replay<br/>Buffer + Warmup Scheduler"]
    G --> E
    H["Real Videos"] --> I["Standard MSE Training"]
    I --> J["L_MSE"]
    E --> K["L_total = λ·L_motion + L_MSE"]
    J --> K

Key Designs¶

1. Self-Imagination Generation: Starting from Pure Noise to Force Semantic Motion Reasoning

In standard diffusion training, even noisy videos preserve coarse motion cues (object trajectories, temporal consistency), causing the model to take a "reconstruction shortcut" by copying residual motion patterns rather than reasoning from the prompt. Diagnostic experiments in Fig. 2 demonstrate that under "original video + correct prompt", "original video + unrelated prompt", and "shuffled frames + correct prompt" setups, the reconstruction loss curves of pre-trained Wan2.1-1.3B are nearly identical. This confirms that the model relies minimally on text content or temporal sequence, completing reconstructions primarily using residual information from the noise. The fourth setup (reconstructing the original ordered sequence from shuffled frames) shows a significantly higher loss, indicating that when the prompt contradicts the residual motion cue in the noise, the model lacks the ability to correct the motion purely based on text, which is direct evidence of the reconstruction shortcut.

To address this, SIFT discards real video inputs and replaces them with pure Gaussian noise $x_T \sim \mathcal{N}(0, I)$. Because pure noise contains zero motion cues, the model cannot path-dependently copy motion and must instead "imagine" the entire video content and motion dynamics from scratch using the text prompt. SIFT generates 10,000 diverse text prompts via GPT-4o, densely covering combinations of camera and object motion, including rare and meticulously decoupled scenarios. While collecting real videos for these scenarios is extremely expensive, generating text prompts incurs virtually zero cost. To optimize computational efficiency, SIFT performs only 3 denoising steps in the high-noise interval ($t = 1000, 980, 960$), as the global motion structure of diffusion models is mainly established during the early denoising stage. Probing only the most critical motion imagination interval greatly reduces computational overhead.

Why it works: Unlike SFT which starts from noisy real-world videos, self-imagination generation completely severs the motion correlation between input and output—there is no "copyable" motion in pure noise. The model must reason the correct motion frame of reference and relative motion from semantic descriptions (e.g., "the camera pans from left to right while the object remains static"). Ablation studies validate this: retaining the reconstruction shortcut (starting from noisy real videos instead of pure noise) drops the camera SA from 3.85 to 3.05 and PC from 4.15 to 3.55, which is the most severe performance degradation among all ablations, proving that the reconstruction shortcut is the primary obstacle to motion reasoning.

2. Decoupled Motion Feedback: Alternating Supervision via Dual Classifiers for Motion-Aware Signals

Even when the reconstruction shortcut is eliminated, the video imagined by the model from pure noise may still contain physical implausibility, as there is no external signal during self-imagination to evaluate correctness. Since pixel-level MSE cannot distinguish whether displacement is caused by the camera or the object, a motion-aware discriminative signal is needed.

The authors train two heterogeneous motion classifiers $\mathcal{C}_\phi$ to classify videos into four categories: camera-only, object-only, both-in-motion, and static. The classifiers are trained on 4,000 manually filtered balanced videos. Crucially, SIFT employs training distribution alignment: the classifiers are trained on $\hat{x}_0$ generated through "noise injection followed by single-step denoising" ($t \in [900, 1000)$), perfectly matching the input distribution seen by the classifiers during SIFT training. R3D and SlowFast achieve 78.4% and 82.8% accuracy, respectively, on a held-out validation set constructed with this distribution. The architectures of the two classifiers are complementary: R3D (3D-ResNet) models spatial-temporal dimensions symmetrically using 3D convolutions, making it effective at capturing local, short-term motion coherence. SlowFast explicitly factorizes temporal and spatial processing using a dual-pathway approach: the slow pathway focuses on fine-grained spatial semantics at a low frame rate, while the fast pathway captures high-frequency motion dynamics at a high frame rate, introducing different inductive biases for motion.

The alternating supervision strategy is the core mechanism of decoupled motion feedback. Instead of simply averaging the outputs of the two classifiers, SIFT alternates between R3D and SlowFast across different training batches (shifting every 25 steps). Fig. 7 shows that under single-classifier supervision, the classification accuracy quickly saturates as the generator learns to "cheat" the specific classifier (e.g., by creating artifacts favored by it) rather than generating physically plausible motion. Alternating between heterogeneous classifiers acts as implicit adversarial ensemble training: the generator must concurrently satisfy complementary motion criteria, preventing it from overfitting to any single inductive bias. In ablation studies, single-classifier variants dropped to 3.20 (R3D) and 3.50 (SlowFast) on object motion SA, both performing worse than the full SIFT score of 3.50. The motion loss uses standard cross-entropy: $\ell(\hat{x}_i, c_i) = -\mathbb{I}(c_i)^\top \log \mathcal{C}_\phi(\hat{x}_i)$, where $\mathbb{I}(c_i)$ is the one-hot vector of the ground-truth motion category.

3. Progressive Hard Case Replay: Easy-to-Hard Curriculum for Stable Motion Learning

In the early stages of training, the self-imagination videos generated by the model are of low quality, and the motion classifiers almost always predict incorrect classes. Updating gradients using all failed samples from the start introduces heavily noisy gradients, which not only hinders motion learning but also degrades visual quality. This is supported by the ablation study where removing progressive replay led to a larger drop in SA (3.85 $\to$ 3.00) than in PC, reflecting collateral damage to visual quality.

Progressive hard case replay resolves this issue: after each self-imagination sequence, samples with incorrect classification predictions ($\arg\max(c_{\text{pred}}) \neq c_i$) are stored in a hard case buffer $\mathcal{B}$. In the early stages of training ($s < S_{\text{warmup}}$), these hard cases are temporarily excluded from loss calculation to avoid unstable noisy gradients. As training progresses, hard cases are introduced into the loss calculation with an increasing probability: $p_s = \min(1, s/S_{\text{warmup}})$, where $S_{\text{warmup}} = 500$. Formally, for each sample $i$ in batch $b_s$, we sample $u_i \sim \text{Uniform}[0,1]$ and only include the hard case in the motion loss if $u_i \leq p_s$: $$\mathcal{L}_{\text{motion}} = \sum_{i \in b_s} \left[\mathbf{1}_{\{i \notin H\}} + \mathbf{1}_{\{i \in H\}} \mathbf{1}_{\{u_i \leq p_s\}}\right] \ell(\hat{x}_i, c_i)$$ Additionally, hard cases are sampled and replayed from the buffer every 10 batches to reinforce the model's command of these challenging motion patterns.

Why it works: This strategy enables a curriculum learning process where the model first establishes basic motion understanding on "easy" samples (correctly classified generations) before gradually tackling hard cases as the parameters stabilize. In the ablation studies, removing the progressive replay ($S_{\text{warmup}} = 0$, exposing hard cases from step 1) degraded the camera SA from 3.85 to 3.00—the largest single drop across configurations. Progressive replay also improves training efficiency, yielding greater progress within the same 1,000 optimization steps, as hard cases are only fully utilized when the model is capable of handling them, avoiding unproductive or destructive gradient updates early on.

A Walkthrough Example¶

Assume a typical training prompt generated by GPT-4o: "A red car is parked on a quiet street. The camera pans smoothly from left to right." The motion category label is camera-only.

First, starting from pure Gaussian noise $x_T \sim \mathcal{N}(0, I)$, the model generates a 16-frame video $\hat{x}_0$ after 3 denoising steps ($t=1000 \to 980 \to 960$). Second, the current batch uses the SlowFast classifier. It takes $\hat{x}_0$ and outputs the category probabilities $[0.82, 0.08, 0.07, 0.03]$. The $\arg\max$ result is camera-only, matching the ground truth. The sample passes classification, bypasses the hard case buffer, and contributes directly to $\mathcal{L}_{\text{motion}}$ via $\ell = -\log(0.82)$. Meanwhile, standard real videos are sampled to compute the standard $\mathcal{L}_{\text{MSE}}$. The total loss $\mathcal{L}_{\text{total}} = 0.01 \times \ell + \mathcal{L}_{\text{MSE}}$ is backpropagated to update the parameters.

Now consider a hard case. Prompt: "The camera remains completely static. A golden retriever runs across a grassy field from left to right." (object-only). The current batch uses the R3D classifier and outputs $[0.15, 0.05, 0.72, 0.08]$, incorrectly predicting both-in-motion. This sample is stored in buffer $\mathcal{B}$. If the current training step is $s = 200$, the replay probability is $p_s = \min(1, 200/500) = 0.4$. When this sample is replayed from the buffer, it has a 40% probability of being included in the loss computation (if $u_i \leq 0.4$); after step $s = 500$, $p_s = 1.0$ and it is always included. The entire process requires no ground-truth motion-decoupled videos, driving motion learning solely through text prompts and classifier feedback.

Loss & Training¶

The total loss of SIFT is a weighted combination of the motion-aware discriminative loss and the pixel-level reconstruction loss: $$\mathcal{L}_{\text{total}} = \lambda \cdot \mathcal{L}_{\text{motion}} + \mathcal{L}_{\text{MSE}}$$ where $\lambda = 0.01$. $\mathcal{L}_{\text{MSE}}$ follows the standard diffusion training objective (velocity prediction MSE for Wan, noise prediction MSE for CogVideoX), computed on real video-text pairs to preserve visual quality and semantic alignment.

The motion loss $\mathcal{L}_{\text{motion}}$ is a cross-entropy loss gated by the progressive hard case scheduler, applied to the self-imagination branch. During classifier training, we match the SIFT inference distribution by adding noise to clean videos up to $t \in [900, 1000)$ and performing a single-step denoising to obtain $\hat{x}_0$, which serves as the input.

Key Hyperparameters: Both backbone models (Wan2.1-T2V-1.3B, CogVideoX) are optimized using AdamW with a learning rate of $5 \times 10^{-6}$ for 1,000 steps. The batch size is 1 (1 self-imagination sample + 1 real sample per step) across 8 NVIDIA H100 GPUs. Self-imagination uses 3 denoising steps ($t = 1000, 980, 960$), with the motion classifiers alternating every 25 steps, the hard cases replayed every 10 steps, and a warmup of $S_{\text{warmup}} = 500$ steps. VLM evaluation is performed using InternVideo2.5 by shuffle-evaluating all method generations under the same prompt. Test prompts are generated by Gemini Pro Vision 1.5 (a different model than the training prompt generator GPT-4o) to prevent train-test leakage.

Key Experimental Results¶

Main Results¶

We compare SIFT against the base models, SFT (fine-tuned on 4,000 manually selected motion-decoupled videos), and VideoREPA (distilling physical understanding from video foundation models via feature alignment) on Wan2.1-1.3B and CogVideoX. Evaluation metrics include Semantic Alignment (SA) and Physical Commonsense (PC), evaluated via VLM (InternVideo2.5, rating 1-5) and human evaluation (20 participants, 16 groups each). The test set contains 100 prompts formatted as "{content description} {camera description}" using 12 camera motion templates and 5 static templates.

Method	Camera SA (VLM)	Camera PC (VLM)	Object SA (VLM)	Object PC (VLM)	Camera SA (Human)	Camera PC (Human)	Object SA (Human)	Object PC (Human)
Wan (base)	3.58	4.06	3.80	3.60	3.00	2.79	2.91	2.75
Wan + SFT	3.94	4.73	4.32	4.30	3.26	3.53	2.91	2.43
Ours (Wan + SIFT)	4.80	4.93	4.75	4.72	3.89	4.24	3.98	3.84
CogVideoX (base)	3.95	3.10	3.75	2.81	3.51	3.20	2.73	2.39
CogVideoX + SFT	4.68	4.32	4.25	3.69	3.50	3.10	3.18	2.86
CogVideoX + VideoREPA	4.53	4.05	4.00	3.50	3.34	4.05	2.20	2.60
Ours (CogVideoX + SIFT)	4.89	4.84	4.38	4.25	3.88	4.25	3.75	3.68

SIFT consistently achieves the best performance across all VLM and human evaluation metrics on both backbones. Notably, SFT underperforms the base model in some human evaluations (e.g., Wan+SFT object PC is 2.43 vs. base Wan's 2.75). This validates the claim that SFT merely mimics the motion distribution in the data without acquiring true motion reasoning; human evaluators easily identify kinematic inconsistencies when SFT models generalize to new scenarios. While VideoREPA improves temporal stability via feature alignment, it lacks explicit motion decoupling supervision, yielding limited gains in PC with persistent relative motion errors (camera vs. object). Human preference studies (Fig. 4) confirm that SIFT is significantly preferred over the baselines. Compared to camera control methods (CameraCtrl, Wan-Move) evaluated on the camera motion subset (Tab. 2), SIFT achieves superior scores with SA 4.10 and PC 4.30 without requiring manually designed trajectories or reference frames.

Ablation Study¶

All ablations are performed on the Wan backbone using VLM evaluation (40 randomly sampled prompts, with relative scoring among all variants under the same prompt; scores are relative within the group and cannot be compared directly to the main results).

Configuration	Camera SA	Camera PC	Object SA	Object PC	Description
Full SIFT	3.85	4.15	3.50	4.00	Full model
w/ Reconstruction Shortcut	3.05	3.55	3.45	3.65	Starting from noisy real videos, retaining the reconstruction shortcut
w/ Single Classifier (R3D)	3.65	3.55	3.20	3.60	R3D classifier only, overfitting its inductive bias
w/ Single Classifier (SlowFast)	3.35	3.25	3.50	3.85	SlowFast classifier only, same as above
w/o Progressive Hard Case Replay	3.00	3.45	3.35	3.70	Exposed to hard cases from the start, noisy gradients disrupt training

The three ablation directions validate the effectiveness of each component. Retaining the reconstruction shortcut results in the most severe performance drop (Camera SA decreases from 3.85 to 3.05), proving that the reconstruction shortcut is the primary obstacle to motion reasoning. Single-classifier variants perform worse than alternating supervision, exposing distinct degradation modes (R3D degrades more on object motion SA, while SlowFast degrades more on camera motion), validating the necessity of heterogeneous alternating supervision to prevent overfitting to a single inductive bias. Deleting progressive replay leads to a larger drop in SA than in PC, indicating that early noisy gradients hinder both motion learning and overall visual quality/semantic alignment.

Key Findings¶

Reconstruction Shortcut is the Main Obstacle: Retaining the reconstruction shortcut in ablation causes the most severe performance drop (Camera SA -0.80). This directly supports the paper's core claim that standard diffusion training's "copying" paradigm is the root cause of motion reasoning failure, rather than a lack of data.
SFT Can Be Detrimental: Human evaluations show SFT can underperform the base model (Wan+SFT object PC 2.43 vs. Wan 2.75). This indicates that simply exposing the model to more motion-decoupled data does not teach it motion reasoning, but instead reinforces incorrect statistical correlations learned from limited data. This counter-intuitive finding empirically supports the reconstruction shortcut analysis.
Alternating Supervision Outperforms Ensemble: Alternating between the dual classifiers is superior to using single classifiers, and the alternating strategy (rather than average blending) is key. It forces the generator to satisfy complementary motion criteria dynamically, acting as an implicit multi-teacher adversarial training process. Single classifiers exhibit distinct degradation modes, confirming they capture different facets of motion quality.
Generalization to Complex Motion: In challenging scenarios involving multi-object motion, articulated motion, and long-horizon motion (double the frames, at least two consecutive motion stages) in Tab. 3 (50 prompts each), SIFT consistently improves SA and PC on the Wan backbone. This demonstrates that the acquired motion reasoning generalizes beyond the simple camera/object binary scenarios present in training.

Highlights & Insights¶

Value of the "Imagination-Driven Training" Paradigm: Shifting diffusion model training from "reconstruction of real data" to "imagination plus discriminative feedback" is a paradigm shift. This approach is applicable not only to motion decoupling but can also extend to other quality dimensions where pixel-space loss is hard to define (e.g., physical plausibility, causality, commonsense consistency). The key insight: when real data contains patterns we do not want the model to copy, discarding real data and relying on discriminative feedback is a superior alternative.
"Pure Noise Initialization" as the Shortest Path to Sever Shortcuts: The most elegant design in SIFT is starting from pure Gaussian noise. It requires no architectural changes, no extra modules, and no specialized dataset; simply altering the initialization entirely eliminates the reconstruction shortcut. This "minimal intervention, maximum impact" design philosophy is highly instructive.
Progressive Scheduling for Hard Case Replay: Incorporating curriculum learning into discriminative feedback training with a simple linear warmup schedule $p_s = \min(1, s/S_{\text{warmup}})$ successfully mitigates early noisy gradient issues. This lightweight, effective mechanism can be readily integrated into other generative training pipelines involving discriminator feedback.
Alignment of Classifier Training Distribution: Training the motion classifiers on "noise-injected and denoised" $\hat{x}_0$ instead of clean videos aligns their training distribution with SIFT training. This crucial engineering detail prevents the classifier from providing unreliable feedback on lower-quality early SIFT generations, which would collapse training.

Limitations & Future Work¶

Coarse Four-Class Motion Labels: The current classifier setting only distinguishes "who is moving" (camera, object, both, static). It does not explicitly model motion direction, magnitude, individual trajectories, or temporal transitions between multiple motion states. In complex scenarios with multiple moving entities, simple categorical labels fail to represent the underlying motion structure, limiting the granularity of the supervision signal.
Classifiers as Robustness Bottlenecks: The quality of motion feedback is strictly bounded by classifier accuracy (R3D 78.4%, SlowFast 82.8%). For blurry or heavily degraded self-generated videos, the classifiers may provide unreliable signals, leading to error accumulation.
Dependence on Backbone Model's Underlying Capabilities: Failure cases in the appendix (e.g., camera direction errors—generating "counter-clockwise" when prompted "clockwise") indicate that SIFT cannot fix fundamental flaws in the backbone's semantic understanding of directions. While SIFT improves "who is moving", the precision of "how to move" (direction, speed, trajectory shape) remains limited by the base model.
Future Directions: Introducing richer decomposed motion representations (e.g., object-level trajectories, optical flow, motion fields) and designing fine-grained feedback mechanisms for complex interactions and longer durations; integrating stronger motion discriminators; and extending the SIFT paradigm to broader physical reasoning dimensions such as collisions, occlusions, and gravity.

vs. Traditional SFT: SFT uses the same data (4,000 motion-decoupled videos) but yields significantly worse results, even underperforming the base model in human evaluations. The fundamental difference lies in the source of the training signal: SFT continues to let the model "observe" and copy motion from data, whereas SIFT encourages the model to "imagine" motion and receive discriminative correction. This suggests that when target abilities (e.g., physical reasoning) are hard to demonstrate directly, discriminative feedback can be more effective than demonstration-driven learning.
vs. VideoREPA: VideoREPA distills general physical knowledge by aligning token-level features with video foundation models, improving temporal stability but lacking explicit motion decoupling supervision. While VideoREPA represents "learning better feature representations from a stronger teacher", SIFT represents "learning by imagining and error correction without a teacher (real videos)". The two are complementary: combining VideoREPA's feature alignment with SIFT's imagination training is a promising direction.
vs. Physics Simulation-Driven Methods (PhysGen): PhysGen employs external physics engines to pre-calculate rigid body motion and interactive trajectories, using the VDM primarily as a renderer. SIFT instead internalizes physical reasoning within the model without external simulators or pre-computed trajectories. In the long run, internalizing reasoning is more scalable, though simulation-driven methods currently offer higher precision. Merging the two—using simulation for precise trajectory supervision and SIFT for internalizing reasoning—presents a compelling path forward.
vs. Camera Control Methods (CameraCtrl, Wan-Move): Camera control methods achieve precise control through dense external conditions (manual trajectories, reference first frames) but do not repair the base model's intrinsic motion priors. SIFT enhances the model's intrinsic motion understanding in standard text-to-video settings. While CameraCtrl focuses on "instructing how the camera moves," SIFT focuses on "enabling the model to understand the correspondence between motion and text." SIFT-fine-tuned models can naturally benefit from camera control modules applied on top of them.

Rating¶

Novelty: ⭐⭐⭐⭐ Inverting diffusion training from "reconstruction" to "imagination + discrimination" offers paradigm-level innovation. Using pure noise initialization to cut off the reconstruction shortcut is elegant and effective. However, the concrete components (classifier guidance, hard case mining) are established, making the overall method a clever reorganization of existing techniques.
Experimental Thoroughness: ⭐⭐⭐⭐ Evaluated on dual backbones with multiple baselines (SFT, VideoREPA, CameraCtrl, Wan-Move), employing both VLM and human evaluations, plus three generalization tests and comprehensive three-way ablations. The experimental design is solid; however, the ablation is conducted only on Wan with a relatively small sample size of 40 prompts, and it lacks analysis on how classifier accuracy limits final performance.
Writing Quality: ⭐⭐⭐⭐ The problem definition is clear (the motion entanglement example in Fig. 1 is highly intuitive), the motivational chain is complete (data bias $\to$ reconstruction shortcut $\to$ diagnostic experiment $\to$ SFT failure $\to$ SIFT solution), and the method section is well-supported by illustrations (comparison in Fig. 3 and pseudocode in Algorithm 1). The appendix provides helpful details on LLM/VLM prompts and failure case analyses.
Value: ⭐⭐⭐⭐ Motion entanglement is a highly visible failure mode in VDMs. SIFT provides a practical, low-cost solution (no specialized video data, only text prompts and classifiers) that carries direct engineering value for the video generation community. The "imagination-driven training" paradigm could inspire future work in injecting higher-dimensional reasoning capabilities into diffusion models.