SHIFT: Motion Alignment in Video Diffusion Models with Adversarial Hybrid Fine-Tuning

Conference: ECCV2026
Paper: ECCV Paper
Area: Video Generation
Keywords: video diffusion models, motion alignment, optical-flow residuals, adversarial advantages, hybrid fine-tuning

TL;DR

SHIFT learns motion rewards from optical-flow residuals and long-term point trajectories, combining advantage-weighted denoising of generated samples with a supervised real-video anchor to reach 86.70 VBench Motion on SVD at 2.46 times the per-epoch cost of SFT, primarily preventing fine-tuning from making videos nearly static.

Background & Motivation

Image-conditioned video models can generate sharp images with consistent subjects, but looking like a good photograph is not equivalent to exhibiting realistic motion. Further supervised fine-tuning of models such as SVD can improve appearance and distributional distance while substantially reducing motion magnitude. In this paper's SFT experiment, VBench Appearance rises from 83.68 to 85.07, whereas Motion drops from 85.89 to 76.09. This is not simply metric noise: the authors also report a dynamic-degree reduction from 0.67 to 0.33, suggesting a tendency toward videos with little change. Selecting models using only frame quality, semantic alignment, or FVD can therefore mistake this degradation for progress.

Correcting that preference requires a reward that distinguishes genuine movement from flickering that merely creates change, while observing both adjacent frames and complete trajectories. Human preference annotations and vision-language models usually assess appealing content more readily than fine-grained pixel motion at low cost. Directly driving an optical-flow constraint to zero is also inappropriate because lighting, occlusion, and sensor noise prevent real videos from strictly satisfying ideal brightness constancy. Meanwhile, video models have long reverse denoising chains, making trajectory storage and stepwise policy-gradient updates expensive. Even an inexpensive reward can be exploited by a changing generator, producing samples that score well but look unnatural.

SHIFT consequently treats motion alignment as matching realistic motion-feature statistics, not simply increasing optical-flow magnitude. It retains real data as a stable anchor, evaluates current generated samples with an evolving discriminator, and incorporates that feedback through ordinary noising-and-denoising training. This avoids sending gradients through the complete sampling process while placing reward optimization and preservation of generative capabilities within the same update. Core Idea: use dual-scale motion discriminators to provide adaptive relative advantages, and jointly train on generated and real videos at a shared noise level so that reward optimization and the supervised anchor do not act at mismatched scales.

Method

Overall Architecture

The inputs are a batch of real videos and conditioning images, with text also provided in the TI2V setting; the generator produces synthetic videos from these conditions. Dual-scale motion rewards read adjacent-frame transport residuals and cross-frame point trajectories to produce motion-realism signals. Adversarial advantages turn these signals into relative weights while refreshing the discriminators using the latest generated videos. Hybrid denoising and noise alignment apply those weights to the diffusion loss on generated samples while retaining the standard supervised loss on real videos. The updated generator supplies new samples to the training loop; inference uses only the fine-tuned generator, without online discriminator or point-tracker scoring.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Real videos and<br/>conditional generated videos"] --> Reward["Dual-Scale Motion Rewards<br/>IMR + LMR"]
    Reward --> Advantage["Adversarial Advantages<br/>Discriminator refresh and centering"]
    Advantage --> Hybrid["Hybrid Denoising and<br/>Noise Alignment"]
    Input -->|Real-data supervised anchor| Hybrid
    Hybrid --> Model["Update generator LoRA"]
    Model -.->|Refresh generated samples during training| Input
    Model -->|Inference: conditions and sampling| Output["Output video"]

Key Designs

1. Dual-Scale Motion Rewards: learn natural motion statistics instead of demanding zero residuals

The Instantaneous Motion Reward (IMR) starts from temporal changes in pixel intensity and compares them with changes explained by optical-flow transport. Spatial derivatives are estimated with Sobel operators, temporal derivatives with adjacent-frame finite differences, and the velocity field with pretrained SEA-RAFT. The authors draw on the continuity equation but deliberately remove its divergence term, retaining only the advective component to construct a transport residual. The reason is specific: a camera zoom geometrically expands the image, but surface brightness need not decrease like the density of an expanding compressible fluid. Directly imposing mass conservation would therefore penalize normal scaling, whereas the desired representation concerns motion in video intensities. The residual is not itself a quantity to minimize; the paper observes that suppressing its squared norm can produce blurry, static results. Instead, the residual map is concatenated with SEA-RAFT's confidence map and passed to a ViT that learns differences between real and generated motion. Confidence gives the discriminator an opportunity to distinguish unnatural video from unreliable flow estimation instead of attributing every estimation error to the generator.

The Long-term Motion Reward (LMR) randomly selects query points in the first frame and tracks them throughout the video using CoTracker3. Differences between consecutive point coordinates yield velocities, which are concatenated with visibility masks and tracking confidence to form a motion state. Dense correlation maps between query points and their temporal neighborhoods add spatial affinity information beyond the trajectories themselves. A separate ViT uses these descriptors to assess whether motion over longer horizons resembles real video. Random queries check different locations across evaluations, reducing opportunities to manufacture acceptable trajectories only at fixed grid positions. IMR can expose local anomalies such as flickering, but adjacent-frame observations alone may tolerate globally stalled motion; LMR adds evidence about sustained movement. Both rewards are trained through real-versus-generated video classification without human preference rankings, but neither should therefore be treated as a strict verifier of physical laws.

2. Adversarial Advantages: track the changing generator distribution without passing discriminator gradients into it

During reward pretraining, the first frame and optional text of each real video condition the base model to generate 4 negative examples. Real and generated videos form binary classification data for the two motion discriminators, trained with binary cross-entropy while the flow and tracking backbones remain frozen. Generator errors change during fine-tuning, so the discriminators periodically continue training on the latest generated samples and real videos. This redirects the reward toward currently present motion artifacts instead of permanently retaining the initial decision boundary. The authors interpret an ideal discriminator's signal as a log-density ratio in motion-feature space; that interpretation does not guarantee reliability for a finite-capacity learned model.

Feedback uses the discriminator's raw logit rather than its sigmoid probability. When a discriminator is confident, probabilities cluster near 0 or 1, making differences among samples poorly reflected in their advantages. Logits retain a wider numerical range and are temperature-scaled, centered, and clipped for training. The centering baseline is the mean reward of generated samples in the batch, not the joint mean of real and generated videos, and no standard-deviation normalization is applied. The paper omits that normalization to avoid difficulty bias associated with low-variance groups. The following restates the readable definition in Algorithm 1 for generated-sample advantages before clipping, where \(B\) is the batch size and \(\beta\) controls reward strength:

\[ \widetilde A_b=\frac{r_\omega(\widetilde x_b)}{\beta}-\frac{1}{B}\sum_{j=1}^{B}\frac{r_\omega(\widetilde x_j)}{\beta}. \]

Generated videos above the batch mean receive positive advantages and those below it receive negative advantages; this is not simply selecting the highest-scoring videos for more SFT. Here, adversarial training means alternately refreshing the real-versus-generated boundary, not propagating discriminator gradients through video pixels and the reverse diffusion chain. The discriminator supplies scalar feedback used as a weight during generator updates, keeping the two optimization processes decoupled.

3. Hybrid Denoising and Noise Alignment: apply real-data constraints at the noise scale where rewards act

SHIFT treats a clean generated video as a training sample, adds noise through the forward diffusion process, and computes a standard denoising error. Video sampling still incurs a cost, but the method need not store and optimize the probability of every reverse denoising transition. Real videos simultaneously pass through an ordinary SFT branch, supplying a supervised target that does not drift with the current policy. Positive-advantage samples have their denoising errors reduced, whereas negative-advantage samples induce the opposite update under this surrogate objective; the real branch continuously constrains the output distribution. When generated-sample advantages are zero, only standard supervised fine-tuning on real data remains. The method thus neither trains an extra video postprocessor nor reranks candidate videos at inference time.

Both branches must also share the sampled noise level within an update, which is Noise Alignment. High noise levels predominantly affect coarse structure and motion, whereas low levels affect details; independently sampling the branches can let rewards strongly alter motion while the real-data constraint acts mostly at another scale. Sharing the level makes SFT constrain the model at the scale of the reward update, reducing the mismatch between a nominal regularizer and where it actually acts. The shared quantity is the noise level; this does not imply that real and generated videos must receive exactly the same Gaussian noise tensor. To display the structure clearly, the following uses \(\widehat A_b\) for the applied clipped advantage and restates the denoising surrogate from Algorithm 1 and Eq. (12):

\[ \mathcal L_{\mathrm{SHIFT}}=\mathbb E_{b,t}\left[\widehat A_b\left\|\epsilon_\theta(\widetilde x_{b,t},t)-\widetilde\epsilon_b\right\|_2^2+\left\|\epsilon_\theta(x_{b,t},t)-\epsilon_b\right\|_2^2\right]. \]

Both terms use the same \(t\); the first comes from generated videos and the second from real videos, with conditioning omitted from the notation for brevity. This optimizes a loss after forward noising and should not be interpreted as eliminating reverse generation altogether. The paper motivates the hybrid objective using a reward-tilted real-data distribution and Forward KL, then substitutes a diffusion ELBO proxy for the likelihood gradient. However, the mixture estimator over real and generated samples already introduces an approximation, so the final linear-advantage loss should not be described as an exact unbiased solution to the target KL objective.

A Worked Example

Consider a conditioning frame of an object undergoing deformation and a video produced from it; this is an explanatory example, not an additional experiment. If the result is almost static, adjacent frames may look smooth, so smoothness alone cannot establish that the intended motion was generated. IMR assesses whether optical-flow residuals resemble real video, while LMR checks for credible sustained displacement and structural relationships among tracked points. The video's reward is compared with other generated videos in the batch to determine its weight in the generated-sample denoising branch. Meanwhile, real deformation videos provide an SFT constraint at the same noise level, discouraging abnormal texture changes used to obtain reward. Newly emerging artifacts become discriminator negatives in subsequent iterations; deployment retains the trained generator alone.

Loss & Training

SVD has approximately 1.2B parameters and is fine-tuned on approximately 4K temporally segmented DAVIS2017 training clips, with rank-32 LoRA applied only to temporal attention. Wan2.2 TI2V has approximately 5B parameters and is fine-tuned on the deformation subset of WISA-80K, with rank-32 LoRA applied to all attention layers. The main SVD comparison uses the same 64 GPUs, and both SHIFT and FlowGRPO use a buffer reuse factor of \(K=10\). Multiple inner updates after generating a batch amortize sampling cost; online training does not mean every gradient step must regenerate videos. Sampler independence means the optimization does not require the stepwise likelihood of a particular discrete reverse chain, not that all possible samplers have experimentally demonstrated equal quality. Several displayed equations in the cached text have missing symbols; the formulas above restate only the advantage and loss structures supported by readable algorithmic and textual descriptions, without reconstructing the entire derivation. The prose is not fully consistent about logit clipping versus advantage clipping and gives no verifiable ranges here; it also does not specify the exact IMR-LMR combination weights. More detailed ablations on temperature, reuse, reward backbones, and query sampling are deferred to supplementary material whose tables are absent from this cache, so their configurations and numbers are not supplied.

Key Experimental Results

Main Results

Source: Table 1, page 10. SVD is fine-tuned on DAVIS2017; VBench metrics use VBench-I2V, and FVD is computed on the DAVIS2017 validation set. Higher Appearance, Motion, Overall, and Motion Score are better; lower FVD is better. Time is per-epoch wall-clock cost on identical hardware, normalized to SFT at 1.00 times. Motion Score is a separate metric from reference [41], not VBench Motion; its full computation is not provided in the main text.

Method	Appearance	Motion	Overall	Motion Score	FVD	Training Time
Base	83.68	85.89	84.41	4.39	395.87	N/A
SFT	85.07	76.09	82.08	2.69	322.81	1.00x
Track4Gen	85.01	76.44	82.15	2.84	316.91	1.07x
DenseDPO	83.73	86.03	84.50	4.34	399.73	2.40x
FlowGRPO	83.61	86.67	84.63	4.44	396.56	19.35x
SHIFT	83.69	86.70	84.69	4.40	396.45	2.46x

SHIFT exceeds SFT by 10.61 Motion points but exceeds Base by only 0.81; preventing fine-tuning degradation is a more appropriate characterization than dramatically surpassing the base model. Against FlowGRPO, it gains only 0.03 Motion and 0.06 Overall points while being approximately 7.9 times faster per epoch; this is not a measurement of total time to reach the same quality threshold. SHIFT's Motion Score of 4.40 is below FlowGRPO's 4.44, and its FVD of 396.45 is slightly worse than Base's 395.87, so the paper's broad claim of no individual metric degradation should not be repeated unqualified. Table 2, page 12 reports Wan2.2 Motion values of 86.25, 83.34, and 86.45 for Base, SFT, and SHIFT, with Overall values of 86.27, 85.87, and 86.40. Their corresponding Motion Scores are 2.96, 1.94, and 2.93, showing that SHIFT largely avoids the SFT loss without exceeding the base model on this metric.

Ablation Study

Source: Table 3, page 13. All variants are evaluated at the same epoch 7; Figure 4 states that they start from the same pretrained SVD checkpoint. NA denotes Noise Alignment and Adv. RM denotes adversarial reward-model updates; A3 adds LMR to A1, not to A2.

Config	Enabled Components	Appearance	Motion	Overall	Motion Score	FVD
A0	IMR	81.62	91.01	84.75	4.29	502.60
A1	IMR + NA	82.53	87.18	84.08	3.75	472.20
A2	IMR + NA + Adv. RM	83.06	87.28	84.47	3.75	451.20
A3	IMR + NA + LMR	82.48	89.31	84.76	4.14	460.00
A4	IMR + NA + Adv. RM + LMR	83.63	86.98	84.75	4.35	401.76

Key Findings

• A0 has the highest Motion at 91.01 but the worst FVD at 502.60, showing that a rising reward or aggregate motion metric does not automatically indicate better video quality.
• Adding adversarial refresh alone from A1 to A2 reduces FVD from 472.20 to 451.20; its main contribution is stability rather than increased motion magnitude.
• Adding long-term rewards from A1 to A3 raises Motion from 87.18 to 89.31 and Motion Score from 3.75 to 4.14, supporting complementarity between trajectory supervision and local residuals.
• A4 improves Appearance over A3 by 1.15 and lowers FVD by 58.24, but Overall is 0.01 lower; the full model represents a multi-metric tradeoff rather than universal superiority.
• The fixed-epoch ablations in Table 3 should not be treated as the same final checkpoint as Table 1; their reported values are different.

Highlights & Insights

• Using a physically inspired residual as a discriminative feature rather than enforcing zero is an important adaptation to real lighting and estimation errors, better suited to video data than simply minimizing a flow residual.
• Noise Alignment shows that an effective supervised anchor depends not only on loss weight but also on whether it operates at the scale where reward optimization changes the model.
• Adversarial training need not require GAN-style end-to-end gradient coupling: an adaptive discriminator can provide relative sample weights while the generator uses its existing diffusion-training interface.
• Reader insight: motion evaluation should jointly report magnitude, coherence, and appearance, preventing a rising dynamic score from concealing texture artifacts or distributional drift.

Limitations & Future Work

• The authors assume real data exhibits correct motion, but compression artifacts, occlusion, camera motion, and acquisition biases can enter the discriminator's target; matching feature distributions is not proof of physical correctness.
• Flow and tracking errors can contaminate rewards. Confidence inputs mitigate rather than eliminate this problem, motivating separate evaluation of rapid motion and heavy occlusion.
• Two generator backbones provide cross-model evidence, but the Wan2.2 comparison includes only Base, SFT, and SHIFT, so superiority over alternative reward methods across all large video models is not established.
• Small VBench gains are not accompanied by repeated-run variance or confidence intervals; computational savings should be discussed separately from statistical significance.
• The main text is insufficient to reproduce every reward-combination and clipping detail and lacks comprehensive physical-task verification; fixed test suites, released configurations, and stratified hard-case analysis would strengthen the evidence.

Related Work & Insights

• vs Track4Gen: it supplies auxiliary correspondence supervision through point tracking; SHIFT turns trajectories and local residuals into refreshed rewards, focusing on preserving motion during reward-driven fine-tuning.
• vs DenseDPO: it uses fine-grained temporal preferences; SHIFT substitutes automatic real-versus-generated discrimination for human preference labels but still relies on real data and pretrained motion estimators.
• vs FlowGRPO: its optimization uses policy updates over the reverse sampling chain; SHIFT re-noises generated videos for training, avoiding stepwise reverse-trajectory optimization and substantially lowering per-epoch cost in the reported SVD setting.
• vs DiffusionNFT and reward-weighted regression: forward-process alignment and weighted denoising have precedents; the more specific combination here is motion representations, a real-data anchor, centered advantages, and online adversarial refresh.
• An extension worth studying is how to distinguish membership in real data from credible dynamics when the real dataset itself contains systematic motion biases, instead of simply reproducing those biases.

Rating

• Novelty: 4/5. The combination of motion representations and stable hybrid updates is targeted, although weighted denoising and adaptive discriminators each have precedents.
• Experimental Thoroughness: 4/5. Two backbones and clear component ablations are provided, but variance estimates and broader physical-correctness evaluation are missing.
• Writing Quality: 3/5. Training roles are clear, but approximation boundaries, clipping details, and claims of no degradation require greater precision.
• Value: 4/5. It offers a reusable approach to motion preservation in video post-training, with efficiency gains more convincing than the small leaderboard advantage.