Inference-time Motion Calibration for Video Generation¶
Conference: ECCV 2026
Paper: ECCV Official Page
Full Cache: /Users/zy/workspace/paper_cache/ECCV2026/eccv-3884.txt
Area: Video Generation
Keywords: Inference-time Calibration, Dynamic Latent Control, Dynamic RoPE Control, Amortization Trajectory Rectification, Vision Foundation Models
TL;DR¶
Addressing motion artifacts, severe temporal flicker, and semantic drift in pretrained video generators, this paper presents a training-free inference-time motion calibration framework that couples external VFM-based velocity guidance (DLC) with online internal attention modulation (DRC), amortized by ATR to incur only 1.3x baseline latency.
Background & Motivation¶
Generative video models built upon diffusion and flow matching architectures—such as Diffusion Transformers (DiTs) including CogVideoX, HunyuanVideo, and Wan 2.1—have achieved impressive advancements in high-resolution, photorealistic per-frame synthesis. However, when deployed for continuous video generation, these models frequently suffer from unnatural physical motions, high-frequency temporal flicker, deformed anatomical structures, and catastrophic semantic drift over time. This deficiency stems from their core training objectives, which predominantly optimize per-frame visual reconstruction; coherent temporal dynamics emerge merely as an indirect byproduct of temporal attention layers rather than explicit physical motion supervision. For frontier video backbones with billions of parameters, full retraining or extensive fine-tuning remains computationally prohibitive.
Existing test-time scaling approaches predominantly tackle quality degradation via black-box heuristic search, such as evolutionary search (EvoSearch) or noise trajectory search guided by external reward verifiers. These methods maintain large candidate populations and select high-scoring samples through discrete ranking. Consequently, they treat the underlying generator as a black box and cannot provide dense, directional gradient feedback to rectify flawed sampling dynamics; worse, their computational overhead scales linearly with candidate population size. In parallel, fixed architectural inductive biases—notably rigid temporal receptive fields and static positional encodings—fail to adapt across diverse motion regimes, yielding oversmoothed fast actions or weakly coupled slow dynamics.
This work addresses this challenge by converting off-the-shelf Vision Foundation Models (VFMs) from passive ranking oracles into differentiable energy functions whose gradients locally steer the sampling trajectory, while simultaneously adapting the internal attention temporal receptive field online. Core idea: formulate inference-time motion calibration as a training-free trajectory energy-shaping problem, deploying Dynamic Latent Control (DLC) to inject VFM motion-semantic gradients, Dynamic RoPE Control (DRC) to dynamically modulate temporal attention kernels, and Amortization Trajectory Rectification (ATR) to cache rectification fields for minimal computational overhead.
Method¶
Overall Architecture¶
The proposed method intervenes in continuous Flow-ODE trajectories of the form \(\frac{\mathrm{d}z_t}{\mathrm{d}t} = v_\theta(t, z_t \mid c)\) (and standard clean-latent ODE approximations in diffusion samplers). Calibration proceeds online along a single trajectory: at sparse calibration steps, the clean terminal latent \(\hat{z}_0(t)\) is estimated via first-order Euler extrapolation and mapped through a lightweight decoder into low-resolution proxy frames; vision foundation models (optical flow via RAFT, geometric structure via DINOv2, and semantic alignment via CLIP) evaluate a composite motion-semantic energy \(\mathcal{L}\) and backpropagate gradients into the latent space; this gradient updates the base velocity field (DLC), while high-order optical flow statistics dynamically reconfigure temporal Rotary Position Embeddings (DRC); finally, ATR caches the velocity rectification field across consecutive intermediate solver steps to bypass repetitive VFM backpropagation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Noise state z_t and condition c"] --> B["Euler clean latent extrapolation z_0(t)"]
B --> C["Decode low-resolution proxy frames x_proxy"]
C --> D["Composite VFM energy evaluation<br/>Optical Flow + DINOv2 + CLIP"]
D --> E["Dynamic Latent Control (DLC)<br/>Backpropagate velocity calibration gradient"]
D --> F["Dynamic RoPE Control (DRC)<br/>Modulate temporal attention receptive field"]
E --> G["Amortization Trajectory Rectification (ATR)<br/>Cache and reuse rectification field"]
F --> H["Calibrated velocity v' and DiT attention update"]
G --> H
H --> I["Output next state latent z_{t-Δt}"]
Key Designs¶
1. Dynamic Latent Control (DLC): differentiable trajectory shaping via VFM feedback To transcend the limitations of black-box sample filtering, DLC targets a Boltzmann-reweighted probability distribution \(p^\star(z_0 \mid c) \propto p_\theta(z_0 \mid c) \exp(-\lambda_{\mathrm{E}} \mathcal{L}(z_0))\). Rather than waiting for full reverse diffusion, DLC estimates the clean latent at intermediate timestep \(t \in [0, 1]\) via a forward Euler extrapolation \(\hat{z}_0(t) \approx z_t - s_\tau \tau_t v_\theta(t, z_t \mid c)\), where \(\tau_t \approx t\) represents the remaining time scale and \(s_\tau \in [1.0, 1.25]\) is a scaling factor. Passing \(\hat{z}_0(t)\) through decoder \(\mathcal{D}\) generates low-resolution proxy frames \(x^{\mathrm{proxy}} = \mathcal{D}(\hat{z}_0(t))\), on which a standardized composite loss is evaluated: $\(\mathcal{L} = w_1 \mathcal{L}_{\mathrm{flow}} + w_2 \mathcal{L}_{\mathrm{struct}} + w_3 \mathcal{L}_{\mathrm{sem}}\)$ where \(\mathcal{L}_{\mathrm{flow}}\) enforces temporal motion vector continuity, \(\mathcal{L}_{\mathrm{struct}}\) retains fine-grained object geometry via DINOv2, and \(\mathcal{L}_{\mathrm{sem}}\) ensures prompt alignment via CLIP. Backpropagating this error yields a corrected continuous velocity field: $\(v^{\mathrm{DLC}}(z_t, t \mid c) = v_\theta(t, z_t \mid c) + \beta_t b_t J_{\mathcal{D}}(\hat{z}_0(t))^\top \nabla_x \mathcal{L}(x^{\mathrm{proxy}})\)$ Crucially, adding the positive velocity correction corresponds to gradient descent in terminal endpoint space because \(\hat{z}_0 = z_t - \tau_t v_t\). To eliminate chaotic pseudo-gradients from early noisy latents, DLC activates strictly after \(t_{\mathrm{start}} = 0.5\) and employs momentum alongside gradient norm clipping (\(\gamma = 2.0\)) to guarantee stability and prevent mode distortion.
2. Dynamic RoPE Control (DRC): motion-adaptive temporal attention modulation While trajectory-level steering corrects macroscopic drift, fixed-weight DiT backbones retain an inflexible architectural temporal receptive field: high-speed, violent motion causes cross-frame visual oversmoothing and high-frequency flickering, whereas gentle, prolonged motions suffer from inadequate long-range temporal binding. DRC dynamically recalibrates temporal Rotary Position Embeddings (RoPE) without modifying any model weights. For temporal lag \(\Delta = r - s\), the modulated relative phase angle is formulated as: $\(\Delta \phi_i = \left(\frac{r - s}{\rho_t} + 2\phi_t\right)\omega\)$ where temporal dilation \(\rho_t\) governs the effective receptive field span and phase bias \(\phi_t\) introduces directional temporal asymmetry. Using optical flow velocity statistics \(\hat{u}_i\), discrete jerk derivatives \(\hat{j}_i\), and inter-frame feature asymmetry \(\Delta s_t\) calculated during the proxy step, DRC updates parameters dynamically: $\(\rho_t = \rho_0 \exp(-\kappa_u \hat{u}_i - \kappa_j \hat{j}_i), \quad \phi_t = \beta \Delta s_t\)$ When jitter or rapid acceleration is detected, \(\rho_t\) contracts toward its lower bound to focus attention on adjacent frames, effectively suppressing flicker; in slow, uniform scenes, \(\rho_t\) expands to reinforce long-horizon temporal coherence.
3. Amortization Trajectory Rectification (ATR): efficient cached rectification fields Evaluating full VFM backpropagation at every solver step imposes an intolerable \(10\times\) latency penalty. ATR exploits the local smoothness and Lipschitz continuity of VFM-induced energy fields along ODE trajectories to decouple sampling into sparse calibration steps and rapid base steps. At slow step \(k_s\) (executed every \(S_{\mathrm{ATR}}\) solver steps), the algorithm computes the exact rectification field residual: $\(R(z_{k_s}, t_{k_s}) = v'_{k_s} - v_\theta(z_{k_s}, t_{k_s})\)$ and caches it. Across all subsequent fast steps \(k_f \in (k_s, k_s + S_{\mathrm{ATR}})\), the solver approximates the calibrated velocity without calling VFMs by simply reusing the cached rectification: $\(v'_{\mathrm{approx}}(z_{k_f}, t_{k_f}) = v_\theta(z_{k_f}, t_{k_f}) + R(z_{k_s}, t_{k_s})\)$ The authors establish a formal Lipschitz error bound: \(\|\mathcal{E}_{\mathrm{approx}}\|_2 \le L \|z_{k_s} - z_{k_f}\|_2 + K |t_{k_s} - t_{k_f}|\). Because the ODE solver step size is small, drift remains tightly bounded, slashing the number of expensive VFM backward passes from 50 to 8 per trajectory while retaining virtually identical generation quality.
Key Experimental Results¶
Main Results¶
On text-to-video (T2V) benchmarks VBench 1.0 and VBench 2.0, the framework is evaluated against leading proprietary platforms, open-source baselines, and test-time search methods. It demonstrates marked superiority on advanced physical motion fidelity metrics.
Table 1: Quantitative comparison on VBench 1.0 & VBench 2.0 (Text-to-Video) (from original Table 1)
| Method | Motion Smoothness ↑ | Temporal Flickering ↓ | Dynamic Degree ↑ | Motion Rationality ↑ | Motion Order ↑ | Human Anatomy ↑ | Mechanics ↑ |
|---|---|---|---|---|---|---|---|
| Sora | – | – | – | 34.48% | 15.15% | 86.45% | 62.22% |
| Kling 1.6 | – | – | – | 38.51% | 29.29% | 86.99% | 65.55% |
| CogVideoX-1.5 (Baseline) | 96.92% | 98.66% | 70.97% | 33.91% | 26.60% | 59.72% | 80.80% |
| CogVideoX-1.5 + EvoSearch | 96.80% | 98.72% | 71.40% | 34.00% | 26.70% | 59.80% | 81.00% |
| CogVideoX-1.5 + Noise Traj. | 97.05% | 98.78% | 71.80% | 34.10% | 26.80% | 59.90% | 81.10% |
| CogVideoX-1.5 + Ours (N=8) | 97.40% | 99.10% | 72.80% | 36.00% | 29.00% | 61.20% | 82.00% |
| CogVideoX-1.5 + Ours (N=16) | 97.20% | 98.90% | 73.20% | 35.20% | 28.50% | 61.40% | 82.30% |
| HunyuanVideo (Baseline) | 98.70% | 99.10% | 57.00% | 34.48% | 26.94% | 88.58% | 76.09% |
| HunyuanVideo + EvoSearch | 98.55% | 99.00% | 57.50% | 34.60% | 27.10% | 88.70% | 76.40% |
| HunyuanVideo + Noise Traj. | 98.50% | 98.95% | 57.70% | 34.70% | 27.20% | 88.65% | 76.50% |
| HunyuanVideo + Ours (N=8) | 96.90% | 98.80% | 68.60% | 37.20% | 29.50% | 90.30% | 78.80% |
| HunyuanVideo + Ours (N=16) | 96.70% | 98.70% | 68.20% | 36.80% | 29.70% | 90.10% | 79.10% |
On image-to-video (I2V) generation, the framework similarly demonstrates broad architecture-agnostic gains across subject alignment, camera motion, and visual quality:
Table 2: Quantitative comparison on VBench-I2V (Image-to-Video) (from original Table 2)
| Method | i2v_subject ↑ | i2v_background ↑ | Camera Motion ↑ | Dynamic Degree ↑ | Motion Smoothness ↑ | Aesthetic Quality ↑ |
|---|---|---|---|---|---|---|
| FramePack (Baseline) | 93.50% | 93.00% | 21.50% | 20.05% | 96.50% | 63.94% |
| FramePack + EvoSearch | 93.60% | 93.10% | 21.60% | 20.20% | 96.60% | 63.98% |
| FramePack + Ours (N=8) | 94.40% | 93.80% | 22.80% | 21.10% | 97.30% | 64.32% |
| Wan 2.1 (Baseline) | 96.30% | 95.60% | 33.50% | 46.02% | 97.20% | 63.12% |
| Wan 2.1 + EvoSearch | 96.40% | 95.70% | 33.60% | 46.20% | 97.35% | 63.20% |
| Wan 2.1 + Ours (N=8) | 98.10% | 97.40% | 36.80% | 47.70% | 98.50% | 63.72% |
| Hunyuan I2V (Baseline) | 94.50% | 94.00% | 23.00% | 17.74% | 96.00% | 62.04% |
| Hunyuan I2V + Ours (N=8) | 96.50% | 95.90% | 26.50% | 18.55% | 97.50% | 62.42% |
Ablation Study¶
Table 3: Component isolation and efficiency comparison on HunyuanVideo (synthesized from Tables 3 & 5)
| Configuration | FVD ↓ | Motion Score ↑ | Temporal Flicker ↑ | Latency (s) ↓ | Description |
|---|---|---|---|---|---|
| HunyuanVideo Baseline | 430.2 | 0.58 | 98.70% | 5.2s | Uncalibrated base generator |
| + Universal Guidance (dense backprop) | 385.4 | 0.63 | – | 48.6s | Dense Jacobian backward pass at every step (9.3x latency) |
| + DLC only (no ATR, per-step) | 358.6 | 0.68 | 99.05% | ~50s | Trajectory energy guidance alone yields major quality jump |
| + DRC only (no DLC) | 401.3 | 0.62 | 99.20% | 5.4s | Internal attention modulation alone without trajectory guidance |
| + DLC + DRC (Full, no ATR) | 318.9 | 0.72 | 99.10% | 55.7s | Theoretical quality upper bound, but high cost (10.7x overhead) |
| + Full Model + ATR Amortization (N=8) | 328.5 | 0.70 | 98.80% | 6.8s | Preserves >97% quality gain with only 1.3x baseline latency |
Table 4: Scalability with Vision Foundation Model (VFM) strength (from original Table 4)
| VFM Configuration (\(\mathcal{L}_{\mathrm{sem}} + \mathcal{L}_{\mathrm{mot}}\)) | Scheduling | FVD ↓ | CLIP Score ↑ | Motion Score ↑ | Latency (s) ↓ |
|---|---|---|---|---|---|
| CLIP ViT-B/32 + Frame Difference | Full DLC | 405.7 | 0.279 | 0.61 | 28.5s |
| CLIP ViT-B/32 + Frame Difference | ATR Amortized | 406.1 | 0.279 | 0.60 | 5.9s |
| DINOv2 ViT-B/14 + RAFT Optical Flow | Full DLC | 329.7 | 0.287 | 0.70 | 48.8s |
| DINOv2 ViT-B/14 + RAFT Optical Flow | ATR Amortized | 330.5 | 0.286 | 0.70 | 6.4s |
| CLIP ViT-L/14 + RAFT Optical Flow | Full DLC | 318.9 | 0.293 | 0.72 | 55.7s |
| CLIP ViT-L/14 + RAFT Optical Flow | ATR Amortized | 319.2 | 0.292 | 0.72 | 6.8s |
Key Findings¶
- Synergy of external and internal control: Deploying DLC alone reduces FVD from 430.2 to 358.6, while DRC alone reduces it to 401.3. When unified, FVD plummets to 318.9, proving that macroscopic velocity field deflection and microscopic temporal attention kernel adaptation act as complementary mechanisms.
- High fidelity of amortized rectification: ATR with only 8 calibration steps produces an FVD of 328.5 versus 318.9 for exhaustive 50-step DLC (preserving >97% of the gain), while slashing latency from 55.7s down to 6.8s (an 8.2x speedup). This empirically confirms the local Lipschitz continuity of VFM guidance fields.
- Search-based methods hit a performance ceiling: Search baselines (EvoSearch, Noise Trajectory Search) yield marginal improvements (\(\le 0.3\%\)) across advanced motion metrics despite consuming comparable forward passes. Without directional gradients targeting continuous motion derivatives, discrete ranking cannot systematically guide models toward physically plausible dynamics.
Highlights & Insights¶
- Transitioning from ranking oracles to continuous guidance fields: The work reframes pre-trained vision foundation models into differentiable energy functions evaluated on intermediate Euler proxy latents, enabling training-free quality scaling without updating backbone weights.
- Low-dimensional dynamic RoPE modulation: Demonstrates that temporal Rotary Position Embedding parameters (\(\rho_t, \phi_t\)) serve as an effective, zero-parameter-overhead control knob to dynamically match attention receptive fields to real-time scene velocity.
- Reusable cached rectification (ATR): Proves that trajectory correction vectors exhibit high temporal correlation along diffusion ODE paths, offering a general recipe for accelerating gradient-guided test-time sampling in diffusion and flow matching models.
Limitations & Future Work¶
- Slight trade-off in superficial motion smoothness: On select backbones like HunyuanVideo, substantial increases in dynamic expressiveness and physical correctness lead to a minor dip in superficial Motion Smoothness (from 98.70% to 96.90%), highlighting an intrinsic tension between high dynamic range and uniform frame-to-frame interpolation.
- Decoder and noise sensitivity of intermediate proxy frames: The accuracy of DLC gradients hinges upon proxy frame clarity past \(t \ge 0.5\). Severe VAE reconstruction degradation or aggressive extrapolation step sizes may impair gradient fidelity.
- Future directions: Investigating latent-space native motion estimators to eliminate the need for intermediate pixel-space VAE decoding entirely.
Related Work & Insights¶
- vs Universal Guidance (UG): UG backpropagates through the entire network at every sampling step, leading to extreme memory footprint and a 9.3x latency overhead (48.6s); in contrast, this work employs Euler proxy estimation and ATR to achieve superior FVD (328.5 vs 385.4) with only a 1.3x latency overhead (6.8s).
- vs EvoSearch / Noise Trajectory Search: Conventional search approaches scale linearly with candidate population size and treat generators as black boxes; this framework injects closed-loop motion gradients along a single trajectory, achieving far superior physical fidelity at lower compute cost.
- vs Motion-conditioned fine-tuning (e.g., FlowVid, DynamicCrafter): Prior conditioned methods require heavy paired training and alter network weights; this work is entirely training-free and plug-and-play across diverse DiT and 3D U-Net architectures.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates continuous inference-time motion calibration combining external trajectory guidance with dynamic RoPE attention modulation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensive evaluations on CogVideoX, HunyuanVideo, Wan 2.1, and FramePack spanning both T2V and I2V, backed by rigorous ablations and Lipschitz error analysis.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear theoretical formulation, well-motivated architectural designs, and thorough empirical validation.
- Value: ⭐⭐⭐⭐⭐ Highly practical, training-free plug-in framework that substantially enhances open-source video generation with minimal latency overhead.