Skip to content

QWERTY: Training-Free Motion Control via Query-Warped Video Diffusion Transformers

Conference: ECCV 2026
Paper: ECCV Official
Code: Pending release
Area: Video Generation
Keywords: Video Diffusion Transformer, Training-free, Motion Control, Query Warping, Attention Manipulation

TL;DR

Addressing the challenge of steering motion in video Diffusion Transformers (DiTs) without expensive retraining, QWERTY introduces a training-free framework that performs Semantic-Temporal Channel Decomposition (STCD) and query-only warping on 3D full attention, combined with self-guided latent optimization to achieve precise object trajectory and camera motion control.

Background & Motivation

Video diffusion transformers (DiTs) with 3D full-attention backbones generate visually stunning videos with superior temporal coherence and expressive visual dynamics. However, existing pretrained models control motion almost exclusively through natural language text prompts, which are fundamentally ill-suited for specifying exact spatial displacements, velocities, or intricate trajectories. To impose explicit trajectory guidance, prevailing approaches rely on spatial prompts such as bounding boxes, masks, or point tracks, integrating them via ControlNet-style auxiliary branches, warped input noise, or finetuning extra tokens. Nevertheless, finetuning demands heavy computational overhead, extensive annotated datasets, and frequently impairs the foundational visual generalization of the pretrained model.

Consequently, training-free motion control strategies developed for U-Net architectures (such as noise warping, attention masking, or latent optimization based on bounding-box feature similarity) have been examined. However, when adapted to DiTs, these U-Net methods fail dramaticallyโ€”noise warping causes severe visual collapse, while feature similarity optimization yields mostly static videos or bizarre stretching artifacts due to the absence of convolutional inductive bias in DiTs. Furthermore, recent DiT-based training-free works primarily focus on motion transfer from an existing reference video, leaving interactive, user-defined motion control (from user-drawn masks or camera optical flows) an unresolved problem.

This paper tackles this gap through a key insight: in image-to-video (I2V) DiTs, the first-frame token features already exhibit a well-structured and stable spatial semantic layout even at early denoising timesteps; meanwhile, the attention softmax operation normalizes independently across keys for each query. Core idea: selectively warp only the query tokens within an isolated frame-consistent semantic subspace from the first frame into target frames, directly concentrating target attention onto corresponding first-frame reference keys without perturbing the attention denominator, and utilize this query-warped prediction to self-guide latent optimization.

Method

Overall Architecture

The QWERTY framework operates through three core stages: first, Semantic-Temporal Channel Decomposition (STCD) applies PCA to separate frame-invariant semantic query channels from frame-variant temporal dynamics; second, semantic-subspace query warping pastes first-frame queries into target frames after spatial RoPE but before temporal RoPE, complemented by specialized hole-handling; finally, the query-warped noise serves both as an early denoising trajectory bias and a self-guidance signal to optimize the input latent via magnitude and Fourier phase alignment.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input image & user control<br/>(Object mask warp / Camera optical flow)"] --> B["Semantic-Temporal Channel Decomposition (STCD)<br/>PCA on spatial and temporal anchor tokens"]
    B --> C["Semantic Subspace Query Warping<br/>Warp & paste queries after spatial RoPE + Hole handling"]
    C --> D["Temporal RoPE injection & 3D full attention forward"]
    D --> E["Dual velocity prediction<br/>Standard prediction u_t & Query-warped prediction u_t^warp"]
    E --> F["Self-Guided Latent Optimization<br/>L2 loss + Fourier phase-consistency constraint"]
    F --> G["Flow matching update x_{t-ฮ”t}<br/>Controlled coherent video output"]

Key Designs

1. Query-Only Warping: Exploiting Attention Softmax Asymmetry for Clean Correspondence The operational goal of motion control is to compel target token \(i\) at frame \(n\) to attend strongly to source token \(s\) at frame 1, thereby drawing semantic value vectors from the reference frame. The attention weight between target token \(i\) and source token \(s\) is governed by: $$ \alpha_{i,s} = \frac{\exp(q_i^\top k_s / \sqrt{d})}{\sum_j \exp(q_i^\top k_j / \sqrt{d})} $$ Conventional intuitions might suggest warping keys (\(k_i \leftarrow k_s\)) or jointly warping queries and keys (\(q_i \leftarrow q_s, k_i \leftarrow k_s\), analogous to hidden-state warping). However, because the softmax denominator sums across key indices \(j\), key warping leaves the numerator \(q_i^\top k_s\) unchanged and disperses first-frame source attention. Joint query-key warping does increase the numerator, but also introduces competing target-key terms into the denominator with magnitudes near self-correspondence, thereby diluting the effective attention pull toward \(k_s\). QWERTY proves both analytically and empirically that setting \(q_i \leftarrow q_s\) alone boosts the target-to-source numerator without inflating the denominator with spurious competitors, delivering the cleanest and most concentrated cross-frame correspondence.

2. Semantic-Temporal Channel Decomposition (STCD): Disentangling Frame-Invariant Subspaces Directly pasting raw queries across frames introduces noticeable visual artifacts because DiT query representations are heavily entangled with temporal dynamics and frame ordering after positional encoding. STCD leverages principal component analysis (PCA) to disentangle these subspaces. Taking first-frame features \(X^{(1)} \in \mathbb{R}^{P \times D}\) as the semantic anchor matrix \(A^{\text{sem}}\) (which captures rich spatial layout), it computes singular value decomposition (SVD) \(A = U \Sigma V^\top\) and derives channel-wise semantic saliency across the top \(k_{\text{sem}}\) components: $$ s_d(\mathbf{A}, k) = \max_{1 \le i \le k} \sigma_i |V_{d,i}| $$ Concurrently, spatial average pooling across all tokens within each frame produces a temporal anchor matrix \(A^{\text{tmp}} \in \mathbb{R}^{F \times D}\), filtering out local spatial details to isolate inter-frame variations and yield temporal saliency \(s^{\text{tmp}}\). A binary mask \(m_{\text{sem}, d} = \mathbb{I}[s_d^{\text{sem}} \ge s_d^{\text{tmp}}]\) partitions channels, ensuring query warping operates exclusively within channels where semantic discriminativeness dominates temporal variance.

3. Decoupled 3D RoPE Warping and Boundary Hole Handling 3D Rotary Position Embeddings (3D RoPE) decompose into independent spatial \((x, y)\) and temporal \((t)\) rotations along separate channel blocks. If queries are warped before applying spatial RoPE, target tokens lose the spatial coordinate shift relative to first-frame keys; conversely, applying temporal RoPE before warping would stamp target tokens with frame 1's temporal index, destroying temporal progression. QWERTY applies warping strictly after spatial RoPE and before temporal RoPE, guaranteeing that target queries match source keys spatially while respecting their native temporal slots. Furthermore, for object motion control, the vacated source region is filled with nearest-neighbor background tokens to prevent "object cloning" artifacts; for camera optical flow, forward-warping holes created by disocclusions are left untouched as native queries, allowing the DiT to inpaint them seamlessly.

4. Self-Guided Latent Optimization: Phase-Constrained Trajectory Steering During early denoising steps (\(t=999\) through the first 5 steps), the query-warped DiT prediction \(\hat{u}_t^{\text{warp}}\) already captures global motion intent. To reinforce this trajectory into the latent space without retraining, QWERTY utilizes \(\hat{u}_t^{\text{warp}}\) as a self-guidance target. Freezing the DiT backbone, gradients backpropagate into the input latent \(x_t\) to minimize the discrepancy between the standard prediction \(\hat{u}_t\) and the warped prediction \(\hat{u}_t^{\text{warp}}\), augmented with a Fourier phase constraint that preserves motion boundaries and high-frequency structural contours: $$ \mathcal{L}_{\text{guide}} = |\hat{\mathbf{u}}_t - \hat{\mathbf{u}}_t^{\text{warp}}|_2^2 + \lambda |\angle \mathcal{F}(\hat{\mathbf{u}}_t) - \angle \mathcal{F}(\hat{\mathbf{u}}_t^{\text{warp}})|_2^2 $$ where \(\mathcal{F}(\cdot)\) is the discrete Fourier transform, \(\angle\) extracts the phase angle, and \(\lambda = 0.1\). This optimization stabilizes camera panning and large non-rigid deformations where pure forward warping might exhibit mild drift.

Key Experimental Results

Main Results

Experiments are conducted on two standard benchmarks: VIPSeg (100 validation videos featuring dominant foreground objects with user-specified polygonal mask warps) for object motion control, and DL3DV (100 complex scene videos with optical flow conditioning) for camera motion control. All models are evaluated on native resolutions resized to 480ร—832 on the Wan 2.2 TI2V-5B backbone across image quality (FID, FVD), trajectory adherence (FTD, lower is better), and VBench consistency metrics.

Dataset / Task Method Paradigm Backbone FID \(\downarrow\) FVD \(\downarrow\) FTD \(\downarrow\) VBench-SC \(\uparrow\) VBench-BC \(\uparrow\)
VIPSeg
(Object Control)		 FreeTraj Training-free VideoCrafter 126.8 0.924 0.528 0.886 0.901
MOFT Training-free SVD 76.04 0.861 0.527 0.955 0.944
SG-I2V Training-free SVD 60.83 0.798 0.579 0.952 0.961
Vanilla Wan Base Model Wan 2.2 46.17 0.677 0.534 0.977 0.973
Noise Warping Training-free Wan 2.2 270.38 3.392 0.569 0.725 0.753
SG-I2V (DiT) Training-free Wan 2.2 107.98 0.704 0.600 0.976 0.960
GWTF Finetuned DiT 40.19 0.645 0.528 0.958 0.975
QWERTY (Ours) Training-free Wan 2.2 45.95 0.667 0.512 0.978 0.981
DL3DV
(Camera Control)		 MotionClone* Training-free AnimateDiff 96.63 1.624 0.683 0.897 0.934
MOFT Training-free SVD 51.84 1.327 0.424 0.955 0.967
SG-I2V Training-free SVD 45.18 1.265 0.578 0.966 0.973
Vanilla Wan Base Model Wan 2.2 32.21 1.362 0.367 0.977 0.974
Noise Warping Training-free Wan 2.2 282.93 5.663 0.455 0.719 0.747
SG-I2V (DiT) Training-free Wan 2.2 42.46 1.524 0.438 0.976 0.970
GWTF Finetuned DiT 30.32 1.576 0.513 0.962 0.978
QWERTY (Ours) Training-free Wan 2.2 35.56 1.007 0.237 0.946 0.956

(Note: MotionClone utilizes reference ground-truth videos for motion transfer; FVD is scaled by \(\times 10^3\); SC denotes subject consistency, BC denotes background consistency)*

Ablation Study

Ablations on Wan 2.2 (DL3DV camera control) and CogVideoX-I2V-5B (VIPSeg object control) validate the impact of each modular component:

Configuration Description DL3DV (Wan) FVD \(\downarrow\) DL3DV (Wan) FTD \(\downarrow\) VIPSeg (Cog) FVD \(\downarrow\) VIPSeg (Cog) FTD \(\downarrow\)
QWERTY (Full) Complete proposed framework 1.007 0.237 0.668 0.473
(a) No STCD Warp all query channels without decomposition 1.348 0.297 0.697 0.542
(b) w/o spat. RoPE Warp queries without spatial RoPE 1.419 0.299 1.578 0.531
(c) w/ temp. RoPE Jointly warp temporal RoPE 1.348 0.285 0.831 0.511
(d) K Warp only keys 1.482 0.293 0.759 0.544
(e) Q, K Jointly warp queries and keys 1.427 0.290 0.727 0.531
(f) Attn. output Warp attention block output features 1.590 0.288 0.745 0.544
(g) No opt. Denoise with query-warped DiT without latent opt. 1.301 0.274 0.669 0.501
(h) Opt. only Latent optimization without query-warped forward 1.423 0.355 0.688 0.536
(i) w/o phase loss Latent optimization without Fourier phase loss 1.440 0.285 0.704 0.514

Key Findings

  • Breakdown of U-Net Paradigm on DiTs: Direct transfer of noise warping to Wan causes catastrophic collapse (FVD soaring to 3.392 and 5.663), while SG-I2V produces mostly static sequences or severe limb stretching due to the lack of spatial inductive bias in DiT.
  • Superiority of Query-Only Warping: Warping only keys (FTD 0.293 on DL3DV) or warping both queries and keys (FTD 0.290) falls distinctly behind query-only warping (FTD 0.237), directly corroborating that query-only replacement avoids inflating competing denominator terms in the attention softmax.
  • Crucial Interplay of STCD and RoPE: Omitting STCD degrades FTD from 0.237 to 0.297 on Wan, while omitting spatial RoPE causes FVD on CogVideoX to explode from 0.668 to 1.578, proving that semantic decoupling and spatial coordinates must be strictly aligned.
  • Practical Single-Image Deployment: When conditioning camera trajectories on depth maps predicted by Depth Anything 3 (DAM 3) or VGGT rather than ground-truth video flow, QWERTY achieves an optical flow error of 0.707โ€“0.710 and an FTD of 0.243โ€“0.251, performing on par with ground-truth optical flow (FTD 0.237) and demonstrating strong viability for real-world user workflows.

Highlights & Insights

  • Mathematical Exploitation of Attention Asymmetry: QWERTY uncovers that the asymmetric nature of softmax in self-attention makes query warping uniquely effective at concentrating attention from any frame onto stable reference keys, without diluting the denominator.
  • Clean Saliency Disentanglement via STCD: Using PCA on first-frame spatial anchors versus spatial-mean temporal anchors provides an elegant, lightweight, training-free mechanism to separate semantic content from temporal RoPE artifacts.
  • Phase-Guided Dual-Path Self-Supervision: Leveraging early-step query-warped velocity as an optimization target while matching Fourier phase angles establishes structural rigidity and prevents boundary distortion.

Limitations & Future Work

  • Extreme Disocclusions and Perspective Warps: Under violent 3D camera rotations or severe self-occlusions, unobserved regions rely solely on the DiT's hallucination capabilities, which occasionally yields blur or stretching.
  • Inference Latency from Dual Passes: Performing dual-path forward predictions and gradient backpropagation across the first five timesteps incurs roughly 30% to 40% additional latency during the initial denoising phase.
  • Future Directions: Integrating explicit 3D Gaussian Splatting (3D-GS) point-cloud priors to guide query warping in 6-DoF complex camera pathways.
  • vs Go-With-The-Flow (GWTF): GWTF finetunes video DiTs with warped noise, achieving strong in-domain control but exhibiting degraded out-of-domain generalization on complex real-world scenes like DL3DV (FVD 1.576 vs. QWERTY's 1.007); QWERTY achieves comparable or superior control without any retraining.
  • vs SG-I2V & FreeTraj: These methods rely on U-Net feature maps and bounding-box overlap losses. QWERTY addresses the 3D full-attention formulation of DiTs directly via query manipulation and STCD, overcoming DiT's lack of spatial locality bias.

Rating

  • Novelty: โญโญโญโญโญ (Elegant theoretical rationale for query-only warping under softmax asymmetry, coupled with STCD)
  • Experimental Thoroughness: โญโญโญโญโญ (Evaluated across Wan 2.2 and CogVideoX, rigorous ablations on attention components, and single-image depth-to-flow tests)
  • Writing Quality: โญโญโญโญโญ (Clear logical progression, precise mathematical exposition, and informative experimental figures)
  • Value: โญโญโญโญโญ (Provides an immediate, high-fidelity, training-free motion control solution for video DiTs)