SA-V2V: Training-Free Subject-Aware Video-to-Video Personalization¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://parksoobin1219.github.io/SA-V2V/
Area: Video Generation
Keywords: Video Personalization, Motion Disentanglement, Training-Free Diffusion, Video Generation, 3D Spatiotemporal Attention
TL;DR¶
Addressing the deep entanglement of motion, subject, and background within DiT spatiotemporal attention, SA-V2V establishes a training-free and inversion-free video personalization framework leveraging intrinsic functional attention decomposition, stochastic temporal injection, and alternating target-aware feature guidance to faithfully transfer subject identity while cloning reference dynamics and background.
Background & Motivation¶
Diffusion transformer (DiT) architectures have established state-of-the-art benchmarks in video synthesis, unlocking new frontiers for controllable video customization. In video-to-video personalization, the fundamental objective is: given a reference video and a subject image, synthesize a customized video that faithfully replicates the motion trajectories and background environment of the reference video while substituting the moving entity exclusively with the specified subject identity. However, existing DiT-based customization schemes (such as JointTuner, DualReal, and DreamRelation) heavily rely on test-time optimization via LoRAs or auxiliary tuning adapters, incurring substantial computational overhead (typically requiring 20 to 40 minutes of per-video optimization) and suffering from poor generalization across varied subject-motion pairings.
Training-free approaches circumvent optimization costs, yet earlier paradigms were primarily tailored for UNet-based backbones that feature explicitly decoupled spatial and temporal attention blocks where feature maps or attention maps can be manipulated independently. Conversely, modern DiTs flatten visual tokens across both space and time into a unified 3D spatiotemporal full-attention mechanism. Within this unified attention matrix, spatial identity, motion trajectories, and background semantics are deeply entangled. Naive cross-attention overrides or direct feature replacement severely damage background integrity, produce spatial misalignment between the subject and motion, or cause catastrophic appearance leakage.
The core tension lies in achieving fine-grained, independent control over reference motion, target subject identity, and reference background without parameter updates, external adapters, or compute-intensive diffusion inversion. Investigating the structural geometry of unified 3D attention matrices in DiTs, the authors observe an inherent functional factorization: intra-frame diagonal attention blocks predominantly encode spatial appearance and subject localization, whereas inter-frame off-diagonal blocks align tightly with dense optical flow dynamics. Core idea: exploit the intrinsic diagonal/off-diagonal functional decomposition of 3D spatiotemporal attention in DiTs, combining stochastic temporal injection for geometry-adaptive motion transfer with alternating Target-aware Feature Guidance (TFG) under spatial masks to achieve completely decoupled, training-free video personalization.
Method¶
Overall Architecture¶
SA-V2V receives a reference video \(z_{ref}\) and a subject image \(z_{sub}\) as inputs. To enforce temporal stability from the start, a single random Gaussian latent is replicated across \(F\) frames to construct the initial generation latent \(z_{gen}\). During reverse diffusion denoising, the framework extracts attention representations from the reference video and subject image within intermediate DiT layers, intervening via two non-invasive, training-free mechanisms across the early critical timesteps: first, stochastic temporal injection selectively transfers inter-frame attention dynamics subject to random spatial Bernoulli masking; second, Target-aware Feature Guidance (TFG) independently conditions subject identity (S-TFG) and background appearance (B-TFG) under spatial segmentation masks via an alternating timestep schedule. The resulting denoised latents are decoded into the personalized video via the pretrained 3D VAE.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Input: Reference Video z_ref + Subject Image z_sub"] --> Split["3D Spatiotemporal Attention Functional Decomposition<br/>Diagonal Aspat (Appearance) + Off-Diagonal Atemp (Motion)"]
Split --> S1["Stochastic Temporal Injection<br/>Inter-frame Atemp + Bernoulli Mask Mtemp Injection"]
Split --> S2["Target-aware Feature Guidance TFG<br/>S-TFG (Subject) and B-TFG (Background) Timestep Alternation"]
S1 --> Fusion["Feature Modulation & Convergence<br/>Active during early Ttemp / TTFG steps, fine rendering later"]
S2 --> Fusion
Fusion --> Out["Output: Personalized Video with Cloned Motion & Background"]
Key Designs¶
1. Functional Decomposition of 3D Spatiotemporal Attention: Isolating Appearance and Dynamics To address the severe entanglement of spatial and temporal features within DiT full attention, the authors empirically analyzed the visual-to-visual attention submatrix \(A_{vis-vis}\) (\(F \times F\) frame blocks). By computing the spatial displacement vector \(\Delta p\) of attended patches in subsequent frames from off-diagonal blocks \(A_{i, i+1}\) against RAFT dense optical flow, they observed that endpoint error (EPE) remains remarkably low in shallow-to-middle layers and stabilizes after the initial 30% of denoising steps. Concurrently, intra-frame diagonal blocks \(A_{i,i}\) exhibit minimal mean squared error (MSE) relative to SAM2-derived subject masks, confirming tight spatial localization. The unified attention is thus formally decomposed into a block-diagonal spatial appearance operator \(A_{spat}\) and an off-diagonal temporal dynamics operator \(A_{temp}\): $\([A_{spat}]_{i,i} = [A_{vis-vis}]_{i,i}, \quad [A_{temp}]_{i,j} = [A_{vis-vis}]_{i,j} \quad (i \neq j)\)$ This formulation establishes an orthogonal intervention interface without altering backbone weights or introducing auxiliary parameters.
2. Stochastic Temporal Injection: Balancing Dynamic Fidelity and Morphological Adaptability Directly transferring full inter-frame attention from the reference video rigidly binds the generation to the spatial footprint and contour of the original moving entity; if the new subject possesses a distinct size or posture (such as substituting a tiger with a wolf), dense motion injection causes severe boundary tearing and spatial artifacts. To resolve this, the temporal guidance representation \(h_{temp}^{ref}\) combines generated appearance with reference trajectories: $\([h_{temp}^{ref}]_i = [A_{spat}]_{i,i}^{gen} V_i^{gen} + \sum_{j \neq i} [A_{temp}]_{i,j}^{ref} V_j^{ref}\)$ To introduce adaptive geometric elasticity, an element-wise Bernoulli mask \(M_{temp}\) sampled with probability \(p\) is applied for sparse injection: $\(h_{temp}^{gen} \leftarrow M_{temp} \odot h_{temp}^{ref} + (1 - M_{temp}) \odot h_{temp}^{gen}\)$ Restricting this operation to the initial \(T_{temp} = 0.3T\) denoising steps faithfully clones the reference motion trajectory while granting the generative model sufficient leeway to mold realistic contours for the new subject.
3. Target-Aware Feature Guidance (TFG): Decoupled Appearance Control via Spatial Masks and Timestep Alternation To replace subject identity while preserving background context, spatial manipulation must extend across global attention dimensions to prevent inter-frame identity drifting. Global guidance representations \(h_{app}^{tgt}\) are calculated by pairing \(Q^{gen}\) with target keys and values \(K^{tgt}, V^{tgt}\). To prevent mutual contamination between foreground and background representations, SAM2-derived masks (\(M^{sub}\) for the subject and \(M^{bg}\) for the background) modulate the features via linear interpolation: $\(\hat{h}_{app}^{gen} = c^{sub} M^{sub} \odot h_{app}^{sub} + (1 - c^{sub} M^{sub}) \odot h_{app}^{gen}\)$ $\(\hat{h}_{app}^{gen} = c^{bg} M^{bg} \odot h_{app}^{bg} + (1 - c^{bg} M^{bg}) \odot h_{app}^{gen}\)$ Crucially, simultaneous application or block-wise alternation of S-TFG and B-TFG causes severe feature interference and background overwriting. SA-V2V introduces an alternating timestep schedule over the first \(T_{TFG} = 0.25T\) steps, executing S-TFG and B-TFG on alternate timesteps to harmonize subject fidelity and background preservation.
A Worked Example¶
Consider transferring a "wolf" image into a reference video of a "white tiger running across a snowfield": 1. Latent Initialization: A single random Gaussian noise tensor is replicated across \(F=49\) frames to populate \(z_{gen}\). 2. Early Phase (\(t \le 0.25T\)): Motion injection and appearance modulation operate concurrently. On even timesteps, S-TFG injects the wolf's fur and head textures into the foreground region using \(M^{sub}\). On odd timesteps, B-TFG injects the snow, trees, and sky features into the background using \(M^{bg}\). Simultaneously, stochastic temporal injection transfers running motion trajectories with probability \(p\) per element. 3. Transition Phase (\(0.25T < t \le 0.3T\)): TFG halts, locking in visual identity and background composition. Stochastic temporal injection continues to refine stride continuity and global dynamics. 4. Late Phase (\(t > 0.3T\)): All external feature modulations are disengaged, enabling the DiT to refine high-frequency spatial details and denoise toward clean video latents. 5. Decoding: The 3D VAE decodes the latent into a coherent video of a realistic wolf running across the snowfield with the exact gait of the original tiger.
Key Experimental Results¶
Main Results¶
The evaluation benchmark comprises 49 diverse reference videos from DAVIS, Pexels, and VBench (25 animal, 14 human, and 5 human-object interaction clips) paired with varied subjects, yielding 200 evaluation pairs. All models are evaluated on NVIDIA A100-80GB GPUs using CogVideoX-5B as the backbone. Evaluation metrics include Masked CLIP-I scores for subject adherence and background preservation, Fréchet Video Distance (FVD), and five fine-grained dimensions from VideoScore.
| Method | Masked CLIP Subject (↑) | Masked CLIP Background (↑) | FVD (↓) | VideoScore VQ (↑) | VideoScore TC (↑) | VideoScore Average (↑) |
|---|---|---|---|---|---|---|
| CtrlX | 0.7491 | 0.8503 | 67.8538 | 2.2641 | 2.0639 | 2.5854 |
| MotionClone | 0.8243 | 0.7604 | 85.8680 | 2.4075 | 1.9452 | 2.4230 |
| DualReal | 0.8116 | 0.7984 | 83.4813 | 2.6268 | 2.5410 | 2.7415 |
| JointTuner | 0.8079 | 0.8240 | 74.7159 | 2.5522 | 2.4721 | 2.7036 |
| SA-V2V (Ours) | 0.8140 | 0.8768 | 66.5532 | 2.6524 | 2.5491 | 2.7646 |
Ablation Study¶
Ablation experiments confirm the necessity of each decoupled component:
| Config | Subject CLIP-I (↑) | Background CLIP-I (↑) | FVD (↓) | Note |
|---|---|---|---|---|
| Full Model (SA-V2V) | 0.8312 | 0.8783 | 82.84 | Complete framework with temporal injection + B-TFG + S-TFG |
| w/o Temporal Injection (temp) | 0.8252 | 0.8146 | 103.82 | Motion guidance lost; FVD degrades sharply (+20.98) |
| w/o Background Guidance (B-TFG) | 0.8257 | 0.8313 | 97.27 | Background degradation and identity bleeding |
| w/o Subject Guidance (S-TFG) | 0.8101 | 0.9041 | 72.30 | Target subject not transferred; original background preserved artificially high |
Analysis of TFG configurations further demonstrates: - w/o Masking: Unconstrained attention modulation bleeds subject appearance into the background, generating blurred composite scenes. - Block-wise Alternation: Alternating guidance across DiT layers within the same timestep causes contradictory feature updates and unstable convergence. - Naive Injection: Directly replacing feature activations instead of masked linear interpolation overwrites the reference background entirely with subject artifacts.
Key Findings¶
- Stochastic temporal injection is essential for dynamic fidelity: Omitting temporal injection causes FVD to jump from 82.84 to 103.82, with qualitative results demonstrating a total failure to replicate reference gait and camera movement.
- Substantial efficiency gains over optimization-based baselines: Compared to DualReal (41.21 minutes training per video, 21.72 GB peak memory) and JointTuner (23.37 minutes training, 20.38 GB peak memory), SA-V2V requires 0 training time and reduces peak memory to 19.11 GB. Despite a minor 5-6 second increase in inference latency (125.38s vs 119.27s), overall wall-clock turnaround is orders of magnitude faster.
- Decisive preference in user studies: Across 29 participants, SA-V2V ranked first in background preservation (76.1%), motion fidelity (69.0%), video quality (72.1%), and text alignment (70.7%).
Highlights & Insights¶
- Discovery of emergent functional decomposition in 3D attention: The paper reveals that without architectural constraints or explicit decoupling, DiT unified attention matrices naturally factorize into diagonal appearance localization and off-diagonal dense motion tracking. This insight offers a foundational principle for training-free DiT manipulation.
- Stochastic masking resolves geometric over-constraint: Dense motion injection forces rigid spatial alignment with the source object; employing a simple Bernoulli sparse mask decouples motion transfer from strict geometric bounding, enabling natural adaptation to subjects of varying proportions.
- Timestep alternation evades multi-condition collision: Rather than forcing simultaneous multi-condition fusion within shared attention spaces, spreading guidance across alternating diffusion timesteps prevents feature interference while preserving global coherence.
Limitations & Future Work¶
- Single-subject operational constraint: Current TFG relies on binary subject-background partitioning and cannot isolate or independently customize multiple interacting entities in a reference video.
- Dependency on segmentation quality: Precise guidance hinges on accurate segmentation masks from SAM2; severe motion blur, low lighting, or complex occlusions in reference videos can degrade mask boundaries and introduce localized bleeding.
- Future Directions: Developing training-free multi-subject personalization via unsupervised attention clustering and extending spatiotemporal decomposition to long-horizon video generation.
Related Work & Insights¶
- vs MotionClone / CtrlX: MotionClone and CtrlX operate on UNet architectures (AnimateDiff) via decoupled spatial-temporal convolutions, which cannot transfer to full 3D DiT attention matrices; moreover, MotionClone's image adapter overfits to subject features, degrading background fidelity (0.7604 vs SA-V2V's 0.8768).
- vs JointTuner / DualReal: Existing DiT customization approaches rely on dedicated adapter weights and expensive test-time optimization (20-40 minutes per video); SA-V2V achieves superior motion replication and background consistency entirely training-free with zero additional parameters.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First framework to discover and exploit the intrinsic spatiotemporal decomposition of DiT full-attention for training-free video personalization.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarking across 200 pairs, optical flow and SAM2 alignment analyses, detailed component/scheduling ablations, and user studies.
- Writing Quality: ⭐⭐⭐⭐⭐ Cohesive theoretical narrative progressing logically from mathematical analysis to modular framework design.
- Value: ⭐⭐⭐⭐⭐ Eliminates per-video optimization overhead in DiT personalization, providing significant practical value for controllable generative media.