Controlling Motion Transfer in Diffusion Transformers via Attention Heads

Conference: ECCV 2026
Paper: Official page · PDF
Project: HALO
Area: Video Generation / Motion Transfer
Keywords: Diffusion Transformers, motion-specific attention heads, semantic correspondence, structural injection, training-free control

TL;DR

HALO identifies motion-specific and structure-specialized heads in video DiTs, supervises generation latents with semantically refined displacements, and preserves layout through low-entropy reference value features, improving main-benchmark motion fidelity MF from DiTFlow's 59.6 to 66.2 without updating model parameters while maintaining target-text alignment.

Background & Motivation

Motion transfer is not simply recoloring a reference video: the generated video must follow a new subject and scene description while preserving the reference trajectories and spatial organization. If a reference vehicle turns, changing its model and surroundings through the target prompt should still produce a turn, not merely a visually smooth clip of straight driving. This creates at least three potentially competing requirements: correct motion, correct subject semantics, and correct relative object positions. Optimizing only one can produce a video that looks dynamic without actually reproducing the reference.

Unified attention in video DiTs combines spatial and temporal relationships within the same architecture, unlike some U-Net video models with explicit temporal modules. GWTF controls motion through warped noise, RoPECraft manipulates rotary positional embeddings, and DiTFlow extracts displacements from cross-frame attention. These methods expose useful control mechanisms but do not fully establish which heads within a DiT carry motion and which are better suited to retaining structure. Aggregating every head can introduce irrelevant spatial associations; even plausible displacement directions can attach background motion to the foreground or change object positions and counts.

The authors therefore narrow the analysis to individual heads, validating motion heads through cross-frame correspondences and investigating structural heads through attention entropy instead of treating an entire attention layer as functionally homogeneous. Core idea: separately control which object the motion belongs to and which structure should be preserved, using motion heads for displacement, diffusion features for semantic correspondence, and low-entropy structural heads for layout constraints within the same generation process.

Method

Overall Architecture

The inputs are a reference video and a target prompt; the output should satisfy the new description while following the reference motion and layout. HALO uses a pretrained video DiT as its generator. The reference branch extracts cross-frame displacements and structural value features, while the target branch optimizes its latents using a displacement loss and receives reference features through selected heads during generation. Here, training-free means that model parameters remain frozen, not that inference takes a single forward pass or requires no gradients.

There are three main control mechanisms. Motion-head selection isolates appropriate motion signals from unified attention. Semantic refinement and latent optimization address the fact that similar textures need not belong to the same object. Low-entropy structural injection supplies spatial layout information that displacements alone cannot constrain. The first two primarily govern cross-frame relationships; the third supports within-frame organization during generation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    inputs["Reference video and target prompt"] --> motion["Motion-head selection"]
    motion --> semantic["Semantic refinement and<br/>latent optimization"]
    inputs --> features["Reference attention and value features"]
    semantic --> structure["Low-entropy structural injection"]
    features --> structure
    structure --> output["Target video aligned in<br/>motion and layout"]

The diagram shows dependencies between control signals, not a replacement for the per-timestep algorithm in the supplementary material. Motion optimization and structural injection jointly support generation; the available main paper does not specify enough detail to reconstruct the complete execution schedule.

Key Designs

1. Motion-head selection: isolate reliable cross-frame correspondences from unified attention

The authors first distinguish heads by similarity between their attention maps and pattern masks: temporal patterns exhibit cross-frame diagonal associations, while spatial patterns emphasize within-frame locality. They then test whether this classification actually predicts motion capability rather than assigning functions from visualizations alone. For each head, queries from one frame and keys from another produce cross-frame attention. Each source patch is matched to the strongest target patch, and subtracting their two-dimensional coordinates produces a head-specific displacement map. This records where each position points in another frame, making it a more direct representation of motion than attention intensity alone.

The resulting displacements are compared with a RAFT optical-flow reference. Directional Alignment uses cosine similarity between motion directions, while Correlation uses Pearson correlation to measure global pattern agreement. Temporal heads perform better on both, motivating HALO to aggregate only the selected motion heads for subsequent motion control. RAFT serves the analysis and validation here; this does not mean that HALO's core control signal is directly supplied by external optical flow. The figure does not provide reliably transcribable DA and Corr values in the cached text, so none are invented here.

Selection is not based on a general claim that fewer heads are always better. Its purpose is to prevent spatial heads from diluting dynamic correlations. The head-configuration experiment tests this directly: all heads yield MF 63.1, motion heads yield 66.2, and spatial heads yield only 53.6. This supports the selected heads' usefulness for this task, but does not prove that motion and structure are completely disentangled in every model or at every timestep.

2. Semantic refinement and latent optimization: identify the object correspondence before turning displacement into a differentiable constraint

The largest cross-frame attention value can point to a visually similar but semantically unrelated region, making a raw attention maximum an unreliable displacement endpoint. HALO builds semantic correspondences from cross-frame cosine similarities between diffusion features. On the reference side, Semantic Correspondence Refinement, or SCR, does not directly accept the semantic match. It first retains the top-k attention candidates, then selects the candidate whose two-dimensional position is closest to the best semantic match. Attention restricts the plausible motion candidates, and semantic information disambiguates among them to form the reference displacement map.

This is better understood as selecting candidates and then reranking them by proximity to a semantic match, not simply averaging two similarity maps. The following per-patch expression clarifies the indexing described in the paper; it is a notation-level reconstruction from the prose, not an additional algorithm:

\[ I^{\mathrm{ref}}_{f,f'}(i) =\underset{j\in\operatorname{TopK}(M_{f,f'}(i,:))}{\arg\min} \left\|g(j)-g\!\left(I^{\mathrm{cor}}_{f,f'}(i)\right)\right\|_2, \qquad D^{\mathrm{ref}}_{f,f'}(i)=g\!\left(I^{\mathrm{ref}}_{f,f'}(i)\right)-g(i). \]

Here, \(i\) is a source patch, \(j\) is a target patch, \(g\) converts a flattened index into two-dimensional coordinates, and \(I^{\mathrm{cor}}\) is the location with the highest diffusion-feature cosine similarity. The main paper refers the specific top-k value to the supplement, which is absent from the cache, so no value is assigned here.

On the target side, hard argmax matching cannot support the entire control path if the latent is to be optimized through the correspondence estimate. HALO instead applies Semantic Reweighting, or SRW: it adds a bias to attention logits at the best semantic match, applies softmax, and computes the expected coordinate difference. The bias is added before softmax; it neither directly edits normalized probabilities nor forces every nonmatching location to zero.

\[ \widetilde M_{f,f'}(i,:) =\operatorname{softmax}\!\left(E_{f,f'}(i,:) +\beta\,\mathbf{1}[\,\cdot=I^{\mathrm{cor}}_{f,f'}(i)]\right), \qquad D^{\mathrm{tgt}}_{f,f'}(i) =\sum_j\widetilde M_{f,f'}(i,j)\,[g(j)-g(i)]. \]

\(E\) denotes cross-frame attention logits, and \(\beta\) controls the bias strength. The original summation indices are damaged in the cached extraction. Following the prose definition of a patch-level displacement map and coordinate expectation, the expression above fixes the source patch and sums over target patches, avoiding an incorrect collapse of the entire map into a single global displacement. The continuous target displacement can then be compared with the reference through an L2 distance to optimize the latent while generator weights remain frozen. SCR improves the supervision target, whereas SRW improves the target branch's displacement estimate; they are not two names for the same operation.

3. Low-entropy structural injection: transfer reference layout while limiting indiscriminate appearance copying

Even if patch displacements follow the correct directions, the target video need not start from the correct positions, shapes, or object count. In the paper's ablation, removing structural injection produces structural distortions and duplicated objects. Motion constraints therefore cannot determine the entire spatial organization on their own, and HALO additionally extracts reference features suited to retaining layout.

The authors select structural heads through the entropy of their visual-token attention distributions. For a normalized distribution, the basic entropy definition is \(H=-\sum_k a_k\log a_k\); lower entropy means more concentrated attention. The paper observes that low-entropy heads with clear diagonal patterns also yield PCA features with lower spatial entropy and clearer reference structure. For example, attention entropy / feature entropy in Figure 4 is 3.34 / 0.44 for L20 H8 and 7.89 / 0.77 for L20 H6. The evidence concerns the association between patterns and entropy validated in this model, not a universal theorem that every low-entropy distribution necessarily represents structure.

During generation, HALO injects reference value features only into the corresponding selected low-entropy heads rather than copying all heads. It uses heads with entropy below 7, with that threshold motivated by the observed median. Selective injection aims to retain layout while reducing the noise and reference-identity leakage associated with indiscriminate injection; it does not guarantee that leakage disappears entirely. Exact head-level entropy aggregation and the layers and timesteps used for injection still require the supplement or implementation for faithful reproduction.

Loss & Training

The central added objective is the semantic motion loss, defined as the L2 distance between reference and target displacement maps. It updates generation latents rather than training a motion adapter or fine-tuning the DiT. Reference-side candidate selection supplies the fixed motion target, while the target-side soft displacement makes that target usable for gradient optimization.

The main-paper configuration uses CogVideoX, 50 denoising steps, 12 optimization steps \(T_{\mathrm{opt}}\), guidance scale 7, and SRW bias \(\beta=0.1\); structural heads are selected when attention entropy is below 7. The statement "12 optimization steps" should not automatically be interpreted as 12 updates at every denoising step: the detailed algorithm is deferred to supplementary Section C.2.

The cache does not include that supplement, so optimizer choice, learning rate, top-k, the exact injection schedule, memory consumption, and runtime are not filled in. The paper also points to additional Wan experiments without reporting their detailed results in the main text; this does not establish that gains on Wan equal those on CogVideoX.

Key Experimental Results

Main Results

The following reproduces the relevant data from Table 1. CLIP measures text-video alignment, TC measures temporal consistency, and MF measures reference-motion fidelity; higher is better for all three. FTD is another reference-motion alignment metric used by the paper, with lower values preferred. The main text does not elaborate its exact computation or the full TC implementation, so this note retains the metric names without inventing missing definitions.

Method	CLIP ↑	TC ↑	MF ↑	FTD ↓
MoFT	30.9	85.8	34.8	23.0
ConMO	29.8	85.3	52.0	17.4
MotionClone	30.4	78.6	55.6	19.8
DiTFlow	31.0	89.5	59.6	23.0
RoPECraft	30.3	85.9	58.2	19.6
GWTF	31.6	88.2	62.5	21.6
HALO	31.7	87.5	66.2	19.4

Relative to GWTF, HALO increases MF by 3.7 points and CLIP by 0.1, and reduces FTD by 2.2, while TC decreases by 0.7. It has the highest CLIP and MF in this table, but not the best FTD overall: ConMO's 17.4 is lower than HALO's 19.4. Since the comparison spans U-Net and DiT methods, it should be read as an evaluation of complete systems, not a single-module improvement with an identical backbone throughout.

On the Movie Scene Dataset in Table 2, HALO obtains CLIP / TC / MF of 30.5 / 88.9 / 52.6, compared with GWTF's 30.2 / 87.6 / 46.6 and DiTFlow's 29.7 / 90.4 / 48.4. HALO improves MF by 6.0 and 4.2 points respectively, but its TC remains below DiTFlow's. Methods requiring video masks are excluded from this table, so it is not a cinematic comparison against every baseline from the main benchmark.

Ablation Study

The following corresponds to Table 3, with displacement optimization as the baseline. Semantic guidance includes both semantic correspondence refinement and semantic reweighting; structural injection transfers features through the selected structural heads.

Configuration	Semantic guidance	Structural injection	CLIP ↑	TC ↑	MF ↑	FTD ↓
Exp. 1: displacement baseline	No	No	31.0	89.5	59.6	23.0
Exp. 2: structural injection only	No	Yes	30.6	88.3	61.8	20.5
Exp. 3: semantic guidance only	Yes	No	30.9	88.5	61.3	20.9
Exp. 4: HALO	Yes	Yes	31.7	87.5	66.2	19.4

Adding the two components individually improves MF by 2.2 and 1.7, whereas combining them improves it by 6.6 while recovering and improving CLIP. Relative to the complete method, removing structural injection lowers MF from 66.2 to 61.3, and removing semantic guidance lowers it to 61.8. This supports complementarity, but Table 3 groups SCR and SRW into one factor and therefore cannot separately quantify their contributions.

Table 6 tests which heads should be selected; the clearly recoverable values are retained below. The first group changes the heads used for motion displacement, while the second changes the entropy range used for structural injection. These are different factors, not a continuous sweep of one variable.

Analysis factor	Configuration	MF ↑	FTD ↓
Motion-head configuration	All heads	63.1	21.6
Motion-head configuration	Spatial heads	53.6	22.7
Motion-head configuration	Motion heads	66.2	19.4
Structural injection entropy range	Above 7	54.1	23.5
Structural injection entropy range	Below 7	66.2	19.4

Key Findings

• The MF advantage of motion-head selection is quantitative, not just a visual impression: it gains 3.1 points over all heads and 12.6 over spatial heads. Low-entropy structural heads outperform high-entropy heads by 12.1 MF points, with FTD also decreasing from 23.5 to 19.4.
• TC should not be interpreted in isolation as proof of successful motion transfer. The authors note that it tends to favor more static outputs. HALO's higher MF and slightly lower TC are compatible with that explanation, but the main paper does not supply enough evidence to attribute the entire TC difference to static-output bias.
• The user study includes 20 participants using a 5-point scale normalized to 0–100. HALO scores 86.5 / 87.8 / 93.3 for editing accuracy / temporal consistency / motion accuracy, versus DiTFlow's 84.1 / 83.1 / 69.0. These are normalized ratings, not win rates or answer-accuracy percentages.

Highlights & Insights

• The analysis determines where to intervene. Motion-head displacement validation and structural-head entropy analysis both enter the control algorithm and are subsequently tested through head-configuration ablations, rather than remaining auxiliary attention visualizations.
• Semantics constrain motion matching rather than replacing it. SCR selects a motion candidate near the semantic correspondence instead of allowing semantic features alone to determine trajectories; SRW retains this preference in a differentiable soft-displacement estimate.
• Layout and motion need different control signals. Displacement supervision governs how positions change across frames, while structural injection helps preserve the spatial organization where those changes occur. Their joint ablation gains suggest that improving trajectories can also require improving the object structure to which those trajectories attach.

Limitations & Future Work

• Author-proposed direction: extend head-level control to explicit editing of motion direction and intensity. The current main task follows a reference and should not be mistaken for an already implemented interface with arbitrary direction or speed controls.
• Reproducibility boundary: the cache contains only the main paper, not supplementary top-k settings, optimization scheduling, runtime, peak memory, or Movie Scene Dataset details. The authors' description of marginal overhead cannot be converted into unverified seconds or speedup factors.
• Head specialization remains an empirical finding. Low entropy correlates with structural retention, and motion heads outperform spatial heads, but this does not establish complete functional independence or direct portability of the threshold 7 across backbones. Stability across layers, timesteps, and models warrants further study.
• Semantic matching can itself introduce errors. As a mechanism-based inference, occlusions, repeated textures, or erroneous diffusion-feature correspondences can affect SCR and SRW. The main paper shows successful complex-motion examples but no systematic semantic-correspondence failure rate, so these cases cannot be considered solved.
• Evaluation uncertainty remains. The main table has no confidence intervals, the user-study description lacks full sampling and variance details, and the cinematic evaluation filters some baselines. A CLIP gain of 0.1 should not be described as established statistical significance; structural retention is also supported mainly by visualizations and related metrics.

Related Work & Insights

• vs DiTFlow: both derive displacements from cross-frame attention and optimize latents. HALO adds motion-head selection, SCR / SRW for semantic correspondence, and a separate structural feature injection mechanism; it is not a new video generator trained from scratch.
• vs GWTF / RoPECraft: the former warps noise, while the latter controls positional embeddings using trajectory guidance. HALO instead intervenes in internal head-level representations, enabling interpretable component analysis but retaining the cost of latent optimization.
• vs Sparse VideoGen and diffusion-feature correspondence research: the former motivates temporal / spatial head-pattern distinctions, while the latter demonstrates the availability of semantic correspondences in diffusion features. HALO combines these findings for motion transfer rather than treating attention sparsification as the objective.

The official page, PDF, and project page are linked at the top. The available material does not establish a separate code-repository URL, so the project page is not presented as a verified code link.

Rating

• Novelty: 4/5. Translating head-level functional analysis into separate motion and structure controls is a clear contribution, although it builds on existing displacement optimization and feature injection.
• Experimental Thoroughness: 4/5. The main benchmark, cinematic scenes, component ablations, head configurations, and user study complement one another; statistical reporting and essential reproduction details are incomplete in the available main text.
• Writing Quality: 4/5. Problems map clearly to components, but some metrics and implementation details require the supplement, and damaged equations in the cache complicate verification.
• Value: 4/5. The method offers interpretable and reusable interventions for video motion control without parameter updates; deployment value still depends on complete efficiency and cross-backbone evidence.