RefAlign: Representation Alignment for Reference-to-Video Generation¶
Conference: ECCV 2026
arXiv: 2603.25743
Code: https://github.com/gudaochangsheng/RefAlign
Area: Video Generation / Diffusion Models
Keywords: Reference-to-Video Generation, Representation Alignment, Diffusion Transformer, Visual Foundation Model, Multimodal Conditions
TL;DR¶
RefAlign proposes a Reference Alignment loss (RA loss) to explicitly align the intermediate features of the DiT reference branch with the semantic space of a Visual Foundation Model (VFM) during training. Specifically, the positive term pulls features of the same subject closer to ensure identity consistency, while the negative term pushes features of different subjects apart to enhance semantic discriminability. The alignment module is removed during inference to achieve zero extra overhead. It achieves SOTA performance with a TotalScore of 60.42% on OpenS2V-Eval, effectively alleviating copy-paste artifacts and multi-subject confusion in reference-to-video generation.
Background & Motivation¶
Reference-to-Video generation (R2V) is an important paradigm for controllable video synthesis: given text prompts and multiple reference images, it generates videos that follow instructions while preserving the identity and appearance of the subjects, with wide applications in personalized advertising, virtual try-on, and other scenarios. Existing R2V methods commonly adopt a "dual-stream reference" paradigm—on one hand, extracting reference features with a 3D VAE to provide low-level details, and on the other hand, introducing an additional encoder (such as CLIP or MLLM) to inject high-level semantic cues, attempting to mitigate pixel-level leakage in the VAE latent space. However, these additional semantic features and the VAE latent features originate from heterogeneous encoders, leading to an inherent modality mismatch: a systematic bias exists between the internal reference features derived from VAE latent representations and the externally injected semantic reference representations within the DiT, which implicit alignment struggles to fundamentally eliminate. This leads to two typical problems: copy-paste artifacts (the generated video excessively copies pixel details from the reference image) and multi-subject confusion (the appearances of different reference subjects interfere and mix with each other).
Through t-SNE visualization, the authors discovered a crucial phenomenon: the feature distributions encoded by the DiT for reference images are highly entangled, with severe overlap between different references, whereas the features extracted by DINOv3 possess significantly stronger inter-class separability—features of the same reference are compact and consistent, while those of different references are well-separated. Inspired by this, RefAlign avoids injecting extra semantic features and instead directly constrains the intermediate features of the DiT reference branch to align with the VFM feature space via an explicit Reference Alignment loss, fundamentally enhancing the semantic discriminability of reference representations. Core Idea: By using VFM features as "semantic anchors", the model pulls the DiT reference features closer to the VFM features of the same subject and pushes them away from those of different subjects during training, enabling the DiT to learn to generate highly discriminative reference representations on its own without increasing inference cost.
Method¶
Overall Architecture¶
RefAlign uses the T2V DiT of Wan2.1 as the backbone and performs fine-tuning. Given a text prompt \(c_{\text{text}}\), \(M\) reference images \(I = \{I_m\}_{m=1}^{M}\), and a target video \(x\), the text is encoded into \(\hat{c}_{\text{text}}\) by a frozen T5 encoder, while reference images and the target video are encoded into \(\hat{I}\) and \(z_0\) by a frozen Wan-VAE. The DiT (consisting of \(L\) transformer blocks) receives the noisy latent \(z_t\), text conditions, reference conditions, and timestep \(t\), and outputs the velocity prediction \(\varepsilon_{\Theta}(z_t, \hat{c}_{\text{text}}, \hat{I}, t)\) trained under a Rectified Flow objective.
The core modification of RefAlign occurs during the training stage: for the reference image token features \(h^{(l)}\) (\(l \le K\)) generated in the self-attention of the first \(K\) layers of the DiT, a lightweight MLP projector \(\Psi_{\text{proj}}\) is used to project them into the same dimension as the VFM features, obtaining \(\hat{h}^{(l)} = \Psi_{\text{proj}}(h^{(l)})\). These projected features then calculate the Reference Alignment loss (RA loss) with the reference image features \(f = \{\varepsilon_{\text{VFM}}(I_m)\}_{m=1}^{M}\) extracted by a frozen VFM. During inference, the VFM encoder and MLP projector are completely discarded, and the model samples using standard classifier-free guidance, introducing zero computational overhead.
Key Designs¶
1. Positive Term of RA Loss: Pulling Same-Subject Features Closer to Enhance Identity Consistency
Since the DiT reference branch features originate from the VAE latent space where pixel-level information dominates, the features of different reference images are highly entangled in space (showing severe overlap in t-SNE plots). This makes it difficult for the model to distinguish "whether this token belongs to subject A or subject B". The positive alignment term of RefAlign directly addresses this pain point: for each subject \(m\), its DiT reference token \(\hat{h}^{(l),m}\) and the corresponding VFM feature \(f^m\) are aligned by minimizing their patch-wise cosine distance:
The role of the positive term is to teach the DiT that the reference tokens of the same subject, regardless of variations in lighting, pose, and background, should stay close to the subject's "semantic anchor" in the VFM space. This is fundamentally different from the alignment in REPA, which aligns target representations recovered from noise to accelerate convergence, whereas RefAlign aligns clean reference condition representations to enhance the semantic quality of the conditions. The positive term degrades to the sole alignment signal when there is only one reference image (\(M=1\)), in which case \(\mathcal{L}_{\text{neg}} = 0\).
2. Negative Term of RA Loss: Pushing Different-Subject Features Apart to Eliminate Multi-Subject Confusion
Relying solely on positive alignment poses a hidden risk: if the VFM features of different subjects already exhibit some similarity (e.g., two different breeds of dogs), the positive term might pull all reference features toward a blurry average region, thereby exacerbating multi-subject confusion. To address this, RefAlign introduces a negative alignment term with a margin \(\delta\): for the DiT reference token of subject \(m\) and the VFM features of a different subject \(m' \neq m\), they are forced to have a cosine distance greater than the margin, otherwise a penalty is applied:
where \([x]_{+} = \max(x, 0)\). Ablation studies show that removing the negative term drops the TotalScore from 55.73% to 51.75%, with FaceSim and NaturalScore decreasing significantly, and the gender attributes of multiple subjects mix up noticeably in qualitative results—demonstrating that the negative term is indispensable for maintaining semantic boundaries between subjects.
3. Training-only Alignment, Zero-Overhead Inference Bypass Design
A straightforward idea is to feed VFM features directly to the DiT as an extra input (such as Configuration D: VAE + DINOv3 dual-encoder input), which indeed brings improvement over pure VAE input (TotalScore 49.93% \(\rightarrow\) 52.15%), but falls far short of using the RA loss (55.73%), and requires running the VFM encoder during inference. The key insight of RefAlign is: the value of VFM features lies not in "providing an extra channel of information", but in serving as "targets for representation learning"—distilling the semantic discriminability of the VFM into the parameters of the DiT reference branch through the RA loss, allowing the DiT to produce highly discriminative reference representations on its own during inference. This design makes RefAlign mathematically equivalent to standard Wan2.1 during inference, with zero additional parameters and zero additional computation. Moreover, since the VFM only serves as an alignment target during training, the choice of its encoder is no longer a constraint at inference time—experiments show that performance fluctuations across three scales (DINOv3-B/L/H+) are only 0.33%-0.43%, indicating robustness to the scale of the encoder.
Loss & Training¶
The complete training objective is a weighted combination of the Rectified Flow loss and the Reference Alignment loss:
where \(\mathcal{L}_{\text{RF}} = \mathbb{E}_{z_0, \epsilon, t}\left[\left\|\varepsilon_{\Theta}(z_t, \hat{c}_{\text{text}}, \hat{I}, t) - (\epsilon - z_0)\right\|^2_2\right]\), and \(\eta\) and \(\lambda\) control the overall weight of the RA loss and the relative weight of the negative term, respectively (both set to 1.0 by default). The alignment depth \(K\) is set to 9 layers based on experiments—TotalScore shows a "climb then fall" trend with depth; too shallow leads to insufficient alignment signals, while too deep monotonically decreases FaceSim (excessively suppressing identity consistency).
Training is split into two stages: the first stage uses 200K regular paired samples (where reference images match the target video subject) from OpenS2V to learn reference condition modeling; the second stage uses 160K cross-paired samples (where reference images do not match the target video subject) from Phantom-Data to alleviate copy-paste artifacts. The ratio of regular to cross-paired samples is 6:4. During training, random data augmentations—including rotation, scaling, horizontal flipping, affine transformation (with shearing), Gaussian blur, and color jittering—are applied to the reference images of regular pairs. The optimizer is AdamW (\(\beta_1=0.9\), \(\beta_2=0.999\), weight decay=0.01), with a learning rate of 5e-5, a global batch size of 128, and a total of 3000 iterations. CFG training randomly drops the text condition, reference condition, or both, with a 10% probability each.
During inference, a 50-step Euler sampler is used with dual-scale CFG: reference image guidance scale \(\mu_1=5.0\), and text guidance scale \(\mu_2=7.5\). At this stage, the VFM encoder and MLP projector are completely removed, aligning the inference pipeline exactly with standard Wan2.1.
Key Experimental Results¶
Main Results¶
Zero-shot evaluation results on the OpenS2V-Eval benchmark (Table 1). Evaluation metrics include Aesthetics (visual quality), MotionSmoothness (motion continuity), MotionAmplitude (motion range), FaceSim (facial fidelity), NexusScore (subject consistency), NaturalScore (naturalness), and GmeScore (video-text alignment).
| Method | TotalScore↑ | Aesthetics↑ | MotionSmoothness↑ | FaceSim↑ | NexusScore↑ | NaturalScore↑ |
|---|---|---|---|---|---|---|
| Kling1.6 (closed-source) | 56.23% | 44.59% | 86.93% | 40.10% | 45.89% | 74.59% |
| Saber-14B (closed-source) | 57.91% | 42.42% | 96.12% | 49.89% | 47.22% | 72.55% |
| VINO (open-source) | 57.85% | 45.92% | 94.73% | 52.00% | 42.67% | 71.99% |
| BindWeave (open-source) | 57.61% | 45.55% | 95.90% | 53.71% | 46.84% | 66.85% |
| Phantom-14B (open-source) | 56.77% | 46.39% | 96.31% | 51.46% | 37.43% | 69.35% |
| RefAlign-1.3B | 56.30% | 42.96% | 94.74% | 53.06% | 43.97% | 66.25% |
| RefAlign-14B | 60.42% | 46.84% | 97.61% | 55.23% | 48.52% | 73.63% |
RefAlign-14B achieves 60.42% in TotalScore, representing the first open-source or closed-source method to break the 60% threshold in public results. It achieves the best results in both subject-related metrics, FaceSim and NexusScore, verifying the improvement of explicit reference alignment on identity consistency and subject discriminability. RefAlign-1.3B also achieves SOTA TotalScore (56.30% ) among models of the same scale, showing the general effectiveness of the method across different model scales.
Ablation Study¶
Ablation study on the design of the RA loss (unified setup at 1800 iterations, Table 2). The Baseline (Configuration C) only encodes reference images with the VAE and feeds them to the DiT without using the RA loss.
| Config | TotalScore↑ | FaceSim↑ | NexusScore↑ | NaturalScore↑ | Description |
|---|---|---|---|---|---|
| A: Full RA loss | 55.73% | 53.15% | 46.23% | 62.96% | Positive + negative terms, default config |
| B: w/o \(\mathcal{L}_{\text{neg}}\) | 51.75% | 48.67% | 46.61% | 53.75% | Removing negative term, NaturalScore plummets |
| C: w/o \(\mathcal{L}_{\text{RA}}\) | 49.93% | 68.45% | 38.63% | 40.46% | No alignment, artificially high FaceSim (copy-paste) |
| D: VAE + DINOv3 dual input | 52.15% | 35.11% | 45.78% | 66.06% | Extra features as input, inferior to explicit alignment |
Key discovery: FaceSim in Configuration C (68.45%) is surprisingly much higher than in Configuration A (53.15%). This indicates that without the RA loss, the model tends to directly replicate the pixels of the reference image (copy-paste), resulting in a high facial similarity but a failure in instruction following. The RA loss, by constraining the reference representation to align with the VFM semantic space, forces the model to learn to "understand what the subject is" rather than "remember what the subject looks like", thereby moderately decreasing FaceSim while substantially improving NexusScore and NaturalScore. Removing the negative term in Configuration B drops NaturalScore from 62.96% to 53.75%, showing that negative alignment is critical for maintaining naturalness in multi-subject scenarios.
Key Findings¶
- RA loss is the core source of performance improvement: Removing the RA loss plummets the TotalScore from 55.73% to 49.93%, drops NexusScore by 7.6 percentage points, and decreases NaturalScore by 22.5 percentage points—a margin far exceeding any other single ablation.
- Negative alignment is indispensable: Keeping only the positive term (Configuration B) decreases the TotalScore by 3.98% and causes clear gender attribute confusion in multi-subject scenarios, demonstrating that simply pulling same-subject features closer is insufficient to guarantee semantic boundaries between subjects.
- An optimal range exists for alignment depth: TotalScore peaks when the alignment layer depth \(K=9\). Going deeper (\(K \ge 11\)) causes FaceSim to monotonically decrease below 25%, as deep-layer features are highly semanticized and forced alignment impairs identity fidelity.
- VFM encoders are scale-insensitive but type-sensitive: The TotalScore variation among DINOv3-B/L/H+ is only 0.33%-0.43%, but cross-type differences are significant—DINOv3 performs best on consistency metrics (FaceSim, NexusScore), SigLIP2 is superior on quality metrics (Aesthetics, MotionSmoothness), and Qwen2.5-VL excels in motion amplitude, indicating that different VFMs prioritize different alignment signal dimensions.
Highlights & Insights¶
- The "alignment-as-distillation" bypass design is highly elegant: Instead of using extra inputs or performing encoding at inference time, it distills the semantic discriminability of the VFM into the parameters of the DiT reference branch using a contrastive-learning-style loss. During inference, the model behaves as a standard Wan2.1, but the quality of its internal reference representations changes qualitatively. This design philosophy can be transferred to any scenario requiring "enhancement of condition encoder quality" (e.g., condition branches in ControlNet, image prompt encoders in IP-Adapter).
- Honest interpretation of the trade-off between FaceSim and copy-paste: In the ablation study, FaceSim is highest (68.45%) when the RA loss is absent. The authors do not shy away from this "counter-intuitive" result, but instead explain it as high facial similarity caused by copy-paste artifacts. This reminds readers that FaceSim cannot be evaluated in isolation in R2V evaluation and must be interpreted alongside instruction-following metrics (such as NaturalScore).
- First successful application of positive-negative contrastive learning paradigm to the conditioning branch of diffusion models: Introducing the pull-push mechanism from contrastive learning to the conditional representation learning of the DiT, rather than the backbone denoising representation learning, paves a new path for "optimizing conditioning quality". This complements REPA—REPA optimizes target representations, while RefAlign optimizes conditional representations.
- The two-stage training strategy (regular pairs \(\rightarrow\) cross pairs) is a practical trick to alleviate copy-paste: First learning "what the reference condition is" with regular pairs, and then learning "not to blindly copy the reference image" with cross pairs, this curriculum learning strategy is simple yet effective, and can be reused in other conditional generation tasks.
Limitations & Future Work¶
- Insufficient training data diversity: Currently, only 360K samples are used, which is far smaller than the training data size of mainstream T2V models, resulting in an suboptimal balance between instruction following and reference fidelity. Expanding the data scale and mixing multi-source data are straightforward directions for improvement.
- Video length limited by the underlying foundation model: Constrained by the Wan2.1 backbone, RefAlign currently only supports generating 81-frame videos and cannot generate long videos. This is a capability boundary of the base model, which requires waiting for stronger backbones or designing frame interpolation/extrapolation strategies.
- Alignment signal from a single VFM may be incomplete: Experiments have shown that different VFMs focus on different metric dimensions (DINOv3 on consistency, SigLIP2 on quality, Qwen2.5-VL on motion). A single VFM cannot cater to all dimensions. Multi-VFM joint alignment—such as using DINOv3 to anchor identity, SigLIP2 to anchor quality, and Qwen2.5-VL to anchor semantic combinations—is a natural extension.
- Hyperparameters of RA loss (\(\eta\), \(\lambda\), \(\delta\), \(K\)) rely on empirical settings: In the paper, the values \(\eta=\lambda=1.0\) and \(K=9\) come from ablations, but training on different backbones or data distributions might require re-tuning. Designing an adaptive weight scheduling strategy (e.g., focusing on alignment in early training and generation quality in later training) could further improve stability.
Related Work & Insights¶
- vs Phantom: Phantom injects CLIP features into the DiT as an extra semantic branch, which is an implicit alignment; RefAlign does not inject extra features and instead explicitly constrains the reference representations themselves through the RA loss. The former's CLIP features still need to be encoded at inference time (increasing overhead), while the latter discards the VFM once training is complete. The essential difference is "adding information" versus "modifying representation".
- vs REPA: REPA aligns the target representations recovered from noise by the DiT (noisy target \(\rightarrow\) clean VFM) to accelerate convergence; RefAlign aligns the DiT's conditional representations of clean reference images (clean reference \(\rightarrow\) clean VFM) to enhance conditioning quality. REPA's positive-only alignment when \(M>1\) may cause different reference representations to collapse into an average region, a problem that RefAlign's negative mechanism is explicitly designed to address. The two are complementary rather than mutually exclusive in R2V scenarios.
- vs BindWeave / VINO: These methods introduce MLLMs (such as Qwen2.5-VL-7B) for cross-modal reasoning, enabling modeling of complex spatial relationships and temporal semantics, but they require running an extra 7B-class model during inference, imposing huge computational overhead. RefAlign proves that a well-designed alignment loss can achieve or even exceed the performance of MLLM-based solutions without adding inference overhead, providing a new methodology for "lightweight conditioning enhancement".
- Broader Implications: Utilizing a VFM as a "representation teacher" and distilling its knowledge into a submodule of a generative model (e.g., condition encoder, discriminator, prior network) via contrastive-style losses is a paradigm not limited to R2V—it can be generalized to image editing (aligning representations of edited regions), style transfer (aligning the semantic space of style representations), or even autoregressive generation (aligning intermediate representations of token sequences).
Rating¶
- Novelty: ⭐⭐⭐⭐☆ This is the first work to introduce explicit feature alignment into conditional representation learning for R2V. The application of the positive-negative contrastive learning paradigm within diffusion conditioning branches is novel, but contrastive learning and REPA are both established technologies, making the core idea more combinatoric than fundamentally original.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ The main experiments cover 12 baselines across both closed-source and open-source models under two model scales. Ablations cover loss designs (4 configurations), alignment depths (8 depth values), and encoder scales/types (6 encoders), complemented by user studies—making it highly systematic.
- Writing Quality: ⭐⭐⭐⭐☆ The motivation is clear (directly backed by intuitive t-SNE visualizations), the methods are thoroughly described, and comparisons with REPA are clearly analyzed (4 differences explained point-by-point). However, some mathematical notations are deeply nested, and there is some repetition between the appendix and the main text.
- Value: ⭐⭐⭐⭐☆ The "alignment-as-distillation, zero-inference-overhead" design offers a highly practical solution for real-world R2V deployment, as the method is simple and reproducible. However, the 81-frame limitation and the 360K data scale constrain its direct practical utility, meaning it primarily serves to open up a new research direction in conditional representation alignment.