DTI: Dynamic Trajectory Initialization for Generative Face Video Super-Resolution

Conference: ECCV2026
arXiv: 2606.29198
Code: https://github.com/MediaX-SJTU/DTI
Area: Image Generation / Face Video Super-Resolution
Keywords: Face Video Super-Resolution, Diffusion Models, Dynamic Trajectory Initialization, Discriminative Guidance, Fidelity-Perceptual Quality Trade-off

TL;DR

DTI proposes a "Dynamic Trajectory Initialization" paradigm, redefining generative face video super-resolution from "full generation" to "input-driven targeted restoration." By coupling a Discriminative Guider (DG) with SNR alignment theory, it dynamically determines the starting point of diffusion sampling for each low-quality input. With only minor fine-tuning, this approach comprehensively outperforms existing methods in fidelity, perceptual quality, and inference efficiency.

Background & Motivation

Face video super-resolution (FVSR) is a classic ill-posed inverse problem, which aims to restore ground-truth-like spatiotemporal information from unknown, complex degradations. Recently, diffusion-model-based generative methods (GFVSR) have achieved state-of-the-art perceptual quality. Leveraging the generative priors of pretrained large models, they reconstruct high-frequency texture details that traditional discriminative methods fail to recover. However, these methods inherit the inherent flaws of diffusion models: computationally heavy multi-step sampling processes. More critically, in pursuit of high no-reference perceptual scores, these models tend to hallucinate details that look realistic but deviate from the ground truth, resulting in repetitive artifacts or unnatural distortions—referred to as the "distribution generation trap."

In-depth analysis reveals an overlooked Key Challenge. Existing GFVSR methods treat the task as a full generation process starting from pure noise. In reality, however, the low-quality (LQ) inputs preserve the vast majority of low-frequency information (global structure, skin tones, regional colors, etc.), with information loss occurring primarily in high-frequency textured regions. Consequently, starting the diffusion process from pure noise is highly inefficient, as substantial sampling steps are wasted on reconstructing low-frequency components that are already present. A more fundamental issue is that while LQ inputs contain rich usable information, they follow complex distributions of real-world degradation rather than the parameterized Gaussian distribution learned by diffusion models. Hence, they cannot be directly utilized as intermediate diffusion states. This necessitates locating a reasonable starting point on the diffusion trajectory for each LQ input, allowing the downstream sampling to leverage the existing low-frequency information of the LQ while exploiting the generative power of the diffusion model for the missing high-frequency details.

The Key Insight of this work is that since the LQ input already contains most of the low-frequency structural info, there is no need for a full generation process starting from pure noise. Instead, the generation should be formulated as a "targeted restoration" where generative sampling is only applied to the actually lost high-frequency parts, while the existing low-frequency parts are directly preserved. Core Idea: Redefine GFVSR from unconditional-style conditional generation to "input-driven targeted restoration." The core Mechanism consists of two designs: extracting enhanced visual features from the LQ (via DINOv3) with efficient attention injection to precisely constrain the conditioning process, and utilizing a Discriminative Guider (DG) to dynamically compute a reasonable starting time step on the diffusion trajectory for each LQ input based on SNR alignment theory. This simultaneously yields improvements across three dimensions: fidelity, perceptual quality, and inference efficiency.

Method

Overall Architecture

The core mechanism of DTI is to leverage the low-frequency information preserved in the LQ input to dynamically determine the starting point of diffusion sampling via a Discriminative Guider, thereby transforming full generation into targeted restoration. The overall framework consists of three synergistic components: an enhanced conditional injection mechanism, a Discriminative Guider (DG), and a lightly adapted DiT backbone.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input LQ Frame"] --> B["VAE Encoder"]
    A --> C["DINOv3<br/>Visual Feature Extraction"]

    B --> D["LQ latent<br/>Condition C1"]
    B --> E["Discriminative Guider DG"]

    C --> F["Visual Feature<br/>Condition C2"]
    E --> G["Predict M_D, M_R<br/>→ t_pred → z_start"]

    D --> H["Concatenate Input Sequence<br/>Conditions (C1, C2) + Noise X"]
    F --> H
    G --> H

    H --> I["Adapted DiT<br/>Bidirectional (Within Conditions)<br/>Unidirectional (X→Conditions)"]
    I --> J["VAE Decoder"]
    J --> K["Output HQ Frame"]

Key Designs

1. Enhanced Visual Conditioning Injection: Extracting "Purified" Visual Features from LQ via DINOv3

Traditional GFVSR methods inject LQ conditions using either ControlNet (which requires massive auxiliary components and training), concat-flatten-MLP (yielding limited quality due to channel-level information exchange only), or by directly blending the LQ and noise linearly (which aligns poorly with diffusion models). DTI departs from a more fundamental observation: as the sole source of input, the LQ frame is actually a degraded signal where rich, useful visual structures are buried under blur and noise, making direct conditioning insufficient. Human faces feature clear main structures, distinct edges, and regional partitions, which are highly suited for visual feature extractors to enhance conditioning representation.

DTI introduces the self-supervised vision model DINOv3 as a feature extractor. Frames are sampled at temporal downsampling intervals of the VAE encoder, and the extracted fine-grained visual features are encoded as auxiliary condition tokens (\(C_2\)), which serve as inputs alongside the LQ latent condition tokens (\(C_1\)). For conditioning injection, DTI concatenates the noise tokens (\(X\)) and the condition tokens (\(C_1\), \(C_2\)) into a single sequence and feeds them into the DiT attention layers. A key design is the asymmetric attention mechanism: bidirectional self-attention is performed among the condition tokens to allow mutual referencing and enhancement, whereas noise tokens only perform unidirectional cross-attention to the condition tokens—reading conditional info while preventing conditions from being contaminated by noise. This not only fully activates the conditioning information but also reduces the attention complexity from \(O((N+M)^2)\) to \(O(M^2 + N \times M)\) (where \(N\) is the number of noise tokens and \(M\) is the number of condition tokens), significantly lowering computational overhead.

2. Discriminative Guider (DG): Predicting the Optimal Diffusion Starting Point via Interpretable Supervised Learning

DG is a lightweight mapping network. Its primary goal is to answer a key question: given an LQ input, at what percentage of the diffusion trajectory should the sampling process begin? DG predicts two variables through supervised learning:

The first is the local information loss matrix \(M_D\), where each element is defined as \(M_D[i,j] = \|z_L[i,j] - z_H[i,j]\| / (1 + \|z_L[i,j] - z_H[i,j]\|)\), measuring the ratio of information loss at each pixel location (0 indicates perfect preservation, while approaching 1 indicates complete loss). The second is the low-frequency residual \(M_R = z_H - z_L\), representing the element-wise difference between the high and low-resolution latents.

\[t_{pred} = \frac{\text{RMSE}(z_L, z_H)}{1+\text{RMSE}(z_L, z_H)} = \frac{\sqrt{\mathbb{E}\left[(\frac{M_D}{1-M_D})^2\right]}}{1+\sqrt{\mathbb{E}\left[(\frac{M_D}{1-M_D})^2\right]}}\]

Here, \(t_{pred}\) represents the predicted starting timestep aligned with the SNR. Because DG is lightweight and has limited capacity, the predicted \(M_R\) carries some errors and is thus only used as a coarse-grained low-frequency refinement (\(z_{anchor} = z_L + M_R\)), while the recovery of high-frequency details is left to the generative prior of the DiT. This division of labor—"discriminative model for coarse restoration, diffusion model for fine generation"—ensures that even if the DG's prediction is biased, it does not degrade the quality of the final output.

3. SNR-Aligned Dynamic Initialization and Adjustable Fidelity-Perceptual Quality Hyperparameter

The predicted \(t_{pred}\) from the DG corresponds to the optimal starting point under SNR alignment. However, practical applications may demand varying degrees of perceptual quality; if a scenario prioritizes sharp and delicate visual appearance (at the expense of some fidelity), the sampling should begin from a later timestep. Therefore, DTI introduces a hyperparameter \(\lambda\) to flexibly control this trade-off:

\[t_{start} = (1-\lambda)t_{pred} + \lambda, \quad \lambda \in [0,1]\]

When \(\lambda = 0\), the process fully adheres to the SNR-aligned prediction (biasing towards fidelity); when \(\lambda = 1\), it degenerates into a full generation process starting from pure noise (maximizing perceptual quality). The final starting noise latent is \(z_{start} = (1 - t_{start}) \cdot z_{anchor} + t_{start} \cdot \varepsilon\). The overall framework provides a controllable external interface, allowing users to tune the balance between fidelity and perceptual quality. Experiments in the paper demonstrate that the noise levels corresponding to practical degradations lie roughly within the 20%-45% range of the diffusion process, meaning that even with \(\lambda = 0\) (the highest-fidelity mode), the number of sampling steps can be reduced by 55%-80%.

Loss & Training

The model is based on the pretrained Wan2.1-1.3B T2V DiT backbone, with existing DiT blocks fine-tuned solely via LoRA. The newly introduced asymmetric attention blocks are trained from scratch. The DG is trained from scratch for 15k iterations, and the entire model is fine-tuned for only 20k iterations, eliminating the need for large-scale auxiliary training or knowledge distillation. The diffusion training objective is the standard flow-matching loss: \(\mathcal{L}_{FM}(\theta) = \mathbb{E}_{t,x_0,x_1}[\omega(t)\|v_\theta(x_t,t,c) - (x_1 - x_0)\|^2]\), and the DG is optimized using an MSE objective. The spatial resolution is 512×512.

Key Experimental Results

Main Results

Dataset	Metric	PGTFormer	SVFR	Vivid-VR	FlashVSR	SeedVR2	DTI
VFHQ	PSNR↑	25.63	25.54	19.82	19.57	19.80	26.35
VFHQ	SSIM↑	0.66	0.72	0.72	0.71	0.73	0.74
VFHQ	LPIPS↓	0.30	0.22	0.21	0.18	0.21	0.17
VFHQ	LMD↓	6.055	4.71	4.64	6.10	4.59	4.20
VFHQ	TLME↓	5.72	4.11	3.98	4.29	4.19	3.72
CelebV-HQ	PSNR↑	22.75	24.95	23.91	24.59	24.66	25.52
CelebV-HQ	LPIPS↓	0.55	0.28	0.40	0.32	0.34	0.22
CelebV-HQ	LMD↓	45.05	7.54	15.30	9.44	16.04	5.26
CelebV-HQ	TLME↓	29.16	5.34	8.59	6.34	11.39	4.01

DTI (without DG) achieves the lowest LPIPS (<0.2) across all datasets with ground truth, and is consistently optimal in all fidelity metrics (PSNR, SSIM, IDS, LMD, TLME). While FlashVSR achieves the best performance on no-reference metrics due to large-scale post-training, its fidelity metrics are the poorest (VFHQ PSNR < 20).

Ablation Study

Configuration	PSNR↑	LPIPS↓	MUSIQ↑	NFE	Description
DTI w/o DG	26.35	0.17	71.54	50	Full condition injection, starting from pure noise
DTI w/ DG	26.85	0.17	65.17	12	With DG, NFE reduced by 76%, fidelity ↑
LQ Condition Only	22.15	0.28	69.20	50	Without DINOv3 features
DTI Dual-Condition	26.35	0.17	71.54	50	LQ + DINOv3 full conditions

Key Findings

• Incorporating the DG reduces the NFE from 50 to 12 (a 76% reduction) while providing an additional 0.5dB boost to PSNR (reaching 26.85), with a corresponding drop in no-reference metrics (MUSIQ). This validates that the discriminative guidance shifts the perceptual-distortion trade-off toward higher fidelity.
• Compared to single-conditioning (LQ latent only), the dual-conditioning injection improves PSNR on VFHQ by up to 4.2 dB, drops LPIPS from 0.28 to 0.17, and accelerates model convergence, demonstrating the immense benefit of the "purified" visual information provided by DINOv3 feature extraction.
• The authors independently analyzed the relationship between metrics and actual quality, finding LPIPS to be the most reflective metric of overall quality: samples with high MUSIQ often exhibit severe visual artifacts, while samples with high PSNR tend to look blurry, and low LPIPS requires solid performance in both aspects.
• The control of the trade-off via parameter \(\lambda\) is experimentally validated, providing users with a practical and tunable external interface.

Highlights & Insights

• The elegance of the paradigm shift: Redefining GFVSR from "generation" to "restoration" may seem like a simple change in perspective, but it systematically reforms the conditioning method, training objectives, and inference strategies. This philosophy of "clarifying the core nature of the task before designing the method" is highly instructive.
• The intuition behind DINOv3 feature enhancement is highly generalizable: not all degraded information is equal. The low-frequency skeleton in the LQ is reliable, and the DG should prioritize preserving it rather than forcing the diffusion model to regenerate it from scratch. This observation can be readily transferred to other degradation-related tasks.
• The training objectives of the DG are physically interpretable (\(M_D\) represents pixel-wise information loss), distinguishing it from the end-to-end blackbox training of most auxiliary networks, and clarifying the physical meaning of each prediction.
• Serving as a controllable external interface, the parameter \(\lambda\) gracefully unifies intuition (more severe degradation implies starting the sampling later) and operational usability (user-defined adjustment), fully supported by rigorous SNR alignment theory.

Limitations & Future Work

• Training the DG from scratch on a single dataset (VFHQ) limits its cross-dataset generalization. The paper notes an insufficient exploration of initializing the DG from pretrained ViT backbones; leveraging pretrained priors (such as DINOv3 itself) could further bolster generalization.
• DINOv3 feature extraction only acts on frames at VAE downsampling intervals, which is sparse in the temporal dimension (sampling one frame every four frames) and may lose some fine temporal details.
• The drop in no-reference metrics (MUSIQ, CLIP-IQA) after introducing the DG suggests that perceptual quality still has room for improvement, indicating the necessity of tuning the \(\lambda\) parameter for different application scenarios in practice.
• While the concept of DTI—"starting from an intermediate timestep" in diffusion models—is generalizable, the design of the DG heavily relies on the structured nature of facial features. Applying this approach to general video super-resolution without clear salient structures may require further adaptations.

Related Work & Insights

• vs FlashVSR: FlashVSR achieves one-step generation via large-scale post-training (on a 160k image-video dataset) to deliver maximum perceptual quality at the cost of poor fidelity. In contrast, with only minor fine-tuning, DTI leads significantly in fidelity, proving the effectiveness of the targeted restoration paradigm.
• vs Vivid-VR: Vivid-VR employs ControlNet to inject LQ conditions (which requires substantial auxiliary network training), whereas DTI's attention-concatenation approach is simpler, more efficient, and free of extra independent modules.
• vs SeedVR2: Based on diffusion adversarial post-training for one-step generation, DTI's multi-step dynamic initialization offers a more flexible and controllable trade-off.
• vs DR2: DR2 also observes that LQs retain low-frequency information, but DTI rigorously derives the quantitative relationship between the starting timestep and the degradation level through SNR alignment theory rather than relying on empirical heuristics.

Rating

• Novelty: ⭐⭐⭐⭐ The paradigm shift of redirecting GFVSR to "restoration instead of generation" brings systematic improvements across the methodology pipeline, although key components (attention-concatenation injection, SNR alignment) are not entirely pioneering.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive comparisons are performed across 3 benchmarks using a wide range of full-reference and no-reference metrics, supported by complete ablation studies (on conditioning types, DG, and the \(\lambda\) parameter) and quality analysis of the metrics.
• Writing Quality: ⭐⭐⭐⭐ The logical flow is clear, and the problem-solution mapping is complete, though some implementation details in the Method section (Sec.4) could be more straightforward.
• Value: ⭐⭐⭐⭐⭐ It dramatically improves fidelity (PSNR +0.7~4.2dB) and efficiency (NFE reduced by 76%) with only minimal fine-tuning, which holds high practical value. The analysis of the trade-off is also highly illuminating.

Related Papers

• [CVPR 2026] DTG-Restore: Training-Free Diffusion Refinement for Generative Video Super-Resolution
• [ECCV 2026] AVSR-Diff: Scale-Agnostic Diffusion Priors for Temporally Consistent Arbitrary-Scale Video Super-Resolution
• [CVPR 2026] VOSR: A Vision-Only Generative Model for Image Super-Resolution
• [NeurIPS 2025] Image Super-Resolution with Guarantees via Conformalized Generative Models
• [CVPR 2026] DUO-VSR: Dual-Stream Distillation for One-Step Video Super-Resolution