ART-VSR: Adaptive Rectified Trajectories for One-Step Video Super-Resolution

Conference: ECCV2026
Paper: Official page / PDF
Code: https://github.com/Roveer/ART_VSR
Area: Video Super-Resolution / Generative Video Restoration
Keywords: One-step generation, adaptive timestep map, latent trajectory rectification, flow matching, temporal consistency

TL;DR

ART-VSR estimates how far each token should travel along a generative path, then rectifies its latent starting state accordingly, enabling one-step restoration with a Wan2.1 video prior; it achieves 0.197 LPIPS on REDS under the authors' unified re-evaluation protocol, without dominating every fidelity or perceptual metric.

Background & Motivation

Video super-resolution must reconstruct missing spatial frequencies while keeping newly generated details consistent across frames. Pixel-oriented regression tends to average over plausible textures, producing smooth results. Diffusion and flow-based video models offer richer generative priors, but repeatedly evaluating a large video network is expensive. One-step generation is therefore valuable not just as a sampling shortcut: it must also preserve the reliable structure already present in the low-resolution input instead of regenerating the entire scene.

Trajectory-adjustment methods treat the low-resolution latent as an intermediate generative state, which is more natural for restoration than starting from pure noise. However, compression artifacts, blur, and sensor noise do not necessarily match the corruption distribution at any particular pretrained timestep. Spatial variation creates another problem: smooth sky and degraded building textures within the same frame need different restoration strengths. A small global timestep can leave textures blurred, whereas a large one may contaminate clean regions. The paper calls these issues initial state mismatch and the global timestep constraint, and uses trajectory drift to describe their combined effect.

The intervention is not merely an additional enhancement network. A spatial schedule can still start from incompatible latents, while a latent filter can move the input to a state inconsistent with the subsequent integration length. Core idea: jointly learn a token-wise generation time and a time-conditioned correction of the latent starting state, with geometric supervision linking the two.

Method

Overall Architecture

The input is a low-resolution video and the output is a restored high-resolution video. A frozen VAE supplies latent representations. The Adaptive Timestep Estimator (ATE) reads both pixels and latents to produce a continuous map aligned with DiT tokens. The Latent Trajectory Rectifier (LTR) uses this map to adjust detail components, after which the DiT receives both the rectified state and the timestep map, performs a single local integration approximation, and passes its output to the VAE decoder.

Training additionally uses paired high-resolution targets to supervise the geometry of the corrected starting state, alongside reconstruction and perceptual objectives that jointly optimize ATE, LTR, and DiT. Inference requires neither an HR reference nor evaluation of the geometric loss. ATE, LTR, and the VAE still incur computation: one step refers to the generative update, not to a system containing only one operation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Low-resolution video<br/>Pixels and VAE latents"] --> ATE["Adaptive Timestep Estimation<br/>Token-wise timestep map"]
    ATE --> LTR["Latent Trajectory Rectification<br/>Preserve structure, modulate detail"]
    INPUT --> LTR
    ATE --> JOINT["Joint State-Time Learning<br/>One DiT update"]
    LTR --> JOINT
    TARGET["High-resolution target<br/>Training only"] -.->|Geometry and reconstruction supervision| JOINT
    JOINT --> OUTPUT["VAE decoding<br/>High-resolution video"]

Key Designs

1. Adaptive Timestep Estimation: replace a global time with locally estimated integration lengths

ATE does not simply select a better scalar for an entire video. It predicts a token-wise map with values in \([0,1]\). Its heuristic branch computes three pixel-space energy cues: local texture standard deviation, Sobel structural gradients, and FFT high-pass residuals. Their reported weights are 0.8, 0.2, and 0.6. Local statistics use a \(7\times7\) window, followed by \(9\times9\) average smoothing and \(\tanh(4E)\) normalization. This parameter-free branch provides a texture-strength anchor at the start of training, although texture energy is not ground-truth degradation severity.

The neural branch addresses that limitation with two paths. One extracts structural semantics from latents. The other applies a spatial 2D Real-FFT to pixels, feeds the real and imaginary spectrum components into a gating network, modulates the spectrum with the resulting real-valued gate, and returns to the spatial domain through an inverse transform. The paper describes this as changing frequency amplitudes while preserving complex phase structure. After spatiotemporal alignment, the pixel-frequency and latent features are fused and passed through a Sigmoid to obtain a neural map. The final map combines the heuristic prediction \(M_H\) and neural prediction \(M_N\):

\[ T_{\mathrm{map}}=\lambda M_H+(1-\lambda)M_N,\qquad \lambda=0.25. \]

This expression is normalized from the explicit prose accompanying Eq. (9). The fixed heuristic contribution stabilizes early learning, while the neural branch learns content-degradation relationships that high-frequency energy alone cannot resolve. The output is a learned local generation condition, not a calibrated probability of noise at each token.

2. Latent Trajectory Rectification: preserve reliable structure and modulate uncertain detail

An LR latent need not lie on a compatible pretrained generative path. Asking the DiT to restore it in one step therefore combines two burdens: correcting an unsuitable state and synthesizing missing detail. LTR first decomposes the input within the frozen VAE latent space into low-frequency structure and high-frequency detail. Structure bypasses stochastic modulation. Detail is processed together with standard Gaussian noise by an adaptive modulation module conditioned on the timestep map. This is neither uniform noise injection into the whole latent nor wholesale removal of its low-frequency structure.

The retained structure and modulated detail pass through composition and nonlinear rectification blocks. A global residual connection from the original LR latent then produces the corrected starting state, encouraging an approximately identity mapping in artifact-free regions. This state is tokenized for the DiT, which also receives the timestep map directly. ATE therefore controls both initialization and the subsequent generative update, rather than serving only as an internal LTR signal.

The cached text establishes this decomposition, modulation, composition, and residual dataflow, but operators and arguments in Eqs. (10) and (11) are damaged. The prose also does not fully specify layer counts or convolution configurations. Accordingly, this note does not invent an additive-noise formula for the modulation block or present an unsupported implementation-ready architecture.

3. Joint State-Time Learning: match the corrected starting position to the update scale

Standard flow matching moves from noise at \(t=1\) toward data at \(t=0\). One-step restoration approximates this evolution with a single Euler update. ART-VSR replaces the global integration length with a token-wise map, but it does not query a frozen global-time model independently for each token. DiT, ATE, and LTR are jointly fine-tuned under the full spatial map, allowing the network to retain cross-token spatiotemporal interactions. Conceptually, the model predicts a vector field at the rectified state and scales movement toward the clean-video end by the local integration lengths.

Geometric supervision distinguishes the rectifier from an unconstrained learned filter. During training, the HR latent \(z_{\mathrm{HR}}\) anchors a reference direction toward standard Gaussian noise \(\epsilon\). A state's displacement from that anchor is projected onto the direction. The readable prose explicitly defines:

\[ u=\frac{\epsilon-z_{\mathrm{HR}}}{\|\epsilon-z_{\mathrm{HR}}\|_2}, \qquad \pi(z)=\langle z-z_{\mathrm{HR}},u\rangle u. \]

The geometric loss \(\mathcal L_{\mathrm{LTR}}\) penalizes displacement orthogonal to this reference path and encourages longitudinal progress to match the proportional distance indicated by the timestep map. The first part addresses being off the path; the second addresses being near the path but at an incompatible temporal position. The internal longitudinal weight is \(\lambda_{\mathrm{align}}=0.1\). This is a directional regularizer in VAE latent space, not a claim that latent channels are physically orthogonal or that real generative trajectories are provably straight.

There is a mathematical verification boundary here. Parentheses and some symbols in cached Eq. (12) are damaged, and the exact implementation-level broadcasting between a spatially varying map and a global directional projection cannot be fully established from that extraction. This note therefore reports the two constraints and weights supported by the prose without manufacturing a precise full loss expression. At inference, the trained nonlinear LTR predicts the rectified state without the HR anchor.

A Worked Example

Consider a low-resolution clip containing both flat sky and a blurred building facade. This is a mechanism illustration, not an additional measured example from the paper. ATE combines energy cues with neural diagnosis to assign different local integration lengths to the two regions; there is no hard-coded rule that a building must receive 0.8 and sky must receive 0.2.

LTR retains the existing low-frequency outlines and adjusts detail components according to the map, so generation need not start directly from uncalibrated compression artifacts. DiT processes the full rectified clip and map in one update, restoring detail while drawing on its video prior for cross-frame relationships. Geometric supervision links initialization and time during training; no HR video is retrieved during inference.

Loss & Training

Training combines MSE reconstruction, LPIPS perceptual supervision, and geometric rectification. Normalizing the relationship explicitly described around Eq. (13) gives:

\[ \mathcal L_{\mathrm{total}}=\mathcal L_{\mathrm{rec}} +\alpha\mathcal L_{\mathrm{perceptual}} +\beta\mathcal L_{\mathrm{LTR}},\qquad \beta=0.5. \]

Initialization uses an official Wan2.1 DiT checkpoint. ATE, LTR, and DiT are jointly fine-tuned on NVIDIA A800 GPUs while the VAE remains frozen. Training combines HQ-VSR/YouHQ videos with Flickr2K/DIV2K images, following the mixed image/video and degradation setup associated with DOVE/FlashVSR.

Stage 1 stabilizes the latent mapping using 33-frame sequences at \(640\times960\), batch size 32, and learning rate \(1\times10^{-5}\). Reconstruction MSE is evaluated in latent space; LPIPS is disabled with \(\alpha=0\), while the geometric loss remains active. Stage 2 refines visible textures using 17-frame clips at \(480\times640\), batch size 4, and learning rate \(6\times10^{-5}\). Outputs pass through the frozen VAE decoder, reconstruction MSE is evaluated in RGB space, and LPIPS is enabled with \(\alpha=1.0\).

Both stages use \(\beta=0.5\) and \(\lambda_{\mathrm{align}}=0.1\), with heuristic scaling \(\tau=4.0\). The authors describe one-factor validation near the default settings, but the cache does not provide complete sensitivity curves, iteration counts, the specific optimizer type, or all degradation parameters. These details are insufficient to claim full reproducibility.

Key Experimental Results

Main Results

Table 1 in the paper covers REDS, YouHQ40, UDM10, MVSR4x, and VideoLQ. The selection below compares DOVE, FlashVSR, and ART-VSR. These are the authors' re-evaluations of official checkpoints with official inference settings under matched test clips and degradations, not a mixture of numbers copied from the original baseline papers. Lower LPIPS is better; higher PSNR, MUSIQ, CLIPIQA, and DOVER are better.

Dataset	Metric	DOVE	FlashVSR	ART-VSR
REDS	LPIPS	0.252	0.273	0.197
REDS	MUSIQ	53.463	63.051	64.965
UDM10	LPIPS	0.270	0.294	0.242
UDM10	PSNR	26.944	24.012	26.302
YouHQ40	LPIPS	0.165	0.246	0.180
YouHQ40	DOVER	0.686	0.708	0.738
MVSR4x	CLIPIQA	0.515	0.595	0.711
VideoLQ	DOVER	0.465	0.476	0.489

ART-VSR improves REDS LPIPS over DOVE by 0.055, but its YouHQ40 LPIPS is worse by 0.015 and its UDM10 PSNR is also lower. In the full table, DLoRAL's VideoLQ MUSIQ of 60.530 exceeds ART-VSR's 55.272. Competitive perceptual quality is therefore a more accurate conclusion than universal superiority, and these metric trade-offs should not be collapsed into an undefined aggregate improvement percentage.

Ablation Study

The following REDS results come from Table 2. Fixed-time variants replace only ATE's adaptive map; they retain LTR and its geometric loss. They are not ordinary baselines with the entire ART-VSR pipeline removed. Only \(M_H\) and Only \(M_N\) use the heuristic or neural map alone. Lower NIQE is better.

Time configuration	LPIPS	MUSIQ	NIQE
Fixed \(T_L=0.2\)	0.251	43.058	4.823
Fixed \(T_L=0.4\)	0.231	42.595	5.154
Only \(M_H\)	0.205	61.684	3.259
Only \(M_N\)	0.201	62.911	2.886
Full ATE	0.197	64.965	2.806

Table 3 retains ATE and distinguishes removal of LTR from removal of its geometric supervision. The REDS subset below uses matched data, degradation, frame settings, and optimization schedules across configurations.

Rectification configuration	LPIPS	MUSIQ	NIQE
Without LTR	0.264	46.437	5.856
With LTR, without \(\mathcal L_{\mathrm{LTR}}\)	0.212	63.497	3.972
Full model	0.197	64.965	2.806

Key Findings

• On REDS, full ATE improves LPIPS over the neural-only branch by 0.004. The physical anchor adds value beyond neural estimation, while larger differences separate spatial adaptation from fixed global times.
• Removing LTR reduces REDS MUSIQ from 64.965 to 46.437, a difference of 18.528. Keeping its network without geometric supervision also falls short, supporting the distinction between an unconstrained filter and a supervised starting-state correction.
• Dedicated temporal evidence is primarily Figure 6's X-t profiles, which follow a fixed horizontal scanline over time to inspect continuity and sharpness. This is a visual analysis, not a reported quantitative flicker benchmark with confidence intervals.

Highlights & Insights

• Time in restoration need not represent one uniform noise level for an entire frame; it can function as a local update scale. Explicitly conditioning generation on spatially heterogeneous restoration needs is more informative than leaving all variation implicit.
• Starting-state correction and time estimation must be learned together. Treating ATE as a scheduler that can simply be inserted into any frozen flow model would miss the paper's reliance on joint spatially conditioned fine-tuning.
• Structure preservation, detail modulation, and residual connections make generation selective. Randomness is directed toward components requiring repair instead of asking the video prior to infer the entire scene anew from noise.

Limitations & Future Work

• The cached paper has no standalone limitations section. The observations below are critical reading of the method and evidence, not limitations explicitly acknowledged by the authors.
• One-step generation does not establish real-time deployment: the paper provides no runtime, FPS, memory, parameter-count, or matched-hardware speed comparison. Omitting the geometric loss at inference does not make ATE/LTR computation free.
• Quality gains are not uniformly dominant, with some PSNR, LPIPS, and reference-free metrics remaining below competing methods. Text, thin structures, and evidentiary video require detail-authenticity checks rather than judging success from increased sharpness alone.
• The time map responds to both content and degradation and should not be treated as a ground-truth degradation map. Controlled experiments with known spatial corruption patterns could clarify what it encodes.
• Long-horizon evidence is limited: the cache does not report dedicated long-video statistics, occlusion-stratified results, or confidence intervals. Independent flicker metrics and random-seed analyses would strengthen temporal claims.
• Several core equations are damaged in the local text extraction, and exact broadcasting conventions, modulation architecture, and some training details remain unresolved. The directional geometric interpretation should not be promoted into a rigorous trajectory-alignment theorem.

Related Work & Insights

• Compared with DOVE: Both exploit an LR starting state for one-step restoration, while ART-VSR explicitly adapts that state and local time. Table 1 compares complete external systems, not same-backbone component interventions; DOVE still has better YouHQ40 LPIPS.
• Compared with FlashVSR: Both address efficient video restoration, but state-time adaptation is ART-VSR's distinguishing mechanism. The cache lacks the latency evidence needed to claim a streaming-speed advantage over FlashVSR.
• Compared with RealViformer and multi-step generative restoration: The former represents strong deterministic structural recovery; the latter offers rich priors at a sequential sampling cost. ART-VSR investigates structure-preserving texture generation within a one-step budget, not the obsolescence of regression or multi-step methods.
• Transferable direction: Jointly adapting a local update scale and its compatible starting state could be explored in image restoration or spatially heterogeneous degradation tasks. Such transfer is a research direction, not an experimentally established result here. The code URL comes from the cached paper and was not checked online during this reading.

Rating

• Novelty: 4/5. Local time estimation and geometrically supervised state correction form a coherent joint design, while building on existing flow matching and one-step restoration.
• Experimental Thoroughness: 4/5. Five benchmarks and separate ATE/LTR ablations support the mechanism, but efficiency measurements and systematic temporal statistics are missing.
• Writing Quality: 3/5. The problem decomposition and controlled protocols are clear, although the relationship between local time and global geometry needs care; damaged cached equations further limit verification.
• Value: 4/5. The state-time coupling is a reusable restoration idea, while deployment value still needs speed and memory evidence.