Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion

Conference: ECCV2026
Paper: ECCV paper
PDF: Full paper
Project: Stream-DiffVSR
Area: Video Generation / Video Super-Resolution
Keywords: video super-resolution, auto-regressive diffusion, rollout distillation, optical-flow alignment, temporal consistency

TL;DR

Stream-DiffVSR combines four-step diffusion denoising, flow-aligned historical output guidance, and temporal decoding to restore video frame by frame without future frames, achieving REDS4 LPIPS of 0.099 and 0.328-second time-to-first-frame at 720p on an RTX 4090, although it does not yet provide conventional real-time HD video processing.

Background & Motivation

Video super-resolution reconstructs high-resolution video from low-resolution input, requiring both sharper individual frames and details that do not fluctuate between frames. CNN and Transformer methods can reconstruct images relatively quickly, but pixel fidelity and perceptual detail do not always agree: higher PSNR does not necessarily imply more natural texture. Diffusion models provide generative priors for realistic details, yet methods such as StableVSR and MGLD-VSR typically require many denoising steps and use bidirectional temporal information. This creates two different sources of waiting: computing an output is expensive, and producing the first output may also require processing subsequent frames.

For live broadcasting, video conferencing, or interactive rendering, average processing time per frame is insufficient to characterize the experience. Even a chunk-based model with reasonable average throughput may need to buffer an entire chunk before displaying its first frame. The paper therefore reports per-frame runtime separately from time-to-first-frame (TTFF), while requiring each output to depend only on the current input and past state. However, independently applying a fast image diffusion model to every frame is also insufficient: textures generated for one frame do not inform the next, leaving temporal flicker unresolved.

Starting from the existing Stable Diffusion ×4 Upscaler, the paper addresses denoising cost and historical conditioning separately. A few-step model preserves the image generation prior, while the previous high-quality output is motion-aligned and introduced into both latent denoising and RGB decoding. This avoids waiting for future frames without assigning all responsibility for temporal consistency to a single representation level. Core Idea: compress per-frame computation through full four-step rollout distillation, then use the flow-aligned previous output as both a denoising condition and a decoding condition to maintain detail continuity through strictly causal data flow.

Method

Overall Architecture

The input is a low-quality video stream and the output is a frame-by-frame ×4 super-resolved video; this is input-constrained restoration, not text-conditioned generation of a new video. At the current time, the system retains the previous low-quality input and its already generated high-quality output, which carries historical textures and accumulated temporal information. Figure 4 estimates motion from upsampled adjacent low-quality frames and warps the previous high-quality result into the current coordinate system. The aligned historical RGB frame supplies Auto-regressive Temporal Guidance, while its encoded features also supply the Auto-regressive Temporal-aware Decoder. After four DDIM denoising updates, the temporal decoder produces the current RGB output and retains it for the next time step.

Dashed edges below indicate parameter dependencies established through training, while solid edges indicate frame-by-frame inference data flow; historical feedback does not mean repeatedly generating a video within one frame.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Train["Training LQ/HQ pairs"] -.-> Rollout["Four-Step Rollout Distillation"]
    Rollout -.->|Frozen denoiser parameters| ARTG["Auto-regressive Temporal Guidance<br/>ARTG"]
    Current["Current LQ frame<br/>and initial latent"] --> ARTG
    History["Previous HQ output<br/>and adjacent LQ frames"] --> Warp["Optical-flow alignment<br/>Historical RGB and encoded features"]
    Warp --> ARTG
    ARTG -->|Latent after four denoising steps| Decoder["Auto-regressive Temporal Decoding<br/>TPM"]
    Warp --> Decoder
    Decoder --> Output["Current HQ output"]
    Output -->|Next time step| History

Key Designs

1. Four-Step Rollout Distillation: train on the short denoising chain used at inference

The base model is the Stable Diffusion ×4 Upscaler, originally used with 50 inference steps; acceleration does not require replacing the U-Net architecture. Each training iteration executes the complete four-step denoising process, obtains the final clean latent, and supervises that endpoint. Unlike training on a randomly selected intermediate diffusion timestep, this optimizes the short trajectory and final output that deployment will use. Here, rollout primarily denotes unrolling the diffusion denoising chain; the name alone does not establish training on self-generated histories across an entire long video. The main paper places the fuller algorithm in an appendix, so it does not provide enough evidence to specify gradient truncation across video frames or historical sampling ratios.

Endpoint supervision combines latent reconstruction, LPIPS differences after decoding, and an adversarial loss. The latent term constrains the target representation, the perceptual term discourages numerical matching that loses visual structure, and the adversarial term encourages natural details. These losses do not add inference steps; after training, online execution consists of four DDIM updates and one decoding pass. The acceleration therefore comes from teaching a short trajectory to perform restoration, not from continuing to invoke a complete 50-step model at runtime. Table 11 further shows that increasing sampling steps in this system does not monotonically improve every perceptual or temporal metric.

2. Auto-regressive Temporal Guidance: use previous details when restoring the current latent

The previous high-quality output cannot serve directly as a stationary template because the camera or objects may have moved. ARTG first aligns the historical output using optical flow estimated from adjacent inputs, then supplies the aligned RGB content as a temporal condition during U-Net denoising. Across the four updates, the denoiser can exploit already restored details instead of performing a completely independent stochastic restoration of the current frame. This provides a source of texture continuity for the same object while the current low-quality input constrains the structure that should appear now.

Causality follows from the direction of information: the current frame receives only completed past outputs, never future high-quality frames. The immediate state requires only the previous frame, but that frame was itself influenced by earlier outputs, allowing historical information to propagate recursively. The same recursion can also propagate historical errors; optical-flow misalignment does not disappear simply because the restoration model uses diffusion. During ARTG training, the denoising U-Net and decoder are frozen, and consecutive LQ/HQ frame pairs train the added temporal conditioning capability. Its objective again combines latent reconstruction, LPIPS, and adversarial terms, introducing temporal information without disrupting previously learned spatial restoration.

3. Auto-regressive Temporal Decoding: constrain continuity again when reconstructing RGB detail

Historical conditioning during latent denoising does not guarantee stable decoded textures; the base upscaler's latent representation remains at one-quarter of the target spatial resolution. Upsampling and mapping to RGB can turn local feature differences into edge jitter or texture changes. The paper therefore sends the aligned previous frame through a frozen encoder and uses its features alongside the current clean latent during decoding. Temporal Processor Modules (TPMs) are inserted after spatial convolutions, fusing current and historical features at multiple scales rather than merely smoothing the final RGB output.

The TPM illustrated in Figure 2 includes average pooling, concatenation, one-dimensional convolutions, activation, temporal processing, and feature splitting, followed by gated fusion and interpolation-based upsampling. The current branch is multiplied by \(1-\alpha\) and the historical branch by \(\alpha\); the gate controls reliance on the previous frame. Using processed features at the same scale, the weighted fusion shown in the figure can be expressed as:

\[ f_{\mathrm{mix}}=(1-\alpha)f_{\mathrm{current}}+\alpha f_{\mathrm{previous}}^{\mathrm{warp}}. \]

This notation explains the fusion operation in Figure 2; it neither reconstructs a corrupted source loss equation nor implies simply blending two RGB images. The main text describes a learnable gate but does not sufficiently specify whether it varies per pixel, so it should not be interpreted directly as explicit occlusion confidence. Besides reconstruction, perceptual, and adversarial terms, TPM supervision compares optical flow from predicted frame pairs with flow from ground-truth pairs to constrain motion relationships. ARTG therefore controls historical conditioning during denoising, whereas TPM controls continuity when those latents become pixels; they address different stages.

A Worked Example

Consider a continuous ×4 super-resolution stream in which the camera moves right, changing the location of the same texture between adjacent frames. The first frame has no ordinary historical high-quality output; the authors identify degraded early frames as a cold-start issue, but the available text does not include initialization pseudocode. It is therefore unsupported to assume that the first frame necessarily copies the input, uses a zero state, or invokes another restoration network. Once the first HQ result exists and the second frame arrives, the two LQ inputs provide motion estimates for aligning the first HQ frame. That aligned history guides four-step denoising for the second frame, while its encoded features also enter TPM decoding to produce the second HQ output. The third frame repeats the process, now using the generated second frame as history rather than the test set's ground-truth second HQ frame. This distinction explains both continuous online operation and the possibility of accumulating alignment or generation errors. Avoiding future-frame waiting does not eliminate computation latency: an RTX 4090 still needs approximately 0.328 seconds to process a 720p frame.

Loss & Training

Figure 3 presents denoising U-Net distillation, decoder TPM training, and ARTG training after freezing the first two components. TPM learns temporal fusion from ground-truth frames while other weights remain fixed; ARTG subsequently learns to exploit aligned historical outputs. Section 3.5 also describes TPM training as occurring "in parallel"; at minimum, the objectives are optimized separately rather than updating all parameters jointly from scratch. The TPM pixel reconstruction term explicitly uses SmoothL1, alongside optical-flow differences, LPIPS, and adversarial supervision. Reconstruction supervision for the U-Net and ARTG acts primarily on the final latent, with perceptual and adversarial terms computed from decoded outputs. The available equation extraction is visibly corrupted, so this note does not guess loss weights, exact norm notation, or complete implementation hyperparameters. The Figure 3 caption assigns Equations (3) and (6) to TPM and ARTG, respectively, whereas the body assigns Equation (3) to ARTG and Equation (6) to TPM; this note follows the module descriptions in the body. Table 10 reports LPIPS 0.124 and tLP 17.816 for joint training, versus 0.099 and 4.265 for separate stages, supporting decoupled training in that experiment.

Key Experimental Results

Main Results

The following selection combines REDS4 quality results under bicubic downsampling from Table 2 (page 10) with RTX 4090 efficiency measurements for the same methods from Table 3 (page 11). Outputs are 720p, and TTFF is measured over 100-frame sequences; higher PSNR is better, while lower LPIPS, tLP, tOF, and time are better. tLP measures temporal perceptual differences and tOF measures temporal optical-flow differences; Table 2 reports values scaled by 100 and 10, respectively, preserved below without mixing them with unscaled metrics. The main text does not supply complete evaluation formulas, so no particular implementation is inferred here.

Method	Temporal access	PSNR	LPIPS	tLP	tOF	Runtime per frame (s)	TTFF (s)
BasicVSR++	Bidirectional / offline	32.386	0.132	9.017	2.490	0.098	9.8
StableVSR	Bidirectional / offline	27.928	0.102	5.755	2.742	46.2	4620
MGLD-VSR	Bidirectional / offline	26.53	0.151	18.139	5.910	43.6	218
TMP	Unidirectional / online	30.672	0.194	10.424	2.480	0.041	0.041
RealViformer	Unidirectional / online	26.763	0.129	11.261	4.037	0.099	9.9
Stream-DiffVSR	Unidirectional / online	27.256	0.099	4.265	3.620	0.328	0.328

Relative to TMP, LPIPS decreases from 0.194 to 0.099, an absolute reduction of 0.095, but PSNR and tOF are worse and per-frame computation is slower. Relative to MGLD-VSR, 43.6/0.328 gives approximately 132.9-fold per-frame acceleration; this is distinct from TTFF improvement and should not be conflated with it. Tables 4 and 5 (pages 11–12) report Vimeo-90K-T LPIPS 0.056, tLP 4.307, tOF 2.689, and 0.041 seconds per frame at 448×256 output resolution, not 720p. The mixed-degradation results in Table 6 (page 12) follow additional fine-tuning on REDS with DOVE's degradation pipeline: Ours has LPIPS 0.220 versus MGLD-VSR's 0.246, so this is not adaptation-free generalization.

Ablation Study

The first ablation table selects metrics revealing temporal module effects from Table 9 (page 13), evaluated on REDS4. "Per-frame" disables both ARTG and TPM; "TPM unwarped" retains TPM but removes optical-flow alignment from its historical input.

Config	LPIPS	DISTS	tLP	tOF	TTFF (s)
Per-frame	0.099	0.071	7.261	4.201	0.208
Without ARTG	0.117	0.070	6.132	3.910	0.235
Without TPM	0.116	0.078	12.847	4.639	0.300
TPM unwarped	0.122	0.082	12.846	5.689	0.320
Full model	0.099	0.062	4.265	3.620	0.328

The second ablation table comes from Table 11 (page 14), comparing DDIM step counts within Stream-DiffVSR rather than directly comparing the four-step method against a separate 50-step baseline system. Section 4.3 states that model weights are held fixed; Figure 7 additionally labels a fine-tuned four-step visualization, which is not used here to infer further training configurations.

Denoising steps	LPIPS	DISTS	tLP	tOF	Runtime per frame (s)
50	0.102	0.068	18.798	3.826	3.460
10	0.122	0.072	9.990	3.625	0.718
1	0.138	0.076	9.899	3.882	0.106
4	0.099	0.062	4.265	3.620	0.328

Key Findings

• Identical single-frame LPIPS can hide different video stability: the per-frame baseline and full model both obtain 0.099, but tLP decreases from 7.261 to 4.265, revealing benefits missed by single-frame perceptual quality.
• Removing TPM gives tLP 12.847 versus 4.265 for the full model; unaligned historical features also give 12.846, showing that decoder-side temporal fusion requires appropriate motion correspondence.
• Four steps are slower than one but improve LPIPS from 0.138 to 0.099; they are faster than 50 steps and yield better tabulated temporal metrics, supporting a short sampling trajectory matched to this model's training.
• Numerical consistency caveat: the page 9 prose gives REDS4 tLP/tOF as 4.198/3.638, whereas Tables 2, 9, 10, and 11 give 4.265/3.620; this note uses the table values while retaining the discrepancy.
• Efficiency consistency caveat: Table 1 gives 3.05 FPS and Table 9 gives 3.408 FPS for the full model, while both report 0.328-second TTFF; the latter FPS does not match the reciprocal per-frame runtime, so these should not be merged into one measurement.

Highlights & Insights

• First-frame latency is an independent design objective. Strict frame-by-frame causality eliminates structural waiting for subsequent frames, but per-frame computation still requires separate optimization.
• Temporal information operates at two representation levels. Latent conditioning constrains generated content, while decoder fusion constrains high-resolution detail formation; module ablations explain more than simply calling a model "temporally aware."
• Complete short-trajectory training places supervision near the deployed output. The transferable lesson is to optimize the outcome of the deployment trajectory instead of assuming that more sampling steps necessarily improve video quality.

Limitations & Future Work

• The authors acknowledge greater computational cost than CNN and Transformer methods, possible optical-flow artifacts under fast motion, and cold-start quality issues in early frames.
• At 720p, 0.328 seconds per frame corresponds to approximately 3.05 FPS, not the 25/30 FPS budget of real-time broadcasting or AR/VR displays; "low latency" is primarily relative to earlier diffusion methods.
• Strict causality also prevents future observations from correcting current occlusions. Occlusion confidence, historical-state resets, and long-sequence error-accumulation evaluation are reader-proposed directions, not validated components of this paper.
• Realistic-degradation experiments use additional fine-tuning, and VideoLQ lacks paired ground truth, so improved no-reference quality cannot fully establish fidelity to the original scene.
• The available text contains the main paper and references but not appendix algorithms or complete experimental settings; corrupted equations and conflicting equation references and metrics leave exact reproduction dependent on the original supplementary material.

Related Work & Insights

• vs StableVSR / MGLD-VSR: These represent slower diffusion VSR methods using future information; Stream-DiffVSR changes both the denoising budget and temporal access, but LPIPS alone does not establish superiority on every fidelity and motion metric.
• vs TMP: TMP is a direct online restoration comparison, with faster REDS4 processing and higher PSNR; Stream-DiffVSR instead emphasizes generative perceptual quality and lower tLP, implying different application trade-offs.
• vs FlashVSR: Chunk-based streaming and frame-by-frame causality should not be compared through average runtime alone; buffering affects TTFF, while memory and speed comparisons must also use the same GPU conditions.
• Connection to CausVid / Self Forcing: These highlight few-step autoregressive diffusion and the importance of training-time historical distributions; Stream-DiffVSR addresses restoration constrained by LQ video, so open-ended video generation training details cannot simply be assumed.

Rating

• Novelty: 4/5. Combines short-trajectory distillation with causal conditioning in latent space and decoding, contributing primarily a system design rather than a new diffusion theory.
• Experimental Thoroughness: 4/5. Covers quality, first-frame latency, temporal modules, and training strategies, but numerical conflicts and long-sequence error boundaries require clarification.
• Writing Quality: 3/5. Motivation and module roles are clear, while inconsistent caption equation references and numerical reports increase verification effort.
• Value: 4/5. Offers a useful design for generative video restoration with reduced waiting, although computational cost remains far from typical real-time deployment requirements.