SparkVSR: Interactive Video Super-Resolution via Sparse Keyframe Propagation¶

Conference: ECCV 2026
Paper: ECCV Paper
Code: https://github.com/taco-group/SparkVSR
Area: Image Restoration
Keywords: video super-resolution, sparse keyframes, interactive restoration, temporal propagation, reference guidance

TL;DR¶

SparkVSR lets users restore a few keyframes and propagate their visual priors under the motion constraints of the low-resolution video, reaching MUSIQ 68.88 and DOVER 0.6212 on MovieLQ with PiSA-SR references while exposing a guidance control for choosing between greater fidelity and richer generated detail.

Background & Motivation¶

Video super-resolution involves more than adding pixels: the same low-resolution sequence can admit several plausible high-resolution textures. Generative restoration methods such as DOVE and SeedVR2 can synthesize details, but usually leave the choice of those details to the model. When text, textures, or appearance differ from the user's intention, correcting them consistently throughout the video is difficult. A text prompt for a video model is not equivalent to an explicit visual target at particular frames.

Single-image super-resolution and image editing already offer strong spatial priors and interactive controls, but applying them independently to every frame produces flicker because each frame can receive a different plausible reconstruction. Simply transferring keyframe-based editing methods to super-resolution is also insufficient: generative video can tolerate some content variation, whereas restoration should preserve the input structure and motion. The model therefore needs two kinds of evidence: keyframes specify the desired appearance, and the complete low-resolution video specifies what happens at each moment.

SparkVSR places appearance decisions in a small set of inspectable, editable anchors and delegates the remaining work to a propagation model that retains low-resolution temporal information. Anchors can come from automatic ISR with PiSA-SR or from prompt-controlled Nano-Banana-Pro, without editing every frame. Core Idea: use sparse high-resolution keyframes as appearance priors and the continuous low-resolution video as a motion constraint, while training the same model with and without references so that reference dependence becomes adjustable.

Method¶

Overall Architecture¶

The inputs are a degraded low-resolution video and a small set of frames selected by a user or policy. An external image super-resolution model first produces high-resolution references. The references and original video are encoded separately, their features are concatenated for a diffusion Transformer based on CogVideoX1.5-5B I2V, and a VAE decodes the restored high-resolution video.

Propagation here does not mean estimating optical flow and copying textures. The model learns to exploit sparse references while observing the complete video condition. Latent-to-pixel training supervises that ability, and reference-free guidance controls the strength of the learned reference condition at inference time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Low-resolution video"] --> Select["Editable Keyframe Anchors"]
    Select --> Fuse["Dual-Branch Sparse<br/>Condition Fusion"]
    Input -->|Continuous video latents| Fuse
    Fuse --> Guide["Reference-Free Guidance"]
    Guide --> Output["One-step denoising and decoding<br/>High-resolution video"]
    Train["Latent-to-Pixel Training"] -.->|Learned propagation capability| Guide
    GT["Ground-truth training<br/>videos and images"] -.-> Train

Solid edges describe inference data flow; dashed edges describe training supervision and the source of the model's learned capability. The two training stages are not online operations repeated whenever a video is restored.

Key Designs¶

1. Editable Keyframe Anchors: turn appearance choices into inspectable images

Reference frames can be selected manually, extracted from codec I-frames, or sampled randomly. Manual selection can prioritize badly degraded or important content; I-frames provide candidates from the video's coding structure; random sampling supports automatic operation. The selected low-resolution frames are first processed by an ISR model to produce the high-resolution references supplied to the video restorer, rather than treating the original low-quality frames as additional detail.

The interactive branch separates task prompts from content prompts. The former specify objectives such as upscaling and deblurring, while the latter describe text or appearance that the user wants corrected. These prompts act on the external image model; the paper does not introduce a separate video-control branch that directly consumes both prompts. Users can inspect the keyframes before propagation, while the PiSA-SR option does not require human intervention.

2. Dual-Branch Sparse Condition Fusion: keep appearance priors together with input motion

The pretrained model's 3D causal VAE encodes the low-resolution video with a temporal downsampling rate of 4 and 16 latent channels. The reference branch places high-resolution keyframes at their corresponding temporal latent indices and fills other positions with zeros, producing another 16-channel condition. Concatenation yields a 32-channel input. A zero reference means that no anchor is available at that position, not that the model loses the video evidence for that time.

Denoising starts from the low-resolution video latent instead of entirely random noise, using a DOVE-inspired one-step strategy with timestep 399. The model can therefore add details from an input structure, while sparse references contribute additional appearance information. This reduces repeated diffusion sampling, but the one-step claim excludes external ISR generation and does not establish real-time performance for the complete interactive workflow.

3. Latent-to-Pixel Training: learn propagation before refining visible textures and flicker

The first stage fixes the VAE decoder and trains the Transformer in latent space using MSE between restored and ground-truth high-resolution latents. Training references are sampled from ground-truth videos, with a random count capped at one quarter of the video frame count and strictly more than 4 frames between selected references. ColorJitter, GaussianBlur, and noise make these otherwise ideal anchors resemble the deviations that external ISR outputs may contain, rather than assuming perfect references.

The second stage decodes outputs into pixel space and jointly trains video and image branches. The video branch retains sparse references and combines pixel MSE, the DISTS perceptual loss, and a frame-consistency loss. The image branch uses zero references and only MSE and DISTS, both retaining the 32-channel input format and strengthening reference-free restoration. References are dropped with probability 0.1 during training so that the model does not require complete, accurate external anchors. The cached text does not clearly define the internal frame-consistency loss, so it should not be identified with an assumed optical-flow loss.

4. Reference-Free Guidance: adjust reference dependence rather than promise gains on every metric

Reference dropout enables the same model to predict with either the original video plus references or the original video plus zero references. At inference time, the difference between those predictions controls the influence of the references. Reference-free prediction still retains the low-resolution video condition; it is not unconditional video generation. A guidance scale of 0 gives blind restoration, 1 gives standard reference-conditioned prediction, values above 1 amplify references, and values below 1 reduce the influence of potentially imperfect external priors.

This control selects a working point on the perception-distortion tradeoff rather than merely changing denoising strength. Table 4 shows that stronger references generally increase texture richness and no-reference quality scores but reduce PSNR and SSIM and can increase LPIPS. Equation (3) is corrupted in the text extraction, so this explanation follows the paper's condition definitions and scale descriptions without reconstructing its exact equation.

A Worked Example¶

Consider a 192-frame MovieLQ film clip. The I-frame configuration in Table 3 selects frames 1, 48, 96, and 144. Nano-Banana-Pro first restores these 4 images; the user can inspect facial details or text before encoding the references into their temporal positions. Unselected positions keep zero references, but the motion information from all 192 low-resolution frames remains available in the video branch.

The learned propagation mechanism carries anchor appearance to other moments, and the guidance scale determines how strongly the result follows external restoration. If an anchor contains unreliable details, users can lower the scale or edit the anchor and rerun the process. This illustrates where interaction occurs; it does not imply that the paper measures average multi-round interaction time or user correction success rates.

Loss & Training¶

The method text describes the second-stage video objective as the following combination. Only the stated relationship is summarized here; the internal frame-consistency term is not reconstructed from the unclear extraction:

\[ \mathcal{L}_{\mathrm{video}}=\mathcal{L}_{\mathrm{MSE}}+\lambda_1\mathcal{L}_{\mathrm{DISTS}}+\lambda_2\mathcal{L}_{\mathrm{frame}},\qquad \lambda_1=\lambda_2=1. \]

The image branch omits frame consistency, and the first stage uses latent MSE alone. The two stages address how references affect the video and whether decoded results have desirable textures with less flicker. High first-stage PSNR is therefore not sufficient evidence that pixel-space refinement is unnecessary.

Training uses 2,055 high-resolution HQ-VSR clips and 900 DIV2K images, degraded with the RealBasicVSR and Real-ESRGAN pipelines, respectively. The setup uses 4 A100-80GB GPUs and a total batch size of 8. Stage 1 trains on 33-frame videos at 320 × 640 for 10,000 iterations with learning rate \(2\times10^{-5}\).

Stage 2 jointly trains on videos and images for 500 iterations with learning rate \(5\times10^{-6}\) and mixing parameter \(\varphi=0.5\). The paper names AdamW but lists three beta parameters, which is inconsistent with the conventional AdamW definition. This note therefore does not present that list as an immediately reproducible optimizer configuration.

Key Experimental Results¶

Main Results¶

UDM10, SPMCS, and YouHQ40 use synthetic inputs with training-matched degradations; RealVSR contains smartphone-captured low-quality and high-quality video pairs. MovieLQ contains 10 film clips from the 1940s to 1950s, each at 360 × 480 resolution, lasting 8 seconds at 24 fps for 192 frames. It assesses authentic historical degradation and does not have the full-reference results reported for other datasets in Table 1.

The short-sequence benchmarks use only the first frame as a reference, whereas MovieLQ uses I-frames. The following values are selected from the paper's Table 1. No reference, NBP, and PiSA denote blind restoration, Nano-Banana-Pro references, and PiSA-SR references, respectively; they are different operating modes.

Dataset / Metric	DOVE	FlashVSR-Full	SparkVSR No Reference	SparkVSR NBP	SparkVSR PiSA
UDM10 PSNR ↑	26.52	23.58	26.62	23.70	23.43
UDM10 SSIM ↑	0.7697	0.6993	0.7756	0.6807	0.6710
UDM10 CLIP-IQA ↑	0.5011	0.5016	0.4303	0.5501	0.6252
UDM10 DOVER ↑	0.5664	0.5317	0.5494	0.6902	0.6411
RealVSR LPIPS ↓	0.1850	0.2316	0.1809	0.1678	0.2165
MovieLQ MUSIQ ↑	60.71	66.38	56.34	65.48	68.88
MovieLQ CLIP-IQA ↑	0.5433	0.5754	0.4622	0.6128	0.6361
MovieLQ FasterVQA ↑	0.7647	0.7822	0.7065	0.797	0.8028
MovieLQ DOVER ↑	0.5101	0.5544	0.5121	0.6194	0.6212

MUSIQ and CLIP-IQA are no-reference image quality metrics; FasterVQA and DOVER assess video quality. They do not independently establish that every restored detail matches the real scene. On UDM10, CLIP-IQA rises from the strongest external baseline's 0.5016 to 0.6252, approximately 24.6%, while DOVER rises from 0.5664 to 0.6902, approximately 21.8%. These gains come from PiSA and NBP modes, respectively.

Ablation Study¶

The following UDM10 results are selected from Tables 2 and 4. The ablation tables use shorter decimal precision than the main table, so their reported values are retained without padding from Table 1.

Source / Config	PSNR ↑	LPIPS ↓	MUSIQ ↑	CLIP-IQA ↑
Table 2: S1 only, no reference	26.73	0.330	44.04	0.331
Table 2: S1+S2, no reference	26.62	0.283	55.79	0.430
Table 2: S1 only, NBP	24.53	0.338	62.57	0.474
Table 2: S1+S2, NBP	23.70	0.338	66.16	0.550
Table 4: PiSA, scale 0.5	25.66	0.312	60.17	0.496
Table 4: PiSA, scale 1.0	23.43	0.355	67.52	0.625
Table 4: PiSA, scale 1.5	20.94	0.399	70.39	0.652

Without references, Stage 2 increases MUSIQ by 11.75 and reduces LPIPS by 0.047 while decreasing PSNR by only 0.11 dB. For NBP, however, LPIPS remains 0.338 at the precision in Table 2. The authors' broad statement that all perceptual evaluations improve should not be interpreted as a strict improvement in every cell.

Key Findings¶

Improved reference-driven quality is not the same as pixel fidelity. Increasing PiSA guidance from 0.5 to 1.5 raises MUSIQ from 60.17 to 70.39 while lowering PSNR from 25.66 to 20.94.
Table 3 shows that a single first-frame reference raises MovieLQ MUSIQ from 56.34 to 61.73. Using 4 uniformly distributed references reaches 65.76, compared with 65.48 for I-frame references; no selection policy is best on every metric.
In the same table, the I-frame configuration achieves DOVER 0.619 versus 0.606 for 4 uniformly distributed references. Temporal coverage and content location matter in addition to reference count.

Highlights & Insights¶

User intent becomes a small set of visible images instead of repeated guesses about how prompts affect a complete video. Anchors can be inspected before propagation, which suits restoration workflows requiring explicit appearance corrections.
Keeping the complete low-resolution condition tells the model both the intended appearance and the original motion. Sparse temporal coverage makes it possible to reuse single-image capabilities without restoring every frame independently.
Reference dropout, zero-reference image training, and inference guidance form a connected design. The reference-free branch is both a fallback for missing conditions and a baseline for adjusting reference strength.

Limitations & Future Work¶

External image models can generate incorrect text or textures, and propagation can spread those errors across more frames. Reducing guidance can weaken their influence but does not guarantee factual correctness.
Evaluation covers 5 benchmarks, but MovieLQ contains only 10 clips. Zero-shot colorization and stylization are primarily qualitative demonstrations, not comprehensive validation of a general video editor.
The supplied full text does not report an interactive user study, end-to-end ISR-plus-VSR latency, or separate motion-error and flicker metrics. Aggregate metrics such as DOVER support improved video quality but do not isolate the contribution of temporal consistency.
Training references come from augmented ground truth, whereas inference references come from external generators, leaving a distribution gap. Reference confidence estimation, re-anchoring after occlusion, and content-change-aware frame selection are useful future directions, not components validated in this paper.

vs DOVE: SparkVSR inherits one-step diffusion and latent-to-pixel training, adding editable sparse references and adjustable reference guidance. Its value lies in the control interface and propagation capability, not in first introducing efficient one-step VSR.
vs PiSA-SR / Nano-Banana-Pro: These models provide single-frame detail or interactive correction; SparkVSR reintegrates their outputs into a continuous video. They are reference generators, not video-propagation baselines.
vs STAR: STAR's text guidance primarily supplies coarse semantic information, whereas SparkVSR uses specific reference images to specify appearance. This requires additional anchor generation and potentially human inspection.
Implications for other restoration tasks: Old-film colorization or local style editing can delegate appearance changes to an image model and temporal preservation to a video model, but motion fidelity and error propagation still require separate evaluation.

Rating¶

Novelty: 4/5. Integrates interactive keyframes, low-resolution motion conditions, and reference-free guidance into controllable VSR, while inheriting the backbone and two-stage approach.
Experimental Thoroughness: 4/5. Multiple benchmarks and training, guidance, and selection ablations are informative, but interactive user evaluation and end-to-end cost measurements are missing.
Writing Quality: 4/5. The main argument is clear, though perceptual improvement should be distinguished from gains on every metric, and the optimizer description is questionable.
Value: 4/5. Provides a practical interface for editable video restoration, especially when users can first correct a few keyframes.