ViBe: Ultra-High-Resolution Video Synthesis Born from Pure Images

Conference: ECCV2026
Paper: ECCV paper
Area: Video Generation
Keywords: ultra-high-resolution video, Relay LoRA, global coarse local fine attention, high-frequency reconstruction, flow matching

TL;DR

ViBe adapts an existing video DiT using images alone, removing the modality adapter while retaining the spatial adapter at inference, and combines sparse attention with high-frequency reconstruction training to reach a 74.4% six-metric mean in Table 1's 4K setting, 0.8 percentage points above CineScale.

Background & Motivation

Video DiT attention jointly processes spatial and temporal tokens, so increasing resolution sharply raises computation and memory requirements; retaining multiple training frames increases the burden further. High-resolution images are easier to obtain than corresponding videos and avoid the temporal dimension of training sequences. This makes them attractive for upgrading existing video generators. The objective, however, is not to learn motion from static images: it is to preserve motion capabilities acquired during video pretraining while adding spatial detail capacity.

Directly training a LoRA on high-resolution images and applying it to video does not guarantee that separation. The adapter learns both a change from video to single-frame images and a change from low to high resolution. Applying both changes during video inference can introduce noise and artifacts. Even after reducing this modality interference, global attention may not recover fine textures effectively, while a conventional flow-matching objective does not explicitly demand recovery of high frequencies lost during upsampling.

Core Idea: let an auxiliary LoRA absorb the video-to-image modality change, train a second LoRA for spatial upscaling on that adapted backbone, and retain only the second adapter for video generation, supported by fine-local attention and clean-latent reconstruction supervision.

Method

Overall Architecture

The input is a text prompt and the output is an ultra-high-resolution video. Inference follows an established coarse-to-fine strategy: generation at the model's native resolution establishes subjects, layout, and motion, after which a high-resolution version is generated from that initial video. ViBe does not require high-resolution videos for its adaptation; Relay LoRA enables learning from still images.

Stage 1 trains LoRA1 on low-resolution images. Stage 2 merges and freezes the adapted backbone, adds LoRA2, replaces self-attention with Global Coarse Local Fine Attention (GCLFA), and uses the High Frequency Awareness Training Objective (HFATO). These designs address modality interference, spatial attention coverage, and detail-recovery supervision, respectively. The middle chain below represents the architecture and objective used in Stage 2, not attention and loss operating as separate generators.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Images["Low- and high-resolution images"] --> Relay["Relay LoRA"]
    Relay --> Attention["Global Coarse Local<br/>Fine Attention"]
    Attention --> Objective["High Frequency Awareness<br/>Training Objective"]
    Objective --> Adapter["Keep LoRA2<br/>Remove LoRA1"]
    Prompt["Text prompt"] --> Native["Native-resolution video"]
    Native --> Refine["High-resolution refinement<br/>Output video"]
    Adapter --> Refine

Key Designs

1. Relay LoRA: separate a training-time modality bridge from deployable spatial capacity

Stage 1 freezes the base video model and trains LoRA1 on low-resolution single-frame images, adapting it to inputs without temporal variation. Stage 2 merges LoRA1 into the base weights, freezes the resulting backbone, and trains a new LoRA2 on high-resolution images. Because this starting point already handles the image modality, the new parameters can focus more on resolution adaptation instead of having to absorb the entire video-to-image shift again.

The key is not merely training twice, but declining to deploy the complete training model. Let \(W^{(0)}\) denote the base weights and \(\Delta W_1\) and \(\Delta W_2\) the two adapter updates. The weight relationships in Eqs. (5) and (6) can be summarized as:

\[ W_{\mathrm{stage2}}=W^{(0)}+\Delta W_1+\Delta W_2, \qquad W_{\mathrm{infer}}=W^{(0)}+\Delta W_2. \]

Only \(\Delta W_2\) is optimized in Stage 2, and \(\Delta W_1\) is removed for inference. LoRA1 acts as a temporary environment for learning the spatial adapter: it helps Stage 2 avoid modality mismatch, but the final task is still video, so it is not deployed alongside LoRA2. This is an empirical disentanglement strategy, not a proof that the updates are orthogonal in parameter space or that motion capabilities remain completely unaffected.

2. Global Coarse Local Fine Attention: retain full local windows and compressed global context

Coarse global representations alone can lose detail, while local windows alone can leave different regions without a shared representation of subjects and layout. GCLFA replaces all self-attention layers in Stage 2. Its local branch retains fine tokens but restricts each query to a nearby window. Its global branch pools keys after RoPE and pools the corresponding values, producing coarse tokens covering the whole image. Fine and pooled keys/values are concatenated, and an attention mask governs access so that each query can use both local textures and global semantics.

The local branch also addresses boundary behavior. A conventional sliding window is truncated near an image edge, leaving boundary queries with fewer visible tokens. ViBe shifts an overflowing window inward to preserve the window size associated with the native scale, rather than shrinking it or wrapping across the boundary. The paper gives a corresponding 480P window for a 480P model as an example. This preserves spatial receptive-field size; it is not an additional mechanism for learning cross-frame motion correspondences.

The global branch remains necessary because local high-frequency information alone cannot reliably determine distant object layout. PyTorch FlexAttention implements the sparse access pattern, reducing full high-resolution interactions. The main results do not report generation latency for a fixed-duration 4K video, so attention sparsity should not be converted into an unreported end-to-end speedup.

3. High Frequency Awareness Training Objective: remove input detail while supervising the complete target

HFATO first downsamples and then upsamples a clean image latent, restoring its spatial dimensions while weakening high-frequency content. Gaussian noise is added only after this degradation. The model predicts a flow from the degraded, noisy input and uses it to reconstruct a clean latent. Crucially, the target is the original clean latent, not its degraded version. Removing random noise without recovering texture is therefore insufficient to satisfy the objective.

This matches the coarse-to-fine inference task: a low-resolution video provides structure but lacks all the desired detail, so the second pass must address more than noise corruption. HFATO combines latent degradation with \(x_0\) reconstruction supervision. Table 2 removes either the entire HFATO or just the \(x_0\) Loss to distinguish making the input harder from explicitly supervising the clean result. The cached transcription of Eq. (11) has missing symbols in its noising expression; this note follows the surrounding prose instead of supplying an unverifiable noise parameterization.

A Worked Example

Consider a video requiring fine facial detail. Native-resolution generation first determines the face's location, identity, and movement, and high-resolution refinement then follows this coarse structure. The deployed model contains the base weights plus LoRA2, not the sum of both adapters, avoiding direct deployment of the single-frame modality adaptation from Stage 1.

During facial refinement, local windows model skin texture while pooled global tokens help keep that texture consistent with the whole face. If the face lies near an image boundary, the window moves inward instead of being truncated. HFATO has already trained the model to recover original details from high-frequency-deficient latents. Figure 7 uses skin wrinkles to illustrate local detail differences, but this does not establish pixel-level fidelity to a real person's appearance.

Loss & Training

The training set contains 2.3K images generated by FLUX 1.1 Pro Ultra at \(2752\times1536\). Stage 1 uses low-resolution images and standard flow matching for 3K iterations. Stage 2 uses high-resolution images and the reconstruction objective in Eq. (11) for another 3K iterations. Adaptation targets parameters in attention layers rather than fully retraining the video backbone.

The authors report that both stages finish in about one day on a single A100. Inference also uses a single A100 unless otherwise specified. LoRA rank, optimizer, and scheduler follow DiffSynth-Studio defaults; the main text does not enumerate these values, so a reference to defaults should not be presented as a fixed, independently verified configuration.

The base models include Wan2.2-5B and 14B. The former has a native resolution of \(1280\times704\) and VAE downsampling factors of \(16\times16\times4\). The latter uses mixed resolutions of \(832\times480\) and \(1280\times720\), with factors of \(8\times8\times4\). These models already rely on video pretraining; the image-only restriction applies to ViBe's additional adaptation.

Key Experimental Results

Main Results

Evaluation randomly selects 100 prompts from the standard VBench prompt suite and uses 5 seeds per prompt for each method, producing 500 videos per resolution setting. Figure 5 and the user study explicitly use Wan2.2-5B. Table 1 labels its backbone only as Wan2.2 and does not separate 5B and 14B results, so it should not be interpreted as establishing the same comparison independently for both sizes.

The table below selects 4K results from the paper's Table 1. All values are percentages, with higher being better. Table 1 specifies \(3840\times2176\), whereas Section 4.1 writes \(3820\times2176\). This note follows the results table's \(3840\times2176\) and does not silently replace the inconsistent dimensions with standard UHD dimensions.

Method	Aesthetic Quality	Imaging Quality	Overall Consistency	Overall Score
Wan2.2	59.1	33.9	13.6	65.5
Real-ESRGAN	59.8	58.3	24.3	72.0
Upscale-A-Video	61.1	65.7	25.8	73.5
I-Max	57.0	61.8	26.9	72.5
HiFlow	56.6	54.4	27.0	71.5
CineScale	60.1	66.3	25.1	73.6
T3-Video	60.8	64.8	24.7	72.8
ViBe	61.4	66.1	27.1	74.4

Table 1 defines Overall Score as the arithmetic mean of six metrics. The other three are subject consistency, background consistency, and motion smoothness. It is neither the mean of the three displayed component columns nor interchangeable with the complete official VBench aggregate. ViBe's remaining scores are 95.2%, 97.1%, and 99.2%, respectively:

\[ S_{\mathrm{overall}}=\frac{1}{6}\sum_{j=1}^{6}s_j. \]

At \(1920\times1088\), ViBe scores 74.3%, compared with Wan2.2 at 72.5% and Real-ESRGAN at 72.6%. At 4K, ViBe exceeds CineScale by 0.8 percentage points, but its imaging quality of 66.1% is below CineScale's 66.3%; it does not rank first on every metric.

Ablation Study

The following results come from Table 2. Its full-model values match the 4K row in Table 1, but the Table 2 caption does not independently restate resolution or model size. These should be treated as the corresponding reported ablations without adding unspecified run settings. DR denotes the GCLFA downsample ratio discussed in Section 4.4, not a confirmed hyperparameter of HFATO's degradation operator.

Config	Aesthetic Quality	Imaging Quality	Overall Score	Score Difference from DR = 1/2
w/o Relay LoRA	46.7	37.4	66.0	-8.4 percentage points
w/o GCLFA	52.2	49.0	69.1	-5.3 percentage points
w/o HFATO	53.7	59.0	70.8	-3.6 percentage points
w/o X0 Loss	57.3	59.2	72.3	-2.1 percentage points
ViBe, DR = 1/2	61.4	66.1	74.4	0.0 percentage points
ViBe, DR = 1/4	61.3	65.6	74.2	-0.2 percentage points

Figure 7 defines w/o Relay LoRA as attaching only LoRA1 to the base model, whereas the prose discusses this comparison in terms of direct high-resolution LoRA adaptation. These descriptions are not fully consistent. The 8.4-point gap is therefore an observed difference between the named row and the full model, not an unambiguously isolated causal effect.

Key Findings

• Removing Relay LoRA corresponds to the largest aggregate gap, with imaging quality falling from 66.1% to 37.4%, supporting modality interference as a major risk.
• Removing \(x_0\) Loss leaves a score of 72.3%, suggesting that degraded inputs alone do not replace explicit reconstruction supervision. The component gaps should not be added because the mechanisms may interact.
• DR = 1/4 scores 74.2%, slightly below 74.4% for DR = 1/2. Based on local visual examples, the authors recommend the former for finer textures and the latter for greater robustness.
• The user study includes 35 participants and reports leading preferences across its criteria. The cache does not expose readable Figure 6 percentages, so no preference rates are supplied. Figure 8 demonstrates integration with 4-step distilled generation and image-to-video generation without a complete numerical comparison.

Highlights & Insights

• A useful training-time component need not be deployed. Relay LoRA uses temporary modality adaptation to support learning, then removes it instead of permanently stacking every learned update onto the base model.
• Boundary handling and global semantics can be addressed together. Inward windows preserve local context for edge tokens, while pooled global tokens keep detail recovery from becoming disconnected regional synthesis.
• HFATO changes what the model must solve. It demands recovery of the original latent despite missing high frequencies, matching the deficiency of coarse-to-fine inputs rather than only predicting flow along a noise trajectory.

Limitations & Future Work

• The main paper has no dedicated limitations section. Video-free adaptation does not imply video-free pretraining, nor demonstrate learning new complex motions or long-range temporal causality from images alone.
• The comparison uses 100 prompts and 5 seeds each without confidence intervals. A 0.8-percentage-point mean advantage alone does not establish statistical significance or consistent superiority across scenarios.
• The cache ends at the references and does not contain the additional DiT appendix results mentioned in the text. The available 4-step and image-to-video demonstrations support qualitative observations, not quantitative generalization claims.
• Resolution discrepancies, ablation definitions, and incomplete hyperparameter disclosure leave reproduction ambiguities. Peak memory, inference latency at a fixed frame count, and a unified training-cost comparison with high-resolution video baselines are also missing.
• A useful follow-up would isolate removal of LoRA1 under identical models, prompts, and seeds, then stratify evaluation by motion magnitude, duration, and occlusion. This would test whether spatial and temporal capabilities are genuinely separated; it is a proposed validation direction, not an experiment already reported.

Related Work & Insights

• vs Real-ESRGAN / Upscale-A-Video: These baselines super-resolve low-resolution Wan2.2 videos, whereas ViBe adapts the video DiT's generation process itself. Its emphasis on synthesizing details does not guarantee that those details are faithful recoveries of reality.
• vs I-Max / HiFlow / FreeSwim: The first two are evaluated as training-free high-resolution methods, while FreeSwim motivates local attention. ViBe incurs image-adaptation cost to additionally address modality interference and high-frequency recovery supervision.
• vs CineScale / T3-Video: The paper categorizes these as methods trained with high-resolution real videos. ViBe reduces the additional adaptation's dependence on video data; it does not establish universal superiority under identical data and training budgets.

Rating

• Novelty: 4/5. Two-stage adaptation that discards the modality bridge at inference is distinctive, while coarse-to-fine generation and local attention build on prior work.
• Experimental Thoroughness: 3/5. Multiple baseline families, component ablations, and a user study are included, but uncertainty estimates, full costs, and quantitative cross-backbone evidence are lacking.
• Writing Quality: 3/5. The main mechanism is understandable, but inconsistent resolution and ablation descriptions hinder precise reproduction.
• Value: 4/5. A practical route to spatially upgrading existing video models, with motion and long-video boundaries requiring further testing.