Next-Frame Decoding for Ultra-Low-Bitrate Image Compression with Video Diffusion Priors¶

Conference: ECCV 2026
arXiv: 2603.15129
Code: https://github.com/UnoC-727/NeFIC
Area: Image Compression / Diffusion Models / Video Generation
Keywords: Ultra-Low-Bitrate Image Compression, Video Diffusion Priors, Next-Frame Prediction, Anchor Codec, One-step Generation

TL;DR¶

This paper proposes NeFIC, which reinterprets generative decoding in ultra-low-bitrate image compression (ULB-IC) as a virtual temporal evolution process from a compact anchor to the final reconstructed image. By leveraging a pre-trained Video Diffusion Model (VDM) as a temporal prior for next-frame prediction, and employing a two-stage training strategy (interval adaptation + one-step generation bypass), NeFIC achieves high-quality and highly efficient image compression at extremely low bitrates.

Background & Motivation¶

Background: The goal of ultra-low-bitrate image compression (ULB-IC) is to generate reconstructed results that are visually realistic and semantically consistent with the source image at extremely low bitrates. Generative codecs based on diffusion models (such as PerCo, DiffC, StableCodec, etc.) leverage the generative priors of large pre-trained diffusion models, demonstrating outstanding perceptual realism at extremely low bitrates. These approaches typically inject compressed conditioning signals into the multi-step denoising process of an image diffusion model via mechanisms such as ControlNet.

Limitations of Prior Work: Existing diffusion-based ULB-IC schemes suffer from two fundamental limitations. First, the influence of conditioning signals on the diffusion process is ambiguous, and the uncertainty of standard Gaussian noise initialization collectively leads to semantic drift, which impairs the faithfulness of the reconstructed results to the source image. Second, the iterative denoising process incurs significantly higher computational overhead and decoding latency compared to traditional VAE codecs.

Key Challenge: Current ULB-IC faces a trilemma—simultaneously achieving strong perceptual realism, source-content faithfulness, and practical inference efficiency. Image diffusion models are limited in faithfulness due to implicit conditioning injection methods, while multi-step sampling hinders efficiency.

Core Idea: The key insight of this paper is that pre-trained video diffusion models naturally possess a "defocus-to-focus" temporal transition capability from blurry to sharp. NeFIC introduces this prior into image compression: during decoding, it first reconstructs a compact anchor frame (retaining scene geometry and semantic layout while discarding high-frequency details) and then uses a VDM to model the transition from the anchor to the original image as a next-frame prediction of a two-frame virtual video. To eliminate the inefficiency of multi-step diffusion, a latent bypass is further designed to collapse the multi-step denoising into one-step generation.

Method¶

Overall Architecture¶

The codec pipeline of NeFIC is divided into an encoder side and a decoder side. Encoder: The original image $x$ is encoded and quantized by the Anchor Codec's encoder $\mathcal{E}_{Enc}$ and quantizer $Q$ to obtain the compressed latent code $l$, which is then transmitted. Decoder: An intelligible anchor image $x_{anchor}$ is first reconstructed from $l$ by the Anchor Decoder $\mathcal{E}_{Dec}$ (retaining scene geometry and semantic layout while discarding high-frequency details). Subsequently, a pre-trained video diffusion model (CogVideoX-1.5-5B) is utilized to predict the next frame of a two-frame virtual video sequence, where the anchor serves as the conditioning frame and the original image serves as the target frame, ultimately yielding a high-quality reconstruction $x̂$. The entire generative decoding process is formulated as $\hat{x} = (\mathcal{G} \circ \mathcal{E}_{Dec})(l; \xi)$, where $\mathcal{G}$ is the VDM adapted for single-interval realistic next-frame prediction, and $\xi$ is the noisy latent input of the target frame.

Training is divided into two stages: Stage I performs joint fine-tuning of the VDM and the Anchor Codec to learn the synthesis of high-frequency details from the compact anchor while suppressing unintended temporal artifacts (object motion, scene flickering). Stage II introduces a Latent Bypass mechanism to collapse multi-step diffusion into one-step generation, while incorporating a CLIP-based GAN Loss to enhance perceptual realism.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Original Image x"] --> B["Anchor Encoder<br/>ℰ_Enc + Quantization Q"]
    B --> C["Compressed Latent l"]
    C --> D["Anchor Decoder<br/>ℰ_Dec"]
    D --> E["Anchor Frame<br/>x_anchor (Preserving Geometry/Semantics)"]
    E --> F["Video-VAE Encode<br/>→ z_anchor"]
    F --> G["VDM (CogVideoX)<br/>Next-Frame Prediction"]
    G --> H["Video-VAE Decode<br/>→ Reconstructed Image x̂"]
    C --> I["Latent Bypass<br/>→ z_bypass"]
    I --> G

Key Designs¶

1. Anchor Frame + Next-Frame Prediction Paradigm: Reconceptualizing Image Decoding as Virtual Temporal Transition

Existing diffusion-based ULB-IC methods (e.g., PerCo, DiffC) inject conditioning signals into image diffusion models via mechanisms like ControlNet. The interaction between conditioning and noise is restricted to implicit channel concatenation, which is prone to semantic drift. NeFIC first defines an explicit intermediate state—the anchor frame—which is an intelligible image (reconstructed by a VAE-based Anchor Codec) that preserves scene geometry, semantic layout, and coarse appearance but discards high-frequency details. It then reinterprets generative decoding as a virtual temporal evolution process from the anchor to the final reconstruction: constructing a two-frame virtual video where the anchor is the first frame (conditioning frame) and the original image is the second frame (target frame). The task of the VDM is to predict the next frame from the conditioning frame.

The advantage of this paradigm is that the anchor frame provides a visible, semantically plausible initial state (rather than uninformative Gaussian noise), fundamentally suppressing semantic drift. Meanwhile, the temporal consistency prior of VDMs is naturally suited for the progressive refinement process from "coarse to fine" without requiring an additional ControlNet. This is fundamentally different from methods that predict residuals (such as DIRAC or Add2)—NeFIC exploits the "blur-to-sharp" temporal transition prior inherent in VDMs rather than motion priors.

2. Why Choose VDM over IDM: Temporal Priors and 3D Architecture Advantages

Pre-trained VDMs naturally possess two key capabilities that image diffusion models (IDMs) lack. First, temporal transition prior: VDMs are trained on large-scale video datasets and have learned numerous "defocus-to-focus" temporal patterns—such as the frame-by-frame refinement of parrot feathers from blurry to sharp. NeFIC treats the highly compressed anchor as an early blurry frame and the final reconstruction as the corresponding sharp frame, directly reusing this prior. Second, architectural advantage: The 3D Attention and 3D RoPE of VDMs offer a more native conditioning injection method. While IDMs typically concatenate conditioning and noise in the channel dimension (implicit cross-channel mixing), VDMs achieve explicit adaptive retrieval through token concatenation and 3D Attention—each noise token can directly attend to any anchor token, with 3D RoPE imparting a correspondence bias along the temporal axis to bias attention toward spatially aligned regions. This makes the synthesis of high-frequency details a "structure-anchored refinement" of the coarse layout rather than a black-box random generation.

3. Stage I Interval Adaptation Training: Adapting Multi-Frame Video Generation to Single-Interval Next-Frame Prediction

Directly applying a pre-trained VDM to ULB-IC faces a domain gap: VDMs are trained on progressive temporal evolution across dozens of frames (containing object motion and scene transitions), whereas NeFIC requires a single-interval "coarse-to-fine" transition (only two frames, with no motion). The Stage I interval adaptation is designed precisely to bridge this gap.

Specifically, the original image $x$ is encoded by a frozen Video-VAE encoder $E_{Vid}$ to obtain $\mathbf{z}_0 = E_{Vid}(x)$, and noise is added as $\mathbf{z}_t = \alpha_t \mathbf{z}_0 + \sigma_t \boldsymbol{\epsilon}$. The decoded anchor $x_{anchor}$ is also encoded by $E_{Vid}$ to obtain $\mathbf{z}_{anchor}$. A generic text prompt $p$ ("generating a high-quality image...") is encoded as $\mathbf{z}_{text}$. The inputs to the DiT blocks are the concatenation of three token sets: $\mathbf{z}_{text} \oplus \mathbf{z}_{anchor} \oplus \mathbf{z}_t$. Using $v$-prediction parameterization, supervision is applied only to the token outputs corresponding to the target frame, while the text and anchor token outputs are discarded. The loss function is formulated as:

\[\mathcal{L}_{noise} = \|\hat{\mathbf{z}}_0 - \mathbf{z}_0\|_2^2\]

Meanwhile, an auxiliary anchor loss $\mathcal{L}_{aux}$ (MSE + LPIPS) is introduced to ensure that the anchor remains within the training distribution of the Video-VAE. Only the Anchor Codec and the attention projections ($W_q, W_k, W_v, W_{out}$) of the DiT are fine-tuned via LoRA while the remaining parts are frozen, ensuring high training efficiency.

4. Stage II One-Step Generation Bypass: Collapsing Multi-Step Denoising into One-Step Inference

Although Stage I achieves adaptation, it still requires multi-step (e.g., 50 steps) diffusion sampling, which yields unacceptable decoding latency. The core of Stage II is to establish a latent bypass to construct an information channel between the compression VAE and the generative VAE, enabling high-quality image generation in a single denoising step.

This comprises two sub-designs. Conditional Anchor Encoding: While extracting features $\mathbf{h}_{enc}$, the Anchor Encoder concatenates the original image encoded by the frozen $E_{Vid}$, $\mathbf{z}_0$, along the channel dimension: $\mathbf{h}_{cond} = [\mathbf{h}_{enc}; \mathbf{z}_0]$, biasing the Anchor Codec to preserve scene information and layout semantics consistent with the diffusion prior. Bypass Refinement: The anchor decoding features $\mathbf{h}_{dec}$ are mapped through a Transformer to obtain $\mathbf{z}_{bypass} = T(\mathbf{h}_{dec})$, which replaces pure Gaussian noise as the initial latent code for one-step generation. Treating $\mathbf{z}_{bypass}$ as a partially denoised state, the denoising step is set to $t^\star = 500$ (half of the maximum steps), and the VDM predicts the noise in a single step to reconstruct $\hat{\mathbf{z}}_0$. Finally, the Video-VAE decoder $D_{Vid}$ outputs the final image $\hat{x} = D_{Vid}(\hat{\mathbf{z}}_0)$. In addition, a CLIP-based GAN Loss is introduced to further bridge distribution discrepancies.

Overall loss: $$\mathcal{L}_{stageII} = \mathcal{L}_{RGB} + \lambda_{aux} \mathcal{L}_{aux} + \lambda_R R$$

where $\mathcal{L}_{RGB} = \lambda_{GAN} \mathcal{L}_{GAN} + \lambda_{MSE} \|\hat{x} - x\|_2^2 + \lambda_{LPIPS} \text{LPIPS}(\hat{x}, x)$.

Loss & Training¶

Both stages of training are conducted on the Flickr2W dataset, with random cropping of 512-768px square patches and a batch size of 8. Stage I is trained for 10k steps with lr=1e-4. Stage II is trained for 50k steps, introducing the CLIP-based GAN Loss at 35k steps and decaying the learning rate to 1e-5 at 45k steps. Hyperparameters: $\lambda_{aux}=0.1$, and $\lambda_R \in \{5, 3, 1.7, 1.0, 0.5, 0.25\}$ to control the bitrate. The default LoRA rank/alpha is 256/256. The total training time for both stages is approximately 3 days on A100 GPUs. The AdamW optimizer is employed.

Key Experimental Results¶

Main Results¶

The following is a comparison of BD-rate (%, lower is better, with DiffC as the baseline) across three datasets between NeFIC and various generative codecs:

Dataset	Metric	NeFIC (Ours)	StableCodec (ICCV'25)	DLF (ICCV'25)	Gain (vs Second Best)
Kodak	LPIPS BD-Rate	-64.03	-42.19	-42.31	Decreased by 21.84%
Kodak	PSNR BD-Rate	-34.26	-22.05	7.77	Decreased by 12.21%
DIV2K	LPIPS BD-Rate	-61.71	-27.31	-20.07	Decreased by 34.40%
DIV2K	FID BD-Rate	-49.94	-52.06	-25.71	— (StableCodec slightly better)
CLIC2020	LPIPS BD-Rate	-64.09	-27.39	-22.51	Decreased by 36.70%
CLIC2020	KID BD-Rate	-75.94	-73.46	-67.93	Decreased by 2.48%

Model	Encoding/Decoding Time (Kodak)	Encoding/Decoding Time (DIV2K)	BD-Rate LPIPS (DIV2K)
PerCo	0.39s / 1.92s	0.62s / 38.29s	266.60
DiffEIC	0.23s / 4.82s	2.42s / 58.46s	106.71
StableCodec	0.11s / 0.14s	18.86s / 40.63s	-27.31
NeFIC	0.17s / 0.88s	0.58s / 6.83s	-61.71
w/o Stage II	0.16s / 30.61s	0.51s / 296.02s	-47.01

Ablation Study¶

Configuration	LPIPS BD-Rate	FID BD-Rate	Description
NeFIC (Full)	-61.71	-49.94	Full model
w/o Interval Adaptation	-52.32	-40.32	Skip Stage I, BD-Rate degrades by 8-10%
w/o One-step Bypass	-47.01	-53.72	Stage I only (50-step sampling), LPIPS decreases by 15%
w/o Conditional Encoding	-56.81	-42.01	Conditional encoding removed, LPIPS decreases by 4.9%
w/o Bypass Refinement	-48.01	-39.36	Bypass refinement removed, LPIPS/FID degrade by 10-14%
w/o Aux Anchor Loss	-19.78	-31.07	Without auxiliary anchor loss, severe degradation
w/o CLIP-based GAN	-63.09	-30.31	Without adversarial training, FID/KID severely degrade (+20/41)
LoRA rank=64/64	-44.18	-59.91	Under low rank, LPIPS degrades but FID improves
LoRA rank=256/256	-61.71	-49.94	Default setting (LPIPS optimal)

Key Findings¶

Auxiliary Anchor Loss is Indispensable: Removing $\mathcal{L}_{aux}$ causes the anchor to deviate from the natural image domain (leading to semantic/color drift), causing the BD-rate LPIPS to drastically degrade from -61.71% to -19.78%. This is the most severe single-item degradation among all ablations, demonstrating that keeping the anchor within the Video-VAE distribution is a prerequisite for effective VDM conditioning.
Effect and Trade-off of Stage II One-Step Bypass: Compared to using Stage I only (with 50 denoising steps), Stage II improves LPIPS/DISTS by approximately 15%, but with a slight degradation in FID/KID of about 4%/7%, showing a minor trade-off between perceptual faithfulness and realism. More importantly, the decoding latency drops from 30.61s (Kodak) to 0.88s—achieving a 35x speedup, which significantly enhances practicality.
VDM Paradigm vs. IDM Paradigm: In comparison to replacing the model with Flux-dev-12B (a 12B parameter IDM), NeFIC (5B parameter VDM) achieved significant leads in LPIPS/DISTS, indicating that the gains from the VDM's temporal architecture and its next-frame prediction paradigm outperform mere model scaling.
CLIP-based GAN Loss is Crucial for Realism: Removing it results in an FID/KID degradation of approximately 20 and 41 BD-rate points, respectively. Although LPIPS/DISTS improve slightly, generative realism drops significantly.
Configuration of LoRA Rank and Alpha: Increasing from 64/64 to 256/256 consistently improves LPIPS (-44.18% to -61.71%) but degrades FID (-59.91% to -49.94%), indicating that higher ranks prioritize pixel-level faithfulness (LPIPS) over distribution-level realism (FID).

Highlights & Insights¶

Redefining image decoding as temporal evolution is the most elegant conceptual leap: instead of "using VDM to generate images", it "treats the decoding process as a virtual temporal transition from anchor to target". This directly reuses the defocus-to-focus temporal prior of VDMs without requiring an additionally designed conditioning injection network. This approach can be transferred to other tasks that require "coarse-to-fine" refinement (e.g., image super-resolution, inpainting).
Leveraging the in-context learning capability of VDM for conditioning injection: achieving explicit adaptive retrieval via token concatenation and 3D Attention is more natural and robust than the channel concatenation in IDMs. The alignment bias of 3D RoPE is an inductive bias unique to VDMs, which is highly worthy of further investigation in future work.
The design of Latent Bypass is simple yet highly efficient: mapping the compressed VAE's latent codes to the generative VAE's latent space collapses multi-step denoising into a single step, while $\mathbf{z}_{bypass}$ acts as a partially denoised state carrying rich semantic information, avoiding the ambiguity of Gaussian noise initialization. This design scheme of "establishing an information channel between two latent spaces" can be generalized to other tasks that merge compression and generative models.
Delicate design of the auxiliary anchor loss $\mathcal{L}_{aux}$: by simultaneously using MSE (preserving pixel-level encodability) and LPIPS (preserving semantic consistency) with weights of $\lambda_{MSE}=5, \lambda_{LPIPS}=1$, it ensures the anchor remains within the Video-VAE distribution while retaining sufficient semantics. This is a small but critical empirical finding for intermediate representation design in pipelined architectures.

Limitations & Future Work¶

Dependency on model scale: The current scheme is bound to CogVideoX-1.5-5B. Although it provides an excellent VDM prior, it also means the model size is large (5B), posing deployment challenges in resource-constrained scenarios. Future work could explore lighter VDM architectures.
Limited backward compatibility with pure image diffusion models: The comparison with Flux-dev-12B in the paper shows that IDMs struggle to reuse the 3D temporal prior of VDMs, indicating that this paradigm requires native VDM support. If VDM inference is not yet widely deployed, the entry barrier for this paradigm remains high.
Room for improvement in the realism of one-step generation: Stage II shows a slight degradation in FID/KID compared to Stage I (with 50-step sampling), indicating an inherent loss in distribution-level realism during one-step generation. Combining distillation or consistency models might further bridge this gap.
Dataset limitations: Training is conducted only on Flickr2W (20,000 images), which is relatively small. Testing is focused on Kodak/CLIC2020/DIV2K, lacking validation of generalization capabilities on unconstrained, open-world natural image distributions.
Bitrate range limited by Anchor Codec: The current scheme is restricted to ultra-low bitrates because the anchor must be sufficiently "compact" to yield the temporal transition value of VDMs. For medium/high bitrates, this paradigm might require adjustment or even redesign.

vs. PerCo / DiffC (IDM-based ULB-IC): These methods inject conditioning into image diffusion models via ControlNet or channel concatenation. The core difference of NeFIC lies in adopting the VDM’s next-frame prediction paradigm, offering a more explicit and controllable conditioning injection (token-level 3D Attention) while avoiding the ambiguity of Gaussian noise initialization. The advantages are superior faithfulness and a 5.5x–8.6x decoding speedup.
vs. StableCodec / DLF (Latest SOTA ULB-IC): StableCodec (ICCV'25) also explores stability, but NeFIC achieves an 18.27% lower (better) PSNR BD-rate on Kodak compared to StableCodec, while decoding 6x–40x faster (avoiding tiling overhead on DIV2K).
vs. DIRAC / Add2 (Residual Prediction Methods): These methods predict the residual between the original image and the reconstruction. Instead of predicting residuals, NeFIC leverages the blur-to-sharp temporal prior of VDMs to perform "structure-anchored refinement"—essentially using the diffusion model to generate plausible high-frequency details, rather than executing pixel-level residual correction.
Inspirations for Applying Video Diffusion Models: This work demonstrates that the value of VDMs is not limited to generating long videos; their temporal transition prior (defocus-to-focus) and 3D architectural advantages can be successfully transferred to single-image coarse-to-fine refinement tasks. Future research could explore the general application of VDMs in other low-level vision tasks such as super-resolution, deraining, and inpainting.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Reintroducing VDMs into image compression and redefining it as a virtual temporal evolution is an elegant conceptual innovation. The design of the two-stage training (interval adaptation + one-step bypass) is complete and self-contained.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Quantitative comparisons are thoroughly detailed across three datasets: Kodak, CLIC2020, and DIV2K, comprising five tables (BD-rate in main experiments, efficiency, and three sets of ablation studies), accompanied by qualitative ablation demonstrations for each design aspect (such as the teeth comparison in Figure 7).
Writing Quality: ⭐⭐⭐⭐ The motivation is clearly articulated, and the methodology contains comprehensive pre-requisites (including a dedicated section explaining why VDM > IDM), supported by rich illustrations. One star is deducted because the formula numbering inside parentheses is slightly cluttered (e.g., Equation 1 in the middle of a paragraph), and some subscripts of notations are quite busy.
Value: ⭐⭐⭐⭐⭐ Ultra-low-bitrate compression is a highly demanded direction in practice. NeFIC simultaneously improves faithfulness, realism, and efficiency under a generalizable paradigm, offering distinct academic and practical value.