DC-Gen: Post-Training Diffusion Acceleration with Deeply Compressed Latent Space

Conference: ECCV2026
Paper: Official page / PDF
Code: https://github.com/dc-ai-projects/DC-Gen
Area: Diffusion Models / Efficient Visual Generation
Keywords: Deeply compressed latents, embedding alignment, post-training, LoRA, video generation

TL;DR

DC-Gen aligns input and output interfaces before using LoRA to adapt a pretrained diffusion backbone to a deeply compressed latent space with fewer tokens, retaining comparable quality across generation tasks; the paper reports up to 53.8 times acceleration for FLUX at 4K, but times only the Transformer backbone.

Background & Motivation

Latent diffusion already avoids denoising directly in pixel space, yet the common 8-fold spatial compression still leaves many visual tokens. FLUX.1-Krea represents a 4K image with 65,536 tokens, while an 81-frame 720P video requires 75,600 tokens in Wan2.1. Every diffusion step processes these representations, so reducing the number of sampling steps does not remove the high cost of an individual step. DC-AE can already maintain good reconstruction quality with 32-fold or 64-fold spatial compression. The remaining problem is how to make existing generators use these compact representations.

Replacing an encoder is not simply swapping a compatible file format. Once latent channels, grid dimensions, and feature statistics change, the original patch embedder and output layer often cannot be reused. Randomly initializing these interfaces and immediately fine-tuning the entire pretrained network feeds unfamiliar representations into a deep Transformer. The paper observes feature discrepancies amplifying with depth, blurrier image details, editing models that copy their inputs, and even collapsed video training. Training from scratch avoids the transfer problem but abandons the capabilities and training investment already embodied in the original model.

DC-Gen therefore establishes interface compatibility before recovering generation details, instead of asking the backbone to accommodate unfamiliar inputs, outputs, and latent statistics simultaneously. Core idea: split deep-latent migration into input embedding alignment, output-layer alignment, and LoRA refinement, allowing the old backbone to read and write the new representation before lightly adapting its generative capabilities.

Method

Overall Architecture

During training, the same image or video passes through the original encoder and a deeply compressing encoder, producing two representations that can be compared. Stage one trains only the new patch embedder to match downsampled embeddings from the original model. Stage two freezes the Transformer and jointly trains the new input interface and output layer. Stage three adapts the generative backbone with LoRA to recover detail and conditioning fidelity.

Inference no longer needs the original encoder or the alignment branch: generation iteratively transforms noise in the new latent space, and the corresponding decoder reconstructs the image or video. DC-AE-V, image-to-video conditioning, and the loss correction for guidance-distilled backbones are adaptations for particular model families, not an additional universal fourth training stage.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Training image or video"] --> Old["Original encoder and embedding<br/>Frozen and downsampled"]
    Input --> New["Deep compression encoder<br/>Frozen"]
    Old --> Align["Input embedding alignment"]
    New --> Align
    Align --> Output["Output-layer alignment"]
    Output --> Tune["LoRA refinement"]
    Tune --> Result["Generation in new latents<br/>Reconstruction with matching decoder"]

Key Designs

1. Input embedding alignment: make new tokens familiar to the pretrained backbone

The spatial compression factor \(f\) specifies the reduction along each spatial axis, while patch size \(p\) further groups latent cells into tokens; the two must not be conflated. The image token count is \(N=HW/(fp)^2\). Moving from \(f8p2\) to \(f32p1\) increases spatial compression by a factor of 4 but reduces token count to one quarter, not one sixteenth, because patch size also changes. The latent channel count may increase, but the hidden dimension presented to the Transformer must still match its pretrained backbone.

For the same input \(x\), the original encoder \(E_o\) and deep compression encoder \(E_d\) produce their respective latents. The original patch embedder \(e_o\) yields a denser embedding grid, which is spatially downsampled to the new token grid before becoming the regression target for the new interface \(e_{d,\phi}\). The following notation restates the MSE described in the text. The cached equation has extraction damage; \(S\) is this note's shorthand for the described downsampling operation, not an additional proposed module:

\[ \mathcal L_{\mathrm{align}}(\phi)=\mathbb E_x\left\|e_{d,\phi}(E_d(x))-S\bigl(e_o(E_o(x))\bigr)\right\|_2^2. \]

This stage fixes both encoders and the original interface, updating only the new patch embedder. Alignment occurs in the hidden embedding space rather than forcing two different encoders to produce identical latents. That permits a shorter sequence while preserving input semantics familiar to the backbone. The layerwise distances and generation examples in Figure 2 support the interface-mismatch explanation, but these observations are not a theoretical guarantee for arbitrary latent-space transfers.

2. Output-layer alignment: establish a usable exit into the new latent space while freezing the backbone

After input alignment, the randomly initialized output layer still does not know how to map backbone features into velocity predictions in the new latent space. The Transformer therefore remains frozen while the new patch embedder and output layer are trained jointly with standard flow matching rather than further regression to the old embeddings. This is more than fitting a readout head: the input interface can also adjust under the actual denoising task.

Let \(z_0\) be a clean latent in the new space, \(\epsilon\sim\mathcal N(0,I)\), \(t\in[0,1]\), and \(c\) the text condition. The text explicitly defines the noisy state as \(z_t=(1-t)z_0+t\epsilon\) and the target velocity as \(\epsilon-z_0\). Using these definitions, the extraction-damaged equation can be restated as:

\[ \mathcal L_{\mathrm{FM}}=\mathbb E_{z_0,\epsilon,t,c}\left\|v_{\phi,\theta}(z_t,c,t)-(\epsilon-z_0)\right\|_2^2. \]

After both interfaces have been aligned, the model can already generate semantically reasonable images without backbone fine-tuning. The main text states that input alignment is crucial for recovering pretrained capabilities, whereas output-layer alignment primarily accelerates subsequent convergence. However, their separate experiments are in the unavailable appendix, so precise individual contributions cannot be reported from this cache.

3. LoRA refinement: recover detail from compatible interfaces while respecting backbone-specific conditioning

The final stage performs end-to-end adaptation, adjusting the pretrained Transformer through LoRA rather than training an entirely new generator. The text-to-image experiments use LoRA with rank and alpha both set to 256 on all linear modules. The paper reports 40 H100 GPU-days for the complete DC-Gen-FLUX post-training process, equivalent to 2.5 days on 16 GPUs. Here, lightweight is relative to industrial-scale pretraining; it does not imply a training-free or trivially reproducible single-GPU method.

Existing conditioning mechanisms also need to be respected. FLUX.1-Krea and Z-Image-Turbo use guidance distillation, encoding the CFG scale as a condition. Applying ordinary flow matching directly introduces a biased velocity estimate. The authors propose a corrected objective involving positive text, negative text, and the guidance scale, but operators in Equation 3 are damaged in the local extraction, and its derivation and ablation are relegated to the appendix. This note therefore does not guess the complete equation. In reproductions, the standard objective above must not be applied unchanged to every stage of every backbone.

For video, DC-AE-V uses spatial compression \(f=32\), temporal compression \(t=4\), and \(c=32\) channels. Its chunk-causal temporal design permits bidirectional information within each chunk and causal information flow across chunks, aiming to improve deeply compressed reconstruction while supporting longer videos. The same three training stages operate on spatiotemporal grids. For Wan2.1 image-to-video generation, the input image is repeated 4 times, followed by blank frames to match the target video shape; DC-AE-V encodes this sequence, and its conditioning latents are concatenated with the generation latents. This accommodates the different temporal modeling of the original VAE and the new encoder.

A Worked Example

Consider a \(1024\times1024\) FLUX image. The original \(f8c16\) encoder produces a \(128\times128\) latent grid, and \(p=2\) tokenization yields \(64\times64=4096\) tokens. DC-AE-\(f32c32\) with \(p=1\) instead produces a \(32\times32=1024\)-token grid.

Training downsamples the original \(64\times64\) embeddings to \(32\times32\) to supervise the new patch embedder. Next, the frozen backbone is paired with trainable input and output interfaces to predict velocities in the new latent space. LoRA refinement follows. At inference, the backbone repeatedly processes only these 1024 tokens, and DC-AE reconstructs the original image resolution. This is not low-resolution generation followed by image super-resolution.

For 2K and 4K, the paper further adapts from DC-AE-f32 to DC-AE-f64. With the same \(p=1\), the derived 4K token count falls from the original \(f8p2\) model's 65,536 to 4096. This is a count derived from the configuration, not an assertion that a 16-fold token reduction necessarily yields either a 16-fold or a 53.8-fold measured speedup.

Loss & Training

Text-to-image post-training combines real images from Pexels and Unsplash with synthetic images from the pretrained model. Video training uses 257K synthetic videos generated by FusionX. Editing uses 201K selected pairs from Pico-Banana, with Qwen-Image-Edit first fine-tuned on high-quality data before the DC-Gen pipeline is applied.

To distinguish gains from training data from gains associated with latent migration, the video and editing experiments include Tuned baselines trained on the same data. Complete learning rates, batch sizes, per-stage step counts, and adaptation hyperparameters for every model are not in the main text; they are assigned to the appendix. The available cache ends with the references, so these settings are not invented, and the text-to-image LoRA configuration is not generalized to every task.

Key Experimental Results

Main Results

All latencies measure only the Transformer backbone on one H100 using TensorRT, excluding complete encoding, decoding, and serving overhead. The following selection comes from main-paper Tables 1 and 2. The 1K evaluation uses MJHQ-30K and GenEval; the 4K evaluation uses the Aesthetic-4K / Diffusion-4K evaluation described in the paper.

Resolution	Model	Steps	Latency (seconds)	CLIP Score	GenEval / Aesthetic Score
1K	FLUX.1-Krea-12B	Not listed in Table 1	4.06	27.93	0.69 / N/A
1K	DC-Gen-FLUX	Not listed in Table 1	0.88	28.02	0.71 / N/A
1K	Z-Image-Turbo	Not listed in Table 1	1.92	27.61	0.84 / N/A
1K	DC-Gen-Z-Image	Not listed in Table 1	0.47	27.58	0.84 / N/A
4K	FLUX.1-Krea-12B	20	218.81	Not reported	N/A / Not reported
4K	Diffusion-4K (FLUX.1-WLF)	20	23.76	32.79	N/A / 5.94
4K	DC-Gen-FLUX	20	4.04	35.07	N/A / 6.31
4K	DC-Gen-Z-Image	9	1.91	34.23	N/A / 6.11

The authors report a 53.8-fold FLUX speedup at 4K, whereas dividing the table values gives approximately 54.2. These are not exactly consistent, so the original timings are retained rather than adjusted to reproduce the headline. Original FLUX and Z-Image do not natively support 4K, and the table omits their 4K quality scores. The timing comparison therefore cannot establish acceleration at equal native 4K quality.

The following cross-task selection comes from main-paper Tables 4 and 5. Video rows report VBench overall and semantic scores, while editing rows report English and Chinese GEdit overall scores. These metric families must not be compared numerically against one another.

Task	Model	Backbone latency	Quality metric 1	Quality metric 2
T2V, 720x1280	Wan2.1-T2V-14B Tuned	27.52 minutes	Overall 83.94	Semantic 74.97
T2V, 720x1280	DC-Gen-Wan2.1-T2V-14B	3.58 minutes	Overall 83.97	Semantic 75.83
Editing, GEdit 1K	Qwen-Image-Edit Tuned	57.63 seconds	English O 7.23	Chinese O 7.02
Editing, GEdit 1K	DC-Gen-Qwen-Image-Edit	11.96 seconds	English O 7.20	Chinese O 7.32

Ablation Study

This table follows the qualitative descriptions in Section 3.4 and Figures 4 and 5 without estimating exact scores from plotted curves. All comparisons target DC-AE-f32. Direct Fine-Tuning reuses the backbone, randomly initializes its interfaces, and immediately fine-tunes them jointly, whereas DC-Gen aligns the interfaces first.

Backbone	Direct fine-tuning without embedding alignment	Full DC-Gen	Evidence boundary
FLUX.1-Krea	Relatively stable training but blurrier details	Recovers visual quality close to the pretrained model	Main-text qualitative comparison; no exact tabulated ablation scores
Qwen-Image-Edit	Recognizable structure, distorted details, tendency to copy the input	Performs effective edits while retaining detail	Main-text qualitative comparison; no exact tabulated ablation scores
Wan2.1-T2V-1.3B	Collapses after some training steps, producing near-noise outputs	Stable subsequent fine-tuning with retained generation capability	Main-text training curves and qualitative descriptions

Key Findings

• Interface alignment is not merely a small scoring improvement: for the Wan video model, it is associated with whether training remains stable. Looking only at the image model's final CLIP score would understate this role.
• At 1K, FLUX's CLIP score changes from 27.93 to 28.02 and GenEval from 0.69 to 0.71. These results primarily support quality preservation, not a large quality breakthrough.
• Editing does not improve on every dimension: relative to Tuned, English O changes from 7.23 to 7.20 and Chinese O from 7.02 to 7.32. In the original table, English PQ drops from 7.36 to 7.12 and Chinese PQ from 7.32 to 7.15, showing that acceleration does not guarantee unchanged quality on every axis.
• The image human evaluation uses 75 prompts and 20 user judgments per image pair. Overall visual-quality preferences are 47.4% for DC-Gen, 44.1% for the original model, and 8.5% ties. This supports comparability, but statistical significance is not reported.

Highlights & Insights

• Treating latent migration as an input/output compatibility problem is more targeted than delegating all adaptation to end-to-end optimization. The frozen-backbone stage also tests whether pretrained capabilities remain accessible through the new representation.
• Alignment on embedding grids, rather than pointwise equality of latents, allows spatial resolution and channel configuration to change. This flexibility is central to reusing a pretrained backbone.
• Shortening the token sequence and reducing sampling steps address different sources of computation. Combining them is a plausible direction, but these experiments do not establish lossless composition with arbitrary quantization, caching, or distillation methods.

Limitations & Future Work

• The authors explicitly restrict timing to the Transformer backbone. Deployment evaluation should include autoencoder and text-encoder computation, data transfer, end-to-end latency, and peak memory.
• The original models do not natively support 4K and lack reported quality scores there, so quality equivalence behind the largest acceleration claim is less well established than in the 1K and same-data Tuned comparisons. This is an evaluation limitation, not an invented failure case.
• The local cache lacks the appendix, preventing verification of separate interface ablations, the guidance-correction derivation, detailed DC-AE-V design, and complete hyperparameters. The main text also supplies no exact ablation scores that can reliably be reported here.
• The method still depends on a high-quality compression autoencoder, real or synthetic training data, and a post-training budget. Fine texture, small text, long-video motion, and out-of-domain editing deserve targeted tests; these are potential risks to investigate, not established failures in this paper.

Related Work & Insights

• Relation to DC-AE / DC-AE 1.5: Those methods provide compact latent spaces. DC-Gen primarily addresses stable migration of existing diffusion backbones into them, with the additional DC-AE-V extension for video. These are distinct research problems.
• Compared with PixArt-Σ: Its replacement of SD-VAE-f8c4 with SDXL-VAE-f8c4 preserves interface shapes and allows reuse. DC-Gen changes spatial compression and channel configurations, preventing direct inheritance of input and output layers.
• Compared with OpenSora-2.0: The main text characterizes its f8-to-f32 adaptation as direct fine-tuning and discusses blur and quality degradation. DC-Gen's central addition is pre-adaptation alignment, not a claim to have first used highly compressed video latents.
• Further research insight: When replacing a visual tokenizer, first test capability retention with aligned interfaces and a frozen backbone, then determine how much backbone adaptation is necessary. This is an experimental strategy inferred from the paper, not a validated universal rule.

Rating

• Novelty: 4/5. Three-stage interface alignment turns a concrete mismatch in deep-latent migration into an actionable training problem.
• Experimental Thoroughness: 4/5. Multiple tasks, backbones, and same-data Tuned comparisons are persuasive, but appendix verification and end-to-end timing remain gaps.
• Writing Quality: 4/5. The main argument and failure behaviors are clear; key implementation details depend on the appendix, while local equation-extraction damage separately limits verification.
• Value: 5/5. A practical route to reusing large generators and reducing high-resolution inference costs, subject to measurement under actual deployment conditions.