SSDD: Single-Step Diffusion Decoder for Efficient Image Tokenization

Conference: ECCV 2026
Paper: ECCV Official Page
Code: https://github.com/facebookresearch/SSDD
Area: Image Generation
Keywords: continuous image tokenization, single-step diffusion decoding, flow matching, perceptual reconstruction, knowledge distillation

TL;DR

SSDD trains a convolutional-transformer pixel-space decoder without a GAN and distills its multi-step teacher into a single-step student, allowing its small model to reduce KL-VAE's rFID from 0.87 to 0.46 while increasing decoding throughput from 705 to 1027 images/s on ImageNet at 256 x 256 with f8c4 latents.

Background & Motivation

Latent image generation typically compresses an image into continuous features, generates those features with a model such as DiT, and decodes them back into pixels. A conventional KL-VAE decoder produces a deterministic output for each latent, using pixel reconstruction, LPIPS, and adversarial losses during training. Lossy compression, however, has already discarded some texture information: several plausible images may share a latent representation. Forcing the decoder to return one answer can average away uncertain details. Two objectives must be distinguished here: matching each source image and making the distribution of reconstructed images resemble the real distribution. PSNR and rFID emphasize these different objectives and are not interchangeable.

Diffusion decoders reformulate reconstruction as generation conditioned on a latent, sampling missing details through noise. DiTo and epsilon-VAE have demonstrated this approach, but iterative denoising increases per-image decoding cost, and some strong results still rely on adversarial training that is difficult to scale stably. Simply reducing sampling to one step is insufficient. The paper finds that one step favors low-level distortion, whereas perceptual metrics often benefit from a moderate number of iterations; skipping these iterations changes the model's output behavior.

The authors therefore focus on the decoder rather than redesigning the entire generator. A hybrid architecture first learns the conditional image distribution stably, after which distillation transfers the behavior of a useful multi-step sampler into one step. A shared encoder keeps different decoder capacities within the same latent space. Core Idea: separate learning a realistic conditional reconstruction distribution from executing it in one forward pass, using GAN-free multi-step teacher training for quality and same-noise, same-latent distillation to retain the teacher's generative behavior.

Method

Overall Architecture

An input image is compressed into a two-dimensional continuous latent by a shared encoder. The hybrid pixel decoder receives the latent, a noisy image, and the noise time, and learns denoising through GAN-free joint training. The trained decoder produces teacher reconstructions with 8 steps, and single-step behavior distillation produces the final SSDD. At inference time, only the latent and one sampled Gaussian noise input are needed, without the teacher, LPIPS network, or DINOv2.

SSDD(8) denotes the multi-step model, whereas SSDD without parentheses denotes the distilled single-step model. Encoder compression and decoding steps are independent choices. In particular, downstream acceleration with f16c4 must not be confused with gains obtained by replacing only the decoder in an unchanged f8c4 latent space.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    X["Input image"] --> E["Shared Encoder"]
    E -.->|Training condition| H["Hybrid Pixel Decoder"]
    X -.->|Noisy image and time| H
    H -.-> L["GAN-Free Joint Training"]
    X -.->|Image and perceptual feature supervision| L
    L -.->|Optimize then freeze teacher| S["Single-Step Behavior Distillation"]
    H -.->|Teacher's 8-step reconstruction targets| S
    E -->|Inference latent| S
    N["Gaussian noise"] --> S
    S --> Y["Single-step student output"]

Solid edges show the data needed for final reconstruction; dashed edges show training and teacher supervision. The Single-Step Behavior Distillation node includes student training and the resulting single-step decoder. For downstream generation, DiT supplies the latent in place of image encoding.

Key Designs

1. Shared Encoder: change decoder capacity without repeatedly changing the latent space

The encoder follows a KL-regularized convolutional design. The notation f8c4 indicates spatial downsampling by 8 and 4 latent channels. The authors first train the encoder jointly with SSDD-M, using multi-scale augmentation to expose it to different image sizes, then freeze it for decoders of different sizes. Sharing occurs within a fixed downsampling-factor and channel-count configuration, not across all compression settings. This avoids creating a different latent space for every decoder experiment and retraining a corresponding downstream generator each time.

SSDD can also be trained for an existing grid-shaped latent space, such as that of a KL-VAE encoder discussed in the paper. Its replacement capability concerns interfaces and adaptation, not a guarantee that arbitrary SSDD weights can decode arbitrary existing encoder outputs. Standard f8c4 provides a 48-fold reduction in element count. Changing to f16c4 while keeping the channel count fixed reduces the number of latent elements by another factor of four, transferring more responsibility for detail reconstruction to the decoder.

2. Hybrid Pixel Decoder: use convolutions for pixels and a middle transformer for capacity

Starting from U-ViT, the model retains four convolutional feature levels with two ResNet blocks per level and three downsampling/upsampling stages. It removes the original attention layers and places a transformer in the middle, operating on tokens corresponding to 8 x 8 pixel regions. The transformer uses GEGLU, an MLP with a 4-fold hidden expansion, and multi-head attention. Relative positional embeddings and a fixed 17 x 17 local attention window allow capacity to increase without full-resolution global attention, while local convolutions handle pixel neighborhoods and detail recovery.

The latent is first upsampled to image resolution and concatenated with the noisy image along the channel dimension. Input-only conditioning can weaken in a relatively deep, narrow network, so the latent also conditions the first AdaGN in ResNet blocks and the first AdaLN in transformer blocks. Time embeddings enter through the second AdaGN in ResNet blocks. Input concatenation directly supplies spatial content, while adaptive normalization keeps conditioning active deeper in the network. Increasing channels and transformer layers scales decoder parameters from 13.4M for S to 153.8M for XL.

3. GAN-Free Joint Training: constrain denoising direction, image perception, and intermediate features

Training linearly mixes a real image with Gaussian noise. Under the paper's time convention, zero corresponds to the image and one to noise, and the decoder predicts a velocity pointing from noise toward the image. The prediction direction and one-step reconstruction relation below come from Section 3.2, rather than being imported from a different diffusion convention:

\[ x_t=(1-t)x+t\varepsilon,\qquad \hat{x}_0=x_t+tD(x_t\mid t,z). \]

Flow matching uses an L2 error to supervise the predicted velocity against the image minus noise. LPIPS constrains the perceptual difference between the one-step image estimate and the source image. REPA uses a two-layer MLP to align tokens from transformer layer 4 with reference DINOv2-B features, helping intermediate representations learn useful visual structure. None of these objectives requires a trained discriminator, but GAN-free does not mean independent of pretrained perceptual networks. The combined objective uses the following weights:

\[ \mathcal{L}=\mathcal{L}_{\mathrm{FM}}+0.5\mathcal{L}_{\mathrm{LPIPS}}+0.25\mathcal{L}_{\mathrm{REPA}}. \]

4. Single-Step Behavior Distillation: learn the teacher's sampled result instead of truncating sampling

LPIPS changes the optimization direction of the original flow-matching objective, so the learned velocity field need not describe straight trajectories. Increasing the sampling step count therefore does not consistently improve results: one step favors PSNR, while metrics such as rFID reach a better compromise at intermediate step counts and may then deteriorate. The authors use 8 steps with a time-spacing parameter of 2, taking larger denoising steps first to establish content and smaller ones later to refine details. The cached gradient derivation and scheduling formula have layout corruption, so this note retains the verifiable mechanism and hyperparameters without reconstructing exact equations.

During distillation, a frozen teacher uses the same latent and Gaussian noise as the student to produce an 8-step reconstruction. Initialized from the teacher weights, the student predicts in one step at fixed noise time 1. Its flow-matching target becomes the teacher reconstruction minus that noise, rather than the source image minus noise, so the student learns how the teacher completes uncertain details. Distillation is not limited to L2 regression: the authors state that they preserve the full training objective. Section 3.3 nevertheless contains an ambiguity: its prose says both flow matching and LPIPS are evaluated against teacher outputs, but its displayed LPIPS expression still uses the source image. The supplied cache does not include the referenced appendix, so it does not establish which LPIPS target the implementation uses.

A Worked Example

For a 256 x 256 RGB image, f8c4 produces a 32 x 32 x 4 latent containing 4096 scalars. The original image contains 196608 scalars, so the stated 48-fold compression is an element-count ratio, not an actual file bitrate.

At reconstruction time, the student receives this latent and an image-sized Gaussian noise tensor. The latent enters both as an upsampled spatial grid and through normalization modulation. The hybrid decoder predicts the velocity in one forward pass and reconstructs at noise time 1. Changing the sampled noise allows different plausible details rather than changing the main content represented by the latent. This stochasticity is intentional and does not imply exact recovery of discarded pixels.

With f16c4, the latent becomes 16 x 16 x 4, or 1024 scalars. DiT then operates on a smaller spatial input, creating an opportunity to accelerate the whole generation pipeline. A corresponding DiT must, however, be trained in that latent space. This is a different experiment from retaining an existing generator and replacing only its final pixel decoder.

Loss & Training

Training noise times follow a logit-normal distribution with location 0 and scale 1, and the minimum noise parameter for flow matching is 0. The authors pretrain on 128 x 128 crops taken after random resizing, fine-tune at the target resolution of 128 x 128 or 256 x 256, and then perform single-step distillation. Evaluations at 512 x 512 and 1024 x 1024 reuse the 256 x 256 model rather than retraining at each resolution.

Joint encoder training followed by encoder freezing, teacher freezing, and student fine-tuning belong to distinct stages, not one continuously end-to-end optimization process. The recipe also includes EMA. Full optimizer and model configuration tables are assigned to Appendix B, but the supplied text ends with the main paper and references, so unverified learning rates or training budgets are not added here.

Key Experimental Results

Main Results

Reconstruction is evaluated on the 50k-image ImageNet validation set. rFID measures the difference between reconstructed and real image distributions; LPIPS and DreamSim measure perceptual distortion, with lower values preferred for all three. Higher PSNR is better. Throughput uses one H200, torch.compile, batch size 128, and 1000 iterations, and must not be treated as a single-image latency measurement.

The following representative results come from Table 1. All listed decoders execute one step.

Latent space	Method	Decoder parameters	Decoding throughput (images/s)	rFID, lower better	LPIPS, lower better	PSNR, higher better
f8c4	KL-VAE	47.2M	705	0.87	0.065	24.11
f8c4	SD-VAE	47.2M	707	0.69	0.061	25.52
f8c4	SSDD-S	13.4M	1027	0.46	0.060	24.08
f8c4	SSDD-M	48.0M	357	0.39	0.055	24.38
f16c4	KL-VAE	36.4M	1176	2.93	Not reported	20.57
f16c4	SSDD-L	85.2M	305	0.96	0.155	17.26

SSDD-S achieves 1027/705, approximately 1.46 times the throughput, summarized as 1.4 times in the paper. SSDD-M has a parameter count similar to KL-VAE but substantially lower throughput, so better quality at matched parameters does not mean faster decoding at matched parameters. With f16c4, rFID improves from 2.93 to 0.96 while PSNR drops from 20.57 to 17.26, illustrating the emphasis on generative perceptual quality rather than pixelwise fidelity.

The generation results in Table 3 measure complete throughput, including DiT-XL/2 latent sampling and pixel decoding. DiT is trained for 400k steps on ImageNet at 256 x 256; classifier-free guidance (CFG), when enabled, uses a scale of 1.375.

Latent space and decoder	gFID without CFG, lower better	Images/s without CFG	gFID with CFG, lower better	Images/s with CFG
f8c4 KL-VAE	19.71	7.05	7.63	3.54
f8c4 SSDD-S	17.77	7.07	8.02	3.55
f8c4 SSDD-M	17.03	6.98	7.63	3.52
f16c4 SSDD-L	16.27	25.95	7.34	13.55
f16c4 SSDD-XL	16.08	25.04	7.12	13.30

With CFG, f16c4 SSDD-L achieves 13.55/3.54, approximately 3.83 times the complete generation throughput of f8c4 KL-VAE, while reducing gFID from 7.63 to 7.34. Without CFG, the corresponding ratio is approximately 3.68, not exactly 3.8 in both settings. The small f8c4 SSDD-S model is worse than KL-VAE with CFG, yielding gFID 8.02 versus 7.63, so the paper's overall advantage should not be restated as superiority in every configuration.

Ablation Study

Table 4 is a cumulative experiment on ImageNet at 128 x 128, not an orthogonal ablation independently removing each component. The selection below covers the architecture, REPA, the complete multi-step model, and distillation. Between the third and fourth rows, KL regularization, time sampling, EMA, and shared training are also introduced.

Config	rFID, lower better	PSNR, higher better	Decoding steps
DiTo-S-LPIPS, 48.3M	3.17	23.10	24
Replace with SSDD-M decoder, 48.0M	2.01	23.38	24
Additionally introduce REPA	1.58	22.82	24
Complete remaining recipe and shared pretraining: SSDD(8)-M	0.97	23.04	8
Add single-step distillation: SSDD-M	1.04	23.28	1

Using the displayed values, the architecture improves rFID by 1.16 and the subsequent REPA addition improves it by 0.43. The original table's delta annotation and prose report 0.44, potentially reflecting unreported precision; these values are not forcibly reconciled here. Distillation increases rFID by 0.07 while reducing 8 steps to 1. In a separate GAN comparison starting from the shared-encoder stage, rFID is 1.07 both before and after adding a GAN. This is not the same comparison performed directly on the final distilled model.

Key Findings

• Architecture and REPA provide the two largest quality improvements in the cumulative ablation, but this does not rule out interactions between components.
• Single-step distillation retains most multi-step quality at a small rFID cost; reducing steps does not automatically produce an equal end-to-end speedup.
• In the cross-dataset evaluation in Table 5, ImageNet-trained SSDD-B achieves rFID 3.62 and LPIPS 0.057 on COCO at 256 x 256, versus 4.65 and 0.063 for KL-VAE. This supports transfer between natural-image datasets, not universal domain generalization.

Highlights & Insights

• Treat reconstruction as conditional generation rather than regression to a unique answer. Sampling missing high-frequency details lets latents focus on structure; this is particularly relevant to generation systems that prioritize visual realism.
• Distill sampling behavior, not merely a denoising function. Sharing noise and latents pairs the student's output with a particular teacher generation, instead of forcing direct one-step recovery of the source image and losing the teacher's learned detail distribution.
• A shared latent space makes decoder comparisons more practical. One encoder supports decoders of different capacities, enabling throughput-dependent deployment choices and reducing the need for a separate downstream model for each decoder configuration.

Limitations & Future Work

• Fidelity and realism are not equivalent. Lower PSNR and SSIM under strong compression may be unacceptable for medical or scientific imaging and exact text recovery; such applications cannot select a generative decoder from rFID alone.
• The main evidence concerns natural images. ImageNet, COCO, and DiT under a fixed training budget do not establish equivalent benefits for video, open-domain text-to-image generation, or other generative backbones. These require separate experiments.
• Inference savings do not make training free. Multi-step teacher training and target generation during distillation still have costs, while REPA and LPIPS require additional networks. The supplied cache does not provide the complete training costs assigned to the appendix.
• Implementation details remain ambiguous. The prose and formula disagree on the LPIPS target during distillation, and some equation layouts and table deltas are problematic. Reproduction should check the original appendix and code rather than treat this account as an exact implementation specification.

Related Work & Insights

• vs KL-VAE / SD-VAE: These use a conventional deterministic pixel-decoding path, whereas SSDD recovers a distribution conditioned on noise. Gains primarily concern rFID and perceptual quality, not every distortion metric.
• vs DiTo / epsilon-VAE: All are diffusion tokenizers. SSDD places capacity in a hybrid architecture, supports training with REPA, and removes iterative deployment cost through distillation. The GAN-free result applies to the tested settings, not a claim that adversarial losses are universally useless.
• vs U-ViT / REPA: The paper does not originate hybrid backbones or representation alignment. It applies them to latent-conditioned pixel decoding and evaluates their combination with single-step distillation. A transferable research direction is jointly optimizing latent compression, decoder capacity, and downstream generation speed rather than minimizing rFID in isolation.

Rating

• Novelty: 4/5. Individual components build on prior work, but GAN-free conditional decoding and single-step behavior distillation form a useful combination.
• Experimental Thoroughness: 4/5. Experiments cover capacity, compression, resolution, generation, and COCO transfer, although the main ablation is cumulative and some details require the appendix.
• Writing Quality: 3/5. The perception-distortion distinction is clear, but the distillation objective and some numerical annotations remain ambiguous.
• Value: 4/5. The approach is directly relevant to efficient continuous image generation, with suitability depending on the application's balance between realistic detail and pixelwise fidelity.