SimFlow: Simplified and End-to-End Training of Latent Normalizing Flows¶

Conference: ECCV2026
Paper: ECCV Paper
Area: Image Generation
Keywords: latent normalizing flows, fixed posterior variance, end-to-end training, representation alignment, classifier-free guidance

TL;DR¶

SimFlow fixes VAE posterior variance to a nonzero constant so that the encoder, decoder, and latent normalizing flow jointly learn generation-friendly representations, achieving 2.15 gFID after 160 epochs on ImageNet 256×256 class-conditional generation and 1.91 with REPA-E.

Background & Motivation¶

A normalizing flow (NF) maps a complex data distribution to a Gaussian through invertible transformations, then generates data from Gaussian noise by reversing those transformations. Directly modeling high-resolution pixels is expensive, so methods such as STARFlow first compress images with a variational autoencoder (VAE) and learn the latent distribution instead. However, a pretrained VAE often predicts very small variances to make reconstruction easier, leaving each image with an almost deterministic latent representation. NF training on such a latent space can generalize poorly, motivating additional noise augmentation to smooth the training distribution. Once the model learns a noisy distribution, generation must also handle that noise, often requiring decoder adaptation or an additional denoising procedure.

A frozen encoder prevents the system from addressing the problem at its source: decoder fine-tuning cannot necessarily recover information already destroyed by latent noise. Even excellent clean-image reconstruction does not imply that a VAE produces a distribution that a generative model can easily fit. Unfreezing everything appears attractive, but introduces an optimization shortcut: latents from different images move closer together, reducing the generative loss while the representation collapses. The conventional joint objective also needs to balance reconstruction, the generative prior, and posterior entropy, which exert different pressures on variance and latent scale. The paper therefore focuses on stabilizing joint training and co-adapting the latent space and NF, rather than introducing a more elaborate invertible architecture.

The authors reinterpret the VAE's own stochastic sampling as a mechanism for smoothing the training distribution, and prevent the encoder from eliminating that randomness. With a sufficiently large fixed posterior variance, the encoder must preserve differences between images under noise, while the decoder learns from the outset to reconstruct images from perturbed representations. Posterior entropy also becomes independent of model parameters, removing one term that would otherwise require balancing. Core Idea: use fixed nonzero variance as an information constraint, allowing reconstruction to resist latent contraction while the NF generation objective directly shapes the encoder.

Method¶

Overall Architecture¶

Training takes real images and their classes; the VAE encoder predicts only latent means, and samples are drawn around those means with fixed variance. The same sampled latents feed the decoder for image reconstruction and the NF for its data-to-Gaussian training objective. The encoder, decoder, and NF are optimized jointly, without stopping gradients between the NF and encoder. Optional REPA-E adds supervision from an external representation model, but is not required by basic SimFlow. Inference no longer requires a real-image encoder: Gaussian noise passes through the inverse NF, guidance refinement, and VAE decoder to produce an image.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Training image and class"] --> B["Fixed-Variance Sampling"]
    B --> C["Joint Reconstruction and Flow Optimization<br/>Decoder reconstruction + NF likelihood"]
    C --> D["Optional Representation Alignment"]
    T["DINOv2-B image features"] --> D
    D -.->|Training gradients| C
    C -.->|Reuse trained NF| E["Gaussian noise + class<br/>Inverse NF and original CFG"]
    E --> F["Full-Model Score Guidance"]
    F --> G["VAE decoding to generated image"]

The upper solid-arrow path represents training data and supervision, while the lower path represents sampling; dashed arrows distinguish training feedback from reuse of the trained model. Reconstruction here is a constraint for learning generative representations, not a downstream image restoration task. This distinction also explains why the paper belongs to image generation rather than image restoration despite reporting reconstruction metrics.

Key Designs¶

1. Fixed-Variance Sampling: place latent perturbation inside the VAE posterior

For an image \(i\), the encoder predicts the mean \(\mu_\psi(i)\), but no longer predicts a freely shrinking variance. The sampling mechanism is the following relation explicitly described in the paper, where \(\bar\sigma^2\) denotes fixed variance:

\[ x=\mu_\psi(i)+\epsilon,\qquad \epsilon\sim\mathcal N(0,\bar\sigma^2 I). \]

In a standard pretrained VAE, KL regularization typically has a small weight, while reducing variance directly simplifies reconstruction, so sampling noise can nearly disappear. Fixing variance preserves a noise floor that the encoder cannot switch off, assigning each image a region of latent space rather than an almost isolated point. The decoder continuously sees samples from these distributions and learns robustness to latent perturbations from the beginning. The basic training pipeline therefore does not require a separately designed noising schedule and corresponding denoising model outside the VAE.

The mechanism does not eliminate noise; it incorporates noise into posterior sampling and joint reconstruction training. If two images have means that are too close, fixed noise overwhelms their differences and prevents the decoder from reconstructing the two targets separately. Reconstruction error consequently forces the encoder to retain distinguishable information, counteracting pressure from the generation objective to contract the means. This is an explanation of the optimization mechanism, not a guarantee against collapse for every variance and architecture. Excessive variance can oversmooth the latent space and damage both information capacity and final generation quality.

2. Joint Reconstruction and Flow Optimization: make latents decodable and learnable

In the training direction, the NF maps latents \(x\) to Gaussian variables \(z=f_\theta(x)\) and computes latent density through change of variables. The invertible transformation's Jacobian determinant corrects for volume changes, so the objective is not merely mean squared regression of all latents toward the origin. The joint VAE objective uses the density learned by the NF as its latent prior rather than insisting on a fixed simple Gaussian prior. The encoder can therefore change the representation distribution while the NF adapts to it, instead of placing the two models in separate stages without feedback.

A general ELBO contains reconstruction log-likelihood, latent-prior log-density, and posterior entropy. The entropy of a Gaussian with fixed covariance does not depend on its mean, making posterior entropy constant with respect to trainable parameters here. The remaining core objective jointly increases reconstruction log-likelihood and NF latent log-density, with both weights set to 1.0. Actual training also retains perceptual and adversarial VAE losses, so “two terms remain” describes the core ELBO decomposition rather than the entire implementation. The cached ELBO equations have damaged formatting; this account follows the components explicitly described in the prose rather than presenting a guessed full equation as the authors' original.

Joint training also changes the spatial structure of latents, rather than merely increasing the decoder's tolerance to errors. Figure 4 on page 7 reports lower spectral entropy, fewer high-frequency components, lower total variation, and higher autocorrelation. These statistics indicate simpler frequency-domain structure, smoother neighboring tokens, and greater spatial consistency in token sequences. The model uses autoregressive flows, where each token's invertible transformation depends on preceding tokens, making such changes favorable for conditional modeling. These are statistical associations observed by the authors, not evidence that any individual statistic alone determines gFID.

3. Optional Representation Alignment: let semantic supervision reach the encoder

The authors extract image features with DINOv2-B and use a projector to align intermediate NF hidden states with these features. Because the NF and VAE are connected, alignment gradients pass through the NF into the VAE encoder, producing REPA-E-style training. Unlike REPA applied over a frozen VAE, this supervision affects both the generative network and its input representations. It also differs from original REPA-E, which stops the generative loss from reaching the VAE to avoid collapse; SimFlow does not require that gradient stop.

The external model supplies training supervision rather than replacing the invertible flow or generating images step by step during sampling. Basic SimFlow and the REPA-E-enhanced variant must be reported separately because the latter uses additional pretrained representations. In Table 1, alignment improves both generation gFID and reconstruction rFID, indicating that its benefits extend beyond the generator output alone. However, the supplied main text neither establishes equal benefits from all representation teachers nor provides the projector's complete implementation details.

4. Full-Model Score Guidance: use all NF blocks at the final latent

Sampling first uses STARFlow classifier-free guidance (CFG) to generate latents, then applies one score-guided refinement to those generated latents. The authors point out that STARFlow's original CFG design operates only on the last block, leaving earlier blocks less directly involved in guidance. The additional refinement uses the difference between the class-conditional and unconditional NF log-density gradients with respect to the latent, with step size \(\gamma\) controlling its magnitude. These gradients come from the density represented by the full NF, allowing all invertible blocks to contribute rather than only modifying the final block's output rule.

Here, a “score” is a density gradient, not classifier confidence or a separately trained scoring model. Only the refined latent is passed to the VAE decoder; neither the encoder nor representation teacher is a required input to the sampling chain. Figure 7 on page 12 shows that \(\gamma=0.25\) improves the best generation quality obtained by sweeping the original CFG. The refinement still adds gradient computation, so pipeline simplification should not be read as the complete absence of extra sampling steps or computational cost. Because operators in cached Equation 6 are incompletely extracted, its precise algebraic form is not reproduced here; supplementary timing values were also not supplied.

A Worked Example¶

For ImageNet 256×256 class-conditional generation, training sends a labeled image through the encoder to obtain latent means at each position. The default analysis configuration uses 64-dimensional latents and \(\bar\sigma^2=0.5^2\), rather than treating both standard deviation and variance as 0.5. After sampling from the fixed-covariance Gaussian posterior, the decoder learns to reconstruct the image, while the NF learns the distribution of that latent under its class. If the NF encourages different images to approach similar means, reconstruction error under noise exerts opposing pressure on the encoder. With REPA-E enabled, teacher features from the same image additionally constrain intermediate NF representations and affect the encoder through the joint network. At test time, the training image is absent: a class and Gaussian noise determine a new latent through the inverse NF and the two guidance components, and the decoder produces a new image.

Loss & Training¶

The basic VAE adopts the MAR architecture, while the NF builds on TARFlow and follows STARFlow's deep-shallow block configuration. The NF contains 6 blocks with hidden dimension 1152; each of the first 5 blocks has 2 layers, and the final deep block has 46 layers. The main text states that the VAE and NF are trained from scratch; adopting MAR's architecture does not mean freezing and reusing pretrained MAR weights. Unless otherwise stated, training lasts 80 epochs with global batch size 256 and constant learning rate \(1.0\times10^{-4}\). Table 1 adds another 80 epochs for 160 in total, with a cosine learning-rate decay to \(1.0\times10^{-6}\) during the latter half. The ablation tables below should therefore not be treated as results from the final Table 1 model under the same training budget.

Key Experimental Results¶

Main Results¶

The following selection comes from Table 1 on page 9 and evaluates ImageNet 256×256 class-conditional generation. gFID measures the discrepancy between generated and real image distributions, while rFID measures the discrepancy for reconstructed images; lower is better for both, and higher is better for IS. Unguided and guided results are separated, and “Not reported” does not mean zero.

Method	Epochs	Parameters	rFID	gFID unguided	gFID guided	IS guided
STARFlow	320	1.4B	2.73	Not reported	2.40	Not reported
DiT	1400	675M	0.61	9.62	2.27	278.2
REPA-E	800	675M	0.28	1.69	1.12	302.9
SimFlow	160	1.4B	1.21	13.72	2.15	276.8
SimFlow + REPA-E	160	1.4B	1.08	10.13	1.91	284.4

SimFlow improves guided gFID over STARFlow by 0.25, or 0.49 with REPA-E, while both variants use half as many training epochs as STARFlow. However, SimFlow has more parameters than DiT, and an epoch advantage does not establish a proportional saving in wall-clock time or compute. “Best among NFs” does not mean best across all generative paradigms: standalone REPA-E in this table still achieves the lower guided gFID of 1.12. Table 2 on page 10 also reports 2.74 gFID for SimFlow + REPA-E on ImageNet 512×512, compared with 3.00 for STARFlow.

Ablation Study¶

The following table comes from Table 3 on page 12, using the default ImageNet 256×256 analysis setup with fixed variance \(\bar\sigma^2=0.5^2\). Higher PSNR indicates better pixel-level reconstruction fidelity; these results use the default 80-epoch setting rather than the final 160-epoch setting in Table 1.

Latent dimension	rFID	PSNR	gFID without CFG	gFID with CFG
16	4.98	19.94	11.00	3.55
32	2.60	21.42	12.77	2.61
64	1.49	23.01	21.44	2.53
128	0.86	24.61	33.43	3.62

Increasing dimensionality consistently improves reconstruction but not generation: 128 dimensions yield the lowest rFID, yet worse guided gFID than 64 dimensions. Without CFG, 16 dimensions perform best, showing that the “best latent dimension” depends on sampling configuration and cannot be discussed independently of guidance.

The following table comes from Table 4 on page 13 and compares fixed variances under the default analysis setup; superscripts were flattened in the cache and are retained in squared form according to the paper's variance notation.

Fixed variance	rFID	PSNR	gFID without CFG	gFID with CFG
\(0.1^2\)	1.90	22.61	25.54	3.01
\(0.25^2\)	1.55	22.94	21.35	2.57
\(0.5^2\)	1.49	23.01	21.44	2.53
\(0.75^2\)	1.54	22.88	17.76	2.39
\(1.0\)	1.76	22.60	19.36	2.66

The authors adopt \(0.5^2\) as a default balancing reconstruction and generation, but the lowest guided gFID in the table is 2.39 at \(0.75^2\). The claim that a moderate variance works best should therefore be read as a trade-off, not rewritten as the default variance winning every metric.

Key Findings¶

The prose on page 11 and Figure 6 on page 12 report unguided gFID of 67.41 with a frozen fixed-variance VAE and 16.97 with end-to-end training. Fixed variance alone does not replace joint adaptation.
In the same figure, end-to-end noise augmentation gives 20.01, versus 16.97 for fixed variance. The authors explicitly avoid claiming fundamental superiority over all noise augmentation because strength tuning or longer training could narrow the difference.
In the 100K-iteration experiment in Table 6 on page 13, no noise causes collapse, whereas several perturbations enable stable training. This supports the importance of retaining perturbation rather than the exclusivity of one noise design.

Highlights & Insights¶

Fixed variance simultaneously changes sampling, decoder robustness, and the ELBO optimization structure, rather than merely removing a prediction head. It makes latent noise a consistent internal training assumption instead of a separately maintained procedure.
Joint training turns the tension between reconstruction and generation into learnable mutual constraints. Scaling the NF improves both in Table 5 on page 13, also suggesting that latent capacity co-adapts with generator capability.
Full-model score guidance exploits the NF's ability to compute densities. It illustrates how sampling can improve through probability information already available in a model rather than necessarily adding another generative network.

Limitations & Future Work¶

The authors identify large-scale text-to-image training as future work. The existing COCO experiments are preliminary and do not establish open-domain generation ability for complex prompts.
The headline results depend strongly on CFG: SimFlow moves from 13.72 unguided to 2.15 guided gFID. A strong best-guidance result does not eliminate room for improvement in the unguided distribution.
Fixed variance and latent dimensionality interact, so neither arbitrarily increasing noise nor simply maximizing reconstruction quality is sufficient. Reusing defaults across datasets still requires validation.
Training the basic model from scratch must be distinguished from the external pretrained supervision in the REPA-E variant. The latter's gains do not arise entirely without external representation learning.
The supplied cache contains the complete main-text method and experiments but no supplement, and several original equations have damaged operators. Detailed proofs, the full projector implementation, and extra guidance runtime were not independently verified here.
Further work could examine a second stage that freezes the VAE after joint training, and comparisons with matched training compute and guidance-search budgets. These are directions for verification, not completed results of this paper.

Versus STARFlow: STARFlow uses a frozen encoder and latent noise augmentation, whereas SimFlow lets representations co-adapt with the NF under fixed perturbation. The key distinction is whether the encoder adapts to the generation objective, not merely whether the decoder is trained.
Versus REPA-E: Original REPA-E updates the VAE through representation alignment while stopping diffusion-generation gradients; SimFlow uses fixed variance to let NF generation gradients reach the encoder. The two standalone models and their combined variant are distinct experimental objects.
Versus fixed-variance VAEs: The paper acknowledges earlier fixed-variance designs, including σ-VAE for continuous-token autoregressive generation. Its contribution is to explain collapse in joint VAE-NF optimization, establish a simplified framework, and validate generation benefits, not to invent fixed variance itself.
Transfer insight: Preliminary MAR, SiT, and REPA-E experiments on pages 14–15 also observe benefits from joint training. The broader question is whether controlled posterior perturbations can improve both representation decodability and downstream distribution learnability.
Resources: Official paper page; project page supplied by the paper. The project link comes from the main text and was not checked online.

Rating¶

Novelty: 4/5. Fixed variance is not a new component, but its integration with joint latent NF training and the anti-collapse explanation is clear and effective.
Experimental Thoroughness: 4/5. Analyses cover resolution, dimensionality, variance, model scale, and perturbations, while supplementary details and strictly compute-matched comparisons require further verification.
Writing Quality: 4/5. The central logic is concise, but the default variance's overall trade-off must be distinguished from individual table optima.
Value: 4/5. The work offers an understandable joint-training approach for latent generative models, with practical benefits still depending on representation capacity, noise, and sampling guidance.