Representation Fréchet Loss for Visual Generation

Conference: NeurIPS 2026
arXiv: 2604.28190
OpenReview: https://openreview.net/forum?id=9OK87o8Wkf
Code: https://github.com/Jiawei-Yang/FD-loss
Area: Image Generation
Keywords: Visual Generation, Fréchet Distance, One-Step Generator, Representation Learning, Post-Training

TL;DR

By decoupling the population size required for statistical estimation of Fréchet Distance from the batch size used for back-propagation, this work introduces representation-space FD-loss as a scalable post-training objective that substantially improves one-step visual generators and repurposes multi-step denoisers into competitive one-step models without teacher distillation or adversarial training.

Background & Motivation

The Fréchet Inception Distance (FID) has stood for nearly a decade as the gold standard metric for evaluating visual generative models, driving the research community toward ever-lower scores on standard benchmarks like ImageNet. However, optimizing for FID has remained strictly indirect: it serves exclusively as an ex-post evaluation tool rather than a training loss. In principle, every mathematical operation within Fréchet Distance is fully differentiable. In practice, directly optimizing FD has long been deemed intractable due to a fundamental discrepancy between statistical scale and optimization scale: accurately estimating Gaussian means and full-rank covariance matrices in high-dimensional feature spaces (e.g., 2048 dimensions for Inception-v3) requires tens of thousands of samples (such as 50k), whereas back-propagating gradients through such a gigantic population at every iteration causes immediate out-of-memory errors and prohibitive compute costs. Conversely, estimating FD naively within small mini-batches (e.g., 64 to 1024) introduces severe finite-sample noise and covariance rank deficiencies, which degrades generator stability and actively worsens sample quality.

At the same time, the visual generation landscape is confronting two core tensions. First, high-fidelity generative models rely predominantly on multi-step iterative denoising processes (such as Diffusion and Rectified Flow). Existing acceleration approaches that compress these models into single-step generators require intricate multi-stage pipelines, such as score distillation against teacher networks, consistency distillation, or adversarial training with unstable discriminators. Second, a profound metric paradox has emerged in recent benchmarks: state-of-the-art generators report FID scores that already surpass real validation images (for instance, real ImageNet-1k validation images achieve an FID of 1.68 against the training set, whereas multiple recent models reach below 1.2), yet their synthetic samples remain visibly artificial and easily distinguishable by humans, highlighting the blind spots and saturation of the single, aging Inception feature manifold.

The central insight of this paper is that the statistical population scale for distributional estimation does not need to be bound to the optimization batch size. Core idea: Decouple the population size for distributional estimation from the batch size for gradient back-propagation by tracking feature statistics across an extended sample window via feature queues or exponential moving averages (EMA) of moments, making Fréchet Distance a scalable, adversarial-free, and distillation-free distribution-matching objective for generative post-training.

Method

Overall Architecture

FD-loss is designed as a minimalistic distribution-level post-training objective. Before training begins, reference feature statistics—namely the mean vector \(\boldsymbol{\mu}_r\) and covariance matrix \(\boldsymbol{\Sigma}_r\)—are extracted once offline from real images using one or more frozen representation backbones (e.g., Inception-v3, SigLIP2, MAE). During post-training, no real images are loaded or processed. At each training iteration, the generator synthesizes a batch of images from random noise vectors; the frozen feature extractors compute intermediate latent representations; a decoupled estimator (FIFO queue or EMA moments) aggregates these features into an effective population to reconstruct the generative mean \(\boldsymbol{\mu}_g\) and covariance \(\boldsymbol{\Sigma}_g\); and finally, a multi-representation normalized Fréchet Distance loss is evaluated, with gradients back-propagated strictly through the current batch.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Z["Input Gaussian Noise Batch<br/>z ~ N(0, I)"] --> G["Generator Forward Pass<br/>G_theta(z)"]
    G --> F["Frozen Feature Extractors<br/>Extract phi(x_hat)"]
    F --> D["Decoupled Population Estimator<br/>EMA Moments / Feature Queue"]
    D --> S["Construct Generated Distribution<br/>mu_g and Sigma_g"]
    R["Offline Real Data Statistics<br/>(mu_r, Sigma_r)"] --> L["Multi-Representation Normalized FD-loss<br/>Symmetric Eigendecomposition Solver"]
    S --> L
    L -->|Backpropagate to Current Batch Only| BP["Generator Parameter Update<br/>Quality Tuning / Step Repurposing"]

Key Designs

1. Decoupling Population Scale from Optimization Scale: Resolving Covariance Degeneracy and Gradient Explosion

To bridge the gap between requiring large populations for stable moment estimation and small batches for tractable optimization, the authors formulate two lightweight statistical estimators: - Queue-based Estimator: An explicit first-in-first-out (FIFO) buffer of size \(N\) (e.g., 50k to 100k) maintains recently generated features. In each training step with batch size \(B\) (e.g., 1024, where \(B \ll N\)), newly extracted features are enqueued while the oldest \(B\) features are evicted. FD is computed over the entire queue, but gradients are detached for historical entries, flowing exclusively through the current batch \(B\). - EMA-based Estimator: To eliminate the memory footprint of storing tens of thousands of feature vectors, the model maintains exponential moving averages (EMA) directly on the first moment (mean) and uncentered second moment: $\(\boldsymbol{\mu}_{\mathrm{batch}}^{(t)}=\frac{1}{B}\sum_{i=1}^{B}\phi(\hat{\mathbf{x}}_{i}),\qquad\mathbf{M}_{\mathrm{batch}}^{(t)}=\frac{1}{B}\sum_{i=1}^{B}\phi(\hat{\mathbf{x}}_{i})\,\phi(\hat{\mathbf{x}}_{i})^{\top}\)$ $\(\boldsymbol{\mu}^{(t)}_{g}=\beta\boldsymbol{\mu}^{(t-1)}_{g}+(1-\beta)\boldsymbol{\mu}^{(t)}_{\mathrm{batch}},\qquad\mathbf{M}^{(t)}_{g}=\beta\mathbf{M}^{(t-1)}_{g}+(1-\beta)\mathbf{M}^{(t)}_{\mathrm{batch}}\)$ The covariance matrix is recovered at each iteration via \(\boldsymbol{\Sigma}^{(t)}_{g}=\mathbf{M}^{(t)}_{g}-\boldsymbol{\mu}^{(t)}_{g}\boldsymbol{\mu}^{(t)\top}_{g}\). The EMA formulation is not only memory-free with respect to sample counts, but also inherently weights recent samples higher, yielding a strictly more on-policy estimate that scales seamlessly to multi-representation setups.

2. Fast Matrix Square Root via Symmetric Eigendecomposition: Ensuring Numerical Stability in Gradient Backpropagation

Computing the Fréchet Distance involves the matrix square root term \(\text{Tr}\big((\boldsymbol{\Sigma}_{r}\boldsymbol{\Sigma}_{g})^{1/2}\big)\). Standard matrix square root operations on asymmetric products are computationally prohibitive and notoriously prone to ill-conditioned gradients. The authors leverage the cyclic permutation property of matrix trace and pre-compute the symmetric matrix square root of the real data covariance \(\boldsymbol{\Sigma}_r^{1/2}\) offline. During training, the trace term is equivalently converted into the eigenvalue sum of a symmetric positive semi-definite matrix: $\(\text{Tr}\Big((\boldsymbol{\Sigma}_{r}\boldsymbol{\Sigma}_{g})^{\frac{1}{2}}\Big)=\sum_{j}\sqrt{\lambda_j\big(\boldsymbol{\Sigma}_{r}^{\frac{1}{2}}\boldsymbol{\Sigma}_{g}\boldsymbol{\Sigma}_{r}^{\frac{1}{2}}\big)}\)$ By utilizing the GPU-accelerated torch.linalg.eigvalsh routine, the method avoids explicit matrix square root computations entirely, achieving substantial training acceleration and preventing numerical instability during gradient back-propagation.

3. Multi-Representation Normalized FD Optimization and Diagnostic Metric FDr: Correcting Single-Metric Distortions

Raw Fréchet Distances vary across different representation models by orders of magnitude (for instance, FD-MAE vs. FD-DINOv2 differ by over 100×). To jointly optimize across multiple representations without one dominating the gradient, each loss term is self-normalized using a stop-gradient magnitude: $\(\mathcal{L}=\sum_{i}w_{i}\cdot\mathcal{L}_{\phi_{i}},\qquad\mathcal{L}_{\phi_{i}}=\frac{\mathrm{FD}_{\phi_{i}}(\mathcal{R},\,\mathcal{G})}{\texttt{sg}\!\left(\mathrm{FD}_{\phi_{i}}(\mathcal{R},\,\mathcal{G})\right)+c}\)$ where \(\texttt{sg}(\cdot)\) is the stop-gradient operator and \(c\) is a small constant. Combining vision-language (SigLIP2), supervised classification (Inception-v3), and masked autoencoding (MAE)—denoted as FD-SIM—ensures simultaneous alignment over high-level semantic categories, discriminative boundaries, and fine-grained visual textures. Furthermore, the paper introduces the multi-representation normalized distance ratio \(\text{FDr}^K\), which normalizes each model's FD by the validation set's FD in that space: $\(\mathrm{FDr}_{\phi_i}(\mathcal{G})=\frac{\mathrm{FD}_{\phi_i}(\mathcal{G},\mathcal{T})}{\mathrm{FD}_{\phi_i}(\mathcal{V},\mathcal{T})},\qquad \mathrm{FDr}^K(\mathcal{G})=\frac{1}{K}\sum_{i=1}^{K}\mathrm{FDr}_{\phi_i}(\mathcal{G})\)$ This diagnostic metric explicitly demonstrates that outperforming validation data under Inception FID does not imply visual perfection, establishing an objective multi-representation standard.

Loss & Training

Generators are fine-tuned from pre-trained checkpoints. The EMA decay parameter is set to \(\beta=0.999\). Before the first training step, 50k images are sampled on-the-fly from the base model to warm-start the EMA moments, providing an on-manifold initialization. Optimization uses AdamW with \(\beta_1=0.9, \beta_2=0.95\), weight decay of 0, and a cosine learning rate schedule with 5 warm-up epochs. A peak learning rate of \(10^{-6}\) is used for MeanFlow models (pMF, iMF) and \(10^{-5}\) for repurposing multi-step denoisers (JiT). Training runs in bf16 precision with a global batch size of 1024.

Key Experimental Results

Main Results

On ImageNet-1k 256×256 class-conditional generation, FD-loss sets new state-of-the-art results across pixel-space, latent-space, and repurposed multi-step models in a single network evaluation (1 NFE).

Method / Generator	NFE	Space	#Params	FDr⁶ ↓	FID ↓	Inception Score ↑
Real Validation Set (50k reference)	N/A	N/A	N/A	1.00	1.68	232.2
Latent-Space One-Step Models
iMF-XL (Base)	1	latent	610M	8.39	1.82	278.9
iMF-XL + FD-loss (SIM)	1	latent	610M	2.45	0.76	301.3
Pixel-Space One-Step Models
pMF-L (Base)	1	pixel	410M	9.09	2.72	261.7
pMF-L + FD-loss (SIM)	1	pixel	410M	2.09	0.78	309.2
pMF-H (Base)	1	pixel	935M	6.87	2.29	267.2
pMF-H + FD-loss (Incep.)	1	pixel	935M	4.86	0.72	298.5
pMF-H + FD-loss (SIM)	1	pixel	935M	1.89	0.77	310.1
Repurposed Multi-Step Models
JiT-L (50-step multi-step base)	200	pixel	459M	10.73	2.59	288.5
JiT-L (Direct 1-step inference)	1	pixel	459M	214.75	291.59	2.0
JiT-L + FD-loss (SIM)	1	pixel	459M	3.24	0.77	317.3
JiT-H (50-step multi-step base)	200	pixel	953M	7.66	1.97	296.0
JiT-H + FD-loss (SIM)	1	pixel	953M	2.65	0.75	313.0

Ablation Study

Ablation experiments on pMF-B/16 post-trained for 50 epochs on ImageNet-1k 256×256:

Config / Variant	Setting / Representations	FID ↓	IS ↑	FDr⁶ ↓	Note
Base Model (No Post-Training)	N/A	3.31	254.6	13.70	Untuned pMF-B baseline
Batch-wise Estimation (\(N=0\))	\(\beta=0\) / batch size 1024 only	3.84	250.9	17.06	Severely degraded due to covariance noise
Queue Estimator	Queue size \(N=5\text{k}\)	1.05	280.0	11.89	Decoupling rapidly stabilizes training
Queue Estimator	Queue size \(N=50\text{k}\)	0.89	288.3	10.91	Optimal balance point for feature queues
Queue Estimator	Queue size \(N=500\text{k}\)	1.22	294.4	17.67	Outdated cached features cause policy drift
EMA Estimator	Decay factor \(\beta=0.9\)	0.98	283.6	11.19	Smooth running moment estimation
EMA Estimator	Decay factor \(\beta=0.999\) (Default)	0.81	294.5	10.81	Minimal memory and best on-policy tracking
Single Representation	Inception only	0.81	294.5	10.81	Superb FID but limited cross-model gain
Single Representation	DINOv2 only	4.89	347.1	8.47	Higher FID but superior structural fidelity
Multi-Representation	SigLIP + Inception	0.89	307.5	5.95	Balances discriminative and text-aligned cues
Multi-Representation	SigLIP + Inception + MAE (SIM)	0.94	307.8	4.20	Broadest perceptual improvement

Key Findings

• Decoupling is an Absolute Prerequisite: Directly computing FD within an optimization batch of 1024 degrades the base model's FID from 3.31 to 3.84 due to severe estimation noise in high dimensions. Decoupling population scale via EMA at \(\beta=0.999\) drives the FID down to 0.81 without extra memory overhead.
• The Pitfall of Single-Metric Inception Tuning: Optimizing strictly against Inception features produces a misleadingly low FID of 0.81 while leaving \(\text{FDr}^6\) elevated at 10.81. Conversely, incorporating modern ViT encoders (MAE, SigLIP) significantly improves structural realism and human visual preference, despite scoring higher in raw Inception FID.
• Distillation-Free Multi-Step Repurposing: JiT models, which completely fail when prompted directly at step 1 (FID 291.59), are converted into top-tier 1-step generators (FID 0.75, \(\text{FDr}^6\) 2.65) solely via FD-SIM post-training, outperforming the original 50-step Heun solver baseline without teacher distillation or adversarial losses.

Highlights & Insights

• Repositioning Evaluation into Minimalist Supervision: The paper challenges the ten-year assumption that Fréchet Distance cannot serve as an effective training objective, proving that simple temporal decoupling transforms it into a robust, scalable loss function.
• Data-Free Post-Training Protocol: Post-training requires zero loading, decoding, or augmentation of real images. Storing only lightweight pre-computed distribution statistics (means and covariances) enables friction-free, I/O-independent model tuning.
• Diagnostic Benchmark FDr: Normalizing Fréchet Distances against the real validation set across multiple architectures exposes the fact that the ImageNet generative problem is far from solved, providing the community with a more holistic and reliable benchmark.

Limitations & Future Work

• Authors' Acknowledged Limitations: The approach relies fundamentally on the Gaussian assumption of Fréchet Distance, meaning it primarily matches first and second moments and cannot explicitly constrain higher-order multi-modal density structures; furthermore, the \(\text{FDr}^K\) metric remains dependent on the chosen set of representation extractors.
• Identified Research Scope: Representation models may inject inductive biases that allow generators to exploit feature-space shortcuts. Scaling this distribution-level matching to ultra-high resolutions or continuous video manifolds requires handling larger feature spatial dimensions.
• Promising Follow-up Directions: Extending the decoupling principle to higher-order tensor moments or Maximum Mean Discrepancy (MMD); and applying FD-loss alongside preference optimization to steer the visual aesthetic of large text-to-image foundation models.

Related Work & Insights

• vs Score Distillation (e.g., DMD, SiD, ADD): Score distillation requires running heavy teacher diffusion models during training and often relies on auxiliary adversarial discriminators. FD-loss requires zero teacher inference and zero adversarial games, depending purely on frozen static feature statistics.
• vs Batch-wise Moment / Fréchet Matching (e.g., Bińkowski et al., Doan et al.): Earlier attempts computed distributional metrics within single training batches, suffering from rank-deficient covariances and unstable gradients. Decoupling population scale via EMA solves this long-standing bottleneck.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Decoupling population estimation from gradient batch size elegantly converts Fréchet Distance into a practical, large-scale training objective.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation spanning pixel and latent spaces, one-step and repurposed multi-step models, extensive ablations, and blind human evaluation.
• Writing Quality: ⭐⭐⭐⭐⭐ Lucid argumentation, crisp conceptual framing, and insightful analysis of metric paradoxes.
• Value: ⭐⭐⭐⭐⭐ Establishes a highly efficient, data-free post-training paradigm with broad utility across generative pipelines.

Related Papers

• [NeurIPS 2026] Improved Baselines with Representation Autoencoders
• [ICCV 2025] Pretrained Reversible Generation as Unsupervised Visual Representation Learning
• [ECCV 2026] MEPA: Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Experts
• [ICLR 2026] Condition Errors Refinement in Autoregressive Image Generation with Diffusion Loss
• [ECCV 2026] Towards Scalable Pre-training of Visual Tokenizers for Generation