On the Diffusibility of High-Dimensional Latents

Conference: ECCV 2026
Paper: ECCV Official
Project: https://cfeng16.github.io/on_the_diffusibility/
Area: Image Generation
Keywords: text-to-image generation, representation autoencoder, high-dimensional latents, flow matching, x0-prediction

TL;DR

Addressing the optimization failure of standard velocity prediction in high-dimensional representation autoencoders caused by effective dimensionality collapse after reconstruction finetuning, this paper analyzes how orthogonal noise components disrupt diffusion learning and demonstrates that clean data (\(x_0\)) prediction successfully isolates the low-dimensional signal manifold, enabling high-fidelity visual generation without latent compression.

Background & Motivation

Diffusion models and flow matching have become the dominant paradigm for high-fidelity visual synthesis, commonly operating in latent space to maintain computational efficiency. Early approaches rely on latent spaces from variational autoencoders (VAEs), which frequently discard fine-grained perceptual details. Recently, representation autoencoders (RAEs) have emerged to conduct generative modeling directly within the feature spaces of pretrained semantic visual encoders such as DINOv2 and SigLIP2. However, off-the-shelf semantic encoders are not trained for reconstruction and tend to lose high-frequency components such as text, small objects, and intricate textures. To overcome this limitation, finetuning semantic encoders with an image reconstruction loss or adopting reconstruction-oriented representations like MAE has become a natural remedy.

Nevertheless, training diffusion models directly on these reconstruction-tuned high-dimensional feature spaces presents severe optimization hurdles. Under standard flow matching formulations, velocity prediction (\(v\)-prediction) converges exceedingly slowly, struggling even on simple single-image overfitting tests. Existing work typically bypasses this bottleneck by employing an adapter to compress the high-dimensional latents into a low-dimensional bottleneck before diffusion. However, this aggressive dimensionality reduction inherently discards crucial visual nuances and semantic information, leaving the true generative capability of uncompressed high-dimensional reconstruction features largely unexplored.

The core tension stems from a profound geometric mismatch: reconstruction objectives force high-dimensional features to concentrate closely around a lower-dimensional signal manifold, inducing a sharp collapse in effective dimensionality. Under this collapsed geometry, the standard velocity target decomposes into a manifold-aligned signal component and a dominant orthogonal isotropic noise component that provides zero information about clean data. Core idea: identify the effective dimensionality collapse induced by reconstruction tuning, and adopt clean data (\(x_0\)) prediction to mathematically bypass orthogonal noise components, thereby achieving efficient diffusion in uncompressed high-dimensional feature spaces.

Method

Overall Architecture

The proposed framework performs text-to-image diffusion directly in high-dimensional, strong-reconstruction representation spaces. The architecture consists of two primary components: a high-dimensional strong-reconstruction tokenizer that captures fine-grained visual details while retaining semantic structures, and a diffusion transformer (DiT) conditioned on text representations to perform denoising in the uncompressed high-dimensional latent space.

During text-to-image synthesis, an input image \(I\) is encoded by the high-dimensional visual encoder into feature \(h_0 \in \mathbb{R}^h\). The text prompt is processed by a frozen autoregressive vision-language model (Qwen3-VL-2B), and learnable query tokens extract text conditioning embeddings \(y\). The diffusion transformer (a 1B Lumina-Next DiT) takes the noisy latent \(h_t = (1-t)h_0 + t\epsilon\) and text condition \(y\) as inputs, directly parameterizing the clean representation \(x_0\). Finally, the predicted clean high-dimensional representation is fed into the image decoder to synthesize the high-fidelity output image.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Text Prompt"] --> B["High-Dimensional Strong-Reconstruction Extraction<br/>Preserving high-frequency details & semantics"]
    B --> C["Effective Dimensionality Collapse Analysis<br/>Features concentrate near low-dimensional subspace"]
    C --> D["Manifold-Decoupled Clean Data x0 Prediction<br/>Bypassing orthogonal noise interference"]
    D --> E["High-Fidelity Pixel Decoding<br/>Without low-dimensional bottleneck compression"]

Key Designs

1. Effective dimensionality collapse analysis: revealing geometric changes from reconstruction tuning

Pretrained visual encoders focused strictly on semantic representations (such as frozen DINOv2 and SigLIP2) maintain a high effective dimensionality across their full feature space, providing favorable conditions for standard velocity prediction. However, when encoders are finetuned for image reconstruction or pretrained with pixel reconstruction objectives (e.g., MAE), the singular value spectrum of patch representations exhibits rapid decay. To quantify this behavior, singular value decomposition (SVD) is performed on 200k patch embeddings sampled from ImageNet, yielding ordered singular values \(\sigma_1, \dots, \sigma_D\). The cumulative explained energy ratio of the top \(k\) principal components is defined as:

\[C(k) = \frac{\sum_{i=1}^k \sigma_i^2}{\sum_{i=1}^D \sigma_i^2}\]

The effective dimensionality \(R_\tau = \min \{ k \mid C(k) \ge \tau\% \}\) denotes the minimum number of principal components needed to capture at least \(\tau\%\) of the total variance. While frozen DINOv2-L maintains \(R_{90} = 672\) in its 1024-dimensional space, finetuning it for reconstruction causes \(R_{90}\) to plummet to 129; similarly, the 768-dimensional MAE-B possesses an \(R_{90}\) of merely 103. Because natural images lie on a low-dimensional manifold, forcing encoders to retain pixel-level details constrains high-dimensional features into a narrow subspace, triggering severe effective dimensionality collapse.

2. Orthogonal noise interference in velocity prediction: the mathematical bottleneck of high-dimensional diffusion

When clean high-dimensional representation \(x_0 \in \mathbb{R}^h\) concentrates near a lower-dimensional subspace, it can be modeled as a true latent vector \(c_0 \in \mathbb{R}^l\) projected via an orthonormal basis matrix \(Q \in \mathbb{R}^{h \times l}\) (\(Q^\top Q = I_l\)) such that \(x_0 = Q c_0\), where \(h \gg l\). Under flow matching interpolation \(z_t^h = (1-t)x_0 + t\epsilon^h\), the high-dimensional Gaussian noise \(\epsilon^h\) splits into an in-subspace component \(Q\epsilon^l\) and an orthogonal complement component \(\epsilon_\perp = (I - QQ^\top)\epsilon^h\). Consequently, the standard velocity prediction target \(v_\theta^h(z_t^h, t) = \mathbb{E}[\epsilon^h - x_0 \mid z_t^h]\) decomposes into:

\[v_\theta^h(z_t^h, t) = Q v_\theta^l(z_t^l, t) + \frac{1}{t}(I - QQ^\top)z_t^h\]

When \(h \gg l\), the uninformative orthogonal noise component \(\epsilon_\perp\) dominates the total norm in high-dimensional space. The diffusion network is forced to waste substantial capacity and gradient updates fitting the diverging term \(\frac{1}{t}\epsilon_\perp\), severely degrading learning efficiency on the underlying signal manifold and resulting in deformed or amorphous sample geometry.

3. Manifold-decoupled clean data (\(x_0\)) prediction: isolating the signal manifold without dimensional compression

To eliminate the destructive impact of orthogonal noise on optimization, the framework replaces velocity regression with direct clean representation prediction (\(x_0\)-prediction). The conditional expectation target is formulated as:

\[x_\theta^h(z_t^h, t) = \mathbb{E}[x_0 \mid z_t^h] = \mathbb{E}[Q c_0 \mid Q z_t^l + t\epsilon_\perp]\]

Because the clean signal \(Q c_0\) is statistically independent of the orthogonal noise component \(\epsilon_\perp\), the conditional expectation simplifies cleanly to:

\[x_\theta^h(z_t^h, t) = Q \mathbb{E}[c_0 \mid z_t^l] = Q x_\theta^l(z_t^l, t)\]

This mathematical derivation demonstrates that \(x_0\)-prediction naturally filters out the orthogonal noise directions in high dimensions. The diffusion transformer is allowed to concentrate its entire capacity on learning the underlying low-dimensional signal manifold \(c_0 \in \mathbb{R}^l\). Consequently, the model achieves the optimization efficiency of low-dimensional diffusion directly within the high-dimensional representation space, discarding the need for lossy dimensionality reduction adapters.

Loss & Training

For text-to-image training on paired data \((I_i, T_i)\), the high-dimensional strong-reconstruction feature \(h_{i,0} \in \mathbb{R}^h\) and text conditioning \(y_i\) are extracted. Training utilizes shifted uniform time scheduling: for base time \(t_n\), shifted time is defined as \(t_m = \frac{\alpha t_n}{1 + (\alpha - 1)t_n}\), where \(\alpha = \sqrt{m/n}\), \(m\) is the total token dimension product, and reference scale is set to \(n = 4096\).

The diffusion transformer \(x_\theta\) takes noisy feature \(h_{i,t} = (1-t)h_{i,0} + t\epsilon\) as input and is optimized using mean squared error to predict the clean target \(h_{i,0}\):

\[\mathcal{L}_{\text{Diff}} = \mathbb{E}_{t, \epsilon, h_{i,0}, y_i} \left[ \| x_\theta(h_{i,t}, t, y_i) - h_{i,0} \|^2 \right]\]

Sampling time \(t\) is clamped to a minimum of 0.05 to avoid numerical instabilities near boundaries. When finetuning encoder \(g'\), the overall training objective combines a semantic preservation loss \(\mathcal{L}_{\text{sem}} = \| g(I) - g'(I) \|^2\) (with frozen original encoder \(g\)) and an image reconstruction loss \(\mathcal{L}_{\text{recon}}\) consisting of pixel \(L_1\), perceptual, and adversarial objectives to prevent semantic collapse.

Key Experimental Results

Main Results

On 256×256 resolution using a 34M subset of the BLIP-3o pretraining dataset (trained for 90k steps), the approach was benchmarked across GenEval, DPG-Bench, and zero-shot COCO-30k FID.

Tokenizer	Latent Dim.	Pred. Type	GenEval ↑	DPG-Bench ↑	COCO-30k FID ↓	Note
DINOv2-L (frozen baseline)	1024	\(v\)-pred	37.63	70.21	18.34	Strong semantics, weak reconstruction
FT-DINOv2-L (reconstruction-tuned)	1024	\(v\)-pred	30.96	67.47	29.97	Collapse impairs \(v\)-prediction
FT-DINOv2-L-low-dim	32 (compressed)	\(v\)-pred	40.99	73.04	17.64	Compression loses fine details
FT-DINOv2-L (Ours)	1024	\(x_0\)-pred	39.48	71.83	16.80	Preserves high-dim details, best FID
MAE-B-RAE	768	\(v\)-pred	36.17	67.54	22.20	\(v\)-prediction lags on MAE
MAE-B-RAE (Ours)	768	\(x_0\)-pred	40.89	73.90	17.24	Substantial gains across all metrics

Ablation Study

The table below details the effective dimensionality \(R_\tau\) and reconstruction quality (PSNR on ImageNet validation set) across different encoder variants, illustrating the geometric impact of reconstruction-oriented training.

Tokenizer / Encoder	Training Dataset	Full Dim.	\(R_{90}\)	\(R_{95}\)	\(R_{99}\)	Recon PSNR (dB)
DINOv1-B	ImageNet-1k	768	502	595	701	—
DINOv2-B-RAE	LVD-142M	768	507	611	726	18.85
SigLIP2-B-RAE	WebLI	768	460	578	715	19.10
MAE-B-RAE	ImageNet-1k	768	103	252	578	28.13
MAE-B-IG-3B	Instagram-3B	768	197	348	595	27.67
DINOv2-L (frozen)	LVD-142M	1024	672	811	965	17.34
FT-DINOv2-L (finetuned)	ImageNet-1k	1024	129	302	726	29.12

In scaling experiments, expanding to 90M pretraining data and 60k SFT steps achieved GenEval 78.62 and DPG-Bench 80.13 at 256 resolution, and 79.69 and 81.15 at 512 resolution, outperforming Scale-RAE (2.4B parameters, GenEval 77.43, DPG-Bench 78.47).

Key Findings

• Reconstruction objectives induce drastic effective dimensionality collapse: In frozen DINOv2-L, \(R_{90}\) accounts for 65.6% of total dimensions, but drops to 12.6% upon reconstruction finetuning, proving that pixel reconstruction forces high-dimensional representations onto a much narrower manifold.
• Parameterization determines generative viability in high dimensions: On finetuned DINOv2-L, standard \(v\)-prediction deteriorates FID to 29.97, whereas \(x_0\)-prediction significantly improves FID to 16.80 and boosts GenEval from 30.96 to 39.48. On MAE-RAE, \(x_0\)-prediction similarly reduces FID from 22.20 to 17.24.
• Retaining high dimensionality is essential for visual fidelity: While compressing features to 32 dimensions (FT-DINOv2-L-low-dim) offers competitive text alignment due to compact search space, loss of high-frequency information results in inferior FID compared to uncompressed \(x_0\)-prediction (17.64 vs 16.80).

Highlights & Insights

• Systematic characterization of geometric collapse: Provides the first comprehensive quantification of effective dimensionality collapse induced by reconstruction tuning, identifying the core tension between representation learning and diffusion parameterization.
• Elegant mathematical noise decoupling: Formulates the subspace projection analysis demonstrating why high-dimensional \(v\)-prediction fails and why \(x_0\)-prediction naturally bypasses uninformative orthogonal noise.
• Dispelling the low-dimensional compression dogma: Establishes that high-dimensional representation autoencoders can be diffused directly without bottleneck adapters by simply choosing the clean data parameterization.

Limitations & Future Work

• Evaluation scale and resolution: Primary comparisons are centered around 1B DiT models at 256×256 resolution. Scaling behavior under extreme foundation-scale regimes (>10B parameters, multi-megapixel outputs) warrants deeper investigation.
• Decoupled two-stage training: Visual encoders are finetuned prior to diffusion training; exploring unified end-to-end training that co-optimizes representation learning and diffusion generation remains an exciting frontier.

Related Work & Insights

• vs RAE / Scale-RAE: RAE employs frozen semantic encoders (DINOv2, SigLIP2) with standard \(v\)-prediction, but yields low reconstruction PSNR (~17-19 dB) and loses fine visual structures. This paper boosts reconstruction PSNR to 29 dB via finetuning, resolving the resulting diffusion optimization roadblock via \(x_0\)-prediction.
• vs AlignTok: AlignTok finetunes pretrained encoders for reconstruction but relies on down-projection adapters to bypass high-dimensional diffusion difficulties. This work shows that \(x_0\)-prediction diffuses high-dimensional latents directly and yields superior image fidelity (FID).
• vs JiT: While JiT observes the utility of \(x_0\)-prediction in pixel space, this paper generalizes and formalizes the insight for high-dimensional representation autoencoders, establishing the link between reconstruction tuning, effective dimensionality collapse, and flow matching dynamics.

Rating

• Novelty: ⭐⭐⭐⭐☆ Solid theoretical foundation linking reconstruction tuning, effective dimensionality collapse, and diffusion parameterization.
• Experimental Thoroughness: ⭐⭐⭐⭐☆ Extensive spectral analysis, single-image overfitting diagnostics, multi-benchmark evaluations, and scaling studies.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical derivation, clean narrative flow, and well-structured empirical validation.
• Value: ⭐⭐⭐⭐⭐ Provides an effective standard recipe for building representation autoencoders without lossy bottleneck compression.