content_hash: 8db4114001481b97

Diffusion Image Generation with Explicitly Modeling of Data Manifold Geometry¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/xddgit/MIND
Area: Image Generation
Keywords: Image Generation, Diffusion Model, Data Manifold, Image Tokenization, Hyperspherical Geometry

TL;DR¶

Addressing the issue that continuous diffusion models diffuse in unbounded Euclidean spaces and ignore data manifold geometry, this paper proposes the Data Manifold-aware Image diffusioN moDel (MIND), which maps discrete tokens onto a compact hyperspherical manifold via a differentiable Soft Top-$k$ bridge and dynamic multi-stage transition sampling, achieving superior image fidelity with only 130M parameters that outperforms billion-parameter discrete and continuous baselines.

Background & Motivation¶

Modern generative models, exemplified by continuous diffusion models (e.g., DDPM, LDM) and Diffusion Transformers (DiT), have achieved remarkable synthetic fidelity. However, conventional continuous diffusion frameworks typically operate within unbounded Euclidean spaces with infinite continuous states. While theoretical analyses of generative models rely heavily on the "low-dimensional manifold hypothesis"—which posits that high-dimensional natural images reside on a dense, low-dimensional topological structure—standard continuous diffusion algorithms rarely model this geometric prior explicitly (apart from a few complex Riemannian diffusion formulations), struggling to leverage data manifold geometry effectively during training and sampling.

Attempting to constrain continuous representations (such as continuous VAE latents) directly onto a strict low-dimensional topological manifold (e.g., a low-dimensional hypersphere surface) inevitably causes catastrophic representation collapse. Continuous autoencoders rely heavily on vector magnitude to encode structural and texture details; the strict $\ell_2$-normalization required by hyperspherical surfaces strips away this magnitude information, resulting in severe perceptual distortion and failure of visual reconstruction. In contrast, discrete image tokenization (e.g., VQ-GAN, IBQ, GigaTok) compresses images into discrete token indices that encode categorical semantics and rely exclusively on angular separation rather than vector magnitude, making them innately immune to bottleneck degradation on compact manifolds. Nevertheless, existing discrete autoregressive or masked generation frameworks force an artificial 1D or random causal decoding order, failing to exploit the global bidirectional context and parallel iterative refinement inherent to continuous visual diffusion.

This paper bridges the representational robustness of discrete tokenization on low-dimensional manifolds with the global bidirectional sampling flexibility of continuous diffusion. Core idea: propose the Data Manifold-aware Image diffusioN moDel (MIND), which explicitly projects discrete image tokens onto a continuous low-dimensional hyperspherical manifold for forward and reverse diffusion, introduces a differentiable Soft Top-$k$ aggregation bridge to enable end-to-end training across discrete and continuous spaces, and incorporates dual-branch high-frequency embeddings and dynamic multi-stage transition sampling.

Method¶

Overall Architecture¶

The overall pipeline of MIND integrates discrete tokenization, hyperspherical manifold geometric parameterization, differentiable soft-sampling forward/reverse diffusion, and dynamic multi-stage transition inference. Given an image $I \in \mathbb{R}^{C \times H \times W}$, a frozen pre-trained discrete tokenizer (such as IBQ or GigaTok) first quantizes it into a sequence of discrete tokens $k \in \mathbb{R}^N$ with vocabulary size $V$. Next, a manifold projection operator $P_\theta$ embeds each token onto a continuous representation $x_0 \in \mathbb{R}^{N \times L}$ residing strictly on a hyperspherical manifold of radius $R$, partitioned into sub-vector normalized subspaces.

During forward diffusion, continuous representation $x_0$ is perturbed on the hyperspherical surface by adding Gaussian noise according to the noise schedule, yielding noisy latent $x_t$. The denoising backbone (implemented via DiT) processes $x_t$ through a dual-branch high-frequency projection module to resolve spectral bias on low-dimensional inputs, predicting unnormalized categorical logits $\tilde{x}_{0t} \in \mathbb{R}^{N \times V}$. During training, a differentiable Soft Top-$k$ aggregation and spherical projection operator maps the unnormalized logits back to the continuous hypersphere to yield denoised latent $\dot{x}_{0t}$, supervised jointly by categorical cross-entropy and latent mean squared error. During inference, a parameterized multi-stage transition sampling strategy dynamically transitions between soft sampling, entropy-driven hybrid sampling, and greedy sampling, before the final sequence is decoded into high-fidelity pixels by the Detokenizer.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image Input and Tokenization<br/>Image -> VQ Tokenizer -> Discrete Tokens k"] --> B["Hypersphere Manifold Parameterization<br/>P_theta Projection onto Sphere Surface x_0"]
    B --> C["Dual-Branch High-Frequency Feature Mapping<br/>RFF Base Branch + Zero-Init Residual Sinusoidal Branch"]
    C --> D["DiT Diffusion Denoising Backbone<br/>Predicts Unnormalized Categorical Logits x_tilde_0t"]
    E --> F["Dynamic Multi-Stage Transition Sampling<br/>Soft Sampling -> Entropy-Driven Hybrid -> Greedy"]
    D --> E["Differentiable Soft Top-k Aggregation Bridge<br/>Top-k Softmax Weighting and L2 Projection to x_dot_0t"]
    F --> G["Detokenizer Output<br/>Synthesized High-Fidelity Image"]

Key Designs¶

1. Hyperspherical Manifold Parameterization: Eliminating Representation Collapse via Discrete Anchors

To address catastrophic representation collapse caused by magnitude destruction when projecting continuous latents onto low-dimensional manifolds, MIND utilizes the angular separation invariance of discrete tokens to parameterize the manifold geometry. The projection operator $P_\theta$ maps each discrete token index from vocabulary $V$ to an $L$-dimensional continuous embedding, which is factored into $d_{\text{sub}}$-dimensional hyperspherical subspaces via sub-vector $\ell_2$-normalization such that $\sum_{l=0}^{L-1} x_0^2(n, l) = R^2$. The forward diffusion process perturbs this geometrically constrained latent: $$x_t = c_1 \sqrt{1 - t} \cdot w + c_2 \sqrt{t} \cdot x_0$$ where $w \sim \mathcal{N}(0, I)$ and $t \in [0, 1]$. Because discrete tokens encode semantics categorically rather than through continuous vector magnitudes, mapping them onto a compact hypersphere preserves complete semantic integrity while providing dense, compact data manifold support for continuous diffusion.

2. Differentiable Soft Top-$k$ Aggregation Bridge: Connecting Categorical Logits to Continuous Manifolds

The denoising backbone network $s_\phi$ outputs unnormalized logits $\tilde{x}_{0t} \in \mathbb{R}^{N \times V}$ over discrete vocabulary $V$, whereas diffusion state progression requires continuous latents $\dot{x}_{0t}$ on the hypersphere manifold. Hard discrete sampling breaks gradient backpropagation, while global Softmax averaging introduces heavy noise from long-tail classes and pulls vectors into the interior of the sphere. MIND designs a soft Top-$k$ aggregation and spherical projection operator: for each token position, it extracts the index subset $I_k$ corresponding to the Top-$k$ logit values and computes normalized weights $\alpha_i = \exp(x_i) / \sum_{j \in I_k} \exp(x_j)$. Because convex combinations of points on a hypersphere fall inside the sphere, an explicit $\ell_2$-projection is applied: $$\dot{x}_{0t}(n) = \frac{\sum_{i \in I_k} \alpha_i P_\theta(i)}{\left\| \sum_{i \in I_k} \alpha_i P_\theta(i) \right\|_2}$$ This differentiable bridge allows smooth gradient backpropagation, enabling dual supervision in both categorical probability space and geometric continuous latent space.

3. Dual-Branch High-Frequency Feature Mapping: Alleviating Transformer Spectral Bias

Diffusion Transformers suffer from severe spectral bias when handling low-dimensional geometric inputs, tending to learn only low-frequency components while neglecting fine structural details. MIND incorporates a dual-branch feature projection module before feeding latents into the DiT backbone: the base branch applies Random Fourier Features (RFF) using a projection matrix $B \sim \mathcal{N}(0, \sigma^2)$ with large variance to project $x_t$ into trigonometric activations $f^{\text{base}} = [\sin(2\pi x_t B), \cos(2\pi x_t B)]$, mapped by a shallow MLP to hidden dimension $h_{\text{base}}$; the residual branch maps $x_t$ through deterministic sinusoidal positional encodings $f^{\text{freq}}$ into an MLP with LayerNorm and SiLU. Crucially, the final linear layer of this residual MLP is initialized to zero, ensuring training stability at initialization while progressively unlocking multi-scale high-frequency geometric representations.

4. Dynamic Multi-Stage Transition Sampling: Balancing Diversity and Structural Sharpness

Sampling Gaussian noise $x_1 \sim \mathcal{N}(0, c_1^2 I)$ back to the manifold requires balancing stochastic diversity at early steps with structural fidelity at late steps. MIND employs a three-phase transition sampling operator $S(\cdot)$ parameterized by thresholds $\rho_1$ and $\rho_2$: - Initial Phase ($t < \rho_1$): Follows the differentiable soft sampling mechanism from Eq. (3), allowing continuous latents to explore semantic directions freely on the hypersphere surface; - Intermediate Transition Phase ($\rho_1 \le t < \rho_2$): Applies an entropy-driven hybrid filtering mechanism. Given categorical distribution $p = \text{Softmax}(\tilde{x}_{0t})$, it computes Shannon entropy $H = -\sum p \log p$ and scales sampling temperature adaptively via $\tau_{\text{adj}} = \tau \cdot (2.5 e^{-H/3} + 0.6)$, dampening temperature when entropy is high. Discrete tokens are then sampled within the intersection of Top-$k$ and Nucleus (Top-$p$) candidate sets; - Terminal Phase ($t \ge \rho_2$): Collapses to greedy sampling $\hat{k}_t = \arg\max(\tilde{x}_{0t})$, eliminating stochastic jitter and locking in crisp local textures.

Loss & Training¶

The network parameters $s_\phi$ and $P_\theta$ are trained end-to-end by minimizing a hybrid objective combining categorical cross-entropy (CE) and continuous mean squared error (MSE): $$\mathcal{L}(s_\phi, P_\theta) = \mathbb{E}_{k \sim p_{\text{data}}, t \in [0, 1]} \left[ \text{CE}(\tilde{x}_{0t}, k) + \lambda \cdot \text{MSE}(\dot{x}_{0t}, x_0) \right]$$ where $\lambda$ balances classification accuracy and geometric feature alignment. Unconditional token dropout with probability $p_{\text{drop}} = 0.1$ is applied to support classifier-free guidance (CFG) at inference time. Optimization uses AdamW with chunked gradient checkpointing on the DiT backbone to minimize memory overhead.

Key Experimental Results¶

Main Results¶

Evaluations were conducted on the ImageNet $256 \times 256$ benchmark. The table below presents performance comparisons under a restricted 80-epoch training budget as well as system-level comparisons against established generative paradigms.

Model	Paradigm	Params	Epochs	Guidance (cfg)	FID ↓	IS ↑	Precision ↑	Recall ↑
DiT-S/2	Continuous Diff.	33M	80	w/o CFG	68.40	N/A	N/A	N/A
SiT-S/2	Continuous Flow	33M	80	w/o CFG	57.64	24.78	0.41	0.60
MIND-S (Ours)	Manifold Diff.	~35M	80	w/o CFG	40.72	31.51	0.48	0.61
DiT-B/2	Continuous Diff.	130M	80	w/o CFG	43.47	N/A	N/A	N/A
SiT-B/2	Continuous Flow	130M	80	w/o CFG	33.02	43.71	0.53	0.63
DiT-L/2	Continuous Diff.	458M	80	w/o CFG	23.33	N/A	N/A	N/A
MIND-B (Ours)	Manifold Diff.	~130M	80	w/o CFG	22.73	56.17	0.55	0.58
DiT-B/2	Continuous Diff.	130M	80	cfg = 1.5	20.01	73.00	0.65	0.56
SiT-B/2	Continuous Flow	130M	80	cfg = 1.5	16.88	84.26	0.66	0.56
MIND-B (Ours)	Manifold Diff.	~130M	80	cfg = 1.5	12.15	100.16	0.71	0.51
DiT-B/2	Continuous Diff.	130M	80	cfg = 2.0	10.95	119.12	0.76	0.47
SiT-B/2	Continuous Flow	130M	80	cfg = 2.0	9.07	137.07	0.77	0.47
MIND-B (Ours)	Manifold Diff.	~130M	80	cfg = 2.0	7.97	154.20	0.81	0.43

System-Level Scaling Comparisons¶

Under extended training and scaled backbone configurations, MIND demonstrates remarkable parameter efficiency against state-of-the-art discrete autoregressive and continuous diffusion models.

Model	Family	Params	Epochs	FID ↓	IS ↑	Precision ↑	Recall ↑
LlamaGen-3B	Discrete AR	3.1B	300	2.18	263.33	0.81	0.58
RQTran.-re	Discrete AR	3.8B	N/A	3.80	323.70	N/A	N/A
VAR-d24	Discrete Multi-Scale	1.0B	200-350	2.09	312.90	0.82	0.59
DiT-XL/2	Continuous Diff.	675M	1400	2.27	278.20	0.83	0.57
SiT-XL/2	Continuous Flow	675M	1400	2.06	270.30	0.82	0.59
SimDiff	Continuous Diff.	2.0B	800	2.77	211.80	N/A	N/A
MIND-B (Ours)	Manifold Diff.	~130M	1000	2.18	258.46	0.80	0.58
MIND-B-G (Ours)	Manifold Diff. + GigaTok	~130M	1600	2.06	268.03	0.78	0.62
MIND-XL (Ours)	Manifold Diff.	~715M	1000	1.97	297.93	0.78	0.61
MIND-XL-G (Ours)	Manifold Diff. + GigaTok	~715M	1600	1.95	293.78	0.75	0.67

Key Findings¶

Fast Convergence in 80 Epochs: Without classifier-free guidance, MIND-B achieves an FID of 22.73 after only 80 epochs, halving the 43.47 FID of vanilla DiT-B/2 and outperforming the 458M DiT-L/2 (23.33).
Extraordinary Parameter Efficiency: With guidance at cfg = 2.0, the 130M MIND-B achieves an FID of 7.97, outperforming DiT-B/2 (10.95) and SiT-B/2 (9.07). With converged training and GigaTok, MIND-B achieves an FID of 2.06, outperforming the 3.1B-parameter LlamaGen-3B (2.18) and the 2.0B-parameter SimDiff (2.77).
High Structural Fidelity: Visual evaluations confirm that MIND effectively eliminates the structural tearing and topological distortions frequently observed in continuous baselines, evidenced by a Precision score of 0.81 at cfg = 2.0.

Highlights & Insights¶

Unified Manifold Geometric Formulation: By identifying that continuous latent spaces suffer from magnitude collapse under manifold normalization while discrete tokens remain resilient due to angular encoding, MIND provides a principled foundation for embedding discrete visual primitives onto continuous manifold diffusion.
Differentiable Soft Top-$k$ Projection: Successfully circumvents gradient truncation from non-differentiable sampling while avoiding interior degradation from unconstrained softmax convex combinations, maintaining manifold geometry integrity.
Effective Spectral Bias Mitigation: Combines random Fourier feature projection with a zero-initialized deep residual sinusoidal pathway, offering a general recipe for applying Transformer architectures to compact low-dimensional coordinate spaces.

Limitations & Future Work¶

Inference Speed: The current implementation relies on a 250-step SDE solver, which exhibits higher latency compared to modern one-step or few-step consistency and flow-matching models.
Quantization Information Loss: While discrete tokenization protects manifold geometry, the intrinsic codebook capacity limit of VQ tokenizers imposes a reconstruction bottleneck, which could be mitigated via continuous residual features.
Trajectory Geometry Optimization: Exploring advanced geodesic trajectories or flow-matching variants on Riemannian manifolds could further accelerate convergence and generation speed.

vs DiT / SiT: Standard DiT and SiT learn score functions across unbounded Euclidean spaces without geometric priors; MIND restricts representations to a compact hyperspherical manifold, allowing a 130M model to surpass a 458M DiT-L model in training efficiency and generation fidelity.
vs RJF (Riemannian Flow Matching): RJF applies Jacobi regularization to continuous representations in high-dimensional space; MIND injects a low-dimensional manifold prior using discrete token anchors, avoiding continuous representation collapse and permitting flexible off-sphere diffusion trajectories.
vs LlamaGen / VAR / MaskGIT: Prior discrete generative models enforce 1D autoregressive or masked causal ordering that sacrifices bidirectional spatial context; MIND marries the robustness of discrete tokens with the parallel iterative denoising capabilities of continuous diffusion.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Pioneering integration of discrete token geometry into continuous hyperspherical diffusion with differentiable soft projection]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 80-epoch budgets, long-schedule convergence, multiple scales, and diverse baseline paradigms]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical derivations, insightful representation collapse analysis, and well-structured experimental presentation]
Value: ⭐⭐⭐⭐⭐ [Provides a compelling alternative to massive parameter scaling by demonstrating the effectiveness of explicit manifold geometric priors in generative vision]