Hi-DiT: Hybrid Latent-Pixel Diffusion Transformer for Image Generation

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/HiDream-ai/Hi-DiT
Area: Image Generation
Keywords: diffusion model, Diffusion Transformer, hybrid latent-pixel, time-gated injection, high-frequency detail synthesis

TL;DR

Hi-DiT unifies latent-space and pixel-space diffusion inside a single parameter-shared Diffusion Transformer, leveraging temporal denoising heterogeneity to activate a time-gated pixel pathway and hierarchical sub-pixel predictor at low-noise timesteps for state-of-the-art visual fidelity.

Background & Motivation

Modern diffusion-based generative models face an intrinsic dilemma between generation efficiency and visual precision. Latent Diffusion Models (LDMs) perform generative trajectories within a compact manifold compressed by a pre-trained Variational Autoencoder (VAE), substantially mitigating computational overhead and stabilizing training. Nevertheless, spatial downsampling (e.g., \(16\times\) compression) and latent channel bottlenecks inevitably discard high-frequency visual cues such as sharp boundaries and fine textures. Consequently, VAE decoders frequently produce blurred outputs and reconstruction artifacts, bounding the ultimate synthesis fidelity by the autoencoder's capacity ceiling. Conversely, pixel-space diffusion models bypass VAE compression artifacts by denoising directly on raw image grids, but they force the network to jointly model low-frequency global structure and high-entropy, high-frequency variations within a high-dimensional space. This induces severe optimization difficulty, capacity competition, and sluggish convergence compared to latent counterparts under comparable compute.

This conflict stems from the temporal heterogeneity of diffusion denoising across the sampling trajectory: early high-noise stages are predominantly governed by coarse semantic layouts and global scene topology, whereas late low-noise stages are tasked with synthesizing fine-grained textures and high-frequency details. Standard uniform denoisers force a single parameter set to reconcile these qualitatively distinct objectives simultaneously across all timesteps, causing severe objective interference and inefficient parameter allocation.

The authors address this tension by recognizing that global semantics should be established in a compact latent pathway early on, while pixel-level modeling capacity should be activated exclusively for late-stage high-frequency synthesis. Core idea: Hi-DiT builds a parameter-shared dual-stream Diffusion Transformer that exploits temporal specialization via a Time-Gated Injection mechanism, dynamically engaging a raw pixel stream and a hierarchical sub-pixel predictor during late low-noise denoising to decouple global structural planning from high-frequency detail recovery.

Method

Overall Architecture

Hi-DiT coordinates a Latent Stream and a Pixel Stream within a single parameter-shared Transformer backbone. Given conditioning input \(c\) (class or text prompt) and diffusion timestep \(t\), the process operates over two temporal regimes. In early high-noise steps (\(t \ge \tau\)), the model runs exclusively in the compressed latent manifold, predicting a velocity field \(v_\theta^z(z_t, t, c)\) to anchor low-frequency global layout with minimal compute. Once crossing the time-gating threshold into the low-noise regime (\(t < \tau\)), the pixel stream activates: the intermediate latent state is decoded to serve as a structural scaffold, and noisy pixel patches are fed alongside latent tokens into the shared Transformer for joint cross-domain reasoning. Finally, a latent predictor yields structural predictions while a dedicated sub-pixel predictor regresses high-frequency residuals, which are linearly fused with the VAE-decoded scaffold into the final image.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Noisy latent z_t and condition c"] --> B["Dual-stream shared backbone<br/>Latent stream establishes global semantic layout"]
    B --> C{"Time-Gated Injection<br/>Evaluate denoising step t < tau"}
    C -->|High noise t >= tau| D["Pure latent velocity prediction"]
    C -->|Low noise t < tau| E["Activate pixel branch & patch embedding"]
    E --> F["Hierarchical sub-pixel predictor<br/>Two-stage PixelShuffle upsampling"]
    D --> G["Output: VAE decoded structure"]
    F --> H["Fusion layer: VAE scaffold + high-frequency residual"]
    H --> I["Output: Final high-fidelity image x_final"]

Key Designs

1. Dual-stream shared backbone: Decoupling global semantics from local textures

To enable seamless multi-domain interaction without doubling model parameters, Hi-DiT shares a single Diffusion Transformer across both streams. Latent tokens are mapped into the hidden space via a Latent Embedder to form \(h_z^0\), modeling macro-level spatial semantics across the trajectory. In the low-noise regime, noisy pixel images \(x_t\) are partitioned into \(p \times p\) non-overlapping patches and projected into \(h_x^0\) via a bottleneck patch embedder. Crucially, patch size \(p\) is matched to the VAE downsampling factor (\(p=16\)), ensuring identical sequence lengths between pixel and latent tokens. Within the backbone, shared self-attention layers naturally exchange context across macro geometry and micro details, achieving structural consistency without external cross-attention overhead.

2. Time-Gated Injection: Eliminating early capacity competition

Naively feeding pixel and latent tokens simultaneously throughout the entire trajectory forces the network to struggle with high-frequency fitting during early high-noise phases. Hi-DiT introduces a Time-Gated Injection mechanism with a hard schedule \(\mathcal{G}(t) = \mathbf{1}(t < \tau)\), constructing the fused input token sequence as: $\(h_{zx}^0 = h_z^0 + \mathcal{G}(t) \cdot h_x^0\)$ With the threshold empirically set to \(\tau = 0.3\), the pixel stream remains silent during \(t \ge \tau\), reserving 100% of network capacity for stable flow matching velocity learning. When \(t < \tau\), the gate opens: the pixel branch is initialized at the transition boundary \(t_0\) by decoding the intermediate latent state \(x^{\text{dec}}_{t_0} = \mathcal{D}(z_{t_0})\). At each subsequent low-noise step, the pixel state is updated by adding the predicted high-frequency component to the decoded latent, adhering to curriculum learning by focusing on macro structures before refining high-frequency textures.

3. Hierarchical sub-pixel predictor: Alleviating high-dimensional regression bottlenecks

Standard pixel-space diffusion models typically rely on a single linear layer to regress large \(p \times p \times 3\) patches from individual tokens, lacking localized inductive bias and often causing checkerboard artifacts. Inspired by sub-pixel convolution, Hi-DiT introduces a High-Frequency Pixel Predictor that reshapes hidden states \(h\) into a 2D feature grid \(\mathbf{G}_0 \in \mathbb{R}^{C_{\text{mid}} \times \frac{H}{p} \times \frac{W}{p}}\) and processes it through two consecutive convolutional refinement blocks: $\(\mathbf{G}_i = \text{SiLU}(\text{GroupNorm}(\mathcal{PS}(\text{Conv}(\mathbf{G}_{i-1}))))\)$ where \(\mathcal{PS}\) denotes PixelShuffle with a scale factor of 2. After two stages of spatial expansion, the feature grid is upsampled by a factor of 4 to \(\mathbf{G}_2\). Following AdaLN modulation conditioned on timestep and class/text embeddings, a lightweight linear head only needs to regress a localized \(\frac{p}{4} \times \frac{p}{4}\) sub-patch at each spatial position. This adds only 7M parameters while significantly easing optimization.

A Worked Example

Consider class-conditional generation on ImageNet \(256 \times 256\) with downsampling factor 16: - High-noise stage (\(t = 1.0 \to 0.3\)): The model executes flow matching sampling in latent space. With \(\mathcal{G}(t) = 0\), only 256 latent tokens traverse the SiT backbone to predict the velocity field, establishing object class geometry, overall color distribution, and coarse boundaries. - Threshold crossing (\(t = 0.3\)): The partially denoised latent \(z_{0.3}\) is decoded via the VAE into an initial pixel scaffold \(x^{\text{dec}}_{0.3} \in \mathbb{R}^{256 \times 256 \times 3}\). - Low-noise refinement (\(t = 0.3 \to 0.0\)): The gate opens (\(\mathcal{G}(t)=1\)). The noisy image is patched into 256 tokens and processed jointly with latent tokens in the shared Transformer. The latent predictor maintains semantic alignment, while the sub-pixel predictor expands the \(16 \times 16\) feature grid to \(64 \times 64\), predicting \(4 \times 4\) high-frequency sub-patch residuals \(x_{\text{HF}}\). The residual is added to the VAE-decoded image at each step until \(t=0\), yielding crisp hair strands, plumage, and fine textures.

Loss & Training

Hi-DiT is trained with a composite objective. Across the full trajectory, the Latent Stream is supervised by the rectified flow matching objective: $\(\mathcal{L}_{\text{lat}} = \mathbb{E}_{t, c, z_0, \epsilon} \left\| v_\theta^z(z_t, t, c) - (\epsilon - z_0) \right\|_2^2\)$ In the low-noise regime (\(t < \tau\)), the Pixel Stream is optimized via a composite reconstruction and perceptual loss on the combined prediction \(x_{\text{final}} = \mathcal{D}(\hat{z}_0) + x_{\text{HF}}\): $\(\mathcal{L}_{\text{pix}} = \lambda_{\text{rec}} \left\| x_{\text{final}} - x_0 \right\|_1 + \lambda_{\text{per}} \text{LPIPS}(x_{\text{final}}, x_0)\)$ The overall loss is defined as: $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{lat}} + \mathbf{1}(t < \tau) \cdot \mathcal{L}_{\text{pix}}\)$ Training follows a two-stage protocol: the latent backbone (SiT-XL/1) is first pre-trained on ImageNet with DINOv2 feature alignment, followed by end-to-end joint fine-tuning of the full hybrid architecture.

Key Experimental Results

Main Results

Hi-DiT achieves state-of-the-art results across ImageNet \(256 \times 256\), ImageNet \(512 \times 512\), and MS-COCO text-to-image synthesis.

Dataset	Model	Paradigm	Params	FID ↓	IS ↑	Precision ↑	Recall ↑
ImageNet 256×256	VAR (Tian et al.)	Autoregressive (AR)	600M	2.57	302.6	0.83	0.56
ImageNet 256×256	DiT-XL (Peebles et al.)	Latent Diffusion	675M	2.27	278.2	0.83	0.57
ImageNet 256×256	SiT-XL (Ma et al.)	Latent Diffusion	675M	2.06	270.3	0.82	0.59
ImageNet 256×256	VA-VAE (Yao et al.)	Latent Diffusion	675M	1.35	295.3	0.79	0.65
ImageNet 256×256	REPA (Yu et al.)	Latent Diffusion	675M	1.29	306.3	0.79	0.65
ImageNet 256×256	PixelFlow (Chen et al.)	Pixel Diffusion	677M	1.98	282.1	0.81	0.60
ImageNet 256×256	Deco (Ma et al.)	Pixel Diffusion	682M	1.62	301.0	0.80	0.62
ImageNet 256×256	JiT-G (Li & He)	Pixel Diffusion	2B	1.82	292.6	0.79	0.62
ImageNet 256×256	Hi-DiT (CFG=2.4)	Hybrid Latent-Pixel	689M	1.06	296.6	0.79	0.66
ImageNet 256×256	Hi-DiT (CFG=3.0)	Hybrid Latent-Pixel	689M	1.10	319.4	0.79	0.65
ImageNet 512×512	DiT-XL	Latent Diffusion	675M	3.04	240.8	0.84	0.54
ImageNet 512×512	REPA	Latent Diffusion	675M	2.08	274.6	0.83	0.58
ImageNet 512×512	JiT-G	Pixel Diffusion	2B	1.78	306.8	-	-
ImageNet 512×512	Hi-DiT (CFG=4.6)	Hybrid Latent-Pixel	689M	1.26	295.0	0.80	0.62
MS-COCO 256×256	Hi-DiT (U-ViT-S)	Hybrid Latent-Pixel	-	5.15	-	-	-

Note: Without CFG guidance, Hi-DiT achieves 1.66 FID on ImageNet 256×256, outperforming DiT-XL (9.62) and VA-VAE (2.17). On MS-COCO, Hi-DiT achieves 5.15 FID, outperforming U-ViT-S/2 baseline (5.48).

Ablation Study

Ablation experiments conducted on ImageNet 256×256 trained for 80 epochs from scratch:

Config	Latent Input	Pixel Predictor Type	Params	FID ↓	Note
Pixel Baseline	✗	Naive MLP	682M	3.38	Direct pixel token regression struggles without semantic guidance
Hybrid + Linear Head	✓	Naive MLP	682M	1.74	Latent scaffold drastically accelerates convergence and quality
Hi-DiT Full Model	✓	HF Pixel Predictor	689M	1.65	Hierarchical sub-pixel upsampling boosts detail reconstruction

Sensitivity of Time-Gating threshold \(\tau\) (80 epochs): - \(\tau = 0.7\): FID 1.73; premature pixel activation reintroduces objective conflict during early high-noise steps. - \(\tau = 0.5\): FID 1.68; stable but retains slight optimization redundancy. - \(\tau = 0.3\): Optimal FID of 1.65 (IS 267.7); perfectly balances coarse semantics and fine details. - \(\tau = 0.1\): FID drops to 1.75; pixel stream engages too late to sufficiently refine high-frequency signals.

Key Findings

• Latent scaffolding catalyzes pixel diffusion: Denoising purely in pixel space yields an inferior 3.38 FID, but injecting latent guidance drops FID to 1.74, proving that global semantic priors are critical for stabilizing pixel-level optimization.
• Hierarchical decoding outperforms flat projection: Replacing the 682M MLP head with the 689M sub-pixel convolutional predictor yields a clear gain from 1.74 to 1.65 FID with only a 7M parameter increase.
• Negligible computational overhead: Hi-DiT requires 1.67 seconds per image at inference (vs. 1.52s for SiT and 1.58s for VA-VAE), and training time rises modestly from 17.7 min/epoch to 18.1 min/epoch with 35.52G peak memory, achieving a superior cost-performance trade-off.

Highlights & Insights

• Temporal specialization resolves frequency interference: Rather than maintaining two disparate networks, Hi-DiT elegantly schedules latent and pixel pathways across time within a shared backbone, strictly matching network roles to the coarse-to-fine nature of diffusion.
• Matched spatial discretization: Equating the pixel patch size to the VAE downsampling factor (\(p=16\)) ensures spatial 1:1 alignment between latent and pixel tokens without needing complex cross-attention projection layers.
• Reusable sub-pixel head for patch models: The cascade of convolution, PixelShuffle, and localized AdaLN modulation serves as a generic, plug-and-play module for recovering continuous high-frequency textures in any tokenized visual generator.

Limitations & Future Work

• Author-noted limitations: Introducing the pixel stream during late timesteps slightly increases inference time per image and raises peak training memory from 32.9G to 35.5G.
• Methodological observations: The hard threshold \(\tau = 0.3\) is a static empirical hyperparameter, lacking adaptiveness to variable prompt complexities or image dynamic ranges; furthermore, initial pixel states still depend on the VAE decoding quality of intermediate latents.
• Future directions: Exploring dynamic soft-gating schedules conditioned on image frequency spectra, and extending hybrid latent-pixel modeling to video generation and unified multimodal autoregressive models.

Related Work & Insights

• vs VA-VAE / REPA (Latent Optimization): VA-VAE and REPA incorporate representation alignment into latent space, yet remain fundamentally bounded by VAE decoding loss. Hi-DiT keeps the latent space for macro semantics while introducing a lossless pixel bypass for high-frequency details.
• vs JiT / PixelFlow (Pure Pixel Diffusion): JiT-G scales to 2B parameters to force pure pixel diffusion to learn global and local features simultaneously, achieving 1.82 FID. Hi-DiT attains 1.06 FID with only 689M parameters, demonstrating the efficiency advantage of hybrid modeling.

Rating

• Novelty: ⭐⭐⭐⭐½ [4.5/5.0 — Elegant dual-stream temporal specialization within a parameter-shared transformer.]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [5.0/5.0 — Exhaustive comparisons across ImageNet 256/512, MS-COCO, diverse CFG sweeps, and runtime costs.]
• Writing Quality: ⭐⭐⭐⭐⭐ [4.8/5.0 — Clear motivation from spectral-temporal perspectives with comprehensive architectural descriptions.]
• Value: ⭐⭐⭐⭐⭐ [4.8/5.0 — Provides a practical, highly scalable blueprint for next-generation Diffusion Transformers.]