PixelU: A U-Shaped Transformer for Efficient End-to-End Pixel Diffusion¶
Conference: ECCV 2026
Paper: ECCV Paper
Code: https://github.com/gzp6688/PixelU
Area: Image Generation
Keywords: pixel-space diffusion, clean-image prediction, U-shaped Transformer, frequency decoupling, spatial downsampling
TL;DR¶
PixelU combines shallow-to-deep skip connections for spatial detail with single-stage, constant-channel downsampling for global semantics under clean-image prediction, reaching FID 1.63 at 136.9 GFLOPs on ImageNet 256×256, improving the quality-compute trade-off over JiT-G's FID 1.82 at 383 GFLOPs.
Background & Motivation¶
Latent diffusion compresses images with a VAE before generating over a shorter sequence, but generation ultimately inherits the limitations of lossy reconstruction. Direct pixel-space generation avoids this bottleneck, yet its Transformer must handle object layout, boundaries, textures, and substantial noise together. Approaches such as DeCo, DiP, and PixelDiT add pixel-level decoding or detail modules to divide semantic and texture modeling across network components, introducing additional computation.
PixelU starts from an observation made by JiT: learning may change fundamentally when the network outputs a clean image instead of a velocity containing high-dimensional noise components. On the same JiT-B/16 baseline, the authors compare prediction targets and decoders. A DeCo decoder reduces velocity-prediction FID from 190.11 to 42.07, but improves clean-image-prediction FID only from 43.65 to 41.20. The value of a complex decoder therefore depends on the prediction target and cannot be assessed independently of output parameterization.
Even with clean-image prediction, deep Transformers can lose fine spatial information. The aim is therefore not to remove all detail modeling, but to preserve it more cheaply. Shallow features bypass a semantic bottleneck, while deeper spatial resolution is deliberately reduced to avoid repeatedly processing high-frequency signals. Core Idea: under clean-image prediction, preserve detail through skip connections and compress the semantic backbone through single-stage, constant-channel downsampling, allowing the two paths to replace expensive auxiliary pixel decoders.
Method¶
Overall Architecture¶
PixelU is an end-to-end, class-conditional pixel-space generator that does not require a separately trained image autoencoder. Each denoising call receives a noisy image, a timestep, and a class condition, and outputs a clean-image estimate. This estimate is converted into the velocity required by a flow-matching sampler, whose repeated calls transform noise into an image.
The network first extracts shallow features at relatively high spatial resolution, reduces the token count through a single-stage constant-channel semantic bottleneck, and then upsamples to the original feature resolution while recovering local structure through shallow-to-deep detail skips. Clean-image prediction is the output parameterization of the entire network, not a preprocessing operation that cleans the noisy input before feature extraction.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Noisy image, timestep<br/>class condition"] --> B["Shallow feature extraction"]
B --> C["Single-stage constant-channel<br/>semantic bottleneck"]
C -->|Deep modeling, then upsampling| D["Shallow-to-deep<br/>detail skips"]
B -->|Spatial features bypassing the bottleneck| D
D --> E["Clean-image prediction"]
E --> F["Clean-image estimate<br/>converted to sampling velocity"]
The diagram shows the data flow within each network call. During training, the clean image participates only as a supervision target; inference has no side channel carrying ground-truth clean textures. Class tokens and REPA serve conditioning and auxiliary training supervision, respectively.
Key Designs¶
1. Single-stage constant-channel semantic bottleneck: perform semantic processing over a shorter sequence
Instead of keeping the same spatial resolution throughout a flat DiT, PixelU halves both dimensions of the feature grid, reducing the spatial token count in the middle section to one quarter. The important choice is not repeated hierarchical compression, but a single downsampling stage and its corresponding upsampling stage, with the same channel width in shallow, middle, and deep sections. This reduces attention and per-token computation in the middle section without the repeated channel expansion of conventional U-Nets, which would offset some savings. The authors interpret this bottleneck as a low-pass bias: layout and coarse shapes can be represented at lower spatial resolution, while high-frequency noise and some detail are attenuated, so deeper layers need not process every frequency on the full grid.
This does not establish that every downsampling operator is a rigorous antialiasing filter. The cached paper does not specify the exact sampling kernel, filter response, or complete operator implementation. What it supports is single-stage compression with constant-channel routing and a low-frequency bias illustrated by feature spectra. Downsampling alone worsens FID from 43.65 to 48.87, showing that the semantic bottleneck can discard information needed for generation. It must be understood together with the detail path described next. The B, L, and H models have channel widths of 768, 1024, and 1152, with encoder/middle/decoder depths of [4,4,4], [8,8,8], and [8,20,8], respectively. Constant channels are a constraint within each model, not a shared width across model scales.
2. Shallow-to-deep detail skips: bypass spatial compression instead of relearning detail through an auxiliary pixel decoder
As the middle bottleneck concentrates on large-scale structure, a spatial information path that avoids this compression becomes more important. PixelU connects shallow encoder features to deep decoder features, giving the output side both semantic modeling results and features at higher spatial resolution. This reduces the burden of repeatedly preserving boundary, position, and texture cues through the deep backbone. Removing the decoder means removing an additional complex pixel-decoding module, not deleting the U-shaped network's own upsampling and decoder sections. The text does not explicitly specify whether individual skips use addition or concatenation, so a familiar U-Net implementation should not be assumed.
Preserving detail also does not mean the shallow features are already noise-free. The network still receives a noisy image, and the skips transmit features extracted from it. The authors' description of uncorrupted high-frequency detail is better read as avoiding further destruction of these cues through deep compression. Table 1 directly illustrates the interaction with the prediction target: with skips alone, clean-image-prediction FID is 38.53, while velocity-prediction FID remains 185.54. Skips are not a universal remedy for every prediction target. They are not literally free either: computation increases from 21.99 to 23.20 GFLOPs in this experiment, although this is cheaper than adding a complex decoder.
3. Clean-image prediction: keep flow-matching sampling while changing what the network directly fits
PixelU follows JiT by having the output head estimate the clean image rather than noise or velocity. During training, a clean image and Gaussian noise are linearly interpolated according to the timestep to construct the input. The true velocity is the clean image minus the noise, \(v=x-\epsilon\). The predicted clean image is not directly integrated as a velocity. Instead, it is combined with the current noisy state to obtain a predicted velocity for the flow-matching error or the sampler. Clean-image prediction and a velocity-error training objective are therefore compatible: the former specifies the network's output parameterization, whereas the latter specifies the optimization objective.
This combination directs the output toward an image estimate rather than requiring a direct representation of a noise-heavy target. Within the authors' ablation setting, it reduces part of the difficulty previously addressed by complex pixel decoders. The resulting objective is not ordinary image-reconstruction error weighted equally across timesteps. The velocity conversion makes it equivalent to a time-reweighted image-prediction error, so replacing it with plain MSE would change the training procedure. Equations (1) through (3) are corrupted in the cached text extraction. This explanation follows the intact prose and does not reconstruct the damaged equations as purported verbatim author formulas.
A Worked Example¶
Consider PixelU-B/16 at 256×256. Interpreting /16 as the patch size gives a 16×16 spatial grid, or 256 spatial tokens, excluding any additional class tokens. The first 4 Transformer blocks extract shallow features and preserve information for later skip connections. One spatial downsampling operation produces an 8×8 grid, or 64 spatial tokens. The middle 4 blocks model semantics at this resolution, while the channel width remains 768. The network then upsamples to 16×16, and the final 4 blocks combine deep and shallow features to refine the output into a clean-image estimate.
These token counts are a structural illustration derived from the patch size and the paper's half-resolution design, not an additional measurement. They describe internal feature compression, not generation of a finished low-resolution image followed by a separate super-resolution diffusion model. During training, the ground-truth clean image supplies supervision. During sampling, only noise and a class condition are available, and the network call above is repeated. For 512-resolution results, Table 4 uses the /32 variant. The resolution comparison therefore also changes patch size and is not an experiment that increases image resolution while keeping tokenization fixed.
Loss & Training¶
The principal loss is mean squared error between predicted and true velocities, although the network directly outputs the clean image. REPA representation alignment is also applied at the middle block with weight 0.01. REPA is a training auxiliary, not an image VAE added to the inference path. The model also uses in-context class tokens to provide conditioning through the sequence. Table 8 adds these general-purpose enhancements incrementally, so the headline result should not be attributed entirely to the bare U-shaped architecture.
Training uses ImageNet-1K, a global batch size of 1024, AdamW with betas (0.9, 0.95), a constant learning rate of 0.0002, and 8 B200 GPUs. Timesteps follow logit-normal sampling with logit mean -0.8 and standard deviation 0.8; the EMA decay is 0.9999. Evaluation generates 50K images and reports FID, sFID, IS, and Precision/Recall. Sampling uses 50 Heun steps, classifier-free guidance (CFG), and a CFG interval.
Table 4 reports PixelU NFE as 100×2. The 50 Heun steps should not be mistaken for 50 network evaluations, and the listed network GFLOPs should not be treated as the total cost of generating an image. Figure 4 labels its samples with CFG=4.0, but this does not establish that every quantitative table uses that same optimal CFG value. Core architectural ablations explicitly use 160 epochs and cfg=1 and must be interpreted separately from the final guided results.
Key Experimental Results¶
Main Results¶
The following results come from Tables 3 and 4 and use CFG for ImageNet class-conditional generation. Lower FID and higher IS are better. GFLOPs are the reported network computation, not measured end-to-end latency.
| Resolution | Model | Epochs | GFLOPs | FID | IS |
|---|---|---|---|---|---|
| 256×256 | JiT-G/16 | 600 | 383 | 1.82 | 292.6 |
| 256×256 | DeCo-XL/16 | 600 | Not reported | 1.69 | 304 |
| 256×256 | PixelU-H/16 | 320 | 137 | 1.77 | 306.86 |
| 256×256 | PixelU-H/16 | 600 | 136.9 | 1.63 | 305.88 |
| 512×512 | JiT-H/32 | 600 | 183 | 1.94 | 309.1 |
| 512×512 | PixelU-H/32 | 600 | 138 | 1.92 | 322.10 |
For the 600-epoch, 256-resolution PixelU, the table uses the more precise 136.9 GFLOPs from Table 3; Table 4 rounds it to 137. This is approximately 35.7% of JiT-G's compute, a reduction of about 64.3%, not a measured threefold speedup. At 512 resolution, the FID improvement over JiT-H is only 0.02, and parameter counts differ. This should not be described as a large gain at equal model size.
Three internal discrepancies in the source deserve preservation: the 320-epoch FID is 1.77 in Table 4 but 1.76 in nearby prose; the 512 model is H/32 in Table 4 but H/16 in the text; and H/32 has 1171.8M parameters in Table 2 but 1152M in Table 4. The table above follows the results table for FID and model naming rather than silently reconciling these differences.
Ablation Study¶
Table 1 compares modules added to the same JiT-B/16 baseline on ImageNet 256×256, with 160 epochs and cfg=1. The PixelDiT decoder entry is the authors' reproduction, not an official result for the complete PixelDiT model.
| Added module | GFLOPs | Clean-image-prediction FID | Velocity-prediction FID |
|---|---|---|---|
| None, JiT-B/16 | 21.99 | 43.65 | 190.11 |
| DDT decoder | 25.35 | 43.48 | 189.50 |
| DeCo decoder | 25.91 | 41.20 | 42.07 |
| PixelDiT decoder, authors' reproduction | 33.78 | 39.20 | 44.85 |
| Skip connections | 23.20 | 38.53 | 185.54 |
Table 5 fixes clean-image prediction and uses the same resolution, training duration, and cfg=1 to test whether the two architectural choices complement one another.
| Skips | Downsampling | GFLOPs | Parameters | FID | IS |
|---|---|---|---|---|---|
| No | No | 21.99 | 131.1M | 43.65 | 34.08 |
| No | Yes | 18.25 | 153.6M | 48.87 | 30.29 |
| Yes | No | 23.20 | 135.8M | 38.53 | 38.57 |
| Yes | Yes | 19.46 | 158.3M | 34.92 | 43.51 |
Key Findings¶
- Downsampling alone hurts FID, but combining it with skips reduces FID by 8.73 relative to the baseline and lowers GFLOPs by approximately 11.5%. Parameters increase, so lower compute does not imply fewer parameters.
- In Table 6, [4,4,4] reaches FID 34.92, versus 35.47 for [5,2,5] and 38.08 for [3,6,3]. Balanced allocation is a conclusion from this B-scale ablation, not a claim that the larger H model should also use equally deep sections.
- Table 7 reports FIDs of 38.53, 34.92, and 42.70 for 0, 1, and 2 downsampling stages. With constant channels, further compression can create an overly narrow bottleneck; a deeper U-Net hierarchy is not necessarily better.
- In Table 8, H/16 at 160 epochs progresses from FID 12.41 for the vanilla model to 8.93 with class tokens, 6.95 with REPA, and 2.42 with CFG and the CFG interval. Reaching 1.63 additionally requires extending training to 600 epochs.
Highlights & Insights¶
- Changing the prediction target before judging architectural necessity is more informative than comparing modules in isolation. Table 1 suggests that some supposedly essential detail modules compensate for difficulties specific to an output parameterization.
- Reducing spatial tokens and providing a detail bypass are interdependent operations. The degradation from downsampling alone and improvement from the combined design in Table 5 provide more direct support than a visually appealing frequency plot by itself.
- Figure 5 uses t-SNE on 10 ImageNet classes with 100 samples per class, while Figure 6 analyzes intermediate features through a two-dimensional FFT. They support the semantic-bottleneck interpretation through class organization and frequency distribution, but do not constitute a complete causal proof.
Limitations & Future Work¶
- Experiments focus on ImageNet class-conditional generation and do not establish the same benefit for open-domain text-to-image generation, complex typography, or video generation.
- Any pixel-space state-of-the-art claim requires a restricted comparison set: the text mentions reference FIDs of 1.61 for PixelDiT and 1.38 for SiD2, neither included in Table 4, while latent-space RAE-XL has FID 1.13 in that table. FID 1.63 is not the absolute best among all reported methods.
- Better global FID does not imply superiority on every metric. At 256 resolution, PixelU has sFID 5.04 versus DeCo's 4.59; at 512 resolution, PixelU has Recall 0.58 versus DeCo's 0.60.
- The final model includes REPA, and parameter counts, training budgets, and sampling settings are not identical across methods. The GFLOPs savings are not accompanied by matched-hardware measurements of throughput, peak memory, or total training cost.
- Low frequency does not contain all semantics, and fine-grained recognition can depend on texture. Future work could test single-stage compression on distributions where high-frequency cues determine classes or small objects matter, and release complete operator and training configurations.
Related Work & Insights¶
- vs JiT: PixelU inherits clean-image prediction but changes internal feature routing. Rather than processing the same spatial resolution throughout, it introduces an explicit semantic bottleneck and detail skips.
- vs U-ViT / U-DiT: Neither U-shaped topology nor skip connections are new. The contribution is the combination of pixel-space generation, clean-image prediction, single-stage downsampling, and constant channels, supported by targeted ablations.
- vs DeCo / PixelDiT: These methods assign responsibilities through specialized pixel decoders or detail-modeling modules. PixelU instead reduces this cost through shallow-to-deep feature transfer. The conclusion applies to the tested settings, not to every possible task that might benefit from a complex decoder.
- vs latent diffusion: PixelU avoids VAE image compression but still compresses internal features. The transferable insight is to distinguish generating in pixel space from requiring full spatial resolution throughout the backbone.
The code link comes from the paper's release statement; the repository's current implementation availability was not checked online for this note.
Rating¶
- Novelty: 4/5. The components are classical, but the analysis of interactions between prediction targets and architectural necessity is valuable.
- Experimental Thoroughness: 4/5. Multiple scales, two resolutions, and targeted ablations are covered; open-domain tasks and measured runtime costs remain missing.
- Writing Quality: 3/5. The argument is clear, but several numbers and model names are inconsistent, and claims such as zero cost and pure semantics are too strong.
- Value: 4/5. The work offers an understandable, computationally economical architecture for direct pixel generation, although reproduction still requires fuller implementation details.