UltraGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attention

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/PeterYYZhang/UltraImageGen/
Area: Image Generation
Keywords: diffusion models, ultra-high-resolution image generation, local attention, hierarchical guidance, computational efficiency

TL;DR

UltraImageGen addresses the quadratic attention explosion and scarce high-resolution training data in text-to-image diffusion transformers by pairing window-first local window attention with position-scaled low-resolution global guidance and lightweight LoRA joint denoising, delivering state-of-the-art 4K/8K synthesis with over 10x acceleration without requiring any native high-resolution training data.

Background & Motivation

As applications across digital art creation, ultra-high-definition commercial advertising, scientific visualization, and virtual environments demand increasingly fine textures alongside rigorous global structural fidelity, synthesizing ultra-high-resolution images (4K, 8K, and beyond) has emerged as a crucial frontier for generative modeling. Nevertheless, despite recent breakthroughs in text-to-image diffusion transformers such as FLUX.1, SD3, and PixArt, state-of-the-art models remain largely confined to sub-2MP resolutions (typically below 1Kx2K). Directly scaling these models to 4K causes the spatial token sequence length to surge into tens of thousands, severely restricting their practical deployment in demanding production pipelines.

The inability of existing frameworks to cleanly scale to ultra-high resolutions stems from two deeply interconnected bottlenecks. First, conventional self-attention mechanisms scale quadratically with token count (\(O(N^2)\)), where stepping from 1K (4,096 tokens) to 4K (65,536 tokens) increases computational overhead by a staggering 256-fold, triggering out-of-memory failures even on flagship GPUs equipped with FlashAttention. Second, native high-quality 4K/8K paired image-text training datasets are exceptionally scarce and financially prohibitive to curate; existing training-based adaptations must rely on synthetic high-resolution datasets or heavy fine-tuning that degrade aspect-ratio robustness and introduce structural artifacts, while training-free extrapolation approaches (e.g., DemoFusion, I-MAX, CLEAR) often suffer from intolerable per-step iteration latency, seam artifacts, or anatomical corruptions such as deformed hands and multiple faces.

The central insight of this work is that long-range dependencies in image generation primarily govern macro-level spatial composition and semantic structure, whereas fine-grained high-frequency details depend almost entirely on localized spatial contexts. Consequently, an ultra-high-resolution model does not require dense global attention across all high-resolution tokens. Instead, the generation process can be hierarchically decoupled: local windows capture sharp textures via lightweight hardware-aligned attention, while a positionally scaled low-resolution latent serves as an anchor injecting macro-level semantic guidance. Core idea: decouple ultra-high-resolution denoising into hardware-aligned window-first local attention and position-scaled low-resolution global guidance, bridged by parameter-efficient LoRA joint denoising on commodity 1K data, achieving near-linear complexity, over 10x inference speedup, and robust structural fidelity across 4K and 8K synthesis without any native high-resolution training data.

Method

Overall Architecture

UltraImageGen introduces a modular hierarchical reformulation into the Multi-Modal Diffusion Transformer (MMDiT) architecture. The input high-resolution noisy latent is partitioned into fixed-size non-overlapping local windows, permuted into a window-first token sequence so that spatial self-attention executes strictly within GPU-friendly dense local blocks while maintaining relative RoPE positions within the familiar range of the pretrained model. Simultaneously, a low-resolution guidance latent is projected into the high-resolution spatial coordinate space via coordinate scaling, acting as a global semantic anchor. Within each transformer block, an asymmetric joint-denoising mechanism with an attention mask and lightweight LoRA updates low-resolution projection layers while freezing high-resolution local generative weights. At test time, a low-resolution guidance warm-start initialization skips early low-frequency denoising steps, further accelerating inference.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Text prompt and ultra-high-resolution target request"] --> B["Stage 1: Window-first local window partitioning<br/>256×256 window reordering and local self-attention"]
    B --> C["Stage 2: Position-scaled low-res global guidance<br/>ρ=4 coordinate scaling and RoPE anchor injection"]
    C --> D["Stage 3: Parameter-efficient LoRA joint denoising<br/>Tune only low-res guidance Q/K/V projections"]
    D --> E["Stage 4: Guidance-guided warm-start sampling<br/>Interpolated noisy reference bypassing early steps"]
    E --> F["Synthesized 4K/8K ultra-high-resolution image"]

Key Designs

1. Efficient local window attention with window-first permuted token sequence: reducing quadratic cost to near-linear while anchoring RoPE

Directly calculating full self-attention over high-resolution image latents \(X \in \mathbb{R}^{H \times W}\) incurs an impractical \(O(N^2)\) complexity, where transitioning from 1024x1024 (4,096 tokens) to 4096x4096 (65,536 tokens) incurs a 256-fold computational spike. UltraImageGen partitions the spatial grid into non-overlapping local windows \(x_i \in \mathbb{R}^{l \times l}\). With a standard 16x VAE downsampling factor, choosing \(l=16\) (corresponding to 256x256 image patches) slashes the computational footprint to \(O(\lceil \frac{H}{l} \rceil \lceil \frac{W}{l} \rceil \cdot l^4) \approx O(N \cdot l^2)\), yielding an exact 256-fold reduction in attention complexity at 4K. Crucially, standard raster-scan token flattening scatters 2D window-adjacent tokens across wide sequence offsets, conflicting with GPU kernel tiling. The proposed window-first token permutation reorders tokens by local window blocks prior to attention calculation, causing the attention pattern to degrade into dense local blocks that directly map onto optimized GPU tiles in FlashAttention-2 and SageAttention (e.g., Q-block=128, K-block=64). Furthermore, this reordering ensures that RoPE relative positional offsets remain strictly bounded within the original 256x256 training window size, eliminating catastrophic position extrapolation failures while permitting cross-window attention along adjoining borders to suppress tiling seams.

2. Low-resolution global guidance via position-scaled anchors: maintaining structural coherence across distant regions

Relying solely on isolated local window attention inevitably causes semantic drift and structural fragmentation, such as multiple disjoint faces or mangled limb anatomy. The core technical mechanism relies on the observation that long-range context guides holistic scene layout rather than fine local brushstrokes. UltraImageGen introduces a parallel low-resolution guidance latent \(X_{lr} \in \mathbb{R}^{h \times w}\), parameterized by an aspect-ratio scaling factor \(\rho = \frac{H}{h}\) (empirically set to \(\rho=4\) to optimize the accuracy-efficiency trade-off). To align low-resolution tokens with the physical high-resolution grid, their coordinates \((m, n)\) are scaled to the target resolution:

\[(\tilde{m}, \tilde{n}) = (\rho \cdot m, \rho \cdot n)\]

These positionally aligned low-resolution tokens serve as spatial anchors. A composite attention mask \(M\) governs interactions across the unified token sequence: text tokens attend to themselves and to high-resolution tokens; high-resolution tokens attend to their local window, adjacent boundary pixels, the text prompt, and the scaled low-resolution anchors; while low-resolution tokens execute global self-attention among themselves. This design preserves long-range semantic coherence across the entire image and naturally supports recursive cascading, where an existing 4K output serves as low-resolution guidance for synthesizing 8K or higher resolutions.

3. Parameter-efficient joint denoising with LoRA adaptation: commodity-resolution training without native 4K data

Concatenating the low-resolution stream produces the joint representation \([C_T; X; X_{lr}]\). Because the positional indices of \(X_{lr}\) are expanded by \(\rho\), the frequency characteristics of the RoPE positional encodings undergo a structural shift. Directly applying pretrained attention projection matrices would distort guidance signals. UltraImageGen introduces an asymmetric adaptation scheme: all original \(Q, K, V\) projection weights for high-resolution tokens \(X\) are completely frozen, preserving the base model's strong generative and texture synthesis priors; simultaneously, lightweight LoRA matrices (rank 16, \(\alpha=16\)) are attached exclusively to the \(Q, K, V\) projections processing \(X_{lr}\), yielding adapted matrices \(\tilde{Q}, \tilde{K}, \tilde{V}\). Omitting text tokens for brevity, the joint multimodal attention is formulated as:

\[\text{MMA}([X; X_{lr}]) = \text{Softmax}\left(\frac{[Q(X), \tilde{Q}(X_{lr})][K(X), \tilde{K}(X_{lr})]^T \cdot M}{\sqrt{d}}\right) [V(X), \tilde{V}(X_{lr})]\]

The framework is trained using the flow matching objective:

\[\mathcal{L}_{\text{FM}}(\theta; C_T, X_{lr}) = \mathbb{E}_{t, X_t}\left[ \| f_\theta(X_t, t, C_T, X_{lr}) - u_t(X_t; C_T, X_{lr}) \|^2 \right]\]

Crucially, training is performed on a modest dataset of only 10,000 synthetic images at standard 1024x1024 resolution paired with 256x256 guidance latents, requiring zero native 4K training samples while allowing seamless zero-shot scaling to 4K and 8K.

4. Guidance-weighted warm-start sampling: bypassing early low-frequency denoising stages

In standard diffusion sampling trajectories, early timesteps are predominantly consumed by laying down macro-level structural layouts, whereas later timesteps resolve fine-grained high-frequency textures. Because the low-resolution guidance image \(X_{lr}\) already provides a coherent global composition, initializing high-resolution sampling from pure Gaussian noise introduces substantial computational redundancy. UltraImageGen employs a warm-start strategy: the low-resolution latent is bilinearly upsampled to target dimensions, sharpened, and perturbed with noise corresponding to an intermediate timestep \(t\), producing \(X_t^{\text{ref}} = \text{Interpolate}(X_{lr})\). The initial high-resolution latent is then constructed via convex noise-reference interpolation:

\[X_t^{\text{hr}} = \alpha X_T + (1 - \alpha) X_t^{\text{ref}}, \quad \text{where } \alpha = \frac{t}{T}\]

By injecting low-frequency compositional priors directly into the initial state, this warm-start mechanism safely bypasses the computationally expensive initial denoising iterations, reducing total inference steps and runtime by over 30% without sacrificing high-frequency texture diversity.

Loss & Training

The framework is optimized under the Flow Matching formulation, minimizing the squared discrepancy between the network-predicted velocity vector field \(f_\theta\) and the ground-truth velocity field \(u_t\). Training employs the Pyriology learning rate scheduler with a base learning rate of 1.0, weight decay of 0.01, bias correction, and safe-guard warmup. The model is trained for 20,000 steps across two NVIDIA RTX A6000 Ada GPUs (48GB VRAM each) with a per-GPU batch size of 1 and gradient accumulation over 4 steps. During inference, the NTK RoPE scaling factor is set to 10 and dynamic scheduler shifting is disabled.

Key Experimental Results

Main Results

Quantitative evaluations are conducted across a benchmark of 1,000 diverse prompts generated by GPT-4o. Image fidelity metrics (FID and patch-based FIDpatch) are calculated against 10,000 real images sampled from the LAION-High-Resolution dataset, alongside CLIP Score, CLIP-IQA, and Inception Score (IS).

Method	Type	CLIPIQA ↑	FID ↓	FIDpatch ↓	IS ↑	ISpatch ↑	CLIPscore ↑
SANA	4K Trained	0.4457	76.31	74.27	16.68	14.02	0.3197
Diffusion-4K	4K Trained	0.3012	121.85	120.59	14.39	10.77	0.2844
UltraPixel	4K Trained	0.4421	77.42	70.94	16.98	13.26	0.3251
URAE	4K Trained	0.4369	67.39	62.56	17.11	12.39	0.3204
FLUX+BSRGAN	Super-Res	0.3897	71.39	63.45	17.08	12.87	0.3210
UltraFlux	4K Finetuned	0.4371	75.89	65.12	17.01	13.24	0.3165
DemoFusion	Training-Free	0.4392	74.89	66.37	16.23	13.02	0.3187
DiffuseHigh	Training-Free	0.4221	81.54	73.35	16.15	13.08	0.3175
HiDiffusion	Training-Free	0.4021	172.46	217.49	9.43	8.12	0.3129
I-MAX	Training-Free	0.4381	70.33	65.67	16.50	12.69	0.3211
HiFlow	Training-Free	0.4407	69.18	63.72	17.13	13.43	0.3113
CLEAR+SDEdit	Linear Attention	0.4352	88.23	87.19	16.84	13.78	0.3221
Ours (FLUX.1)	Hierarchical Local	0.4505	67.03	61.78	17.21	13.31	0.3231
Ours (FLUX.2)	Hierarchical Local	0.4594	61.38	59.82	18.02	14.93	0.3267

Hardware runtime and memory overhead when generating a single 4K latent (65,536 tokens) across FLUX.1-based models demonstrate substantial practical gains:

Method	Total Latency (s)	Iteration Time (s)	Memory (GB)
FLUX (Vanilla FlashAttention)	659.35	23.20	57.16
URAE	509.88	18.21	52.81
I-MAX	299.10	17.89	47.81
HiFlow	272.79	15.00	47.83
CLEAR + SDEdit	536.79	21.57	50.16
Ours	152.21	7.81	39.71 (Overhead: 7.3GB)

Ablation Study

The ablation study validates the optimal configuration of window size (\(ws\)) and guidance downsampling ratio (\(\rho\)).

Config	Window Size \(ws\)	Ratio \(\rho\)	FID ↓	FIDpatch ↓	Note
Default Optimal	256	4	67.23	61.46	Highest efficiency with competitive fidelity
Smaller Downsampling	256	2	67.19	62.28	Higher guidance cost with marginal gain
Medium Window	512	4	66.31	63.48	Negligible FID gain; slight FIDpatch drop
Large Window	1024	4	67.83	62.61	16x attention FLOPs without quality gains

Ablation over the attention connectivity confirms two vital architectural conclusions: - Excluding Low-Res Guidance (attending only to text): Global composition collapses completely, leading to disjointed patches, unnatural repetitions, and multi-face artifacts. - Excluding Adjacent Window Attention (attending strictly within isolated windows): Noticeable boundary seams emerge across the grid; enabling cross-window boundary interactions costs less than 2% additional computation while entirely eliminating grid seams.

Key Findings

• Hardware-aligned sweet spot for local window sizes: Setting local windows to 256x256 (16x16 in latent space) saturates visual fidelity (FIDpatch of 61.46). Expanding window dimensions exponentially inflates FLOPs without perceptual improvement, whereas smaller windows drop below the 128/64 tile granularity of modern GPU tensor cores.
• Massive throughput scaling at 8K resolutions: Under 8K synthesis stress-testing, naive attention runs out of memory, while UltraImageGen achieves a 130.4-fold FLOPs reduction in attention layers, a 6.93x per-iteration speedup, and a 10.77x total inference speedup over FlashAttention-2.
• Strict resolution-agnostic compositionality: On the GenEval benchmark, overall performance remains rock-solid between 1K and 4K (0.66 to 0.67), with object counting (0.74–0.75), color binding (0.79–0.81), and single-object fidelity (0.98–0.99) exhibiting zero degradation across scaling tiers.

Highlights & Insights

• Window-first token permutation bridges hardware and position encoding: By mapping 2D spatial windows directly into contiguous sequence blocks, the architecture maximizes GPU tensor-core efficiency while confining relative positional offsets within the familiar 256x256 training window, elegantly solving the long-sequence extrapolation challenge.
• Scale decoupling eliminates native 4K data dependencies: Decoupling macroscopic layout from microscopic texture enables training exclusively on commodity 1K synthetic images via lightweight LoRA, dismantling the long-standing assumption that ultra-high-resolution models require massive native 4K training corpuses.
• Orthogonal compatibility with distilled fast-samplers: The framework seamlessly pairs with distillation pipelines like FLUX.2-Klein (4-step generation), yielding sub-30-second 4K generation and sub-2-minute 8K generation on commodity hardware.

Limitations & Future Work

• Author-admitted limitations: Extreme scaling tiers beyond 16K necessitate recursive multi-stage cascading; excessively small guidance ratios may occasionally lose subtle binding on microscopic sub-objects. Additionally, custom cross-window boundary attention requires specialized kernel compilation.
• Broader perspectives: While highly effective for static text-to-image synthesis, scaling this hierarchical local-global paradigm to ultra-high-resolution 4K video diffusion models represents an exciting next step, though it introduces new challenges in inter-frame temporal coherency.

Related Work & Insights

• vs CLEAR / GRAT (linearized and neighbor attention): CLEAR relies on expanding local neighbor radii to preserve quality at higher resolutions, leading to ballooning computation and degraded global coherence that demands SDEdit patching. UltraImageGen maintains a fixed small window size and relies on position-scaled low-resolution anchors, preserving true near-linear complexity.
• vs SANA / PixArt-Σ (native 4K pretraining): SANA requires millions of expensive native 4K images and complex multi-stage pretraining regimes, often suffering from attention blurriness inherent in linear attention. UltraImageGen achieves superior visual quality with zero native 4K data by adapting existing pretrained foundation models.
• vs HiFlow / I-MAX (training-free extrapolation): Training-free methods frequently introduce diagonal frequency streaks or anatomical artifacts and still compute heavy full-attention passes. UltraImageGen's parameter-efficient LoRA joint denoising ensures artifact-free synthesis with over 10x end-to-end acceleration.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Decoupling local window attention via window-first permutation paired with scaled low-resolution semantic anchors is an exceptionally clean and hardware-conscious design.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive 4K benchmark against 14 competitive baselines, GenEval multi-resolution evaluation, 8K scaling stress tests, and rigorous ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Highly coherent narrative with lucid technical motivations, transparent hardware profiling, and rigorous experimental presentation.
• Value: ⭐⭐⭐⭐⭐ Delivers an accessible, scalable, and highly practical blueprint for democratization of 4K/8K image synthesis on commodity hardware.