title: >- [Paper Note] HRDiT: Training-Free High-Resolution Image Generation with Off-the-Shelf Diffusion Transformer Models description: >- [ECCV 2026][Image Generation][Diffusion Transformer] Addressing spatial disorder and quadratic latency in training-free high-resolution DiT generation, HRDiT introduces Spatial Position Alignment (SPA) and Head-adaptive Attention Pruning (HAP) to achieve high-fidelity 2K/4K/8K synthesis with substantial speedups. tags: - ECCV 2026 - Image Generation - Diffusion Transformer - Training-Free Generation - Attention Pruning - High-Resolution Synthesis date: 2026-09-19 content_hash: 1b3ac7dfd387eb97

HRDiT: Training-Free High-Resolution Image Generation with Off-the-Shelf Diffusion Transformer Models¶

Conference: ECCV 2026
Paper: ECCV Official
Full Text Cache: /Users/zy/workspace/paper_cache/ECCV2026/eccv-4252.txt
Area: Image Generation
Keywords: Text-to-high-resolution image generation / Diffusion Transformer / Training-free adaptation / Spatial position alignment / Head-adaptive attention pruning

TL;DR¶

Addressing the two fundamental bottlenecks of off-the-shelf Diffusion Transformers (DiTs) when generating high-resolution images in a training-free manner—spatial disorder and quadratic generation latency—HRDiT introduces Spatial Position Alignment (SPA) via token index bundling and sliding to restore positional expressiveness, alongside Head-adaptive Attention Pruning (HAP) via one-pass Taylor expansion and integer linear programming to prune redundant attention computations, achieving superior visual fidelity and substantial inference speedups across 2K, 4K, and 8K resolutions without any model fine-tuning.

Background & Motivation¶

Text-to-high-resolution image generation has recently attracted substantial research attention due to its vast potential in digital art, commercial advertising, and game development. Existing approaches generally fall into training-based and training-free categories. Training-based methods fine-tune or train dedicated generators for high-resolution synthesis, but require immense compute and large-scale paired high-resolution datasets. In contrast, training-free methods aim to unlock the generative capability of off-the-shelf diffusion models originally trained at standard resolutions (e.g., 1024×1024). However, prior training-free literature has predominantly centered on U-Net architectures (such as ScaleCrafter, HiDiffusion, and FreeScale). With modern Diffusion Transformers (DiTs) such as Stable Diffusion 3 and FLUX substantially outperforming U-Nets at standard resolutions, adapting off-the-shelf DiT models to high-resolution generation in a training-free regime represents a critical yet largely underexplored challenge.

When attempting to scale off-the-shelf DiT models directly to higher resolutions—either by feeding high-resolution initial noise or by employing existing DiT-tailored flow methods like I-Max—two severe challenges emerge that critically hinder practical deployment. First is spatial disorder: synthesized images suffer from noticeable object duplications, geometric distortions, and anatomical dismemberment. Second is long generation time: due to the quadratic complexity ($O(T^2)$) of multi-head self-attention, generating an 8K (8192×8192) image with FLUX requires over 1,385 seconds, where multi-head attention accounts for more than 90% of the total inference latency.

A rigorous theoretical and empirical investigation reveals the underlying causes of both issues. Theoretically, the pseudo-dimension of the attention position embedding mechanism imposes a strict upper bound on its capacity to distinguish pairwise token positions ($h(T) \le |S_{pe}|$). As the token count $T$ scales up with resolution, this upper bound fails to grow proportionally, leaving fewer than 10% of tokens properly distinguishable in 4K FLUX synthesis and directly triggering spatial disorder. Empirically, different attention heads develop specialized, stable roles: some handle long-range semantic coherence, while others attend strictly to local textures. Uniformly applying generic local-window pruning inevitably disrupts global semantics unless fine-tuned. The core idea is: manipulating token indices prior to the position function via complementary bundling and sliding operations to restore positional expressiveness (SPA), while employing a one-pass Taylor expansion combined with integer linear programming to adaptively allocate optimal attention receptive fields across heads (HAP), enabling high-fidelity, high-efficiency high-resolution DiT generation without any retraining.

Method¶

Overall Architecture¶

HRDiT is specifically designed to adapt off-the-shelf DiT models to high resolutions without touching model weights or training pipelines. The pipeline comprises two decoupled, orthogonal components: 1. Spatial Position Alignment (SPA): Operates during attention position embedding injection. It intercepts token indices, groups them into sequential bundles to constrain the diversity of the position function input space, and executes a sliding boundary mechanism across multiple parallel passes to restore intra-bundle uniqueness. 2. Head-adaptive Attention Pruning (HAP): Conducts an efficient one-time preparatory step prior to inference. Leveraging a single full-attention forward pass and first-order Taylor expansion on the attention matrix, it quantifies image quality degradation and computational cost for every candidate window across all heads, solves for the Pareto-optimal window allocation under a global budget constraint via Integer Linear Programming (ILP), and compiles the plan into FlashInfer GPU kernels for execution.

The overall pipeline and data flow are illustrated in the flowchart below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["High-Resolution Latent Noise Input<br/>T visual tokens"] --> B["Spatial Position Alignment (SPA)<br/>Bundle grouping & Slide boundary shift"]
    B --> C["Attention Contribution Computation<br/>Parallel average c_ij^(SPA)"]
    C --> D["One-Time Profiling & ILP Solver (HAP)<br/>Taylor expansion loss estimation + ILP"]
    D --> E["Head-Adaptive Window Kernel Execution<br/>Per-head optimal receptive fields"]
    E --> F["High-Resolution Image Latent Output<br/>Decoded by VAE to high-fidelity image"]

Key Designs¶

1. Spatial Position Alignment (SPA): Restoring Positional Expressiveness via Index Mapping In mainstream DiT backbones (FLUX, SD3), spatial awareness is injected into query, key, and value vectors via position function $f_{pe}$. Let $c_{i,j} = g(x_i, x_j, f_{pe}(i,j))$ represent token $j$'s contribution to the attention output at position $i$. The paper proves that the number of distinguishable pairwise positional subsets $h(T)$ under distance metric $dis_g$ is bounded by the pseudo-dimension $\xi$ of function class $\mathcal{G}$: $$h(T) \le \lambda \cdot \Big(\sup_{S_{pe}}|g(x_i, x_j, f_{pe}(i,j))|\Big)^{2\xi}$$ When scaling to 4K/8K, token count $T$ increases dramatically. Ideally, $h(T) = |S_{pe}|$ is required to prevent spatial overlap. However, in practice, less than 10% of test samples in 4K FLUX satisfy this bound, causing pairwise distinctions to drop below discrimination threshold $\epsilon$ and precipitating spatial disorder.

To address this without retraining, SPA operates directly on token indices: - Bundle: Groups adjacent tokens sequentially into bundles, passing bundle indices $\phi_{bundle}(i)$ instead of raw indices $i$ into $f_{pe}$. Middle bundles have size $N$, while the first bundle has size $N_1$ ($1 \le N_1 \le N$). This reduces the input diversity from $T$ to $(T+N-N_1)/N$, shrinking $|S_{pe}|$ and restoring macro-level positional discrimination. However, a single bundling operation eliminates positional distinctions among tokens within the same bundle. - Slide: To recover fine-grained intra-bundle distinctions, SPA constructs $N$ distinct mapping functions $\{\phi_{bundle}^{(N_1=1)}, \dots, \phi_{bundle}^{(N_1=N)}\}$ by progressively varying the first bundle size $N_1$ from 1 to $N$. Across the $N$ mapping functions, the bundle index combination assigned to each token index is strictly unique.

During inference, SPA computes the attention contribution by averaging the $N$ parallel bundled passes: $$c_{i,j}^{SPA} = \frac{1}{N} \sum_{n=1}^{N} g\bigg(x_i, x_j, f_{pe}\Big(\phi_{bundle}^{(N_1=n)}(i), \phi_{bundle}^{(N_1=n)}(j)\Big)\bigg)$$ Because these $N$ mappings execute in parallel across tensor dimensions, SPA incurs negligible latency overhead (e.g., rising from 203s to only 212s for 4K FLUX) while completely eliminating structural collapse and object duplications.

2. One-Pass Taylor Expansion Estimation (HAP Step 1): Zero-Cost Head Sensitivity Quantification Uniformly enforcing a fixed local window across all heads severely impairs visual fidelity because attention heads exhibit marked specialization: certain heads capture global structure and cross-modal alignment, whereas others concentrate on localized texture synthesis. Brute-force evaluation across $N_{head}$ heads and $N_{scope}$ candidate scopes would require $N_{head} \times (N_{scope}-1)$ full model passes, which is computationally prohibitive.

Inspired by MoA, HAP formulates an analytical approximation based on first-order Taylor expansion. In a one-time calibration step on a few validation prompts, only a single full-attention pass is executed. Let $L$ denote the generated image loss and $A \in \mathbb{R}^{T \times T}$ be the attention matrix of head $n_{head}$. For candidate scope $n_{scope}$, let $S_{omit}^{(n_{scope})}$ denote the set of omitted token index pairs $(u, v)$. Accounting for the Softmax normalization derivative, quality degradation $I_q(n_{head}, n_{scope})$ is analytically estimated as: $$I_q(n_{\text{head}}, n_{\text{scope}}) \approx \sum_{(u,v) \in S^{(n_{\text{scope}})}_{\text{omit}}} \left( \frac{\partial L}{\partial A(u,v)} \cdot (-A(u,v)) - \sum_{w \in \{1\dots T\} \setminus \{v\}} \frac{\partial L}{\partial A(u,w)} \cdot \frac{A(u,w) \cdot A(u,v)}{1 - A(u,v)} \right)$$ Meanwhile, computational cost $I_c(n_{head}, n_{scope})$ is defined as the total number of participating tokens within the scope. The entire calibration takes under an hour, eliminating expensive iterative forward evaluations.

3. Integer Linear Programming Optimization (HAP Step 2): Pareto-Optimal Scope Assignment Given quantitative estimates of quality drop $I_q$ and computational cost $I_c$, HAP casts window allocation into a constrained combinatorial optimization problem. For a target computation ratio $r_c \in [0, 1]$ (set to $r_c = 0.1$ in main benchmarks, retaining only 10% of theoretical attention operations): $$\min_{\{n_{\text{scope}}^{(1)}, \dots, n_{\text{scope}}^{(N_{\text{head}})}\}} \sum_{n_{\text{head}}=1}^{N_{\text{head}}} I_q\big(n_{\text{head}}, n_{\text{scope}}^{(n_{\text{head}})}\big) \quad \text{s.t.} \quad \sum_{n_{\text{head}}=1}^{N_{\text{head}}} I_c\big(n_{\text{head}}, n_{\text{scope}}^{(n_{\text{head}})}\big) \le r_c \times \sum_{n_{\text{head}}=1}^{N_{\text{head}}} I_c(n_{\text{head}}, N_{\text{scope}})$$ This formulation is solved via Integer Linear Programming (ILP) using standard solvers (e.g., Gurobi) within minutes. The optimal per-head window configuration is precompiled into FlashInfer GPU kernels, enabling runtime execution to eliminate 90% of redundant attention computation while preserving global semantic fidelity.

A Worked Example¶

Consider generating a 4K image using a DiT model with latent token count $T=4096$ and middle bundle size $N=4$: 1. SPA Step: For token index $i=5$, the mapping evaluates across $N_1 \in \{1, 2, 3, 4\}$. When $N_1=1$, $\phi_{bundle}^{(1)}(5) = \lceil (5-1)/4 \rceil = 1$; as $N_1$ shifts to 2, 3, and 4, the bundle boundaries shift rightward. Tokens $i=4$ and $i=5$ receive distinct multi-pass bundle signatures $\{1, 1, 1, 0\}$ vs $\{1, 1, 1, 1\}$, preventing ambiguous collisions in $f_{pe}$ without expanding the effective input index domain. 2. HAP Step: Across 24 attention heads with $N_{scope}=50$ discretized candidate ratios, the ILP solver allocates full 100% receptive fields ($n_{scope}=50$) to critical global cross-attention heads, while background detail heads are assigned compact 5% or 10% local neighborhoods. FlashInfer kernels dispatch directly against this static schedule, reducing per-step attention latency by over 50%.

Key Experimental Results¶

Main Results¶

Experiments were conducted on NVIDIA H200 GPUs using 1,000 randomly sampled captions from LAION-5B. The benchmarks evaluate U-Net baselines (SDXL + FreeScale), direct DiT scaling, and training-free methods including DemoFusion, DiffuseHigh, I-Max, FreCaS, and HiFlow. Metrics include overall FID/KID, patch-level FIDp/KIDp, CLIP score, and per-image generation latency.

Table 1: Quantitative results on 2K and 4K image generation (reproduced from Table 1 of the paper)

Resolution	Backbone	Method	FID ↓	FIDp ↓	KID ↓	KIDp ↓	CLIP Score ↑	Latency (s) ↓
2K	SDXL	+ FreeScale	73.23	65.04	0.0093	0.0132	30.57	26
	SD3	Direct	81.34	70.88	0.0215	0.0194	30.23	17
	SD3	+ DemoFusion	72.21	59.24	0.0132	0.0125	31.40	35
	SD3	+ DiffuseHigh	70.57	57.39	0.0119	0.0112	31.59	20
	SD3	+ HiFlow	70.49	57.35	0.0126	0.0110	31.62	18
	SD3	+ HRDiT (Ours)	67.90	52.59	0.0081	0.0073	31.81	11
	FLUX	Direct	73.96	69.56	0.0173	0.0164	29.63	47
	FLUX	+ DemoFusion	68.95	60.53	0.0127	0.0120	30.83	94
	FLUX	+ DiffuseHigh	67.40	58.63	0.0115	0.0108	31.01	52
	FLUX	+ I-Max	68.37	56.96	0.0121	0.0106	30.67	85
	FLUX	+ HiFlow	67.45	56.79	0.0117	0.0105	30.73	49
	FLUX	+ HRDiT (Ours)	64.42	51.55	0.0074	0.0067	31.27	35
4K	SDXL	+ FreeScale	74.85	68.89	0.0112	0.0157	30.49	248
	SD3	Direct	84.29	74.45	0.0243	0.0231	30.18	108
	SD3	+ DemoFusion	73.39	60.75	0.0141	0.0133	31.16	229
	SD3	+ DiffuseHigh	71.88	58.76	0.0130	0.0126	31.35	115
	SD3	+ HiFlow	71.39	58.40	0.0133	0.0118	31.59	112
	SD3	+ HRDiT (Ours)	68.63	53.04	0.0089	0.0079	31.79	58
	FLUX	Direct	75.68	72.73	0.0192	0.0188	29.57	203
	FLUX	+ DemoFusion	71.18	62.38	0.0150	0.0142	30.48	649
	FLUX	+ DiffuseHigh	69.23	59.13	0.0132	0.0125	30.97	215
	FLUX	+ I-Max	70.11	59.83	0.0142	0.0133	30.52	484
	FLUX	+ HiFlow	68.36	57.43	0.0124	0.0114	30.62	209
	FLUX	+ HRDiT (Ours)	64.91	51.84	0.0078	0.0074	31.17	116

At the extreme resolution of 8K (8192×8192), HRDiT demonstrates even larger advantages (Table 3 of the paper): on SD3, HRDiT achieves 69.51 FID in 454s (compared to DemoFusion's 76.07 FID in 2720s); on FLUX, HRDiT achieves 65.73 FID in 827s, outperforming direct scaling (78.47 FID in 1708s) and DemoFusion (75.72 FID in 5749s) by wide margins.

Ablation Study¶

To isolate the distinct contributions of SPA and HAP, the authors evaluated ablation variants on 4K image generation using FLUX:

Table 2: Ablation study on key components of HRDiT (reproduced from Table 2 of the paper, 4K FLUX)

Configuration	FID ↓	FIDp ↓	KID ↓	KIDp ↓	CLIP Score ↑	Latency (s) ↓	Note
FLUX (Direct)	75.68	72.73	0.0192	0.0188	29.57	203	Baseline with severe spatial disorder and high latency
Ours (w/o SPA)	75.85	72.83	0.0195	0.0192	29.55	110	HAP pruning only; fast but spatial structure collapses
Ours (w/o HAP)	64.82	51.81	0.0076	0.0074	31.19	212	SPA alignment only; best quality but latency remains high
Ours (Full)	64.91	51.84	0.0078	0.0074	31.17	116	Full model: negligible quality drop, ~45% latency reduction

Key Findings¶

Strict orthogonality and complementarity: Removing SPA degrades FID by over 10.9 points (from 64.91 to 75.85), proving that positional alignment is mandatory to maintain coherent spatial geometry. Introducing HAP slashes 4K FLUX latency from 212s to 116s (a 45.3% reduction) with a negligible FID change of 0.09, proving the precision of adaptive scope allocation.
Decisive advantage over diffusion-path modifications: Compared to leading methods like HiFlow and I-Max, HRDiT reduces 4K FLUX FID by 3.45 to 5.2 points and runs 4.1× faster than I-Max (116s vs 484s).

Highlights & Insights¶

Theoretical grounding via pseudo-dimension: Rather than resorting to trial-and-error heuristics, the paper translates learning theory to DiT position embeddings, diagnosing the root cause of spatial disorder as an expressiveness bound breach ($h(T) \le \lambda \cdot (\sup |g|)^{2\xi}$).
Non-invasive index manipulation: SPA rectifies positional representations purely by transforming token indices fed into $f_{pe}$, leaving internal weights, architecture, and attention computations entirely intact.
One-pass Taylor profiling and ILP for visual pruning: Eliminating the empirical trial-and-error of conventional vision pruning, HAP determines per-head receptive fields analytically in under an hour, offering a generic paradigm transferable to video DiTs and long-sequence multi-modal models.

Limitations & Future Work¶

Static resolution calibration: While HAP's calibration step is one-off, the derived configuration is tied to a specific target resolution and cost ratio ($r_c$), requiring separate profiling runs for dynamic resolution shifts.
Extreme aspect ratios: The bundling and sliding intervals are primarily validated on square aspect ratios; non-standard wide panoramas (e.g., 8:1) may require anisotropic geometric adaptation.
Absence of timestep-adaptive windowing: Currently, HAP applies a uniform scope configuration across all diffusion timesteps. Since early denoising steps govern global layout while later steps refine local textures, timestep-aware dynamic pruning could yield even greater acceleration.

vs DemoFusion / DiffuseHigh / HiFlow: These methods alter the reverse sampling trajectory or fuse multi-scale features, often multiplying sampling latency. HRDiT targets the DiT architecture's inner attention and position encoding mechanisms, resolving structural distortion while accelerating inference.
vs I-Max: I-Max is restricted to Rectified Flow models and introduces significant latency overhead via projected flow. HRDiT is universally compatible with diverse DiTs (SD3 and FLUX) and outperforms I-Max in both speed and fidelity.
vs Classical Local-Window Attention (e.g., Neighborhood Attention): Classical methods enforce identical geometric windows across all heads and require retraining to suppress boundary artifacts. HAP preserves global receptive fields on sensitive heads without any fine-tuning.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Pioneering theoretical diagnosis of DiT position embedding breakdown at scale, paired with elegant training-free SPA and HAP designs.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 2K, 4K, and 8K resolutions on multiple modern DiT backbones with rigorous ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Clear progression from mathematical problem formulation to engineering execution, supported by precise figures and tables.
Value: ⭐⭐⭐⭐⭐ Provides an immediate, production-ready framework for low-cost, ultra-high-resolution synthesis with off-the-shelf DiT models.