Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer¶

Conference: ECCV 2026
arXiv: 2511.19778
Code: None
Area: Diffusion Models / Image Generation
Keywords: RoPE, Mixed-Resolution Attention, Diffusion Transformer, Position Encoding Phase Alignment, Inference Acceleration

TL;DR¶

This paper reveals for the first time the root cause of RoPE's failure in mixed-resolution DiTs: namely, attention scores are a sinusoidal periodic function of the relative token distance \(\kappa(\Delta)\). When tokens of different resolutions are mapped to a unified coordinate space, they land on different phases of the \(\kappa(\Delta)\) curve, causing systematic attention score distortion. Based on this, a training-free Phase-Aligned Mixed-Resolution Attention (PMA) is proposed. PMA calculates position offsets for each query-key pair at the original scale of the query, recovering a consistent phase reference. Combined with a lightweight Boundary Enhancement and Refinement module (BER), PMA achieves generation quality comparable to or even better than full resolution with 4x acceleration on the Wan video model and the FLUX image model.

Background & Motivation¶

Diffusion Transformers (DiTs) have become the mainstream architecture for image and video generation, with RoPE serving as their standard positional encoding. As generation resolution continues to scale, the quadratic complexity of attention computation has become a major bottleneck. An intuitive speedup strategy is mixed-resolution processing: assigning high-resolution (HR) tokens to salient regions while allocating low-resolution (LR) tokens to background or non-critical regions, thereby reducing the total token count without losing details. However, in practice, directly mixing LR and HR tokens for self-attention results in blurriness and artifacts, even when mapping token positions across different resolutions into a unified coordinate space via linear position interpolation (PI)—either the LR regions look normal but HR regions collapse, or vice versa.

This diagnostics reveal that the root cause of this conflict lies not in the choice of interpolation scheme, but in the strong position-scale bias that RoPE itself imposes on attention scores. The authors measure the relationship between attention scores and the relative token distance \(\Delta\) in pretrained DiTs, denoted as \(\kappa(\Delta) := \mathbb{E}_{(q,k)}[\langle \hat{q}, \mathcal{R}(\Delta) \hat{k} \rangle]\). They find that \(\kappa(\Delta)\) does not decay smoothly but exhibits clear sinusoidal oscillations: a sharp peak at \(\Delta \approx 0\), followed by alternating high and low values across different distance intervals. This curve remains highly stable across layers and timesteps, and is significantly amplified by RoPE-dominant heads—essentially acting as a sinusoidal phase filter learned by each attention head, with its frequency determined by the pre-defined RoPE frequencies. In single-resolution training, all tokens share the same distance scale, allowing the model to naturally adapt to this bias. However, once mixed-resolution forcibly maps LR/HR tokens to a unified coordinate space, the pairwise distance of at least one token set is compressed or stretched, causing token pairs to land in incorrect phase regions of \(\kappa(\Delta)\). Consequently, some unrelated pairs receive deceptively high attention while relevant pairs are suppressed, leading to systemic distortions in the output. Theoretical analysis further expands the attention scores as \(\sum_i C_i(q,k) \cos(\omega_i \Delta + \phi_i)\), confirming that each attention head implements a learned sinusoidal phase filtering.

Core insight: The key to stabilizing mixed-resolution attention is not finding a better global interpolation scheme, but ensuring that the relative distance of each query-key pair is evaluated at its original scale during pretraining—meaning aligning one pair at a time, rather than aligning all tokens to a single coordinate system.

Method¶

Overall Architecture¶

This paper presents a complete mixed-resolution DiT inference pipeline: during the denoising process of a standard DiT (such as Wan for video / FLUX for image), a lightweight saliency model is employed to identify important regions and assign HR tokens, while secondary regions remain in low resolution. In self-attention calculation, standard RoPE positional mapping is replaced with PMA to guarantee stable cross-resolution attention. At the LR-HR boundary, the BER module performs local content swapping to eliminate texture boundaries' discontinuities. The entire approach is training-free (PMA requires zero parameters, and BER's latent resizer has only 25M parameters), allowing for plug-and-play integration with any pretrained DiT, and naturally stacks with orthogonal acceleration methods like feature caching and step distillation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Noise + Text"] --> B["Coarse Stage: Low-Resolution Denoising<br/>Establish Global Structure"]
    B --> C["Saliency Detection<br/>Select High-Resolution Regions"]
    C --> D["PMA: Query-Scale Aligned<br/>Mixed-Resolution Attention"]
    D --> E["BER: Boundary Content Exchange<br/>Smooth Resolution Transition"]
    E --> F["Fine Stage (Optional)<br/>Full-Resolution Refinement"]
    F --> G["Output Denoised Latent"]

Key Designs¶

1. PMA: Query-Key Pairwise Scale-Aligned Positional Alignment

The motivation of PMA stems directly from the core finding in Section 3—\(\kappa(\Delta)\) is a periodic function, and any global position remapping will push some token pairs into incorrect phases. Therefore, instead of global coordinate unification, PMA centers on each query and re-expresses the key's position at the query's original scale.

Specifically, let \(S_q\) and \(S_k\) denote the original resolution scales of the query and the key, respectively (where \(S\) of HR is larger than LR), and define the scale ratio as \(\alpha_{k \rightarrow q} = S_q / S_k\). For each query-key pair, the key's position is scaled by this ratio: \(p_k^{(q)} = \alpha_{k \rightarrow q} p_k\), and the RoPE is subsequently computed using the scaled key positions. Mathematically, two scenarios exist in practice: (1) HR query to all keys—using the HR grid as the reference, the LR key positions are stretched (e.g., in the toy example, LR \([0,1,2][5,6,7,8]\) is stretched to \([0,2,4][10,12,14,16]\)); (2) LR query to all keys—using the LR grid as the reference, HR key positions are compressed and downsampled via stride sampling (e.g., HR \([6,7,8,9]\) is compressed to \([3,4]\) with stride=2). This design ensures that \(\Delta\) in each dot-product attention is computed at the pretrained scale, thus maintaining the correct phase of \(\kappa(\Delta)\)—at the cost of zero extra parameters, modifying only position indices.

2. BER: Content Exchange and Local Smoothing at Resolution Boundaries

While PMA addresses the phase misalignment of position scales, minor inconsistencies in texture density may still occur at LR-HR boundaries. BER performs lightweight local coordination at the boundaries: during each denoising step, it first expands the LR/HR masks outward by \(n_{\text{pad}}\) (default 2) tokens to form an overlapping band, followed by a bidirectional content exchange. The LR side utilizes a learned latent upsampler to upsample the LR portion of the current clean estimation \(x_0\), which is then re-noised to step \(t-1\) to replace the HR tokens in the overlapping band; the HR side symmetrically performs downsampling and replaces LR tokens. Since the swapped tokens only serve as attention context rather than final outputs, an extremely small resizer model (3D CNN with 25M parameters for Wan, 2D CNN with 6M parameters for FLUX) is sufficient. The resizer is trained using paired data—initially scaled in pixel space and then re-encoded by VAE to obtain the target latents, optimized with a loss consisting of latent-space \(\ell_1\) plus pixel-space \(\ell_1 + \text{LPIPS}\).

3. Saliency-Guided Coarse-to-Fine Three-Stage Inference Scheduling

PMA is a contribution at the attention mechanism level and does not inherently designate which regions should receive high resolution. This paper demonstrates a specific use case: employing an off-the-shelf lightweight saliency model, DeepGaze, on the coarse low-resolution outputs to predict salient regions. Based on the HR token budget (15% for video, 30-60% for image), the highest-scoring regions are selected to form the HR mask before entering the mixed-resolution denoising stage. The entire pipeline comprises three stages: (1) Coarse stage: running on full low-resolution for a few steps to quickly establish global structure and motion; (2) Mixed-resolution stage: employing PMA + BER to stabilize cross-resolution attention while denoising salient regions at high resolution; (3) Optional fine stage: a final few steps of full-resolution refinement. The saliency model only infers once at the end of the coarse stage (taking only 0.27s for video), introducing a negligible overhead to the total latency.

A Full Example: Two Query Modes of PMA in a 1D Toy Case¶

Taking the toy case in Section 3.2 of the paper, the original 1D sequence indices are \([0,1,2,3,4,5,6,7,8]\), and the middle segment \([3,4]\) is upsampled into an HR block. After integerized unification, the indices of the three regions become: LR prefix \([0,2,4]\), HR segment \([6,7,8,9]\), and LR suffix \([10,12,14,16]\). Let us examine how PMA processes the attention of an HR query (at position 7) over all keys: using the HR scale as the reference, the LR key positions are stretched by \(\alpha = S_{\text{HR}} / S_{\text{LR}}\). If the scale ratio from LR to HR is 2, the LR prefix \([0,2,4]\) stretches to \([0,4,8]\), and the LR suffix \([10,12,14,16]\) stretches to \([20,24,28,32]\). Thus, the distances from the HR query 7 to its neighboring HR keys 6, 8, and 9 remain on the original training scale (\(\Delta = -1, 1, 2\)). The distances to LR keys are scaled up but maintain a consistent linear scaling, landing on the correct phase of \(\kappa(\Delta)\). Conversely, for an LR query (at position 0) over all keys: referencing the LR scale, the HR keys \([6,7,8,9]\) are first compressed to \([3,3.5,4,4.5]\), and then downsampled to \([3,4]\) via stride=2. Since the LR query only requires coarse-grained context, the downsampled HR keys provide exactly this information.

Loss & Training¶

PMA itself is training-free as it only modifies the positional index mapping and introduces no learnable parameters. The latent up/downsampler in BER needs to be trained separately. For Wan (3D CNN, hidden dim 384, 25M), the Pexels + Aesthetic-Train-V2 dataset is used, wherein paired targets are obtained through pixel-space scaling followed by VAE re-encoding; the same applies to FLUX (2D CNN, hidden dim 128, 6M). The objective function consists of latent-space \(\ell_1\) (weight 0.01) + pixel-space \(\ell_1\) (weight 1) + LPIPS (weight 0.1), with a batch size of 1. The training overhead is minimal—the resizer parameter size is significantly smaller than the DiT itself, and it only needs to be trained once to be reused across all subsequent inference scenarios.

Key Experimental Results¶

Main Results¶

Video generation is evaluated using Wan2.1-1.3B on the full VBench prompts, and image generation is evaluated using FLUX.1-dev on 5K samples of the MSCOCO 2014 validation set.

Table 1: Comparison of RoPE interpolation methods in mixed-resolution denoising for Wan video

Method	DOVER Aesthetic ↑	DOVER Technical ↑	DOVER Overall ↑	VBench Quality ↑	VBench Semantics ↑	VBench Total ↑	Time (s) ↓
HR (Full HR)	99.83	10.43	79.12	80.12	62.30	76.56	172.1
PI-LR	98.10	8.01	63.39	75.93	54.92	71.73	—
PI-HR	86.52	4.94	35.04	70.38	49.41	66.18	—
NTK	92.76	5.89	44.52	71.80	52.93	68.02	43.2
YaRN	98.56	8.96	66.38	76.39	56.72	72.46	—
PMA (Ours)	99.63	10.01	75.34	80.76	62.17	77.04	43.2

PMA approaches or even slightly outperforms the full-resolution HR baseline across almost all metrics, with DOVER Overall slightly decreasing from 79.12 (HR) to 75.34 (only a 4.8% drop), whereas the best interpolation baseline, YaRN, scores only 66.38 (a 16.1% drop). Concurrently, inference time is slashed from 172.1s to 43.2s (a 4.0x speedup). PI-HR nearly collapses in Technical and Overall scores (4.94 and 35.04), illustrating the fundamental damage caused by phase misalignment on attention quality.

Table 2: Comparison of RoPE interpolation methods in mixed-resolution denoising for FLUX image

Method	ImageReward ↑	CLIP-IQA ↑	MUSIQ ↑	CLIP Score ↑	Time (s) ↓
HR (Full HR)	1.062	0.621	70.47	31.12	3.4
PI-LR	0.659	0.411	53.96	31.41	—
PI-HR	0.935	0.523	70.94	31.41	—
NTK	0.953	0.542	70.62	31.37	2.4
YaRN	0.926	0.548	69.99	31.29	—
PMA (Ours)	0.978	0.623	71.81	31.31	2.4

PMA is only 7.9% lower on ImageReward compared to HR, remains on par or slightly higher on CLIP-IQA (0.623 vs 0.621), and outperforms HR on MUSIQ (71.81 vs 70.47). PI-LR declines significantly by 38% on ImageReward (0.659 vs 1.062), once again validating the systematic failure of position interpolation.

Ablation Study¶

Table 3: Ablation of BER boundary padding width \(n_{\text{pad}}\) (Wan video)

\(n_{\text{pad}}\) (LR)	\(n_{\text{pad}}\) (HR)	DOVER Aesthetic ↑	DOVER Technical ↑	DOVER Overall ↑	VBench Total ↑
0	0	98.90	8.94	68.43	74.74
2	2	99.63	10.01	75.34	77.04
2	4	99.62	9.88	75.18	76.90

When removing BER (\(n_{\text{pad}} = 0\)), DOVER Overall drops from 75.34 to 68.43, and Technical drops from 10.01 to 8.94—confirming that minor texture inconsistencies indeed persist at the boundaries and can be effectively mitigated by BER. \(n_{\text{pad}} = 2\) is sufficient, and further scaling the width yields no benefits.

Key Findings¶

PMA is the core contribution: Relying solely on PMA (training-free, zero-parameter) boots DOVER Overall from YaRN's 66.38 to 75.34 (a 13.5% relative improvement). This demonstrates that phase alignment, rather than global interpolation, is the key to scaling and stabilizing mixed-resolution attention.
BER is the icing on the cake: Removing BER incurs a drop of ~6.9 points in DOVER Overall, particularly on the Technical dimension—indicating that boundary smoothing is sensitive to technical quality metrics but does not fundamentally derail the global structures.
Orthogonal to other acceleration methods: PMA + TeaCache reaches a 7.2x speedup, while PMA + DMD 4-step distillation achieves 30.7x speedup, both while maintaining robust generation quality. This highlights that mixed-resolution acceleration is compatible with other speedup paradigms.
Robust to different saliency models: Employing different saliency models (DeepGaze I / UNISAL / DeepGaze IIE) or even a fixed central bounding box results in minor DOVER Overall fluctuations between 74.22 and 76.26, indicating that PMA's stability does not heavily rely on the choice of saliency detector.
Naturally supports multiple resolutions: A 3-mixed-resolution setup (480p+960p+1920p video) achieves 288s while maintaining a VBench Total score of 77.73, compared to 1995s for native 2K (a 6.9x speedup). The 4-mixed-resolution variant is similarly stable.

Highlights & Insights¶

Attributing engineering failure of RoPE to a quantifiable, visual mathematical structure: Representing and measuring the \(\kappa(\Delta)\) curve is the most elegant aspect of this work—a seemingly "abnormal" behavior (a sinusoidally oscillating attention bias) is naturally exploited by the model in single-resolution training but becomes fatal in mixed-resolution setups. This diagnostic framework can be generalized to any scenario involving cross-resolution/cross-scale RoPE attention, far beyond mixed-resolution denoising.
Contrarian thinking of "bypassing coordinate unification, and only assuring phase consistency at the dot-product level": Traditional RoPE interpolation schemes (PI/NTK/YaRN) try to find a superior global mapping \(p \mapsto \phi(p)\). This paper does the opposite—it acknowledges that no single mapping can simultaneously satisfy phase consistency across multiple scales. Thus, it relinquishes global unification in favor of independently calculating position offsets for each dot-product. This "local-to-local" formulation might be highly applicable to other attention scenarios requiring cross-domain alignment (e.g., cross-modal tokens, multi-frame rate timestamps).
Training-free and plug-and-play features yield practical value exceeding academic contribution: PMA is zero-parameter, zero-training, and only requires a few lines of code to modify positional indices. It can be immediately deployed to any RoPE-based DiT (e.g., Wan, FLUX, CogVideo, LTX), which is highly valued in industrial pipelines. The paper demonstrates this via seamless integration with TeaCache, MagCache, and DMD.
The dependency of \(\kappa(\Delta)\) on modality is worth further exploration: The appendix reveals that \(\kappa(\Delta)\) along the text axis in FLUX is significantly smoother than along the height/width axes, indicating that attention heads learn different phase modifications for different modalities. A potential research direction is explicitly parameterizing modality-specific phase biases to replace RoPE's standard fixed frequencies.

Limitations & Future Work¶

Complex texture transitions remain a weak spot: The authors highlight failures in the appendix—when the LR-HR boundary intersects complex textures (e.g., boundaries between human skin and clothing), BER's local exchange is insufficient to completely resolve transition boundaries, sometimes leaving visible patch boundaries. A potential improvement is leveraging semantic segmentation masks instead of purely relying on saliency masks, confining HR boundaries to semantic borders.
Reliance on the quality and inference cost of the saliency model: Although the DeepGaze series' computational overhead is extremely low (0.01s~0.27s), the quality of saliency detection directly scales with the final generation quality. If the saliency model misses critical regions, the budget for HR tokens is spent poorly. This is ill-suited for long-tail scenarios where saliency models fail (e.g., custom, irregular user-specified regions).
Absence of training-stage mixed-resolution support: PMA functions as an on-the-fly hotfix during inference. The definitive solution would be training DiTs to encounter mixed-resolution tokens during pretraining to learn scale-invariant phase representations. While the authors identify this as future work, it requires redesigning the positional encodings or training paradigms.
Evaluation is limited to Wan and FLUX: Although these represent the state-of-the-art video/image DiTs, the generalizability to other RoPE structural variants (e.g., CogVideo's grouping strategies, LTX's 3D RoPE layouts) remains to be validated.

vs. RoPE Interpolation Methods (PI / NTK / YaRN): These approaches originate from the context length extrapolation domain in LLMs, universally assuming an optimal global frequency scaling strategy tailored for self-consistent extrapolation within a single sequence. However, in mixed-resolution scenarios, both LR and HR tokens coexist in the same layer, meaning their pairwise distances must simultaneously conform to multiple scales—a problem that cannot be addressed by uniform frequency scaling parameters. PMA's key insight is abandoning global scaling in favor of localized pairwise determination.
vs. RALU: RALU also adopts a coarse-to-fine mixed-resolution pipeline. However, it still uses linear position interpolation during the mixed-resolution phase and relies on extra noise injections and multi-step denoising to resolve artifacts induced by phase misalignment—essentially swapping denoising steps for quality. PMA eliminates phase misalignment at its source, requiring no extra compensation steps.
vs. Token Merging (ToMe) / Token Pruning: These methods reduce attention computations by merging or discarding tokens (token-reduction paradigms). Mixed-resolution is a complementary paradigm that "retains token count but lowers the resolution of selected tokens". The two paradigms can be combined (though direct combinations with ToMe are not evaluated in this paper and warrant future study).

Rating¶

Novelty: ⭐⭐⭐⭐☆ (4/5) First to systematically analyze and quantify RoPE's failure under mixed-resolution configurations, presenting \(\kappa(\Delta)\) as an elegant diagnostic utility. While the training-free PMA approach is clean and powerful, the concept of aligning to the query scale is relatively intuitive.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ (5/5) Spans both video and image modalities. Includes four main comparative tables (interpolation comparisons ×2 + acceleration scaling ×2), comprehensive BER ablations, check of robustness against different saliency models, multi-resolution scaling, integration with orthogonal acceleration schemes, computational overhead breakdowns, and 16-dimensional VBench evaluations. The appendix could almost stand alone as its own paper.
Writing Quality: ⭐⭐⭐⭐⭐ (5/5) Exceptionally coherent narrative pathway: theoretical analysis \(\rightarrow\) empirical measurement \(\rightarrow\) method formulation \(\rightarrow\) experimental validation. Section 3 is particularly strong, establishing clear causation from a toy example and the \(\kappa(\Delta)\) curves to the full mathematical expansion.
Value: ⭐⭐⭐⭐⭐ (5/5) The training-free, zero-parameter, plug-and-play characteristics hold substantial industrial landing potential. It serves as an out-of-the-box speedup solution for any RoPE-based DiT contemplating mixed-resolution inference. Furthermore, the \(\kappa(\Delta)\) analytical paradigm can easily generalize to other RoPE application landscapes.