WARP: Wide Attention with Rich Projections for Image Super-Resolution

Conference: ECCV 2026
Paper: ECCV Official
Area: Image Restoration
Keywords: Image Super-Resolution / Vision Transformer / Wide Attention / Rotary Position Embedding / Parameter Sharing

TL;DR

Addressing the inefficient multi-hop communication of deep shifted-window super-resolution transformers, WARP introduces a wide-attention architecture leveraging 2D RoPE and a cross-layer shared rich nonlinear projection, matching or outperforming deep 30-block SOTA models with only 12 blocks and 20M parameters.

Background & Motivation

Single-image super-resolution (SISR) aims to reconstruct faithful high-resolution (HR) fine details from low-resolution (LR) inputs. Since SwinIR introduced 8×8 shifted local windows into super-resolution, stacking deep networks with narrow windows (e.g., HAT, CAT, ATD, and PFT, which stack 16 to 30+ blocks) has emerged as the dominant paradigm. These models attempt to enlarge their receptive field and capture degradation context via cross-window interaction, stripe attention, or successive layer shifts.

However, shifted small-window attention suffers from an inherent bottleneck of indirect multi-hop communication: features far apart must traverse multiple attention layers through local window bounds. After two layers of 16×16 shifted windows, a given query reaches only 7 of 256 cells across the feature map (a mere 2.7%). This multi-hop propagation is remarkably inefficient when reconstructing natural images abundant in long-range self-similarities, regular lattices, and repeating periodic textures, preventing the network from capturing global structural coherence in early layers.

The fundamental technical barrier preventing prior SR transformers from adopting wide attention over large windows is their reliance on learned relative position bias (RPB). RPB forces the materialization of a full \(N \times N\) dense bias matrix in memory (amounting to ~16.8M entries per head for a 64×64 window), which negates the \(\mathcal{O}(N)\) memory savings of exact attention kernels like FlashAttention and causes out-of-memory errors. The core idea of this paper is to replace additive RPB with 2D Rotary Position Embeddings (RoPE), eliminating the quadratic bias memory bottleneck to enable 64×64 to 128×128 fixed-window wide attention; simultaneously, a high-capacity nonlinear QKV projection module is shared across all 12 blocks to provide 115M-equivalent representational richness within a compact 20M parameter footprint, coupled with inverted-temperature YaRN scaling for resolution-adaptive inference.

Method

Overall Architecture

WARP adopts the standard encoder-reconstruction pipeline consisting of shallow feature extraction, a deep Transformer backbone, and sub-pixel convolution upsampling. Given a low-resolution input \(I_{LR} \in \mathbb{R}^{H \times W \times 3}\), a single \(3 \times 3\) convolution first extracts shallow features \(F_0 \in \mathbb{R}^{H \times W \times C}\) with channel dimension \(C=320\). These features then pass through \(L=12\) cascaded WARP Transformer blocks to yield deep features \(F_L\). The final high-resolution output \(I_{SR}\) is reconstructed through a global residual connection, a \(3 \times 3\) convolution, and PixelShuffle upsampling, stabilized by an additive bilinear upsampling skip path: $\(I_{SR} = \text{PixelShuffle}(\text{Conv}_{3\times3}(F_0 + F_L)) + \text{Upsample}(I_{LR})\)$ where the final \(3 \times 3\) convolution weights are initialized to zero, ensuring that the network operates as an exact bilinear upsampler at the beginning of training.

Each WARP block comprises a dual spatial-channel attention module and a depthwise-convolution-augmented SwiGLU feed-forward network with pre-LayerNorm and residual connections. Crucially, all blocks share a single high-capacity nonlinear QKV projection module, while maintaining lightweight per-head QK-normalization and output projections per block, providing massive spatial context at a modest parameter cost.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Low-Resolution Input ILR<br/>3-Channel Image"] --> Shallow["Shallow Feature Extraction<br/>3×3 Conv (C=320)"]
    Shallow --> Block["12 Cascaded Transformer Blocks<br/>Input Feature F"]

    subgraph CoreBlock["WARP Core Block"]
        direction TB
        BlockIn["Block Input with Pre-LN"] --> SharedQKV["1. Shared Rich Nonlinear Projection<br/>1×1 Conv expand 12C + 3×3 DWConv + 6C output"]
        SharedQKV --> DualSplit["Split into spatial Qs,Ks,Vs and channel Qc,Kc,Vc"]
        DualSplit --> DualAttn["2. Dual Spatial-Channel Wide Attention<br/>Spatial: 2D RoPE + Wide Window Attention<br/>Channel: Transposed Self-Attention O(C²)"]
        DualAttn --> AttnMerge["Element-wise Sum Fusion<br/>1×1 Conv Projection + Residual"]
        AttnMerge --> SwiGLUFFN["SwiGLU Feed-Forward Network<br/>3×3 DWConv + Gated SiLU (Expansion 8/3)"]
    end

    Block --> CoreBlock
    CoreBlock --> BlockOut["Deep Feature FL"]
    BlockOut --> GlobalRes["Global Residual Sum F0 + FL"]
    Shallow -.-> GlobalRes
    GlobalRes --> Reconstruct["PixelShuffle Reconstruction<br/>3×3 Conv + Sub-Pixel Convolution"]
    In -.->|Bilinear Upsampling| AddBilinear["Residual Sum"]
    Reconstruct --> AddBilinear
    AddBilinear --> Out["High-Resolution Output ISR"]

    subgraph InferExtrap["Inference Resolution Adaptation"]
        direction TB
        TileEngine["3. YaRN-Based Window & Inverted Temperature Extrapolation<br/>Tiled Processing + 16px Overlap + Sharpening Temperature τ<1"]
    end
    CoreBlock -.->|Inference Adaptation| InferExtrap

Key Designs

1. Shared Rich Nonlinear Projection: Replacing Depth Stacking with High-Capacity Feature Projections Standard vision transformers rely on a single linear transformation \([Q, K, V] = X W_{QKV}\) for query, key, and value generation. While computationally simple, linear mappings offer limited expressive capacity, forcing conventional models to stack 20 to 30+ blocks to accumulate multi-step nonlinear representations. In a shallow 12-block wide-attention model, each block directly faces thousands of tokens and requires far more discriminative projection features. WARP introduces a nonlinear projection module with spatial mixing: input features \(X \in \mathbb{R}^{H \times W \times C}\) are expanded to dimension \(E = r \cdot C\) (with expansion ratio \(r=12\), resulting in 3,840 channels) via a \(1 \times 1\) convolution, activated by GELU, processed by a \(3 \times 3\) depthwise convolution with a residual shortcut, and compressed back to \(6C\) channels via another \(1 \times 1\) convolution: $\(H = \text{GELU}(\text{Conv}_{1\times1}(X)), \quad H' = \text{GELU}(\text{DWConv}_{3\times3}(H)) + H, \quad Z = \text{Conv}_{1\times1}(H')\)$ This rich module contains approximately 8.6M parameters. If instantiated independently per block, it would balloon the total model size to 115M parameters. Because WARP uses large position-fixed windows where all blocks operate over structurally aligned feature maps, a single nonlinear projection module is shared across all 12 blocks. Each block retains only lightweight LayerNorm and a per-block \(1 \times 1\) output projection (\(C^2\) parameters), amortizing the parameter cost and holding the entire network at just 20.0M parameters.

2. Dual Spatial-Channel Wide Attention with 2D RoPE: Bias-Free O(N) Global Communication Existing shifted-window models are constrained by relative position bias (RPB) tables, whose \(\mathcal{O}(N^2)\) memory footprint becomes prohibitive at window sizes beyond 16×16. WARP generalizes Rotary Position Embeddings to 2D image coordinates (2D RoPE), splitting each head's channels \(d_h = C / n_h\) (\(n_h=10, d_h=32\)) equally into horizontal (\(x\)) and vertical (\(y\)) sub-dimensions: $\(\text{RoPE}(q, x, y) = [R(x) \cdot q_{1:d_h/2}] \,\|\, [R(y) \cdot q_{d_h/2+1:d_h}]\)$ where rotary matrices \(R(m)\) apply frequency rotations \(\theta_i = b^{-2i/(d_h/2)}\) with base \(b=10000\). Because the inner product \((R_m q)^\top (R_n k) = q^\top R_{n-m} k\) depends strictly on the relative coordinate displacement \(n-m\), no external \(N \times N\) bias matrix is materialized. This allows wide attention over 64×64 windows (4,096 tokens) to directly leverage FlashAttention with exact \(\mathcal{O}(N)\) memory complexity. Concurrently, the shared projection's \(6C\) output is partitioned into spatial and channel triplets. Spatial attention computes dense affinities across the broad window, while channel attention calculates inter-channel covariances \(\mathcal{O}(C^2)\). Both branches are merged via element-wise addition and a \(1 \times 1\) convolution, capturing global spatial context and cross-channel semantics simultaneously.

3. YaRN-Based Window & Inverted Temperature Extrapolation: Removing Window Size Boundaries During training, WARP uses fixed 64×64 patches. At test time, to handle arbitrary HR resolutions without stitching artifacts, feature maps are divided into overlapping tiles (e.g., 128×128 tiles with 16-pixel overlap for ×2 SR, processing 16,384 tokens per tile). To resolve RoPE degradation caused by unseen high-frequency rotary angles on enlarged windows, WARP adapts YaRN's NTK-aware frequency scaling to 2D RoPE: $\(\theta'_i = \theta_i \cdot s^{-i/(d_h/4 - 1)}\)$ where \(s = T / T_0\) is the spatial extrapolation ratio. Crucially, whereas LLM YaRN applies a temperature factor \(\sqrt{0.1 \ln s + 1} > 1\) to flatten attention distributions over long texts, doing so in SR causes severe blurring. In image upscaling, doubling the tile size mostly adds redundant background tokens, while high-frequency textures remain localized. Attention distributions must be sharpened rather than flattened. WARP therefore introduces an inverted temperature factor \(\tau < 1\): $\(\text{Attn}(Q, K, V) = \text{softmax}\left(\frac{\text{RoPE}(Q)\text{RoPE}(K)^\top}{\tau \sqrt{d_h}}\right) V\)$ Optimal temperatures found via validation sweeps are \(\tau = 0.80, 0.83, 0.89\) for ×2, ×3, and ×4 scales (corresponding to tile sizes \(T=128, 112, 96\)), enabling flexible, high-fidelity inference across diverse dimensions.

Loss & Training

WARP follows a two-stage training scheme: 1. Base Pretraining Phase (500K iterations): Trained on DF2K (DIV2K 800 images + Flickr2K 2,650 images) using bicubic degradation. Low-resolution training patches are sized 64×64 (HR 256×256 for ×4). Optimized using Adam (\(\beta_1=0.9, \beta_2=0.99\), zero weight decay) with batch size 32 under a Warmup-Stable-Decay (WSD) schedule: 1K warmup steps to \(2 \times 10^{-4}\), a stable phase, and a cosine decay over the final 10% steps down to \(10^{-6}\). The objective is standard \(L_1\) pixel reconstruction loss: $\(\mathcal{L}_1 = \|I_{SR} - I_{HR}\|_1\)$ Maintained with an Exponential Moving Average (EMA) decay factor of 0.999. 2. YaRN Extrapolation Fine-Tuning Phase (50K iterations): Initialized from Phase 1 EMA weights, the model is fine-tuned for 50K steps with learning rate \(1 \times 10^{-4}\) at scale-specific tile sizes (128 for ×2, 112 for ×3, 96 for ×4) and inverted temperatures \(\tau\), allowing seamless adaptation to long-range rotary angle distributions.

Key Experimental Results

Main Results

Quantitative evaluations are conducted on the luminance (Y) channel of five benchmarks (Set5, Set14, BSD100, Urban100, Manga109), reporting PSNR (dB) and SSIM across ×2, ×3, and ×4 upscaling factors:

Scale	Method	Train Set	Params	Set5	Set14	BSD100	Urban100	Manga109
×2	SwinIR [16]	DF2K	11.8M	38.42 / 0.9623	34.46 / 0.9250	32.53 / 0.9041	33.81 / 0.9433	39.92 / 0.9797
	HAT [5]	DF2K	20.6M	38.63 / 0.9630	34.86 / 0.9274	32.62 / 0.9053	34.45 / 0.9466	40.26 / 0.9809
	ATD [32]	DF2K	20.1M	38.61 / 0.9629	34.95 / 0.9276	32.65 / 0.9056	34.70 / 0.9476	40.37 / 0.9810
	PFT [18]	DF2K	19.6M	38.68 / 0.9635	35.00 / 0.9280	32.67 / 0.9058	34.90 / 0.9490	40.49 / 0.9815
	WARP (Ours)	DF2K	20.0M	38.71 / 0.9633	34.99 / 0.9278	32.67 / 0.9058	34.99 / 0.9494	40.52 / 0.9813
×3	SwinIR [16]	DF2K	11.9M	34.97 / 0.9318	30.93 / 0.8534	29.46 / 0.8145	29.75 / 0.8826	35.12 / 0.9537
	HAT [5]	DF2K	20.8M	35.07 / 0.9329	31.08 / 0.8555	29.54 / 0.8167	30.23 / 0.8896	35.53 / 0.9552
	ATD [32]	DF2K	20.3M	35.11 / 0.9330	31.13 / 0.8556	29.57 / 0.8176	30.46 / 0.8917	35.63 / 0.9558
	PFT [18]	DF2K	19.8M	35.15 / 0.9333	31.16 / 0.8561	29.58 / 0.8178	30.56 / 0.8931	35.67 / 0.9560
	WARP (Ours)	DF2K	20.0M	35.24 / 0.9337	31.16 / 0.8560	29.59 / 0.8179	30.75 / 0.8952	35.79 / 0.9564
×4	SwinIR [16]	DF2K	11.9M	32.92 / 0.9044	29.09 / 0.7950	27.92 / 0.7489	27.45 / 0.8254	32.03 / 0.9260
	HAT [5]	DF2K	20.8M	33.04 / 0.9056	29.23 / 0.7973	28.00 / 0.7517	27.97 / 0.8368	32.48 / 0.9292
	ATD [32]	DF2K	20.3M	33.10 / 0.9058	29.24 / 0.7974	28.01 / 0.7526	28.17 / 0.8404	32.62 / 0.9306
	PFT [18]	DF2K	19.8M	33.15 / 0.9065	29.29 / 0.7978	28.02 / 0.7527	28.20 / 0.8412	32.63 / 0.9306
	WARP (Ours)	DF2K	20.0M	33.18 / 0.9054	29.30 / 0.7987	28.04 / 0.7524	28.26 / 0.8431	32.73 / 0.9312

Ablation Study

1. YaRN Resolution Extrapolation and Temperature Scaling (×2 SR, Tile 64 → 128, \(s=2.0\))

Configuration	DF2K (1,000 Patches)	Urban100 (Full)	Note
Baseline: 500K Pretrained (Tile = 64)	38.64 dB	34.85 dB	Training resolution baseline
Naïve Extrapolation (Tile = 128, No Scaling)	38.58 dB	34.72 dB	Direct extrapolation drops performance (-0.13 dB)
+ Standard YaRN Flat Temperature (\(\tau \approx 1.03\))	38.54 dB	34.58 dB	LLM attention flattening hurts SR (-0.14 dB)
+ Inverted Temperature YaRN (\(\tau = 0.80\))	38.67 dB	34.95 dB	Attention sharpening yields immediate gain (+0.23 dB)
+ Inverted Temperature + 50K Fine-Tuning	38.70 dB	34.99 dB	Adaptation yields additional +0.04 dB

2. From-Scratch Window Size Ablation (×4 SR, C=320, 200K Iterations, Identical Backbone)

Window Size	Set5 (dB)	Set14 (dB)	BSD100 (dB)	Urban100 (dB)	Manga109 (dB)
16 × 16	32.65	28.93	27.74	26.84	31.40
32 × 32	32.96	29.12	27.90	27.49	32.05
48 × 48	33.05	29.18	27.97	27.83	32.31
64 × 64	33.13	29.27	28.01	28.08	32.51

3. QKV Projection Module Architecture Ablation (C=96, Iso-parameter \(\sim 0.9M\), ×4 SR, Set5)

Configuration	Params	Blocks (L)	Set5 PSNR	Performance Gap (\(\Delta\))
Full Model (Shared + Nonlinear + Dual)	\(\sim 0.9\text{M}\)	4	32.48 dB	—
−Nonlinear (Standard Linear QKV)	\(\sim 0.9\text{M}\)	4	32.18 dB	-0.30 dB
−Nonlinear (Compensated with Depth L=9)	\(\sim 0.9\text{M}\)	9	32.47 dB	-0.01 dB (Requires 2.25× attention passes)
−Shared (Independent Projections, \(r=2\))	\(\sim 0.9\text{M}\)	4	32.41 dB	-0.07 dB
−Dual (Spatial-Only Attention, \(r=14\))	\(\sim 0.9\text{M}\)	4	32.43 dB	-0.05 dB

Key Findings

• Wide windows are the decisive causal driver for structural restoration: The from-scratch window ablation confirms that scaling the window size from 16×16 to 64×64 yields modest gains on smooth images (Set5 +0.48 dB) but dramatic improvements on texture-rich scenes (+1.24 dB on Urban100, +1.11 dB on Manga109), proving that direct long-range communication resolves the multi-hop bottleneck.
• Image super-resolution requires sharpening temperature rather than flattening: Adapting YaRN with an LLM-like flattening temperature factor (\(\tau > 1\)) degrades SR quality due to excessive background token distraction. Inverting the temperature to \(\tau < 1\) (e.g., \(\tau=0.80\)) focuses attention on localized repetitive cues, unlocking a substantial +0.23 dB gain on Urban100.
• Nonlinear projection effectively trades pointwise compute for expensive attention layers: The iso-parameter study reveals that a single rich nonlinear projection module provides the representational power of 5 extra transformer layers, replacing costly quadratic attention operations with efficient pointwise convolutions.

Highlights & Insights

• 2D RoPE unlocks memory-efficient wide attention: By encoding relative spatial coordinates directly into query-key rotations, WARP circumvents the \(\mathcal{O}(N^2)\) memory bottleneck of relative position bias tables, enabling large-window attention to run natively inside FlashAttention kernels with linear \(\mathcal{O}(N)\) memory.
• Cross-layer shared high-capacity projection: Challenging the standard practice of using independent linear QKV projections per block, WARP amortizes an 8.6M-parameter \(12C\)-expanded nonlinear module across all 12 blocks, packing 115M-equivalent expressive power into a lean 20M parameter envelope.
• Domain-aware adaptation of LLM context extension: Successfully importing YaRN into vision transformers, the authors uncover the fundamental difference between text and image context expansion, inventing inverted temperature scaling (\(\tau < 1\)) to sharpen attention over spatially redundant image tiles.

Limitations & Future Work

• Quadratic latency scaling at large extrapolation tiles: While 2D RoPE mitigates peak memory usage, the quadratic attention complexity over 128×128 tiles (16,384 tokens) causes ×2 inference latency on an RTX 4090 to rise to 3,664 ms, compared to 366 ms for ×4 (tile 96×96). Incorporating sparse attention or token pruning could address this trade-off.
• Heuristic temperature selection: Inverted temperatures \(\tau\) are currently selected through discrete grid searches on validation sets. Formulating an end-to-end adaptive mechanism that predicts \(\tau\) dynamically based on image content represents a promising next step.
• Broader image restoration tasks: WARP is primarily evaluated on standard bicubic SISR. Extending the wide-attention and shared nonlinear projection framework to image denoising, deblurring, real-world degradation, and video restoration remains an open direction.

Related Work & Insights

• vs SwinIR / HAT / PFT: Prior transformer SR architectures rely on deep networks (16 to 30+ blocks) with narrow shifted windows and multi-hop communication. WARP replaces this paradigm with a shallow 12-block network utilizing wide position-fixed attention, achieving direct long-range context capture in a single step.
• vs ELAN / DRCN: While earlier parameter sharing in SR focused on repeating convolutional filters across time (DRCN) or sharing attention maps between adjacent layers (ELAN), WARP shares an entire high-dimensional nonlinear projection module across all blocks, leveraging position-fixed window alignment to achieve high representational capacity with low parameter overhead.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneers the use of 2D RoPE and inverted-temperature YaRN in image super-resolution, overturning the deep narrow-window paradigm]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive comparisons across 5 benchmarks, 3 scales, with meticulous ablations on window sizes, projection architectures, and temperature dynamics]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, well-structured mathematical formulations, and compelling narrative flow]
• Value: ⭐⭐⭐⭐⭐ [Opens a promising new design space for vision transformers balancing network depth against per-block projection width]