CUST : Clustered Unit-level Similarity Transformer for Lightweight Image Super-Resolution

Conference: ECCV 2026
Paper: ECCV Official Page · PDF
Code: https://github.com/jwgdmkj/CUST
Area: Image Restoration / Lightweight Image Super-Resolution
Keywords: Similarity clustering, cross-window attention, high-frequency residuals, overlapping windows, hardware efficiency

TL;DR

CUST groups distant but similar patches using pooled window representatives, applies attention within the resulting groups, and guides local refinement with downsample-upsample feature residuals: CUST-Base improves average PSNR over CATANet by 0.094 dB across five ×4 benchmarks, while the separate Base+ configuration demonstrates a practical memory-latency trade-off.

Background & Motivation

Single-image super-resolution must recover edges, lines, and repeated textures from a low-resolution input. Convolutional networks handle local patterns well but cannot directly retrieve similar structures far away. Global attention connects distant locations, at a cost quadratic in the number of spatial tokens. Window methods such as SwinIR make attention cheaper but constrain information exchange. For building facades and manga linework, a useful reference texture need not be spatially adjacent, while simply enlarging windows increases memory use and runtime.

Superpixel and representative-token approaches organize interactions by content, but generating representatives, calculating image-wide similarities, and moving data can introduce their own bottlenecks. CUST therefore targets more than parameter counts or FLOPs. It separates two complementary information needs: locating similar structures within a bounded region larger than a window, and recovering local high-frequency details removed by smoothing. The former need not construct all image-wide pairwise relationships, and the latter need not rely on continuously enlarging attention windows.

Core idea: route patches across windows using pooled representatives, preserve relevant connections through masked neighboring-chunk expansion, and compensate for the detail limitations of small local windows with downsample-upsample feature residuals.

Method

Overall Architecture

A 3×3 convolution extracts shallow features from the low-resolution RGB input. Stacked deep extraction blocks then process these features in the order CANA, ConvFFN, MEDA, and ConvFFN, with four LayerNorm layers per block. CANA handles cross-window content interactions, while MEDA repairs local textures using error guidance. A reconstruction module upsamples the final features, and the result is combined with the input's global residual branch to produce the high-resolution image.

The diagram separates CANA's routing and neighboring-chunk attention into two designs, while grouping MEDA's error extraction, gating, and overlapping attention into a third. These contributions follow the actual processing order; the intervening ConvFFNs are standard feature-processing components within each block.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Low-resolution input<br/>3×3 shallow convolution"] --> B["Window-Representative Clustering<br/>CANA routing"]
    B --> C["Neighbor Expansion and Masking<br/>CANA attention"]
    C -->|ConvFFN| D["High-Frequency Error Compensation<br/>MEDA overlapping attention"]
    D -->|Continue stacking after ConvFFN| B
    D -->|Final block followed by ConvFFN| E["Reconstruction and upsampling<br/>Global residual fusion"]
    A -->|Input residual branch| E

Key Designs

1. Window-Representative Clustering: replace spatial adjacency with content adjacency inside a search region

CANA partitions the feature map into search regions of size \(GS\times GS\), each containing windows of size \(ws\times ws\). Non-overlapping average pooling, with both kernel size and stride equal to the window size, produces one representative token per window. Each search region consequently contains \((GS/ws)^2\) representatives. These representatives provide reference points for content matching; they do not permanently replace all original patches with compressed outputs. Every patch is compared with every window representative in its search region using dot-product similarity. An argmax assigns a representative ID, and argsort brings patches with the same ID together even when they originated in different spatial windows.

The important distinction concerns the search scope and the objects that interact. Routing crosses original window boundaries but remains bounded by the search region in a single operation; it is not unrestricted full-image attention. The default \(ws=8\) and \(gs=GS/ws=10\) give an \(80\times80\) region and 100 representatives. The sorted sequence is divided into fixed-capacity chunks for attention. A representative with many assigned patches can therefore occupy several neighboring chunks, so a semantic group and a fixed-capacity computation chunk are not necessarily the same object. Pooling, one similarity calculation, and sorting avoid iterative representative generation, but routing and sorting still have costs.

2. Neighbor Expansion and Masking: repair connections cut by fixed-capacity chunks

If patches sharing a representative are separated by a computation-chunk boundary, attention confined to each chunk still misses relevant structure. CANA uses the current chunk as Query and concatenates the rear half of the previous chunk, the entire current chunk, and the front half of the next chunk as expanded Key/Value candidates. For equally sized interior chunks, this doubles the candidate length instead of constructing dense attention over the entire search region. Here, a neighbor is adjacent in the sorted sequence, not immediately above or beside a patch in the image. The cached main text does not specify the exact padding or boundary policy for the first and last chunks.

Sorting by representative ID does not make numerical distances between different IDs semantically meaningful, so expansion requires filtering. The attention mask permits an interaction only when Query and Key have the same assigned representative ID. It does not test whether they originally came from the same spatial window. Using the prose definition of the mask, the core operation can be written in readable form as:

\[ Y_n=\operatorname{GELU}\!\left[\operatorname{Softmax}\!\left(\frac{Q_nK_{\mathrm{exp}}^\top}{\sqrt d}+M\right)V_{\mathrm{exp}}\right]\odot G_n, \qquad M_{ij}=\begin{cases}0,&a_i=a_j,\\-\infty,&a_i\ne a_j.\end{cases} \]

Here, \(a_i\) is the assigned window-representative ID, \(d\) is the attention scaling dimension, and \(G_n\) is a learned gate. Some operators are missing from the cached equation's text extraction; the expression above restores the additive-mask notation according to the accompanying prose rather than introducing a new method. The gate modulates aggregated responses, but the main text does not give its complete parameterization, so its activation and projection structure should not be invented. Clustering changes the organization of interactions, and the output must correspond to spatial positions again before subsequent spatial processing; the exact reordering implementation is not expanded in the main text either.

3. High-Frequency Error Compensation: identify information lost to smoothing before local overlapping attention

MEDA does not primarily attempt another expansion of global context. Instead, it helps bounded local attention identify what needs repair. It downsamples and upsamples the features following CANA, then subtracts the smoothed result from the original features: \(X_{\mathrm{error}}=X-\operatorname{Up}(\operatorname{Down}(X))\). This error is a feature-space residual, not a supervised difference between the super-resolved output and the high-resolution ground truth. It highlights edges and fine textures that the smoothing operation struggles to preserve, without constituting an exact frequency-domain decomposition.

The residual feeds two branches. The Refiner applies a 3×3 dilated convolution, GELU, and a 1×1 convolution to turn pointwise differences into contextual corrections. The Spatial Gate uses a 1×1 convolution and sigmoid to determine where and how strongly to compensate. Their elementwise product is scaled by a learned coefficient \(\alpha\) and added back to the original features. Overlapping-window self-attention, adopted from HPI-Net, then restores local relationships in the compensated features. The compensation is:

\[ X_{\mathrm{refined}}=X+\alpha\left[\operatorname{Conv}_{1\times1}\!\left(\operatorname{GELU}(\operatorname{DilConv}_{3\times3}(X_{\mathrm{error}}))\right)\odot\sigma\!\left(\operatorname{Conv}_{1\times1}(X_{\mathrm{error}})\right)\right]. \]

Steps 1–5 of Algorithm 1 constitute this error extraction and compensation, called MFA in the ablation study; overlapping attention is step 6. Removing MFA therefore does not remove the entire MEDA module. CANA's representative window defaults to 8, whereas MEDA cycles through windows of 12, 14, 16, and 18 across blocks. These are distinct settings. The main text does not specify the downsampling factor, interpolation method, dilation rate, or overlap ratio. The term "multi-frequency" does not justify assuming additional parallel frequency-band branches that the algorithm does not show.

A Worked Example

Consider one complete \(80\times80\) search region. Default CANA forms 100 window representatives from its 6,400 spatial patches, then assigns each patch to its most similar representative among those 100. Suppose two distant vertical window-frame patches in a building image receive the same representative ID. Sorting moves them into the same sequence segment. This is an illustrative assignment, not an actual clustering result reported by the paper.

If that segment crosses a computation-chunk boundary, half-chunk expansion can expose same-ID patches on the other side, while masking excludes any different IDs that were also included. After a ConvFFN, MEDA processes the aggregated features. Sharp frame edges are smoothed by downsampling and upsampling, so their residuals highlight locations needing correction. The Refiner supplies neighboring texture context, the Gate controls compensation strength, and overlapping attention integrates local structure. The reconstruction head finally produces the ×4 image. If this region alone were used as an input, its output dimensions would be \(320\times320\); this only illustrates the scale relationship and does not imply that actual inference independently crops every search region.

Loss & Training

Training uses the 800 training images of DIV2K, which also contains 100 validation images. Input patches are 64×64, the batch size is 32, and augmentation uses random horizontal flips and 90-degree rotations. Training runs for 500,000 iterations with Adam, \(\beta_1=0.9\) and \(\beta_2=0.99\). The initial learning rate is \(5\times10^{-4}\) with 20,000 warm-up iterations; it is subsequently halved at iterations 250,000, 400,000, 450,000, and 475,000. The gradient-clipping threshold is 0.1.

Base uses 40 channels and 12 blocks, while Small uses 30 channels and 8 blocks. The "+" variants preserve these channel and block counts but use a uniform MEDA window size of 18; the suffix does not indicate test-time self-ensemble. The cached main text does not explicitly state the supervised loss, degradation kernel, or the color-channel and border-cropping conventions for PSNR/SSIM. This note therefore does not silently fill in L1 loss or a standard evaluation protocol from other super-resolution papers.

Key Experimental Results

Main Results

The following selection from Tables 1 and 2 reports ×4 performance. Each quality cell contains PSNR in dB / SSIM, with higher values better. The authors state that the lightweight comparisons use DIV2K training. FLOPs retain the source table's units and values, but the main text does not fully specify the FLOPs input dimensions at that table.

Method	Parameters / FLOPs	Set5	Set14	B100	Urban100	Manga109
SwinIR-light	930K / 64G	32.44/0.8976	28.77/0.7858	27.69/0.7406	26.47/0.7980	30.92/0.9151
CATANet	535K / 34G	32.58/0.8998	28.90/0.7880	27.75/0.7427	26.87/0.8081	31.31/0.9183
CUST-Base	740K / 91G	32.70/0.9008	29.00/0.7899	27.81/0.7441	26.92/0.8085	31.45/0.9190
CUST-Base+	740K / 98G	32.59/0.9000	28.94/0.7893	27.81/0.7444	26.91/0.8086	31.48/0.9191
LMLT-Base	672K / 41G	32.38/0.8971	28.79/0.7859	27.70/0.7403	26.44/0.7947	31.09/0.9139
CUST-Small	309K / 42G	32.46/0.8982	28.85/0.7862	27.73/0.7411	26.60/0.7995	31.14/0.9145

Base improves over CATANet by 0.12, 0.10, 0.06, 0.05, and 0.14 dB across the five datasets, yielding an arithmetic mean of 0.094 dB. Its parameter count and FLOPs are nevertheless higher, so the contribution is not a universal reduction in computation. The listed Small-versus-LMLT-Base differences average 0.076 dB. The paper's prose reports 0.063 dB, which does not match those five entries; this note uses the value reproducible from the table.

The next table selects the ×4 hardware results from Table 3. Measurements use an NVIDIA RTX 3090 with 320×180 inputs; latency is averaged over 50 randomly selected images. Memory retains the original Mem(M) scale and is monitored using torch.cuda.max_memory_allocated().

Method	Peak Memory Mem(M)	Inference Time (ms)
SwinIR-light	350.57	195.89
SRFormer-light	329.08	197.18
HIT-SIR	1331.16	143.74
MambaIR-light	438.25	130.08
CATANet	1818.34	144.89
CUST-Base+	328.17	146.04

At this scale, Base+ uses approximately 81.95% less memory than CATANet but is slightly slower. Its latency is approximately 25.94% lower than SRFormer-light's. The authors' approximately 82.8% mean memory reduction across ×2, ×3, and ×4 uses a different aggregation scope. This table does not provide Base latency, so Base's strongest accuracy must not be paired with Base+'s runtime as though they describe one configuration.

Ablation Study

The following results are from Table 4, all at ×4. The deeper single-module variants use 12 blocks to approach Small's parameter budget; they do not exactly match its parameters or FLOPs. Full Small uses 8 blocks.

Configuration	Parameters / FLOPs	Urban100 PSNR/SSIM	Manga109 PSNR/SSIM
Full CUST-Small	309K / 42G	26.60/0.7995	31.14/0.9145
CANA only, original depth	227K / 29G	26.31/0.7914	30.80/0.9118
MEDA only, original depth	214K / 27G	26.21/0.7871	30.61/0.9087
CANA only, increased depth	307K / 38G	26.51/0.7966	30.98/0.9133
MEDA only, increased depth	288K / 35G	26.38/0.7926	30.81/0.9113

Removing CANA costs 0.39/0.53 dB on Urban100/Manga109, more than the 0.29/0.34 dB lost by removing MEDA. The full model remains stronger even when single-module depth is increased. This supports complementarity between distant content interactions and local detail refinement, although the remaining computation-budget differences are not completely controlled away.

Key Findings

• In Table 5, replacing CANA with Swin Attention lowers Urban100 from 26.60/0.7995 to 26.53/0.7975. Removing KV expansion gives 26.58/0.7985, and reducing the search-region factor from 10 to 5 gives 26.59/0.7987. The overall routing mechanism has a larger PSNR effect than neighboring-chunk expansion alone.
• In Table 6, windows [14,16,18,20] with MFA achieve 26.64/0.8002 at 314.6M memory, while larger windows [16,18,20,22] without MFA achieve 26.62/0.7994 at 343.2M. The former uses approximately 8.3% less memory, suggesting that compensation can replace some window expansion.
• The trend is not universal dominance on every metric. Default windows with MFA give 26.60/0.7995, versus 26.61/0.7989 for windows enlarged by 2 without MFA: slightly lower PSNR but higher SSIM. Base+ likewise does not consistently outperform Base.

Highlights & Insights

• Clustering is valuable because it reorganizes interactions between original patches, not merely because it produces a small set of representatives. Fixed-capacity chunks support computation, while masked neighboring-chunk expansion repairs relevant connections broken by chunking.
• MEDA separates high-frequency localization from local relationship modeling. Residuals identify what smoothing discarded before overlapping attention integrates details, explaining why smaller windows can sometimes provide a better accuracy-memory trade-off.
• Hardware evaluation reveals deployment value that parameter counts alone miss. Despite more FLOPs than CATANet, CUST-Base+ achieves similar latency and substantially lower peak allocated memory on the measured GPU, so FLOPs alone do not determine the ranking.

Limitations & Future Work

• The author acknowledges that stochastic or irregular textures can make patch-to-window affinity ambiguous and reduce clustering effectiveness. Texture-stratified failure analysis and local fallback for low-confidence assignments are plausible next steps, not validated results of this paper.
• Deblurring, denoising, and diverse real degradation processes remain insufficiently explored. Results on five standard super-resolution benchmarks do not establish robustness for medical images or real smartphone photography.
• This note's assessment: a single CANA operation has a bounded search region, and hard representative assignments can discard useful secondary matches. Group size, sorting cost, and throughput at different resolutions warrant more detailed component-level analysis.
• This note's assessment: RTX 3090 memory and latency measurements are not mobile-device deployment validation, and Base+ is not the fastest method in the hardware table. The cache contains the main paper through its references but not the cited Appendices B–E; Test2k results, scaling curves, and additional visualizations therefore cannot be checked here, and no numerical claims from them are invented.
• Reproduction still requires checking the unspecified loss, resampling operators, overlap ratio, and evaluation preprocessing. The inconsistency in the reported Small average gain is another reason to inspect individual table entries.

Related Work & Insights

• Compared with CATANet: CUST explicitly adapts its similarity-clustering paradigm, combining window representatives with bounded search regions and same-ID neighboring-chunk expansion. The contribution is better understood as a hardware-oriented improvement to content aggregation than as the first proposal of semantic clustering attention.
• Compared with SwinIR / SRFormer: These approaches reduce costs through window-based or permuted attention, whereas CANA reorganizes possible interactions according to content affinity. This relaxes original window restrictions without eliminating search-region boundaries or routing costs.
• Compared with HPI-Net: MEDA adopts its overlapping attention; the added focus is high-frequency error refinement and gated compensation before attention. Transferring this mechanism to other restoration tasks first requires checking whether the residual still identifies useful detail in the presence of noise.

Rating

• Novelty: 3.5/5. A targeted combination of existing content clustering and overlapping attention, with explicit incremental contributions from neighbor masking and error compensation.
• Experimental Thoroughness: 4/5. Three scales, five benchmarks, hardware measurements, and several ablation levels provide substantial evidence, but runtime variance, mobile hardware, and real degradation tests are missing.
• Writing Quality: 3.5/5. The algorithm and ablations support the main argument, although some reproduction details are unspecified and one average improvement conflicts with the table.
• Value: 4/5. Useful for memory-constrained lightweight super-resolution, particularly as an example of combining content routing with error-guided compensation.