Freqformer: Image-Demoiréing Transformer via Effective Frequency Decomposition¶
Conference: ECCV 2026
Paper: ECCV Original
Code: https://github.com/xyLiu339/Freqformer
Area: Image Restoration
Keywords: Image Demoiréing, Frequency Decomposition, Transformer, Spatial-Aware Channel Attention, Color Distortion
TL;DR¶
Freqformer introduces a smoothness-preserving, anti-aliasing frequency decomposition that splits moiré patterns into local high-frequency textures and scale-robust low-frequency color shifts, processed via asymmetric crop/resize branches and unified by a learnable Frequency Composition Transform (FCT), setting a new state-of-the-art across multiple high-definition demoiréing benchmarks with only 6.065M parameters.
Background & Motivation¶
When digital screens are captured using digital cameras, the spatial frequency interference between the screen's subpixel layout and the camera's color filter array (CFA) inevitably introduces severe moiré patterns. These artifacts manifest as complex, non-stationary ripple textures tightly intertwined with broad color distortions, significantly degrading visual quality and impairing downstream computer vision models. Most deep learning-based approaches treat demoiréing as an end-to-end holistic image restoration task. However, because corruptive moiré ripples mimic natural fine textures while color shifts severely contaminate the scene's intrinsic tone, monolithic spatial-domain restoration networks struggle to disentangle them, frequently yielding over-smoothed structural details or lingering chromatic shifts.
To achieve effective decoupling between structural textures and color distortions, recent studies have explored frequency-domain transformations such as the Haar Discrete Wavelet Transform (DWT) and block-based Discrete Cosine Transform (DCT). Unfortunately, conventional discrete transforms exhibit notable fundamental limitations when applied to non-stationary moiré images. First, recursive wavelet downsampling is shift-variant and prone to secondary aliasing, producing discrete representations that lack spatial smoothness. Second, patch-based DCT assumes local stationarity, thereby failing to capture large-scale curvilinear ripples and introducing conspicuous block boundary artifacts. Third, multi-level wavelets fragment features across directional subbands while block DCT expands features into up to 64 isolated channels, creating massive channel redundancy and high computational overhead. Finally, traditional methods rely on rigid, non-trainable inverse transforms where independent restoration errors across frequency bands are compounded during reconstruction, precipitating severe pixel-level structural distortions.
A physical analysis of moiré degradation reveals a clean frequency division: high-frequency components predominantly contain localized, high-variance moiré fringe textures while carrying minimal macroscopic color; in contrast, low-frequency components are smooth, virtually devoid of fine fringes, and carry broad color distortions while exhibiting remarkable scale robustness (downsampling the low-frequency component to 0.1× resolution and reconstructing it preserves nearly 50 dB PSNR, whereas the high-frequency component yields only ~25 dB). Core Idea: Decouple moiré patterns into spatially-localized high-frequency textures and scale-robust low-frequency color distortions via anti-aliasing recursive convolutions, process them through an asymmetric crop/resize dual-branch Transformer equipped with Spatial-Aware Channel Attention (SA-CA), and adaptively recombine features via a learnable Frequency Composition Transform (FCT) to achieve compact, high-fidelity ultra-high-definition image demoiréing.
Method¶
Overall Architecture¶
Freqformer adopts a two-stage frequency-decoupled restoration and composition architecture. Given an input moiré image \(I_m\), the model first performs a multi-level anti-aliasing frequency decomposition that separates \(I_m\) into a high-frequency component \(I_h\) and a low-frequency component \(I_l\) while strictly maintaining the full spatial resolution. The high-frequency branch targets local ripple removal and is trained using a localized cropping strategy. The low-frequency branch focuses on global color correction and operates on downscaled representations during both training and inference. Both branches employ an encoder-decoder architecture driven by Spatial-Aware Channel Attention (SA-CA) blocks, complemented by multi-scale hierarchical feature fusion in the high-frequency decoder. Finally, the low-frequency feature is bilinearly upsampled to align with the high-frequency feature and fed into the learnable Frequency Composition Transform (FCT) module, which adaptively fuses multi-frequency representations to reconstruct the clean image \(\hat{I}\).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Moiré Image $I_m$"] --> B["Multi-Level Smooth Frequency Decomposition<br/>Recursive dilated convolutions without downsampling"]
B -->|High-frequency $I_h$ (local texture)| C["Asymmetric Dual-Branch Strategy<br/>High-frequency branch: crop-based training"]
B -->|Low-frequency $I_l$ (global color)| D["Asymmetric Dual-Branch Strategy<br/>Low-frequency branch: resize-based training"]
C & D --> E["Spatial-Aware Channel Attention & Hierarchical Fusion<br/>RDDB large receptive field + SCP channel attention + Gated FFN"]
E --> F["Learnable Frequency Composition Transform (FCT)<br/>Adaptive dual-branch feature interaction + PostFusion"]
F --> G["Reconstructed Clean Image $\hat{I}$"]
Key Designs¶
1. Multi-Level Smooth Frequency Decomposition: Recursive anti-aliasing convolution for artifact disentanglement Conventional discrete wavelet downsampling and block DCT representations suffer from a severe lack of spatial continuity and introduce irreversible phase-sensitive aliasing and block artifacts. Freqformer replaces orthogonal basis projections with a shift-invariant, smoothness-preserving recursive decomposition. For an input image \(I_m\), dilated convolutions with kernel \(k\) are recursively applied across \(L\) decomposition levels. At level \(i \in \{1, \dots, L\}\), the low- and high-frequency components are calculated as: $\(I^i_l = \text{Conv}(I^{i-1}_l, k), \quad I^i_h = I^{i-1}_h + I^{i-1}_l - I^i_l = I^0_l - I^i_l\)$ where \(I^0_l = I_m\), \(I^0_h = 0\), and the convolution adopts an expansion rate of \(2^i\) to exponentially enlarge the smoothing receptive field. After \(L\) levels, the final representations are obtained as \(I_l = I^L_l\) and \(I_h = I^L_h\), satisfying \(I_m = I_l + I_h\). This formulation entirely avoids spatial downsampling, ensuring that both \(I_h\) and \(I_l\) retain the original image dimensions. Consequently, \(I_h\) cleanly isolates local, high-dynamic moiré interference fringes, while \(I_l\) filters out high-frequency fluctuations to smoothly capture broad illumination tones and color shifts.
2. Asymmetric Dual-Branch Strategy: High-frequency cropping and low-frequency resizing Directly processing ultra-high-definition (4K) images through Transformer networks incurs prohibitive computational and memory costs, whereas naive downsampling irreparably degrades high-frequency moiré ripples and sharp text edges. Empirical validation reveals that downsampling and reconstructing the high-frequency component \(I_h\) achieves a PSNR of only ~25 dB, confirming its sensitivity to localized spatial structures. Conversely, downsampling the low-frequency component \(I_l\) to 0.1× resolution and restoring it preserves an impressive ~50 dB PSNR, highlighting exceptional scale robustness. Based on these distinct characteristics, Freqformer implements an asymmetric strategy: - High-frequency branch: Encodes high-variance textures with minimal absolute color information. Because demoiréing relies primarily on local spatial context, this branch is trained on randomly cropped patches (\(512\times 512\) or \(768\times 768\)) and applied directly to full-resolution or sliding-window inputs during inference. - Low-frequency branch: Focuses on large-scale color shifts caused by moiré modulation and imaging conditions. Resizing the low-frequency component to a compact resolution (\(256\times 256\) or \(512\times 512\)) condenses dispersed color discrepancies into small, textureless patches, drastically simplifying color correction. Crucially, maintaining an identical compact resolution across both training and testing completely eliminates scale-variation discrepancies.
3. Spatial-Aware Channel Attention & Hierarchical Fusion: Capturing wide context and inter-channel priors Moiré artifacts corrupt color channels unevenly—the green channel, for example, is noticeably more resilient to moiré distortion—motivating the use of channel attention to leverage complementary spectral cues. However, pure channel self-attention cannot perceive spatial ripple phases, which require adjacent spatial context to determine wave directionality and frequency. To resolve this, every encoder and decoder block incorporates a Spatial-Aware Channel Attention (SA-CA) module featuring three core components: - Residual Dilated Dense Block (RDDB): Placed before the attention layers, the RDDB employs dense connectivity and dilated convolutions to establish an extensive receptive field, capturing large-scale ripple formations. - Spatial-Channel Projection (SCP) & Channel Attention: Rather than computing standard linear projections, \(Q, K, V\) are generated via a \(1\times 1\) convolution (for channel pattern mapping) followed by a \(3\times 3\) depth-wise convolution (for local spatial aggregation). The reshaped features \(Q' \in \mathbb{R}^{c \times hw}, K' \in \mathbb{R}^{hw \times c}, V' \in \mathbb{R}^{c \times hw}\) then perform multi-head channel self-attention with linear complexity relative to image resolution. - Gated Feed-Forward Network (GFFN): Conventional FFNs struggle with spatially non-stationary artifacts. The GFFN applies dual-path spatial-channel projections, modulating features through a non-linear gating mechanism via element-wise multiplication to selectively filter out spurious high-frequency responses. - Hierarchical Fusion in the High-Frequency Decoder: In the high-frequency branch, shallower features \(\hat{F}_3\) and intermediate features \(\hat{F}_2\) from lower decoder stages are sequentially concatenated into the middle layers (\(N/2\)) of higher decoder stages, supplying fine-grained structural anchors that guide the reconstruction of intricate ripple patterns.
4. Learnable Frequency Composition Transform (FCT): Adaptive feature aggregation without error compounding Traditional frequency-based restoration methods rely on rigid mathematical inverse operations that simply sum decoupled subbands (\(\hat{I} = \hat{I}_l + \hat{I}_h\)). However, independent restorations inevitably suffer from slight spatial phase mismatches or localized luminance errors, which compound upon direct pixel-level addition to produce halo artifacts and blurred edges. To address this, Freqformer designs a learnable FCT operating in the latent feature space. Denoting the high-frequency penultimate feature as \(\hat{F}_h\) and the bilinearly upsampled low-frequency feature aligned with the high-frequency crop region as \(\hat{F}'_l\), the FCT module aggregates them via adaptive linear projections: $\(F = \text{Conv}(\hat{F}'_l) + \text{Conv}(\hat{F}_h)\)$ This addition is mathematically equivalent to channel concatenation followed by convolution while doubling only the bias parameters. The fused representation \(F\) is then processed by a PostFusion module consisting of \(N_f\) Transformer layers to resolve boundary inconsistencies, followed by a \(3\times 3\) projection and PixelShuffle to directly synthesize the clean, artifact-free image \(\hat{I}\).
Loss & Training¶
The network is trained in two sequential phases: - Phase 1 (Independent Branch Pretraining): The high- and low-frequency branches are trained independently using crop-based and resize-based strategies, respectively. Deep supervision is applied across all three decoder stages using \(L_1\) loss and VGG16-based perceptual loss \(L_p\): $\(L_{\text{stage}_1} = \sum_{i=1}^3 \left( L_1(\hat{I}_i, I^{GT}_i) + L_p(\hat{I}_i, I^{GT}_i) \right)\)$ - Phase 2 (Joint End-to-End Fine-Tuning): The dual branches and the learnable FCT module are jointly optimized to restore the full image \(\hat{I}\) against the ground-truth \(I^{GT}\): $\(L_{\text{stage}_2} = L_1(\hat{I}, I^{GT}) + \lambda_2 L_p(\hat{I}, I^{GT})\)$ where the perceptual loss weight is set to \(\lambda_2 = 0.1\). Both stages employ the Adam optimizer with a cyclic cosine learning rate schedule over 150 epochs each.
Key Experimental Results¶
Main Results¶
Freqformer was evaluated across four widely recognized image demoiréing benchmarks: TIP2018, LCDMoire, FHDMi (1080P), and UHDM (4K). Quantitative comparisons against leading demoiréing models and restoration baselines are summarized below.
| Dataset | Metric | Input | MBCNN [53] | FHDe2Net [10] | ESDNet [45] | ESDNet-L [45] | Freqformer (Ours) |
|---|---|---|---|---|---|---|---|
| TIP2018 [31] | PSNR ↑ SSIM ↑ |
20.30 0.7380 |
30.03 0.8960 |
27.78 0.8950 |
29.81 0.9160 |
30.11 0.9200 |
30.63 (+0.52) 0.9269 (+0.0069) |
| LCDMoire [53] | PSNR ↑ SSIM ↑ |
10.44 0.5717 |
44.04 0.9948 |
41.40 N/A |
44.83 0.9963 |
45.34 0.9966 |
45.62 (+0.28) 0.9967 (+0.0001) |
| FHDMi [10] | PSNR ↑ SSIM ↑ LPIPS ↓ |
17.97 0.7033 0.2837 |
22.31 0.8095 0.1980 |
22.93 0.7885 0.1688 |
24.50 0.8351 0.1354 |
24.88 0.8440 0.1301 |
25.26 (+0.38) 0.8518 (+0.0078) 0.1253 (-0.0048) |
| UHDM (4K) [45] | PSNR ↑ SSIM ↑ LPIPS ↓ |
17.12 0.5089 0.5314 |
21.41 0.7932 0.3318 |
20.34 0.7496 0.3519 |
22.12 0.7956 0.2551 |
22.42 0.7985 0.2454 |
22.24 (-0.18) 0.8021 (+0.0036) 0.2424 (-0.0030) |
| Parameters (M) | - | - | 14.192M | 13.571M | 5.934M | 10.623M | 6.065M |
When compared against general-purpose restoration models on FHDMi, Freqformer clearly outperforms the 28.785M AdaIR (21.87 dB) and 11.478M MoCE-IR (21.80 dB), illustrating the necessity of specialized frequency decoupling for non-stationary moiré artifacts.
Ablation Study¶
1. Frequency Decomposition Variants (Evaluated on FHDMi) Comparison between the proposed smooth recursive decomposition, 2-level Haar DWT, and block DCT.
| Decomposition Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | Params (M) | Observation |
|---|---|---|---|---|---|
| Freqformer (Ours) | 25.26 | 0.8518 | 0.1253 | 6.065M | Sharp text edges, clean texture recovery, no block artifacts |
| Freqformer w/ 2-level Haar DWT [18,24] | 23.01 | 0.8036 | 0.1810 | 10.921M | Severe edge blurring and jagged distortions around text |
| Freqformer w/ Block DCT [10,53] | 23.22 | 0.8073 | 0.1569 | 12.715M | Conspicuous block boundary artifacts, residual moiré ripples |
2. Low-Frequency Branch Training Strategy (Low-branch isolated evaluation)
| Low-Branch Configuration | PSNR ↑ | SSIM ↑ | LPIPS ↓ | Params (M) | Note |
|---|---|---|---|---|---|
| Freqformer (Low branch) w/ crop | 23.13 | 0.9220 | 0.0695 | 2.695M | Lacks global context for effective color tone restoration |
| Freqformer (Low branch) w/ crop & tile | 25.90 | 0.9363 | 0.0486 | 2.695M | Visible boundary seams across tiled patch borders |
| Freqformer (Low branch) w/ resize | 29.90 | 0.9568 | 0.0291 | 2.695M | Compact representation boosts PSNR by +6.77 dB |
3. Frequency Decomposition and Learnable FCT Ablation
| Model Architecture | PSNR ↑ | SSIM ↑ | LPIPS ↓ | Params (M) |
|---|---|---|---|---|
| Freqformer-L (w/o frequency decomposition) | 23.92 | 0.8409 | 0.1284 | 5.851M |
| Freqformer w/ fixed FCT (pixel addition) | 25.13 | 0.8478 | 0.1298 | 5.876M |
| Freqformer w/ learnable FCT | 25.26 | 0.8518 | 0.1253 | 6.065M |
| ESDNet-L baseline (w/o frequency decomposition) | 24.88 | 0.8440 | 0.1301 | 10.623M |
| ESDNet w/ learnable FCT (dual-branch adaptation) | 25.20 | 0.8511 | 0.1244 | 12.567M |
Key Findings¶
- Smooth decomposition is foundational: Compared to Haar DWT and block DCT, the proposed decomposition eliminates secondary aliasing and block artifacts while cutting parameter count by roughly 50% through the elimination of subband redundancy, delivering an advantage of over 2.0 dB in PSNR.
- Resize strategy unlocks low-frequency potential: Downsampling preserves ~50 dB fidelity in the low-frequency component, allowing global color correction on compact inputs and boosting isolated low-frequency restoration by 6.77 dB.
- Learnable FCT resolves error compounding: Replacing rigid pixel addition with feature-level learnable fusion adds merely 0.189M parameters while significantly improving reconstruction consistency and eliminating boundary halos.
Highlights & Insights¶
- Physics-Informed Asymmetric Compute Allocation: Exploiting the physical dichotomy between high-frequency local textures and low-frequency global color shifts enables asymmetric patch-based vs. resized training, outperforming models several times larger while processing ultra-high-definition inputs efficiently.
- Learnable Inverse Reconstruction in the Latent Space: Replacing static mathematical inverse operators with a feature-level learnable transformer resolves inter-band error propagation, offering an extensible design pattern for other frequency-decoupled restoration tasks such as defogging, deraining, and super-resolution.
- Dual Spatial-Channel Modeling with Minimal Overhead: Leveraging the green channel's resilience to moiré patterns via channel attention, while supplying crucial wave-phase spatial context through RDDB and depth-wise convolutions, achieves an optimal trade-off between expressive capacity and computational cost.
Limitations & Future Work¶
- Attention Operator Inference Latency: When inferring on 4K images, Freqformer requires only 2.46 TFLOPS (lower than ESDNet-L's 3.68T), but its wall-clock latency is 0.58 s/image compared to ESDNet-L's 0.25 s/image. This stems from attention operations being less optimized for hardware kernel fusion than pure convolutions, highlighting a need for operator acceleration via FlashAttention or reparameterization.
- Temporal Consistency in Video Demoiréing: Freqformer is tailored for single-image demoiréing. In handheld video sequences, dynamic camera motion causes rapid phase shifts in high-frequency ripples, where frame-by-frame application could introduce temporal flickering.
- Extreme Geometric Distortions: For images recaptured from sharply curved displays or captured at acute angles, fixed dilation rates in the recursive decomposition may struggle to match rapid local variations in ripple frequency.
Related Work & Insights¶
- vs WDNet [18] & MBWDN [42]: Early wavelet-based methods directly concatenate directional subbands or stack multi-level branches, incurring heavy parameter redundancy, shift-variant aliasing, and high memory footprints. Freqformer avoids downsampling during decomposition, halving parameter costs while substantially outperforming them in fidelity.
- vs MBCNN [53] & FHDe2Net [10]: Block DCT approaches are constrained by local sliding-window receptive fields, breaking continuous curvilinear patterns and creating block boundary artifacts. Freqformer captures wide-range periodic ripples via RDDB and spatial-aware channel attention, completely avoiding block artifacts with its learnable FCT.
- vs ESDNet [45]: As a representative lightweight UHD demoiréing baseline, monolithic spatial restoration in ESDNet still leaves residual color stains and ripples. By marrying frequency domain priors with lightweight Transformers, Freqformer achieves cleaner artifact suppression and superior perceptual quality with ~40% fewer parameters.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Well-conceived asymmetric frequency decomposition and learnable FCT tailored to the physical characteristics of moiré patterns]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated across four standard and 4K benchmarks with comprehensive ablations across operators, strategies, and decomposition variants]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous technical formulation, and informative visual and quantitative evidence]
- Value: ⭐⭐⭐⭐ [Provides practical architectural insights for frequency decoupling and asymmetric computation in ultra-high-definition restoration]