CogSENet: Blind Image Deblurring with Blur-Conditioned Semantic Routing and Explicit Frequency Fusion

Conference: ECCV2026
arXiv: 2606.30030
Code: To be confirmed
Area: Image Restoration
Keywords: Blind image deblurring, State Space Models, semantic routing, frequency decomposition, CLIP prior, continuous blur field

TL;DR

CogSENet reformulates blind image deblurring from a passive pixel regression into an active, semantically aligned reconstruction process. By mimicking the active visual scanning, retinal functional differentiation, and focal adaptation of the eagle visual system, it proposes three core designs: a Semantic-Driven State Space Module (SDSSM), a Bi-Frequency Fusion Block (BFFB), and joint modulation based on a Continuous Blur Field (CBF) and CLIP semantics. With only 8.9M parameters, it outperforms state-of-the-art (SOTA) methods such as EVSSM and FFTformer on the GoPro, HIDE, and RealBlur datasets.

Background & Motivation

The objective of blind image deblurring is to recover sharp images from unknown and spatially non-uniform blur. This field has transitioned from CNNs (NAFNet) to Vision Transformers (Restormer) and recently to State Space Models (MambaIR, VMamba). CNNs focus on local feature extraction but lack global context; Transformers capture long-range dependencies but incur quadratic computational costs; SSMs achieve effective global modeling with linear complexity. However, a fundamental limitation persists: most existing methods treat image restoration as static pixel-mapping regression, processing features indiscriminately without explicitly modeling semantic boundaries and structural frequencies. Real-world blur is highly non-uniform spatially—the intensity, direction, and regions of motion blur vary throughout a single image. Nonetheless, current methods apply a uniform scanning strategy across all regions, failing to distinguish semantic boundaries or decouple texture from structural frequencies.

The core challenge of this dilemma lies in: to restore a blur-damaged region, the network needs to know what it is restoring (semantic recognition) to decide how to restore it (strategy selection). Static regression lacks this "understanding" capability. This work takes a unique biological perspective: the visual system of an eagle maintains extremely high clarity in complex environments, relying on four adaptive mechanisms to actively scan targets, differentiate high and low frequencies using functional retinal zones, and adjust the lens focus continuously to adapt to different distances. The authors translate this biological framework into neural networks, shifting deblurring from "treating all pixels equally" in passive regression to "semantic-driven active reconstruction."

Core Idea: Inspired by eagle vision, the blind deblurring process is decomposed into three core challenges corresponding to the biological mappings—active scanning via semantic grouping (SDSSM), dual-path fusion with explicit high-low frequency decoupling (BFFB), and joint modulation of physical blur fields with high-level semantic priors (CBF+CLIP). These mechanisms work synergistically within a U-shaped encoder-decoder to achieve semantic-aware, frequency-interpretable deblurring.

Method

Overall Architecture

CogSENet is a three-level symmetric U-shaped encoder-decoder. An input blurry image is first processed by a \(3\times3\) convolution to extract shallow features, which are then passed into a three-level backbone network constructed by stacking CogSF blocks. Each CogSF block contains two complementary modules: SDSSM for semantic context aggregation and BFFB for explicit bi-frequency refinement. Cross-scale transitions between the encoder and decoder utilize bilinear scaling and \(3\times3\) convolutions, while jump connections along the vertical path preserve spatial details.

Unlike a standard U-Net, CogSENet introduces two layers of guidance at the bottleneck. First, a lightweight CNN predicts a Continuous Blur Field (CBF) from the input image, which is fused with the bottleneck features to generate a blur-aware descriptor. Second, a frozen CLIP image encoder extracts global semantic embeddings, and spatial attention maps are computed via cosine matching to apply residual gated modulation to the bottleneck features. The final bottleneck representation (containing both physical blur awareness and high-level semantic guidance) is progressively restored to the original resolution through the decoder and jump connections. Finally, a \(3\times3\) convolution outputs a residual image, which is added to the input to produce the sharp output.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Blurry Image"] --> B["Shallow Feature Extraction"]
    B --> C["Three-level Encoder-Decoder<br/>CogSF Block Stacking"]

    C --> D["Bottleneck Features"]
    E["CNN Blur Head"] --> F["Continuous Blur Field CBF"]
    F --> G["Blur-aware Descriptor"]
    D --> G
    H["Frozen CLIP<br/>Image Encoder"] --> I["Global Semantic Embedding"]
    I --> J["Cosine Matching Attention"]
    G --> J
    J --> K["Residual Gated Modulated Bottleneck"]
    D --> K
    K --> L["Decoder<br/>+ Skip Connections"]
    L --> M["Residual Reconstruction"]
    M --> N["Sharp Image"]

    subgraph SDSSM["SDSSM (Semantic Grouping Scanning)"]
        S1["Gumbel-Softmax<br/>Semantic Routing → K Clusters"] --> S2["Reorder Sequence by Cluster"]
        S2 --> S3["Semantic Prompt Modulates SSM Scanning"]
        S3 --> S4["Inverse Permutation Restores Space"]
    end

    subgraph BFFB["BFFB (Bi-Frequency Fusion)"]
        B1["Wavelet Decomposition<br/>High/Low Freq Separation"] --> B2["Low Freq: Lightweight U-Shape"]
        B1 --> B3["High Freq: Dense Convolution"]
        B1 --> B4["Implicit Branch<br/>Fourier Domain Filtering"]
        B2 --> B5["Adaptive Fusion"]
        B3 --> B5
        B4 --> B5
    end

Key Designs

1. SDSSM: Active Scanning by Semantic Grouping instead of Uniform Scanning

Existing SSMs perform causal scanning on 2D images in a fixed spatial order (row-by-row or column-by-column), which forces adjacent regions into the same scanning sequence even if they are semantically unrelated. The core motivation of SDSSM is to align the scanning order of SSM with semantics rather than space. First, a differentiable semantic router (Gumbel-Softmax) assigns feature tokens to \(K\) semantic clusters, where the assignment probability is determined by a learnable linear projection \(w_{\theta,k}\). The semantic prompt \(\mathbf{z}_i\) of each token is obtained by calculating the expected codebook value of its assigned cluster. Then, tokens are sorted based on their primary cluster ID and reassembled into a sequence, ensuring that tokens within the same semantic region are sequentially aligned in the scanning path. During scanning, the semantic prompt \(\mathbf{z}_i\) is added to the SSM output matrix \(\mathbf{C}_j\), modulating the state transitions with semantic information (Eq. 2). Finally, the spatial order is restored via inverse permutation. The entire process is end-to-end differentiable: Gumbel-Softmax ensures forward differentiability, and the gradient of the argmax sorting is bypassed using straight-through estimation (STE), propagating only through the continuous prompt \(\mathbf{Z}\). This enables the causal scanning of SSM to achieve non-causal global perception over semantically reordered sequences—semantically related tokens that are spatially distant can be repositioned closely to interact with each other, while unrelated tokens are kept separate.

2. BFFB: Explicit High-Low Frequency Separation via Wavelets + Dual-Path Fusion with Implicit Fourier Modulation

An inherent contradiction in blind deblurring is that recovering sharp textures requires enhancing high-frequency details, which may concurrently amplify high-frequency noise or artifacts. BFFB addresses this contradiction via explicit frequency separation, processing features along two parallel paths: ① Using Haar wavelets to explicitly decompose input features into low-frequency approximation coefficients \(\mathbf{X}_{LL}\) and high-frequency detail coefficients \(\mathbf{X}_H\). The low-frequency branch models global structure with a lightweight U-shaped module, while the high-frequency branch enhances local details using dense convolutions, followed by reconstruction via inverse wavelet transform. ② An implicit Fourier-domain refinement branch maps signals to the frequency domain, adaptively weighting them using a learnable spectral filter \(\mathbf{M}\) before transforming them back to the spatial domain. The outputs of both paths are adaptively fused using a learnable scalar \(\beta\) (Eq. 4), granting the network the flexibility to prioritize either high frequencies (for textured regions) or low frequencies (for flat regions) based on the content. The key insight of this design is that wavelet transforms provide physically interpretable frequency decoupling, while the Fourier branch offers flexible spectral modulation—the two complement each other to preserve structures and regularize the spectrum.

3. Joint Modulation of Continuous Blur Field and CLIP Semantics

Real-world blur is spatially non-uniform and continuous (e.g., motion blur varying progressively from the top to the bottom of an image), yet current methods typically process it using discrete kernel predictions or global classification, failing to capture continuous variations. CogSENet introduces two layers of injection at the bottleneck: first, a lightweight CNN directly predicts a dense 2D displacement field \(\mathbf{u} \in \mathbb{R}^{2 \times H \times W}\) (Eq. 5) from the input image, constrained by \(\tanh\) in magnitude, and fuses it with the bottleneck features to yield a blur-aware descriptor. Second, a frozen CLIP image encoder extracts global semantic tokens to calculate pixel-wise cosine similarity against the blur-aware descriptor (Eq. 6). Applying a spatial softmax produces attention maps, which are used to apply residual gated modulation (Eq. 7) via a learnable scalar \(\gamma\) (initialized to 0). This initialization ensures identity mapping in the early stages of training and progressively introduces joint semantic-physical guidance. This method mimics the focal adaptation mechanism of the eye: the attention map dynamically assigns high responses to heavily blurred semantic regions (e.g., text, facial contours) while suppressing responses in flat regions (e.g., sky, walls) to prevent noise over-amplification. Experimental results show that using CBF or CLIP alone provides limited improvements, whereas joint modulation yields significant gains.

Loss & Training

The model is supervised and trained using the Charbonnier loss function. A three-stage progressive training strategy is adopted: Stage 1 trains on \(128 \times 128\) patches with a batch size of 32 for 300k steps (learning rate decayed from \(10^{-3}\) to \(10^{-7}\) using cosine annealing); Stage 2 scales the patches up to \(256 \times 256\) with a batch size of 8 for another 300k steps; Stage 3 scales to \(320 \times 320\) with a batch size of 4 for 150k steps. The optimizer used is AdamW. For non-deblurring tasks (deraining, dehazing, denoising), the blur field branch is disabled to prevent introducing misaligned task priors.

Key Experimental Results

Main Results

Dataset	Metric	CogSENet (Ours)	Prev. SOTA	Parameters
GoPro	PSNR / SSIM	34.72 / 0.9744	34.51 / 0.9713 (EVSSM)	8.9M vs 17.1M
HIDE	PSNR / SSIM	32.18 / 0.9514	31.99 / 0.9503 (EVSSM)	8.9M
RealBlur-R	PSNR / SSIM	41.83 / 0.9799	41.27 / 0.9776 (EVSSM)	8.9M
RealBlur-J	PSNR / SSIM	34.54 / 0.9473	34.34 / 0.9456 (EVSSM)	8.9M
Rain100H (Deraining)	PSNR / SSIM	32.15 / 0.9099	32.17 / 0.9080 (IDT)	—
SOTS (Dehazing)	PSNR / SSIM	28.72 / 0.9692	28.21 / 0.9672 (MPRNet)	—

CogSENet comprehensively outperforms SOTA methods such as EVSSM and FFTformer across four deblurring benchmarks with only half the parameter count of EVSSM (8.9M vs 17.1M). An enhanced version, Ours+ (introducing FullBFFB at the bottleneck), further improves performance to 34.91 dB on GoPro with 19.4M parameters. Cross-task generalization is also highly competitive: achieving the highest SSIM on Rain100H for deraining and ranking first in both PSNR and SSIM on SOTS for dehazing.

Ablation Study

Configuration	GoPro PSNR/SSIM	Description
Full Model	34.31 / 0.9719	Baseline
w/o SDSSM	33.95 / 0.9677	Removing semantic-grouped scanning, showing the most significant performance drop
w/o Semantic Prompts	34.15 / 0.9685	Removing prompt modulation, causing SSM to lose context awareness
w/o Semantic Reordering	34.21 / 0.9691	Reverting to fixed spatial scanning, losing semantic correlation
w/o High-Low Freq Separation	34.01 / 0.9680	Removing explicit wavelet decomposition in BFFB
w/o Implicit Freq Branch	34.15 / 0.9689	Removing Fourier-domain modulation in BFFB
w/o BFFB	33.92 / 0.9677	Removing the entire Bi-Frequency Fusion Block
Only CBF	33.80 / 0.9668	Using blur field only, without CLIP
Only CLIP	33.78 / 0.9667	Using semantics only, without blur field
CBF+CLIP (Unjointed)	33.88 / 0.9673	Adding them separately without joint modulation

Removing semantic reordering and prompts in SDSSM leads to a PSNR drop of 0.10 dB and 0.16 dB, respectively, demonstrating that semantic grouping and prompt modulation complement each other. In BFFB, removing the explicit high-frequency separation has the greatest impact (-0.30 dB), validating the critical role of frequency decoupling in deblurring. Using CBF or CLIP in isolation yields minimal improvement, whereas joint modulation achieves a gain of over 0.4 dB, demonstrating the complementarity of physical priors and high-level semantics.

Key Findings

• Visualization of the routing in SDSSM shows that semantic grouping aligns closely with blur masks: large smooth/motion-dominated areas are allocated to a few major clusters, while texture-rich areas (shrubs, structural edges) activate multiple token IDs. This indicates that SDSSM learns content-adaptive representation capacity allocation.
• The contribution of explicit high-frequency separation in BFFB is more significant than that of the implicit frequency branch—physical interpretability introduced by wavelet decomposition is more critical than pure frequency domain filtering.
• Failure Case Analysis: When extreme motion blur severely destroys semantic cues, CLIP fails to extract meaningful semantics, causing the attention maps to degrade into a uniform distribution and the blur field to lose local micro-details. This indicates that the current system depends heavily on the semantic robustness of CLIP.

Highlights & Insights

• Deep Alignment of Biological Inspiration and Network Architecture: This is not an ornamental biological metaphor. Instead, three specific mechanisms of the eagle eye are mapped directly to SSM routing (active scanning), frequency decomposition (retinal differentiation), and feature modulation (focal adaptation). Each design’s independent contribution is verified in the ablation studies.
• Gumbel-Softmax Semantic Routing + Straight-Through Gradient: Relaxing the discrete token sorting problem into a differentiable routing process, where the sorting gradient propagates through continuous prompts instead of direct argmax, is an elegant engineering trick to resolve the conflict between discrete sampling and end-to-end training.
• Learnable Bi-Frequency Fusion Coefficient \(\beta\): Allows the network to adaptively balance the outputs of the explicit wavelet path and the implicit Fourier path based on content, avoiding the need for hand-crafted, static fusion weights.
• Residual Gating with Learnable Scalar \(\gamma\) Initialized to 0: Setting the initialization of \(\gamma=0\) in joint modulation guarantees an identity mapping during the early stages of training. Semantic guidance is introduced progressively as training advances, stabilizing optimization.

Limitations & Future Work

• Extreme motion blur can damage the semantic extraction capabilities of CLIP, degrading joint modulation into uniform attention, which leaves residual blur and vertical artifacts. This represents the most direct bottleneck of the proposed scheme.
• Although effective, the three-stage progressive training strategy (128 \(\rightarrow\) 256 \(\rightarrow\) 320 patch sizes) requires a total of 750k training steps, incurring high computational costs and hyperparameter tuning complexity.
• The blur field is only injected at the encoder-decoder bottleneck. Exploring hierarchical injection across multiple scales to capture multi-scale blur patterns is a worthwhile direction.
• The authors explicitly highlight video deblurring as a future direction, where extending the semantic routing of SDSSM into the spatiotemporal domain is a natural progression.

Related Work & Insights

• vs EVSSM (ECCV2024): EVSSM first introduced SSMs to image deblurring but relied on uniform scanning. CogSENet uses semantic reordering to align SSM scanning with semantics rather than space, cutting parameters in half.
• vs FFTformer: FFTformer introduces Fourier attention into Transformers. CogSENet’s BFFB goes a step further: a dual-path fusion of explicit wavelet separation and implicit Fourier modulation, combining physical interpretability with flexible modulation capabilities.
• vs AdaRevD: AdaRevD uses a reverse distillation strategy for deblurring (152.4M parameters). CogSENet achieves a higher PSNR (34.72 vs 34.50) with less than 1/15 of the parameters (8.9M), showcasing the efficiency advantages of combining semantic and frequency priors.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Intuitively maps three biological mechanisms of eagle vision to a unified framework integrating SSM, wavelets, and CLIP, with robust validation in ablation studies.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 4 deblurring benchmarks and 3 deraining/dehazing/denoising tasks, complete with comprehensive ablations, visual analyses, and failure investigations.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-structured; biological motivations and technical mechanisms are tightly linked and consistently explained. Equations and figures are clearly presented.
• Value: ⭐⭐⭐⭐ Introducing semantic awareness into low-level vision tasks represents a meaningful paradigm shift; the parameter efficiency of 8.9M also offers practical deployment value.

Related Papers

• [ECCV 2024] Blind Image Deblurring with Noise-Robust Kernel Estimation
• [ICCV 2025] Blind Noisy Image Deblurring Using Residual Guidance Strategy
• [CVPR 2026] OMoBlur: An Object Motion Blur Dataset and Benchmark for Real-World Local Motion Deblurring
• [CVPR 2026] DPGF-Net: Dual-Prior Guided Fusion Network for Joint Assessment of Perceptual Quality and Semantic Consistency in AI-Generated Images
• [ECCV 2026] UHD-MFF: Shattering Barriers in Multi-Focus Ultra-High-Definition Image Fusion via Learnable Lookup Tables