FUMO: Prior-Modulated Diffusion for Single Image Reflection Removal¶

Conference: ECCV 2026
arXiv: 2603.19036
Code: https://github.com/Lucious-Desmon/FUMO
Area: Image Restoration / Diffusion Models
Keywords: Single Image Reflection Removal, Diffusion Models, Prior Modulation, Coarse-to-Fine Restoration, Gated Mechanism

TL;DR¶

FUMO proposes a prior-modulated diffusion framework for single image reflection removal. It extracts VLM-driven reflection intensity priors and multi-scale high-frequency priors from the mixed image to spatially and adaptively modulate the conditional residual injection of ControlNet via a gated mechanism. Guided by a coarse-to-fine two-stage process (one-step denoising diffusion coarse restoration + a fine-grained refinement module), it achieves competitive quantitative results and consistent perceptual quality improvements on three standard benchmarks and real-world scenes.

Background & Motivation¶

Background: Images taken through glass often contain unwanted reflections, which obstruct the background scene and degrade visual quality. Single Image Reflection Removal (SIRR) aims to recover a clear transmission image from a single mixed image, which is an ill-posed mathematical problem. Learning-based methods have made significant progress by implicitly learning priors from data, but robust reflection removal in real-world scenarios remains challenging. Diffusion models have shown strong potential in image restoration tasks, but they require appropriate conditional guidance to avoid content drift and structural distortion.

Limitations of Prior Work: Reflection removal faces an inherent trade-off—the contradiction between aggressive reflection suppression and faithful structure preservation. In real-world scenarios, reflection intensity varies drastically across space, and reflection patterns are tightly entangled with transmission structures. Existing methods often exhibit three typical failure modes on real mixed images: incomplete reflection suppression, color inconsistency, and loss of detail. These three types of problems stem from the same fundamental contradiction: completely removing strong reflection regions requires applying strong conditional constraints, which often sacrifices the geometric fidelity of edge and texture regions; conversely, a conservative approach focusing on structural preservation leaves obvious reflection residues.

Goal: To address the aforementioned contradiction, FUMO aims to introduce an explicit spatial modulation mechanism that can spatially distinguish reflection intensities in different regions and adjust the restoration strength accordingly, while simultaneously preserving details in structure-sensitive regions.

Key Insight: The core observation is that although the mixed image itself lacks direct clues for separating the two layers, complementary guidance signals can be extracted from it: on one hand, the spatial intensity distribution of the reflection is estimated through the semantic understanding capability of VLMs; on the other hand, high-frequency responses sensitive to texture details are captured through multi-scale residual decomposition. These two prior signals serve complementary roles: one "indicates where aggressive reflection removal is needed," while the other "warns where structural details must not be destroyed."

Core Idea: These two complementary priors are combined into a spatial gate to adaptively modulate the conditional residual injection of the diffusion model spatially. This enhances conditional intervention in reflection-dominated regions while maintaining detail sensitivity in structure-sensitive regions. A fine-grained refinement module is then employed to correct geometric distortions and recover fine structures.

Method¶

Overall Architecture¶

The overall pipeline of FUMO is as follows: The input mixed image M passes through a dual prior extraction pipeline to obtain the intensity prior Pint and the high-frequency prior Phf. These two priors are combined through element-wise multiplication to construct a spatial gate g = 1 + beta * Pint * Phf, which modulates the multi-scale conditional residual vectors output by ControlNet. This gating ensures that modulated conditional injections receive higher weights in regions with high reflection intensity and rich structures. The modulated conditions are then fed into the U-Net denoiser of Stable Diffusion 2.1. Utilizing a one-step denoising strategy, the denoiser directly predicts the target latent variable z_t from the fully noisy latent variable z_N, which is then decoded by the VAE to produce the coarse restoration result. Finally, a fine-grained refinement module receives the mixed image, coarse restoration result, and the two priors via channel concatenation, executing a deterministic refinement in image space to output the final transmission image.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Mixed Image M"] --> B["Dual Prior Extraction<br/>VLM Intensity Prior + Multi-Scale High-Frequency Prior"]
    B --> C["Gated Modulation g=1+β·Pint⊙Phf"]
    A --> D["ControlNet<br/>Mixed Image Conditional Encoding"]
    C & D --> E["Gated Residual Injection →<br/>One-Step Denoising Diffusion Coarse Restoration"]
    E --> F["FGRM Fine-Grained Refinement<br/>U-Net + SimpleGate"]
    F --> G["Final Transmission Image T̂"]

Key Designs¶

1. VLM-Driven Reflection Intensity Prior: Estimating Spatial Reflection Severity via Semantic Understanding of Foundation Models

The first branch of prior extraction estimates a pixel-level reflection intensity map from the mixed image. The mixed image M is partitioned into non-overlapping image patches, and a fixed VLM (such as InternVL3 or Qwen2.5-VL) is queried patch-by-patch to estimate reflection severity. Rather than relying on free-text output, this method utilizes the model's logits for the next token to restrict the probability mass to a predefined set of five ordered levels C = {None, Minor, Mid, Major, Critical}. Probabilities p(c) for each category are obtained via softmax, and a continuous severity score is calculated via weighted expectation s = sum_{c in C} w(c) * p(c) (with weights w(c) set to {1,2,3,4,5}). The patch-level scores are then mapped back to the image plane to form an initial intensity field S. However, patch-level scoring may miss visually prominent reflection areas on a global scale. Therefore, an image-level analysis is further introduced: the VLM is also employed to locate reflection-dominated regions and return bounding boxes, and a multiplication factor is applied to the intensity values in the corresponding regions to obtain an enhanced map \(\tilde{S}\). On 100 real images manually annotated with reflection-dominated regions, this localization step achieves 0.65 mIoU, providing a coarse-grained spatial cue. Finally, edge-aware guided filtering is applied to denseify \(\tilde{S}\), removing blocky artifacts and aligning intensity transitions with the primary structures of the image to obtain the final continuous pixel-level intensity prior \(P_{\mathrm{int}} \in [0,1]^{H \times W}\). The ingenuity lies in utilizing the VLM's semantic reasoning capability to perform "scoring + localization" rather than direct image generation, maintaining output controllability while gaining richer semantic context awareness than pure CNN-based methods.

2. Multi-Scale High-Frequency Prior: Extracting Structure-Sensitive Detail Responses from Mixed Images

While the intensity prior provides semantic instructions on "where reflections are strong," signal-level guidance is required in structure-sensitive regions. The high-frequency prior branch extracts detail responses from the same mixed image via multi-scale residual decomposition. A smoothing operator \(B_r(\cdot)\) is implemented using dilated convolutions. At each scale i, the smoothed image \(L^{(i)} = B_{r_i}(M^{(i)})\) and residual \(H^{(i)} = M^{(i)} - L^{(i)}\) are iteratively calculated, with \(L^{(i)}\) serving as the input to the next scale. The scale parameter \(r_i = 2^i\) grows exponentially with i. The final high-frequency prior is obtained by summing the residuals across all scales: \(P_{\mathrm{hf}} = \sum_{i=0}^{L-1} H^{(i)}\). Since this prior is also extracted from the mixed image and contains high-frequency components of both transmission and reflection layers, it does not attempt to separate the two. Instead, it serves as a structure-sensitive guidance signal—informing the model which regions possess local structural details that need preservation. The simplicity of this design is notable: the entire decomposition relies solely on dilated convolutions and element-wise subtraction, requiring no additional learnable parameters.

3. Gated Modulation + Coarse-to-Fine Two-Stage: Spatial Adaptation of Conditional Injection

This serves as the bridge between the priors and the diffusion model, representing the core mechanism of the methodology. The two priors are combined into a spatial gate \(g = 1 + \beta \, P_{\mathrm{int}} \odot P_{\mathrm{hf}}\): the additive identity item guarantees degradation to standard conditional injection when either guidance signal is weak, while the multiplicative interaction emphasizes their joint presence, maximizing the gate value in regions with "heavy reflections and rich structures" while remaining simple and differentiable. In the coarse restoration stage, ControlNet outputs a set of multi-scale conditional residual tensors \(\mathbf{c}_m = \{\mathbf{c}_s\}\). For each scale s, the gate is scaled to the corresponding resolution and modulated element-wise: \(\tilde{\mathbf{c}}_s = \mathrm{clip}(\mathcal{I}_s(g), 1, 1+\beta_{\max}) \odot \mathbf{c}_s\). The parameter \(\beta\) is gradually warmed up from 0 to \(\beta_{\max}=0.25\), allowing the gating effect to be introduced progressively.

Training in the coarse restoration stage adopts a one-step denoising strategy: unlike standard diffusion models that predict noise across multiple steps, the network takes the heavily perturbed latent variable \(z_N\) (N=1000) as input and directly predicts a less perturbed latent variable \(z_t\). The loss is defined as \(\mathcal{L}_{\text{coarse}} = \|z_t - \mu_\theta(z_N, t, \tilde{\mathbf{c}}_m)\|_2^2\). During inference, setting t=0 yields \(\hat{z}_{\mathbf{T}}\) which is then decoded. Only the ControlNet and the upsampling blocks of the U-Net are optimized, while the VAE and remaining parameters are frozen to maintain the pre-trained generative priors.

The Fine-Grained Refinement Module (FGRM) corrects geometric deviations and local inconsistencies after the completion of coarse restoration. FGRM features a U-Net architecture, replacing standard activations with SimpleGate (which splits features into two halves along the channel dimension and multiplies them element-wise). It receives the mixed image, coarse restoration result, and the two priors via channel concatenation, outputting the refined transmission image \(\hat{\mathbf{T}}\). The comprehensive loss of the refinement stage comprises an L1 pixel loss, an LPIPS perceptual loss, and an edge-aware gradient loss.

Loss & Training¶

The two stages are optimized separately. Coarse restoration stage (100k steps, lr=5e-5): only the ControlNet and U-Net upsampling blocks are trained, utilizing the one-step denoising regression loss \(\mathcal{L}_{\text{coarse}}\). Refinement stage (10k steps, lr=1e-4): the coarse restoration part is frozen, and only the FGRM is trained. The loss function is regulated by \(\mathcal{L}_{\text{refine}} = \lambda_{\text{pix}} \mathcal{L}_{\text{pix}} + \lambda_{\text{perc}} \mathcal{L}_{\text{perc}} + \lambda_{\text{grad}} \mathcal{L}_{\text{grad}}\), where \(\lambda_{\text{pix}}=0.5, \lambda_{\text{perc}}=0.25, \lambda_{\text{grad}}=0.25\). Here, \(\mathcal{L}_{\text{pix}}\) represents the L1 pixel loss, \(\mathcal{L}_{\text{perc}}\) is the LPIPS perceptual loss, and \(\mathcal{L}_{\text{grad}}\) represents the Sobel gradient difference L1 loss. The training data consists of real-world pairs (approximately 35,000 pairs across Real / Nature / RR4k / RRW / DRR) and synthetic data (16,000 pairs generated via COCO blending under the formulation \(M = \gamma_1 T + \gamma_2 R - \gamma_1 \gamma_2 T \odot R\)). All images are resized to 768x768.

Key Experimental Results¶

Main Results¶

Evaluation is conducted on three standard benchmarks: Nature (20 pairs), Real (20 pairs), and SIR2 (500 pairs). Comparisons are made against seven SOTA methods including IBCLN, Dong et al., YTMT, DSRNet, Zhu et al., RDNet, and DAI. All baselines are fine-tuned on the same training dataset.

Dataset	Metric	IBCLN	DSRNet	RDNet	DAI	FUMO
Nature	PSNR↑	23.77	24.58	25.74	26.81	26.93
Nature	LPIPS↓	0.145	0.120	0.109	0.203	0.088
Real	PSNR↑	21.55	23.37	24.81	25.21	25.95
Real	LPIPS↓	0.210	0.157	0.118	0.150	0.097
SIR2	PSNR↑	23.89	25.51	26.46	27.35	27.22
SIR2	LPIPS↓	0.127	0.094	0.080	0.093	0.067
Average	PSNR↑	23.79	25.39	26.36	27.24	27.15
Average	LPIPS↓	0.131	0.098	0.083	0.100	0.069
Average	MUSIQ↑	58.19	59.02	58.71	54.57	59.88

FUMO performs comparably to DAI on PSNR (outperforming it on Nature/Real, but slightly lower on SIR2), while leading comprehensively across all perceptual metrics (LPIPS, CLIPIQA, MUSIQ). This indicates that the gated modulation and refinement module significantly improve visual quality and structural fidelity.

Ablation Study¶

Configuration	PSNR↑	SSIM↑	LPIPS↓	CLIPIQA↑	Description
w/o gate	26.55	0.897	0.076	0.390	No spatial modulation for conditional injection
Concat priors	26.87	0.906	0.072	0.406	Priors concatenated with conditional features instead of modulated
Intensity prior only	26.94	0.905	0.073	0.403	Gate driven only by Pint
High-frequency prior only	26.78	0.906	0.071	0.393	Gate driven only by Phf
Coarse restoration only	26.27	0.865	0.133	0.385	No FGRM refinement
Refinement decoder	26.75	0.891	0.084	0.398	DAI-style fine-tuned decoder
Full FUMO	27.15	0.918	0.069	0.424	Dual-prior gating + FGRM

Key Findings¶

Gated modulation is core to perceptual quality: Removing the gate increases LPIPS from 0.069 to 0.076 (-10%), while PSNR only decreases from 27.15 to 26.55 (-2%). This suggests the gate mainly improves perceptual structural fidelity rather than pixel-level errors.
Distinct focus of both priors: The intensity prior improves reflection removal in high-severity regions, while the high-frequency prior enhances detail response in edge and texture regions; joint utilization yields the best result.
Coarse restoration is indispensable but insufficient alone: The coarse-only variant exhibits an LPIPS as high as 0.133 (93% degradation), demonstrating that the directly decoded output suffers from notable blur and restoration artifacts, rendering FGRM refinement crucial for structural consistency.
Prior modulation outperforms concatenation: Concatenating priors into the conditional feature path is less effective than modulating the residual injections with them, showing that the core value of the priors lies in signal-level spatial weighting rather than being extra input features.

Highlights & Insights¶

Degradation estimation via VLM: Utilizing VLM logits for ordered classification before converting them to continuous numerical scores, rather than letting the VLM generate free text, is a reusable technique across multiple image enhancement tasks (e.g., degradation estimation in dehazing, denoising, and low-light enhancement).
Multiplicative interaction design of the gate: In \(g = 1 + \beta P_{\mathrm{int}} \odot P_{\mathrm{hf}}\), the additive identity terms ensure degradation to baseline conditions, while the multiplicative interaction emphasizes the joint activation of "heavy reflection and rich detail" regions. This controls the strength of conditional intervention across space more rationally than simple concatenation or addition.
Role separation in coarse-to-fine restoration: Bold removal using a generative model in the coarse stage combined with careful recovery using a discriminative model in the fine stage. Although this philosophy is common in image restoration, the concrete implementation (one-step denoising instead of multi-step sampling in the coarse stage, SimpleGate U-Net in the fine stage) reflects pragmatic design trade-offs.
Perceptual-quality-first design orientation: FUMO's comprehensive lead in LPIPS/MUSIQ/CLIPIQA validates the success of its design goal—prioritizing the improvement of visual perceptual quality through spatially adaptive modulation.

Limitations & Future Work¶

Semantic confusion between reflection and transmission content: When reflection regions and transmission content are highly similar in intensity and structure, the method struggles to decide which content should be preserved. User-provided spatial guidance (such as coarse masks or sparse edge annotations) is a practical and viable direction for mitigation.
Inference overhead of VLM priors: On an RTX 4090 with 11K resolution, prior extraction takes 3.40s, while the entire restoration network only requires 0.63s. The paper suggests replacing it with a lightweight predictor, which is not implemented in this work.
Prior extraction quality bottleneck: The mIoU of the VLM localization step is only 0.65, which may lack precision in complex scenarios and subsequently affect the gating performance. Moreover, the intensity prior relies on a frozen VLM and cannot be trained end-to-end, limiting the joint adaptation of the prior and downstream tasks.
Limitations of the blending model for synthetic data: The synthetic data continues to employ the quasi-linear blending assumption \(M = \gamma_1 T + \gamma_2 R - \gamma_1 \gamma_2 T \odot R\), which still deviates from the complex non-linear blending of the real world.

vs DAI (Hu et al. 2025): DAI similarly uses ControlNet conditional injection + a diffusion backbone, employing a fine-tuned decoder to improve decoding quality. FUMO introduces explicit dual-prior gated modulation in the middle and replaces simple decoder tuning (LPIPS 0.084) with FGRM (LPIPS 0.069), with the Ablation Study corroborating this improvement.
vs L-DiffER (Hong et al. 2024): L-DiffER first introduced diffusion models to reflection removal, using prior predictions as conditions for iterative denoising. FUMO employs one-step denoising instead of multi-step iterations, offering superior advantages in inference efficiency and training stability.
vs Implicit Prior Networks (DSRNet / RDNet): These methods implicitly learn the differences between reflection and transmission through meticulously designed network architectures. FUMO differs by making the priors explicit and utilizing them for spatial modulation, which is more controllable and interpretable than implicit learning.
Transfer Insights: The paradigm of dual-prior + gated modulation can be transferred to other image restoration tasks (dehazing, deraining, low-light enhancement), where "degradation-level prior + structure-sensitive prior + spatial gating" can form a general design pattern. The method of converting VLM logits to continuous scores can also be widely reused.

Rating¶

Novelty: ⭐⭐⭐⭐ Introducing explicit VLM priors into diffusion conditional modulation is a novel design, although the overall architecture of diffusion + ControlNet + coarse-to-fine is not entirely new.
Experimental Thoroughness: ⭐⭐⭐⭐ The three benchmarks plus real-scenario qualitative evaluations and ablation studies (covering 4 gating forms and 2 refinement forms) are relatively comprehensive, but cross-dataset transfer experiments and comparisons with more diffusion baselines are absent.
Writing Quality: ⭐⭐⭐⭐ The method is described clearly, and the framework diagram in Figure 3 combined with the text is easy to understand, though the ablation analysis section could be further condensed.
Value: ⭐⭐⭐⭐ Single image reflection removal has wide practical demands; the significant improvement in perceptual quality provided by this work holds practical value, and the design framework is generalizable.