Diffusion-based dual-view reflection removal

Conference: ECCV 2026
Paper: ECCV Original
Cached Fulltext: /Users/zy/workspace/paper_cache/ECCV2026/eccv-3182.txt
Area: Image Restoration
Keywords: Reflection removal, Dual-view diffusion, Frame-aware spatial attention, Camera pose prior, Physical parallax disentanglement

TL;DR

Addressing the severe ill-posedness of single-image reflection removal and the vulnerability of multi-frame sequence methods to reflection-corrupted flow estimation, this paper presents a dual-view diffusion framework featuring directed frame-aware spatial attention to prevent cross-view noise leakage and soft smartphone IMU pose embeddings to disentangle reflection layers via physical parallax.

Background & Motivation

Reflection removal is a long-standing inverse problem in computational photography and computer vision, physically modeled as decomposing an observed image into an additive mixture of a transmission layer and a reflection layer. Historically, single-image reflection removal (SIRR) has remained the most user-friendly paradigm due to its minimal operational constraints. However, decomposing a single composite frame into two unknown source images is fundamentally ill-posed, admitting infinitely many plausible solutions. Even with the integration of large-scale generative diffusion priors or vision-language foundational models, state-of-the-art single-image algorithms consistently struggle when encountering strong, overexposed, or structurally intricate reflections—often misidentifying intense virtual reflection patterns as foreground content and erroneously preserving or enhancing them.

To mitigate this mathematical ambiguity, researchers have explored auxiliary physical constraints, including polarization filters, near-infrared imaging, flash/no-flash captures, and multi-view video sequences. While specialized optical sensors impose impractical hardware and operational burdens on casual mobile photography, video-based multi-frame setups demand deliberate continuous camera movements and rely heavily on explicit optical flow estimation or test-time 3D reconstruction. Because physical transmission and virtual reflections reside at distinct spatial depths, they induce fundamentally conflicting parallax motions across viewpoints. When intense reflections corrupt image gradients and feature correspondences, classical optical flow algorithms suffer catastrophic tracking errors, causing severe boundary distortion and artifact accumulation. Meanwhile, modern smartphones routinely feature dual-camera systems capable of capturing paired viewpoints simultaneously via a single shutter press. Yet, transitioning from dense multi-frame sequences to a dual-view setup drastically shrinks observation redundancy, rendering naive multi-view architectures ineffective.

To overcome these roadblocks without resorting to brittle correspondence matching or rigid 3D formulations, this paper introduces an end-to-end generative perspective. Core idea: build a dual-view diffusion reflection removal framework featuring directed frame-aware spatial attention—where noisy transmission latents attend exclusively to clean mixture conditions to eliminate cross-view noise leakage—while conditioning the denoising process on soft IMU-derived relative camera pose embeddings to guide the network in disentangling depth-induced parallax motion.

Method

Overall Architecture

Building upon the latent diffusion architecture of Stable Diffusion v2.1, the proposed framework takes a pair of mixture images \(M_1, M_2\) along with their respective camera poses \(p_1, p_2\) as input to jointly recover clean dual-view transmission layers \(T_1, T_2\). Both mixture and transmission images are initially mapped into the latent space via a frozen VAE encoder. During diffusion forward noising, Gaussian noise is injected solely into the transmission latents, while the dual-view mixture conditions remain pristine. These components are concatenated along the channel dimension to construct a unified 16-channel tensor fed into a modified denoising U-Net. In the deep upsampling stages of the U-Net, specialized frame-aware spatial attention modules mediate cross-view feature interaction. Concurrently, normalized relative camera pose embeddings are merged with diffusion timestep embeddings, serving as global geometric modulators.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Dual-view Input<br/>Image pairs (M1, M2) + Poses (p1, p2)"] --> B["Relative Pose Encoding<br/>Quaternion & normalized translation into timestep"]
    A --> C["VAE Latent Space Channel Concatenation<br/>Noisy latents (z1_t, z2_t) + Clean mixtures (c1_M, c2_M)"]
    B --> D["Denoising U-Net Backbone"]
    C --> D
    D --> E["Frame-Aware Spatial Attention<br/>Noisy transmission Q queries clean mixture K, V"]
    E --> F["Two-stage Curriculum Training<br/>Parallax learning → Robustness enhancement"]
    F --> G["Clean Transmission Recovery<br/>Reflection-free dual outputs (T1, T2)"]

Key Designs

1. Frame-aware spatial attention: Directed cross-attention preventing cross-view noise leakage
In a dual-view formulation, exploiting viewpoint parallax is essential to isolate layers across different depths. However, naively applying unrestricted bidirectional self-attention across viewpoints allows noisy transmission latents to attend to one another, amplifying mutual noise and degrading image synthesis. Furthermore, incorporating 3D convolutions modifies the latent tensor dimensions and destroys the rich 2D generative priors stored in pretrained diffusion weights. To resolve this, the authors introduce frame-aware spatial attention within the \(16 \times 16\) and \(32 \times 32\) upsampling layers of the U-Net. The hidden feature tensor is partitioned into a query component \(l_q \in \mathbb{R}^{B \times 2C' \times H \times W}\) originating from the noisy transmission paths of both views, and a condition component \(l_c \in \mathbb{R}^{B \times 2C' \times H \times W}\) from the clean mixture paths. After sequence flattening, pretrained projection matrices generate unified projections \(Q_{\text{all}}, K_{\text{all}}, V_{\text{all}}\). Crucially, a directed attention flow is enforced: queries are derived strictly from noisy transmission positions, while keys and values are extracted solely from clean mixture conditions: $\(\mathbf{Q} = \mathbf{Q}_{\text{all}}[:, :2HW, :], \quad \mathbf{K} = \mathbf{K}_{\text{all}}[:, 2HW:, :], \quad \mathbf{V} = \mathbf{V}_{\text{all}}[:, 2HW:, :]\)$ The computed attention \(\text{Softmax}(\mathbf{Q}\mathbf{K}^\top / \sqrt{d_k})\mathbf{V}\) is added back to the query features via a residual link, whereas clean mixture features remain unaltered. This asymmetric architecture permits every noisy patch to globally query complementary context across both views without introducing any cross-frame noise contamination.

2. Relative camera pose encoding: Drift-tolerant geometric soft conditioning
Human observers intuitively shift their perspective to distinguish reflections from real objects through depth parallax. However, translating this to mobile devices is complicated by consumer-grade IMU sensors, which suffer from cumulative translation drift and measurement noise, causing strict 3D geometric back-projection to collapse. To establish robust geometric guidance, the system discards noisy translation magnitudes and retains only the highly reliable translation directions and relative rotation unit quaternions. For poses \(p_1 = [\mathbf{t}_1, \mathbf{q}_1]^\top\) and \(p_2 = [\mathbf{t}_2, \mathbf{q}_2]^\top\), the 7-dimensional relative pose is formulated as: $\(\mathbf{p}_{\text{rel}} = \left[ \frac{\mathbf{t}_2 - \mathbf{t}_1}{\|\mathbf{t}_2 - \mathbf{t}_1\|}, \, \mathbf{q}_2 \otimes \mathbf{q}_1^{-1} \right]\)$ This vector is processed by a 2-layer MLP into a geometric embedding \(e_p\) and fused with the diffusion time embedding via \(e_c = e_t + \gamma e_p\) (with scaling factor \(\gamma = 0.15\)). Injected as a global conditioning signal, this lightweight prior informs the network of the exact physical camera trajectory, steering spatial attention to amplify features adhering to transmission parallax while actively suppressing virtual reflection artifacts that violate the scene motion.

3. Two-stage curriculum training: Progressing from parallax learning to extreme reflection robustness
Directly training the network on extreme reflections prevents it from learning subtle physical parallax cues, whereas training exclusively on mild reflections causes catastrophic failure on overexposed real-world images. The authors establish a two-stage curriculum training protocol. Stage 1 operates on semi-synthetic images at \(768 \times 768\) resolution with moderate reflection intensity \(\alpha_r \sim \mathcal{U}(0.3, 0.7)\). During this stage, stochastic view replacement is applied with a 15% probability per view (setting relative pose to identity) alongside a 25% pose dropout, compelling the model to maintain strong single-view fallback capabilities. Stage 2 ramps up reflection intensity to \(\alpha_r \sim \mathcal{N}(0.8, 0.2)\) (clipped to \([0.4, 1.2]\)) and incorporates aggressive real-world perturbations including motion blur, chromatic aberration, and ghosting effects, hardening the network against scenarios where reflections dominate the transmission layer.

Loss & Training

The overall model is trained end-to-end using standard DDPM noise-prediction mean squared error conditioned on view dropout mask \(m_v \in \{0, 1\}^2\) and pose dropout mask \(m_p \in \{0, 1\}\): $\(\mathcal{L} = \mathbb{E}_{t, \mathbf{z}_0, \boldsymbol{\epsilon}, m_v, m_p} \left[ \left\| \boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\mathbf{z}_T^t, t, m_v \odot \mathbf{c}_M, m_p \cdot \mathbf{e}_{\text{rel}}) \right\|^2 \right]\)$ Optimization is performed using AdamW with batch size 8 on top of pretrained Stable Diffusion v2.1 weights.

Key Experimental Results

Main Results

The method is evaluated on the real-world trajectory dataset DualRef-V, the semi-synthetic DualRef-R benchmark, and the real-world Niklaus dataset against leading single-image and multi-image methods (methods marked with * were fine-tuned on the DualRef-R training set):

Dataset	Metric	Ours	Prev. SOTA	Runner-up	Gain
DualRef-V (Real-world)	PSNR↑	24.17	21.57	DAI (21.57) / WSeat (21.44)	+2.60 dB (+12.0%)
DualRef-V (Real-world)	SSIM↑	0.7591	0.7481	DSIT* (0.7481) / WSeat (0.7478)	+0.0110
DualRef-V (Real-world)	LPIPS↓	0.1638	0.1692	WSeat (0.1692) / DAI (0.1762)	-0.0054
DualRef-V (Real-world)	CLIPIQA↑	0.4705	0.4691	WSeat (0.4691) / RDNet (0.4569)	+0.0014
DualRef-V (Real-world)	MUSIQ↑	64.81	64.22	RDNet (64.22) / ZF24* (64.01)	+0.59
DualRef-R (Semi-synthetic)	PSNR↑	28.19	27.11	WSeat (27.11) / DSIT* (26.71)	+1.08 dB
DualRef-R (Semi-synthetic)	SSIM↑	0.8378	0.8389	DSIT* (0.8389) / WSeat (0.8369)	-0.0011
DualRef-R (Semi-synthetic)	LPIPS↓	0.0983	0.0990	DSIT* (0.0990) / RDNet (0.1017)	-0.0007
Niklaus (Real-world)	PSNR↑	25.09	23.72	WSeat (23.72) / DSIT* (23.40)	+1.37 dB
Niklaus (Real-world)	SSIM↑	0.7973	0.7999	DSIT (0.7999) / DSIT* (0.7971)	-0.0026
Niklaus (Real-world)	LPIPS↓	0.0865	0.0934	RDNet (0.0934) / Niklaus* (0.1112)	-0.0069

Ablation Study

Ablations conducted on DualRef-R training set and tested across DualRef-V and DualRef-R evaluate the individual impact of architectural decisions:

Config	DualRef-V PSNR↑	DualRef-V SSIM↑	DualRef-V LPIPS↓	DualRef-R PSNR↑	Note
Single-view	21.59	0.7236	0.1720	26.07	InstructPix2Pix baseline without cross-view attention
Dual separate	21.44	0.7245	0.1703	26.12	Evaluated with identical duplicate input images
Dual w/o pose	23.53	0.7468	0.1694	27.47	Dual-view attention model with pose encoding removed
SD2.1 + RAFT	21.60	0.7369	0.1744	26.33	Baseline with explicit optical flow warping
Cat3D-Style Attn	22.09	0.7267	0.1717	26.52	Naive bidirectional cross-attention (causes noise leakage)
SIR-Style Attn	20.87	0.7036	0.2069	25.99	3D ResNet layers (destroys 2D pretrained prior)
Width-concat	22.03	0.7354	0.1838	25.83	Horizontal spatial concatenation rather than channel stacking
Full model (Ours)	24.17	0.7591	0.1638	28.19	Full dual-view directed attention + relative pose encoding

Key Findings

• Crucial role of camera pose encoding: Incorporating relative pose guidance boosts DualRef-V performance from 23.53 dB (Dual w/o pose) to 24.17 dB (+0.64 dB). In visual inspections, pose guidance disambiguates cases where virtual reflections exhibit noticeable parallax that would otherwise be confused with transmission content.
• Superiority over explicit optical flow: The SD2.1 + RAFT baseline drops by 2.57 dB PSNR compared to the full model on real data. In reflection-corrupted regions, transmission and reflection layers exhibit conflicting motions, causing explicit optical flow algorithms to fail and propagate warped artifacts.
• Sensitivity of attention topologies: The SIR-Style 3D ResNet setup performs worst (20.87 dB PSNR) due to the destruction of pretrained 2D generative features. Meanwhile, the unconstrained bidirectional attention in Cat3D-Style underperforms (22.09 dB PSNR) due to mutual noise amplification across noisy latents, validating the proposed directed querying scheme.
• Diminishing returns beyond dual views: Increasing view count from single-view to dual-view yields massive quantitative jumps (+2.58 dB PSNR on DualRef-V). However, scaling from 2 to 4 views offers negligible marginal improvement while increasing capture friction and computational costs.

Highlights & Insights

• Directed cross-attention architecture: Confining queries to noisy transmission latents while restricting keys/values to clean mixture representations zero-overhead reuses 2D diffusion attention layers while structurally guaranteeing zero cross-view noise contamination.
• Drift-resilient geometric soft conditioning: Normalizing IMU translation vectors into scale-invariant direction unit vectors combined with rotation quaternions provides effective geometric grounding without collapsing under smartphone sensor drift.
• Ground-truth trajectory benchmark DualRef-V: Developing a programmable camera dolly system with removable glass panels enables the first reproducible real-world benchmark with exact ground-truth transmission images for dual-view reflection removal.

Limitations & Future Work

• Degradation under zero parallax: The framework fundamentally relies on viewpoint parallax to separate conflicting visual layers. In scenarios where camera movement is negligible or scenes reside at optical infinity, performance naturally degrades to single-image diffusion baselines.
• Latent diffusion color drift: In rare cases with extensive high-intensity reflection saturation, latent diffusion generation occasionally exhibits slight color casting or hue shifts.
• Future directions: Extending this dual-view soft pose conditioning scheme to short dynamic video bursts (such as Apple Live Photos) and integrating high-dynamic-range (HDR) sensor pipelines to recover fully clipped transmission regions.

Related Work & Insights

• vs Single-image diffusion methods (DAI [27] / WindowSeat [87]): DAI employs single-step generative priors on isolated frames, often hallucinating or accentuating strong reflections; the proposed framework resolves the underlying ill-posedness by incorporating geometric multi-view cues.
• vs Flow-based multi-view methods (Niklaus [46] / TF15 [81]): Traditional multi-view methods compute explicit optical flow fields that degrade catastrophically under semi-transparent reflections; this approach implicitly aggregates multi-view correspondences inside diffusion attention without explicit correspondence commitments.
• vs Burst-based neural spline optimization (Burst Reflection Removal [9]): Neural spline fields require continuous test-time 3D optimization taking ~3 minutes per scene; the proposed feed-forward diffusion framework performs inference in just 3 seconds (~60× speedup) while matching or exceeding restoration fidelity.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Formulates the first dual-view diffusion framework for reflection removal with directed frame-aware attention and soft IMU pose conditioning.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Introduces the rigorously captured DualRef-V real-world benchmark alongside extensive ablations across attention topologies and geometric baselines.
• Writing Quality: ⭐⭐⭐⭐⭐ Highly structured, mathematically rigorous, with lucid explanations of physical motivations and architectural decisions.
• Value: ⭐⭐⭐⭐⭐ Perfectly tailored to modern dual-camera mobile hardware, offering immediate utility for consumer photography applications.