Reflection-aware generative novel view synthesis

Conference: ECCV 2026
Paper: ECCV Paper
Project: https://kim-geonu.github.io/Ref-GeNVS/
Area: 3D Vision
Keywords: mirror reflections, generative novel view synthesis, multi-view diffusion models, training-free, cross-view attention

TL;DR

Ref-GeNVS is the first training-free, reflection-aware generative novel view synthesis framework that treats a mirror image as two complementary viewpoints, combining mirror-gated cross-view attention with latent reflection injection to synthesize photorealistic, reflection-consistent views from single or sparse inputs.

Background & Motivation

Mirrors substantially extend human perceptual fields by revealing physical areas that would otherwise remain occluded or out-of-frame. When humans observe a scene containing a mirror, they naturally disentangle reflected visual content from the surrounding background, treat the mirror as an auxiliary complementary viewpoint, and extrapolate global 3D geometry and contextual structures accordingly. Such reflection-aware reasoning is indispensable in practical scenarios ranging from embodied agent navigation (e.g., inspecting blind spots around tight corners) to immersive AR/VR 3D scene reconstruction. However, contemporary state-of-the-art generative novel view synthesis (NVS) models (such as SEVA, FlexWorld, and MVGenMaster) treat an image containing a mirror simply as an ordinary single view, ignoring the underlying physical laws of reflection. Consequently, when synthesizing novel views along unobserved trajectories, these models produce severe artifacts: "baked" mirror surfaces where reflections behave like static 2D stickers, leakage artifacts where non-reflective textures erroneously pollute the scene, and glaring inconsistencies between the generated 3D room and the reflection visible in the mirror.

Naively fine-tuning large generative multi-view diffusion models on mirror-containing datasets is not only hindered by the scarcity of high-quality paired captures, but also fails because reflection-aware NVS is fundamentally a physically constrained problem—data-driven training alone does not guarantee that networks implicitly learn strict reflection geometry. Meanwhile, existing neural radiance field and Gaussian splatting approaches for specular scenes (such as Mirror-NeRF or MirrorGaussian) require dozens of dense, calibrated camera views and focus solely on reconstructing reflective objects themselves, rather than synthesizing the unobserved scene revealed only through the mirror from sparse or single-image inputs.

This paper tackles the challenge from a distinct angle: since pre-trained multi-view diffusion backbones already possess formidable cross-view reasoning priors, one can physically inject reflected evidence as an explicit complementary virtual viewpoint without modifying any network weights. Core idea: treat a mirror-containing image as two complementary views via geometric camera reflection, use mirror-gated cross-view attention to isolate pure reflected cues during scene generation, and perform latent reflection injection with boundary guidance to achieve reflection-consistent, training-free novel view synthesis.

Method

Overall Architecture

Ref-GeNVS is built upon an "N-in M-out" multi-view diffusion backbone (such as SEVA or MVGenMaster) equipped with cross-view self-attention. Given single or sparse input images and camera poses containing a mirror, the goal is to synthesize target novel views that faithfully reproduce both the scene geometry revealed by the mirror and the mirror surface itself. The pipeline consists of an automated preprocessing stage followed by a two-stage generative process. In the preprocessing stage, the mirror plane is fitted from 3D point regressions to compute the Householder reflected virtual camera pose and flipped mirror image. In Stage 1, the mirror region in the original inputs is masked with a uniform color, and Mirror-gated attention forces target queries to attend exclusively to valid tokens inside the mirror of the reflected virtual view during scene generation. In Stage 2, Reflection injection fills the masked mirror surface at the target pose by injecting latent features from the corresponding virtual target pose, stabilized by SDEdit-style guidance to produce seamless mirror boundaries.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Images & Camera Poses<br/>Is, Ps"] --> B["Mirror Plane Estimation & Virtual View Construction<br/>DAM segmentation + 3D point regression + Householder reflection"]
    B --> C["Stage 1: Scene Generation with Mirror-gated Attention<br/>Uniform color masking + Gated cross-view attention"]
    C --> D["Stage 2: Mirror Surface Generation with Reflection Injection<br/>Latent_Flip injection + SDEdit-style boundary guidance"]
    D --> E["Reflection-Consistent Novel Views<br/>Io (accurate scene & mirror surface)"]

Key Designs

1. Mirror Plane Estimation & Virtual View Construction: formulating reflections as complementary virtual cameras Addressing the inability of off-the-shelf diffusion models to parse reflection geometry, this step transforms mirror physics into explicit multi-view conditioning. In the automated preprocessing pipeline, an off-the-shelf mirror segmenter DAM extracts the binary mirror mask \(M_s\). To circumvent erroneous depth estimation caused by virtual reflections inside the glass, the mirror region is masked with a uniform color before per-pixel 3D surface points are predicted via Depth Anything 3. The 3D points belonging to the mirror surface are isolated, and the mirror plane equation \(e = [n, d]\) is fitted robustly using RANSAC. The input camera pose \(P_{ms}\) is then reflected across plane \(e\) via the Householder reflection matrix:

\[H = I - 2 n n^\top\]

yielding the extrinsic pose of the virtual camera \(P'_{ms}\). Both the input image and its mirror mask are horizontally flipped to preserve the standard right-handed coordinate system, yielding the reflected tuple \((I'_{ms}, P'_{ms}, M'_{ms})\). Thus, an image with a mirror is converted into a dual-view system: the physical viewpoint and an aligned virtual viewpoint looking out from inside the mirror.

2. Stage 1: Mirror-gated Cross-view Attention: eliminating non-mirror leakage and baked surfaces Simply feeding the reflected virtual view as an additional conditioning input into a multi-view diffusion model causes severe failures: non-mirror pixels in the flipped image represent flipped, unphysical background structures that corrupt the target scene via cross-view attention, while retaining raw mirror pixels in the original input causes the model to "bake" the reflection as a flat surface texture.

Stage 1 resolves this by first painting the mirror area in the original view with a solid neutral color, removing the misleading flat texture. Then, during the cross-view self-attention computation, Mirror-gated attention is enforced. Let \(Q \in \mathbb{R}^{N_q \times d}\) denote target query tokens, and \(K, V \in \mathbb{R}^{N_k \times d}\) denote key and value tokens from the reflected input views. Given the downsampled binary mask \(m \in \{0, 1\}^{N_k}\) of the reflected mirror region, the attention logit matrix \(S_{ij} = \frac{q_i^\top k_j}{\sqrt{d}}\) is gated prior to softmax:

\[S'_{ij} = \begin{cases} \frac{q_i^\top k_j}{\sqrt{d}}, & \text{if } m_j = 1 \\ -\infty, & \text{if } m_j = 0 \end{cases}\]

\[\mathrm{Attn}_{\text{mirror}}(Q, K, V) = \mathrm{softmax}(S') V\]

Setting the logits of all tokens outside the mirror mask to \(-\infty\) zeroes out their softmax attention weights completely. Consequently, the target view attends strictly to pixels inside the mirror, drawing on valid reflected content while preventing any flipped non-mirror background from leaking into the generated 3D scene.

3. Stage 2: Reflection Injection with SDEdit-style Guidance: photorealistic mirror completion in latent space After Stage 1, target novel views contain high-quality 3D structures, but their mirror surfaces remain blank or uniformly masked. Stage 2 inpainting synthesizes reflection-consistent mirror surfaces aligned with the newly generated scene. First, SAM 2 tracks the mirror mask \(M_o\) in the Stage 1 outputs. During the reverse diffusion process from timestep \(t \to t-1\), the model simultaneously predicts latent features \(Z_t^m\) at the target pose \(P_{mo}\) and \(Z'^m_t\) at the reflected target pose \(P'_{mo}\). The clean latent \(Z'^m_{0|t}\) is estimated via Tweedie's formula, decoded to image space, horizontally flipped, and re-encoded back into latent space to bypass the lack of horizontal flip-equivariance in pre-trained VAE latents:

\[\hat{Z}_t^m = \tilde{M}_o^m \odot \mathrm{Latent\_Flip}(Z'^m_t) + (1 - \tilde{M}_o^m) \odot Z_t^m\]

\[\mathrm{Latent\_Flip}(Z'^m_t) = E(\mathrm{Flip}(D(Z'^m_{0|t}))) + \sigma_t \epsilon_t\]

To eliminate seam artifacts and structural dissonance around the mirror frame caused by abrupt latent replacement, SDEdit-style guidance is deployed for \(t > t_0\): the Stage 1 output is diffused with noise to form \(Z^{\text{init}}_t\), anchoring the non-mirror context. When \(t \le t_0\), reflection injection takes over, allowing the diffusion generative prior to smoothly harmonize the boundary between the injected mirror content and the surrounding room.

Loss & Training

Ref-GeNVS is an inference-time, training-free framework. It requires zero parameter updates or additional training stages, directly operating on frozen off-the-shelf multi-view diffusion backbones (MVGenMaster and SEVA), along with pre-trained segmentation and depth estimation networks (DAM, SAM 2, Depth Anything 3). All modifications occur inside the sampling loop via attention logit masking and latent feature injection.

Key Experimental Results

Main Results

Ref-GeNVS is evaluated across 6 photorealistic synthetic indoor scenes (bathrooms, living rooms with physically modeled mirrors) and the real-world Mirror-NeRF dataset. Testing trajectories are deliberately sampled from angles that observe regions visible only via mirror reflection. Evaluation measures both perceptual similarity (DreamSim, CLIP) and pixel-level geometric fidelity (PSNR, SSIM, LPIPS).

Dataset	Method	DreamSim ↓	CLIP Similarity ↑	PSNR ↑	SSIM ↑	LPIPS ↓
Real dataset [58]	MVGenMaster	0.361	0.867	12.75	0.413	0.536
Real dataset [58]	Ours (MVGenMaster)	0.194	0.930	13.67	0.455	0.474
Real dataset [58]	SEVA	0.275	0.920	12.03	0.394	0.564
Real dataset [58]	Ours (SEVA)	0.129	0.951	14.19	0.455	0.466
Synthetic dataset	MVGenMaster	0.222	0.930	16.90	0.735	0.350
Synthetic dataset	Ours (MVGenMaster)	0.106	0.964	17.95	0.748	0.291
Synthetic dataset	SEVA	0.264	0.929	13.65	0.623	0.482
Synthetic dataset	Ours (SEVA)	0.151	0.948	15.51	0.682	0.397

In the single-image input setting (Table 2), where scale ambiguity makes pixel-level metrics non-viable, Ref-GeNVS substantially outperforms VistaDream, FlexWorld, and raw SEVA on perceptual metrics averaged over input and ground-truth reflected views:

Method	DreamSim (DS) ↓	CLIP Similarity ↑
VistaDream	0.569	0.858
FlexWorld	0.364	0.885
SEVA	0.381	0.902
Ours	0.333	0.915

Ablation Study

On the synthetic dataset with the SEVA backbone, components are added incrementally to inspect their individual contributions (Table 3 of the original paper):

Virtual View	Mirror-gated Attn.	Two-step Gen.	DreamSim ↓	CLIP ↑	PSNR ↑	SSIM ↑	LPIPS ↓	Note
✗	✗	✗	0.264	0.929	13.65	0.623	0.482	Baseline SEVA; treats mirror as ordinary view
✓	✗	✗	0.236	0.922	13.48	0.647	0.549	Adds virtual camera; background leaks into scene
✓	✓	✗	0.147	0.954	15.51	0.683	0.390	Restricts attention to mirror tokens; huge perceptual gain
✓	✓	✓	0.151	0.948	15.51	0.682	0.397	Full pipeline; harmonizes mirror surface with generated scene

Key Findings

• Mirror-gated attention is the primary defense against leakage artifacts: Adding the virtual reflected view without gating improves geometric reasoning slightly but worsens LPIPS (0.482 to 0.549) due to flipped non-mirror background leakage. Introducing Mirror-gated attention slashes DreamSim from 0.236 to 0.147 and drops LPIPS to 0.390, confirming that blocking non-mirror tokens is essential.
• Two-step generation achieves holistic physical coherence: While two-step generation yields comparable quantitative scores to Stage 1 alone (since mirrors occupy a modest fraction of image pixels), qualitative comparisons show it resolves inconsistencies where mirror reflections mismatch newly generated room elements.
• Enables downstream 3D reconstruction from single images: When generated novel views are fed into Pi³ for feedforward 3D reconstruction, baseline methods produce fragmented or hollow point clouds behind mirrors, whereas Ref-GeNVS reconstructs complete, physically accurate 3D geometries in previously occluded areas.

Highlights & Insights

• Decoupling geometric constraints from diffusion priors: Rather than forcing a neural network to memorize physical reflection laws via brute-force fine-tuning, Ref-GeNVS enforces rigid optical symmetry at the input pose level and delegates contextual hallucination to diffusion priors, achieving training-free zero-shot generalization.
• Tweedie-guided Latent_Flip overcoming VAE non-equivariance: Because VAE latent spaces are not naturally equivariant under spatial reflection, the authors elegantly apply Tweedie's formula to predict \(Z_0\), perform spatial flipping in image space, and re-encode to inject coherent features into target latents.
• Seamless extension to curved/distorted mirrors: By unwarping distorted reflections from mirror balls or convex mirrors into an equivalent planar representation using known surface geometry, Ref-GeNVS generalizes directly to non-planar mirror setups without architectural modification.

Limitations & Future Work

• Reliance on accurate planar geometry: The automatic pipeline presumes a planar mirror. Substantial inaccuracies in 3D surface point prediction or extreme occlusion on mirror boundaries can degrade RANSAC plane fitting, leading to misaligned virtual camera poses.
• Increased sampling latency from two-stage generation: Generating the unobserved scene first and subsequently inpainting the mirror surface requires two successive reverse diffusion cycles, roughly doubling inference time compared to standard multi-view diffusion.
• Future directions: Integrating reflection constraints and mirror detection as explicit inductive biases into end-to-end multi-view diffusion pre-training represents an exciting avenue to achieve single-pass reflection-aware generation.

Related Work & Insights

• vs. Mirror-NeRF / MirrorGaussian: Traditional radiance field/Gaussian splatting techniques require dozens of dense calibrated views and focus narrowly on rendering specular surfaces via ray tracing or reflected Gaussian kernels. They cannot generate unobserved 3D scenes from single or sparse inputs. Ref-GeNVS is a generative method capable of hallucinating blind-spot environments.
• vs. FlexWorld / SEVA / VistaDream: Standard generative NVS frameworks fail to understand specular physics, treating mirror content as planar wall art that moves incorrectly with the camera. Ref-GeNVS equips these backbones with reflection awareness in a plug-and-play, training-free manner.

Rating

• Novelty: ⭐⭐⭐⭐☆ (Creative formulation of mirror reflections as virtual camera views with gated attention in generative NVS)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Rigorous quantitative and qualitative validation on real and synthetic datasets, including ablation and downstream 3D reconstruction)
• Writing Quality: ⭐⭐⭐⭐⭐ (Clear problem exposition, crisp mathematical definitions, and compelling visual evidence)
• Value: ⭐⭐⭐⭐☆ (A practical, plug-and-play solution for specular scene generation and embodied robotic perception)

Related Papers

• [ECCV 2026] REON-NVS: Real-Time Online Novel-View Synthesis from Sparse-View Videos
• [ECCV 2026] Global Pose Control for Generative View Synthesis in Normalized Object Coordinate Space
• [ECCV 2026] Nexels: Neurally-Textured Surfels for Real-Time Novel View Synthesis with Sparse Primitives
• [ECCV 2026] RefracGS: Novel View Synthesis Through Refractive Water Surfaces with 3D Gaussian Ray Tracing
• [ECCV 2026] MoCam: Unified Novel View Synthesis via Structured Denoising Dynamics