PanoLess: Environment Reconstruction from Partial Reflective Views¶

Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://vb-glee.github.io/panoless/
Area: 3D Vision
Keywords: 2D Gaussian Splatting, Specular Reflection, Environment Reconstruction, Deferred Shading, Visibility Map

TL;DR¶

PanoLess reconstructs high-frequency distant environment illumination from partial reflective views on a single side of shiny objects via surface-aligned 2D Gaussian surfels, unattenuated neural cubemap deferred shading, and an online visibility confidence map, improving environment map PSNR by nearly 5 dB over prior reflection-aware baselines.

Background & Motivation¶

Specular reflections from shiny objects and glass façades naturally serve as indirect wide-angle imaging channels, extending the field of view of a camera into its physical blind spots without requiring camera panning or panoramic capture. However, the observed radiance of a reflective surface is jointly governed by surface normal geometry and ambient illumination, creating an inherent physical ambiguity where multiple combinations of geometry and lighting can yield identical observations. In practical scenarios—such as robotics, autonomous driving, and handheld AR capture—cameras are typically restricted to only one side of an object. This setup captures partial, incomplete reflection cues while lacking the 360° multi-view geometric anchoring that conventional reconstruction pipelines rely upon, causing both classic Lambertian stereo and modern neural inverse rendering to fail.

Recent radiance field methods based on 3D Gaussian Splatting (3DGS) offer remarkable novel-view synthesis speed, but volumetric ellipsoidal Gaussians lack well-defined physical surfaces. Their heuristic shortest-axis normals are unstable and noisy, leading to severely perturbed reflection rays. While surface-aligned 2D Gaussian Splatting (2DGS) provides geometrically sound tangent planes, existing specular extensions still depend on low-frequency spherical harmonics, heavy BRDF integration, or dense omnidirectional captures. Crucially, when applied to partial-view inputs, standard roughness attenuation mechanisms exponentially dampen photometric gradients back to the environment representation, causing the reconstructed environment to collapse in sparsely observed regions.

PanoLess resolves this dilemma by treating the specular object as an outward-facing curved mirror whose geometry, once accurately estimated, can directly trace reflected camera rays into the distant environment. Core idea: by coupling surface-aligned 2D Gaussian surfels with an unattenuated neural cubemap deferred shading pipeline and tracking ray-sampling density online into an explicit visibility map, PanoLess enables geometrically consistent, high-fidelity environment illumination reconstruction from incomplete single-sided reflective captures.

Method¶

Overall Architecture¶

PanoLess takes multi-view images captured from only one side of a specular reflector and disentangles the reflector's 3D geometry from the surrounding high-frequency distant illumination. The overall framework operates across three key stages: first, a 2D Gaussian splatting pass rasterizes surface-aligned surfels into screen-space geometric buffers and diffuse albedo; second, a deferred specular shading pass derives reflection rays from composite normals and queries a learnable neural cubemap without attenuation; third, an online visibility accumulation pass records ray hit frequencies across the sphere to output a per-texel confidence map indicating observed versus unobserved directions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Single-Sided Partial Views"] --> B["Surface-Aligned 2DGS Rasterization<br/>Screen-space geometry & albedo buffers"]
    B --> C["Composite Geometry & Normal Extraction<br/>Depth blending & coordinate transformation"]
    C --> D["Unattenuated Neural Cubemap Deferred Shading<br/>Direct bilinear query via reflection rays"]
    D --> E["Online Visibility Accumulation<br/>Opacity-weighted ray hit tracking"]
    E --> F["Joint Photometric & Geometric Optimization<br/>End-to-end update of splats & cubemap"]
    F --> G["Output: 3D Geometry + Environment Map + Visibility Map"]

Key Designs¶

1. Surface-Aligned 2DGS Rasterization & Composite Geometry Extraction: Foundation of Accurate Normals

Specular reflection rays are acutely sensitive to surface normal errors, with first-order perturbation analysis revealing $\|\delta \boldsymbol{\omega}_r\| \le 4 \|\delta \mathbf{n}\|$, which heavily magnifies angular errors at grazing angles. Standard 3DGS produces fragmented normal fields that corrupt reflection indexing. PanoLess represents the surface with $K$ planar 2D Gaussian primitives $G_i$, each parameterized by a center $\boldsymbol{\mu}_i$ and tangent vectors $\mathbf{t}_{u,i}, \mathbf{t}_{v,i}$, yielding exact analytical normals $\mathbf{n}_i = \frac{\mathbf{t}_{u,i} \times \mathbf{t}_{v,i}}{\|\mathbf{t}_{u,i} \times \mathbf{t}_{v,i}\|}$. During rasterization, the model outputs expected depth $D_{\text{exp}}$, median depth $D_{\text{med}}$, accumulated opacity $\alpha$, and rasterized normals $\mathbf{N}_{\text{rast}}$. To mitigate semi-transparency noise while preserving crisp silhouette edges, composite surface depth is computed as: $$ D_{\text{surf}}(\mathbf{p}) = \lambda_d D_{\text{exp}}(\mathbf{p}) + (1 - \lambda_d) D_{\text{med}}(\mathbf{p}) $$ with $\lambda_d = 0.5$. Screen-space differentiation of $D_{\text{surf}}$ provides depth-derived normals $\mathbf{n}_{\text{surf}}$, while world-space normals are transformed from rasterized normals via $\mathbf{N}_{\text{world}} = \mathbf{V}_{3\times3}^\top \mathbf{N}_{\text{rast}}$. $\mathbf{N}_{\text{world}}$ directly drives the reflection calculation, while $\mathbf{n}_{\text{surf}}$ acts as a geometric regularizer to enforce spatial smoothness.

2. Unattenuated Neural Cubemap Deferred Shading: Direct Photometric Gradient Pathway

Conventional inverse rendering pipelines modulate specular environment reflections using roughness parameters $\rho$ via attenuation factors like $(1 - \rho)^2$ or evaluate multi-level pre-filtered Mipmaps. Under incomplete single-sided views, this parameterization is counterproductive: non-zero predicted roughness suppresses specular radiance and starves the environment map of gradient supervision in sparsely observed angular regions. For highly specular objects, PanoLess represents distant illumination as six trainable $L \times L$ RGB grids forming a neural cubemap tensor $\mathcal{E} \in \mathbb{R}^{6 \times 3 \times L \times L}$. For each pixel $\mathbf{p}$ with camera ray $\mathbf{v}(\mathbf{p})$, the reflection direction is computed as: $$ \boldsymbol{\omega}r(\mathbf{p}) = \text{normalize}\bigl(\mathbf{v}(\mathbf{p}) - 2\,(\mathbf{v}(\mathbf{p}) \cdot \mathbf{N})\bigr) $$ The radiance is bilinearly queried directly as }}(\mathbf{p}))\,\mathbf{N}_{\text{world}}(\mathbf{p$\mathbf{L}_{\text{env}}(\boldsymbol{\omega}_r) = \sigma(\mathcal{E}(\boldsymbol{\omega}_r))$ without an intermediate MLP or roughness attenuation. The composite color is simply $\mathbf{C}_{\text{out}}(\mathbf{p}) = \mathbf{A}(\mathbf{p}) + \mathbf{L}_{\text{env}}(\boldsymbol{\omega}_r(\mathbf{p}))$, granting the photometric loss an unattenuated, clean backpropagation path straight to the cubemap texels.

3. Online Visibility Accumulation: Quantifying Observability Boundaries

Because partial-view captures only interrogate a subset of the complete environment sphere, unobserved directions receive zero photometric gradient. Existing methods often impose uniform global priors that cause over-smoothing or spurious artifacts. PanoLess introduces an explicit online visibility map $\mathcal{V}: S^2 \to [0, 1]$ instantiated on the cubemap grid. For each training pixel satisfying $\alpha(\mathbf{p}) > \tau$, an increment weighted by pixel opacity is scattered into $\mathcal{V}$ at the queried reflection direction $\boldsymbol{\omega}_r(\mathbf{p})$. After training, $\mathcal{V}$ serves as a calibrated confidence score: high values signify directions directly constrained by empirical reflection rays, whereas values near zero flag unsupported regions. This provides a principled mask for gating downstream generative inpainting or relighting without fabricating ungrounded details.

Loss & Training¶

The 2D Gaussian primitives and the neural cubemap parameters are optimized jointly end-to-end from iteration zero using a composite objective: $$ \mathcal{L} = \mathcal{L}{\text{color}} + \lambda}}\,\mathcal{L{\text{n-d}} + \mathbb{I} $$ The photometric loss }\,\mathcal{L}_{\alpha$\mathcal{L}_{\text{color}} = (1 - \lambda_s)\mathcal{L}_1 + \lambda_s \mathcal{L}_{\text{D-SSIM}}$ with $\lambda_s = 0.2$ compares composited color $\mathbf{C}_{\text{out}}$ with ground-truth $\mathbf{I}_{\text{gt}}$. The depth-normal consistency regularizer $\mathcal{L}_{\text{n-d}} = 1 - \mathbf{N}_{\text{world}}^\top \mathbf{n}_{\text{surf}}$ aligns rasterized surfel normals with depth gradients. To compensate for the lack of 360° geometric triangulation, an early silhouette loss $\mathcal{L}_{\alpha} = \text{BCE}(\alpha(\mathbf{p}), m_{\text{gt}}(\mathbf{p}))$ supervises accumulated opacity against ground-truth alpha masks for the first 3000 steps, preventing boundary Gaussian floaters from polluting reflection angles.

Key Experimental Results¶

Main Results¶

The method is evaluated on the newly introduced Shiny Partial benchmark featuring three challenging scenes (Vase, Cola, and Mirror) restricted to a single hemisphere of camera poses (100 training and 200 testing images) with ground-truth environment maps and surface normals.

Scene	Method	Env. Map PSNR (dB) ↑	Env. Map SSIM ↑	Env. Map LPIPS ↓	Env. Map L1 ↓	Normal MAE (°) ↓	Normal Median (°) ↓	Normal < 5° (%) ↑
Vase	3DGS-DR	7.96	0.204	0.665	0.346	14.76	5.06	49.82
	GaussianShader	9.62	0.121	0.668	0.264	28.45	22.74	0.08
	Ref-Gaussian	13.63	0.397	0.535	0.150	23.11	17.15	1.03
	MaterialRefGS	14.05	0.474	0.461	0.155	10.67	6.23	30.39
	Ours (PanoLess)	18.51	0.635	0.487	0.082	3.97	2.73	88.10
Cola	3DGS-DR	4.52	0.270	0.700	0.526	15.46	4.49	52.16
	GaussianShader	6.47	0.118	0.736	0.402	33.56	23.75	0.22
	Ref-Gaussian	7.26	0.249	0.660	0.372	22.35	16.91	3.47
	MaterialRefGS	6.81	0.302	0.587	0.416	10.23	8.06	21.55
	Ours (PanoLess)	9.26	0.442	0.477	0.325	6.25	3.21	82.98
Mirror	3DGS-DR	6.22	0.274	0.711	0.420	46.59	68.18	0.11
	GaussianShader	8.54	0.145	0.710	0.297	52.15	52.14	0.00
	Ref-Gaussian	5.56	0.226	0.746	0.457	14.76	14.19	0.00
	MaterialRefGS	5.82	0.291	0.755	0.449	14.04	13.46	0.00
	Ours (PanoLess)	13.31	0.337	0.655	0.177	4.64	4.62	97.00
Average	3DGS-DR	6.23	0.249	0.692	0.431	25.61	25.91	34.03
	GaussianShader	8.21	0.128	0.705	0.321	38.05	32.88	0.10
	Ref-Gaussian	8.82	0.291	0.647	0.326	20.07	16.08	1.50
	MaterialRefGS	8.89	0.356	0.601	0.340	11.64	9.25	17.31
	Ours (PanoLess)	13.69	0.471	0.540	0.195	4.95	3.52	89.36

Ablation Study¶

The contribution of each design component is analyzed on the Vase scene across both novel-view synthesis (Render) and environment map reconstruction (Env. Map).

Config	Env. Map PSNR (dB) ↑	Env. Map SSIM ↑	Env. Map LPIPS ↓	Env. Map L1 ↓	Render PSNR (dB) ↑	Render SSIM ↑	Render LPIPS ↓	Render L1 ↓	Note
Full model (Ours)	18.51	0.635	0.487	0.082	38.56	0.988	0.022	0.002	Full framework with 2DGS, direct cubemap, $\mathcal{L}_n$, and early $\mathcal{L}_\alpha$
$\rho$-weighted $\mathcal{E}$	13.94	0.569	0.546	0.146	36.16	0.982	0.036	0.003	Reintroducing roughness attenuation $(1-\rho)^2$ drops Env. PSNR by 4.57 dB
w/o $\mathcal{L}_\alpha$	12.91	0.466	0.559	0.177	33.75	0.977	0.045	0.004	Omitting early mask loss allows boundary floaters, reducing Env. PSNR by 5.60 dB
w/o $\mathcal{L}_n$	8.68	0.098	0.621	0.337	35.82	0.978	0.042	0.004	Removing depth-normal consistency collapses Env. SSIM to 0.098

In an additional fairness check where baseline MaterialRefGS was retrained with the identical early silhouette loss $\mathcal{L}_\alpha$, its normal MAE remained inferior at 10.42° (Cola), 9.73° (Vase), and 14.79° (Mirror), with 0.0% of normals under 5° on Mirror, confirming that the performance gain is grounded in the 2D surfel formulation.

Key Findings¶

Geometric Normal Fidelity Is Indispensable: Removing normal consistency regularizer $\mathcal{L}_n$ causes environment PSNR to plunge by 9.83 dB and SSIM to collapse to 0.098. Because reflection vectors magnify normal errors linearly up to a factor of 4, small normal inaccuracies send query rays into mismatched texels, producing fatal gradient conflicts.
Roughness Parameterization Harmful in Sparse Settings: While roughness-based attenuation works in dense 360° setups, in partial-view regimes even slight non-zero roughness dampens gradients and starves sparsely sampled texels, leading to a 4.57 dB drop. Direct unattenuated lookup is essential.
Surface Curvature Governs Angular Reach: Curved geometries (Cola, Vase) sweep reflection rays across extensive angular horizons under single-sided views. In contrast, planar reflectors (Mirror) produce narrow directional cones; baselines fail entirely on Mirror (0% normals within 5°), whereas PanoLess maintains 97.0% normal precision and reconstructs clean specular content.

Highlights & Insights¶

Turning Specular Reflectors into Passive Spherical Cameras: Instead of treating specular reflections as annoying view-dependent artifacts to be filtered out, PanoLess converts shiny surfaces into indirect wide-angle sensors, unlocking spatial awareness without dedicated panorama rigs.
Unattenuated Gradient Channels for Sparse Data: Identifies the fundamental flaw of traditional BRDF roughness modulation under incomplete views and adopts a minimal, direct neural cubemap lookup that maximizes gradient transmission efficiency.
Native Physical Confidence Quantification: Rather than generating speculative hallucinations in unseen directions, the online visibility map precisely delineates observation boundaries, delivering an actionable control mask for downstream neural editing and relighting.

Limitations & Future Work¶

Strict Dependence on High Specularity: On diffuse or matte materials (such as liquid coffee inside a cup), absence of specular reflections deprives the cubemap of gradient updates, causing those directions to fade.
Vulnerability to Pose Inaccuracy: Due to the first-order angular amplification of reflection rays, even minor camera rotation jitter ($\sigma = 0.5^\circ$) degrades environment map PSNR from 18.4 dB to 12.0 dB, necessitating high-accuracy COLMAP poses.
Far-Field Infinite Distance Assumption: Modeling environment illumination as a distant directional cubemap discards near-field parallax and occlusions; future work could integrate multi-layer depth-aware radiance fields or diffusion-based priors.

vs 3DGS-DR / GaussianShader: Volumetric Gaussian reflection methods rely on covariance heuristics for normals, yielding 25°–38° normal MAE under partial views; PanoLess uses rigorous 2D planar surfels to achieve sub-5° normal MAE.
vs MaterialRefGS: MaterialRefGS applies complex BRDF decomposition and roughness filtering that stalls under single-sided views; PanoLess streamlines the gradient pipeline through direct cubemap lookup.
vs Ref-NeRF: Ref-NeRF optimizes reflection rays over implicit coordinate MLPs requiring hours of computation; PanoLess leverages explicit rasterization to achieve sharp geometries with real-time novel-view synthesis.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Pioneers partial-view environment reconstruction by marrying 2D Gaussian surfels with direct neural cubemaps and visibility maps.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations on the custom Shiny Partial benchmark, baseline alignment, ablations, and real-world captures.
Writing Quality: ⭐⭐⭐⭐⭐ Rigorous error-propagation derivations, cohesive narrative, and well-structured empirical validation.
Value: ⭐⭐⭐⭐☆ Highly valuable for spatial intelligence, embodied robotics, and handheld AR where panoramic sensors are unavailable.