When 3D Gaussian Splatting Recovers Real Surfaces¶

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: Pending
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Geometric Identifiability, Spherical Harmonics, Parallax Aliasing, Billboard Failure

TL;DR¶

This paper develops a mathematical framework based on a first-hit rendering abstraction to investigate the geometric identifiability of 3D Gaussian Splatting (3DGS), proving that geometric misalignment forcefully converts spatial textures into high-frequency angular signals via parallax, and establishing an intermediate spherical harmonic capacity window where true surface recovery is strictly preferred over opaque billboard collapse.

Background & Motivation¶

3D Gaussian Splatting (3DGS) has rapidly emerged as a foundational representation for novel view synthesis, 3D surface reconstruction, and downstream interactive physics thanks to its real-time, high-fidelity rendering capabilities. Recent research increasingly moves beyond treating 3DGS as a black-box 2D image synthesizer, directly extracting explicit triangle meshes or contact manifolds from the learned Gaussian primitives under the implicit premise that the optimized field faithfully reflects the underlying physical scene geometry. However, all existing geometric reconstruction successes rest primarily on empirical heuristics, leaving an unresolved foundational question: under what formal conditions does the standard 3DGS objective actually guarantee recovery of the true scene surface?

This absence of theoretical grounding introduces a severe geometric trustworthiness crisis. Photometric accuracy across training views does not guarantee geometric convergence to the ground truth. When the representation is endowed with excessive angular capacity to capture view-dependent radiance, the optimization landscape permits degenerate shortcut solutions: the model can place dense, opaque Gaussian primitives at incorrect spatial locations and exploit high-degree view-dependent coefficients to reproduce rays passing through them. Under such conditions, rendering loss drops to near-zero and PSNR remains high, yet the reconstructed point cloud and mesh drift drastically away from the actual surface.

This paper tackles the challenge by formulating the interaction between spatial geometry and view-dependent appearance through the lens of frequency analysis and projection theory. The authors discover that spatial diffuse textures, when sampled across viewpoints through a misaligned geometric proxy, are forcefully modulated by parallax into high-frequency angular color variations. Core idea: construct an analytically tractable first-hit rendering abstraction to establish the parallax frequency transport law $\omega_{\text{fake}} \approx k\Delta z$, proving that bounded angular capacity ($L < \omega_{\text{fake}}$) renders the true surface strictly easier to fit, whereas unbounded capacity triggers catastrophic "opaque billboard" geometric collapse, thereby delineating a well-defined intermediate capacity window for faithful 3D surface recovery.

Method¶

Overall Architecture¶

The proposed framework abstracts the standard 3DGS rendering process into a minimal first-hit rendering model, decoupling the entangled depth-ordering and visibility issues of volumetric alpha-compositing into geometric intersection and local angular projection. For any given camera ray, the rendering operator identifies the first-hit point on an opaque candidate surface and evaluates its emitted color via an orthogonal projection onto the subspace $V_L$ spanned by real Spherical Harmonics (SH) up to degree $L$. Within this abstraction, the geometric identifiability of any candidate surface against the true ground-truth manifold reduces to comparing their respective projection residuals in the angular frequency domain.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Camera Ray Input (y, v)"] --> B["First-Hit Abstraction & Pointwise Projection"]
    B --> C["True Surface Sobolev Smoothness & Upper Bound"]
    B --> D["Misaligned Surface Parallax Frequency & Lower Bound"]
    C --> E["Identifiability Window & Billboard Failure Characterization"]
    D --> E
    E --> F["Output: Surface-Consistent Recovery or Billboard Collapse"]

Key Designs¶

1. First-Hit Abstraction & Pointwise Projection: Decoupling Volumetric Compositing into Local Projections Standard 3DGS composites colors via volumetric alpha-blending across numerous semi-transparent primitives along each ray, entangling depth order, occlusion boundaries, and geometric identifiability. Profiling empirical 3DGS checkpoints reveals extreme depth-wise weight concentration: for 92.6% of foreground rays, the front-most 3 contributing Gaussians systematically account for over 90.2% of the total compositing mass. Motivated by this, the authors introduce a first-hit opaque surface abstraction where the ray color is determined entirely by the appearance function at the initial intersection point. For any candidate surface $U$ with hit point $x$, the expected angular color demanded by the training rays is formalized as the through-seen target $F_U(x, \hat{v}) := \mathbb{E}[I^*(Y, \hat{V}) \mid X_U = x, \hat{V} = \hat{v}]$. Proposition 1 proves that for a fixed geometry $U$, the population loss minimizes pointwise as an orthogonal projection onto the SH subspace: $$g^*(x, \cdot) = \Pi_{V_L}^{(x)}(F_U(x, \cdot))$$ The optimal achievable loss $B_U(L)$ is precisely the expected squared projection distance onto the orthogonal complement of $V_L$. This converts the geometric identifiability problem into a rigorous signal approximation question.

2. True Surface Sobolev Smoothness & Upper Bound: Zero Angular Overhead from Spatial Diffuse Textures On the true scene surface $S$, physical appearance decomposes into a view-independent diffuse texture $T(x)$ and a view-dependent specular residual $R(x, \hat{v})$. Crucially, however complex or high-frequency the spatial diffuse texture $T(x)$ is across the surface manifold, at any fixed point $x \in S$ it remains constant with respect to viewing direction $\hat{v}$, thus residing entirely within the degree-0 SH subspace $V_0 \subset V_L$ and consuming zero angular bandwidth. Under the realistic physical assumption that the specular residual is Sobolev-smooth ($\|R(x, \cdot)\|_{H^{s_{\text{true}}}(S^2)} \le M_{\text{true}}$), spectral properties of the Laplace-Beltrami operator dictate that the projection truncation error decays algebraically with degree $L$: $$A(L) \le (1 + L(L + 1))^{-s_{\text{true}}} M_{\text{true}}^2$$ This ensures that at moderate SH capacities (such as the default $L=3$), the true surface achieves rapid loss reduction, acting as an attractive, easily optimizable basin.

3. Misaligned Surface Parallax Frequency & Lower Bound: Spatial Textures Forged into Angular Oscillations When an optimizer posits a geometrically misaligned surface $U$ separated from the ground truth by depth discrepancy $\Delta z = z_{\text{gt}} - z_{\text{fake}} > 0$ (such as an opaque billboard plane placed ahead of the actual geometry), parallax alters the signal structure fundamentally. Sweeping the camera viewing angle by $\alpha$ shifts the true background intersection point according to $x_{\text{hit}}(\alpha) = x_0 + \Delta z \tan \alpha \approx x_0 + \Delta z \alpha$. If the background possesses a spatial diffuse texture component $a \sin(ku)$ with frequency $k$, the candidate surface must emit the transported signal $a \sin(k x_0 + (k\Delta z)\alpha)$ across views to explain the observations. This derives the fundamental parallax frequency transport law: $$\omega_{\text{fake}} \approx k \Delta z$$ The product of spatial frequency $k$ and depth error $\Delta z$ dictates the required angular frequency. Because an SH basis of degree $L$ restricted along viewing circles cannot exceed frequency $L$, whenever $L < \omega_{\text{fake}}$, trigonometric polynomial approximation inevitably incurs an irreducible non-zero residual, guaranteeing a stubborn lower bound $B_U(L) \ge c_0 > 0$.

4. Identifiability Window & Billboard Failure Characterization: Intermediate Capacity Guarantees True Recovery Combining the rapidly decaying upper bound $A(L)$ of the true surface with the irreducible lower bound $B_U(L)$ of misaligned candidates, Theorem 1 establishes the intermediate-capacity identifiability window: for any degree $L$ satisfying $(1 + L(L + 1))^{-s_{\text{true}}} M_{\text{true}}^2 < c_0$ and $L < \omega_{\text{fake}}$, the true surface achieves strictly lower loss ($A(L) < B_U(L)$), making it mathematically favored by the optimizer. Conversely, when the angular capacity is excessively large ($L \ge \omega_{\text{fake}}$), the SH basis possesses sufficient expressive bandwidth to memorize the parallax-induced false textures. In this regime, the optimizer is mathematically permitted to place an arbitrary opaque billboard in empty space and fit the training images with zero population loss, triggering catastrophic geometric collapse despite high photometric fidelity.

Loss & Training¶

The framework analyzes the population image-space mean squared error: $$\mathcal{J}(U, g) = \mathbb{E}_{(Y, \hat{V}) \sim \mathcal{P}}\left[ |I_{U,g}(Y, \hat{V}) - I^*(Y, \hat{V})|^2 \right]$$ The theoretical derivation directly validates the practice of Spherical Harmonics capacity scheduling: initializing training with low capacity ($L=0$, pure diffuse) before progressively enabling higher degrees ($L=1, 2, 3$). This schedule enforces $L < \omega_{\text{fake}}$ during the early phases, leveraging the identifiability window to pull Gaussian centers onto the true physical surface before complex view-dependent specularities are fitted.

Key Experimental Results¶

Main Results¶

To evaluate whether opaque billboard collapse emerges in practice under standard multi-view capture protocols, experiments were conducted across 10 real-world datasets with ground-truth geometry, comparing standard capacity ($SH=3$) against high capacity ($SH=24$). Geometric accuracy is evaluated via normalized symmetric Chamfer distance (nCD := Chamfer Distance / bounding box diagonal, lower is better), while rendering fidelity is measured by PSNR (higher is better).

Dataset	nCD@SH=3 (↓)	PSNR@SH=3 (↑)	nCD@SH=24 (↓)	PSNR@SH=24 (↑)	Geometric Drift Ratio (nCD Ratio)
DTU	0.0090	31.6 dB	0.0100	32.3 dB	1.11×
MobileBrick	0.0600	28.9 dB	0.0700	29.5 dB	1.17×
YCB-Video	0.0420	28.1 dB	0.0500	28.8 dB	1.19×
T-LESS	0.0650	27.6 dB	0.0750	28.1 dB	1.15×
ScanNet	0.0110	29.3 dB	0.0120	29.9 dB	1.09×
ScanNet++	0.0070	29.8 dB	0.0080	30.3 dB	1.14×
Matterport3D	0.0045	30.4 dB	0.0050	31.0 dB	1.11×
HM3D	0.0047	30.8 dB	0.0053	31.4 dB	1.13×
Tanks & Temples	0.0080	29.0 dB	0.0090	29.7 dB	1.13×
ETH3D	0.0090	30.1 dB	0.0100	30.6 dB	1.11×
Average	0.0220	29.6 dB	0.0254	30.2 dB	1.15×

Ablation Study¶

On a controlled synthetic stress-test benchmark consisting of 100 shapes (25 base geometries with 4 controlled frequency/lighting variations), the authors systematically assessed the emergence of billboard failures as SH capacity increases. A billboard failure is strictly labeled when PSNR remains within 2 dB of optimal while nCD degrades by more than 5× relative to the best geometry.

Failure Mode / Regime	Percentage	nCD Trajectory Profile	Critical Failure Threshold	Representative Performance (SH=3 vs SH=24)
Burst Failure	~53%	Stays low at small degrees, then explodes abruptly once SH crosses critical threshold	SH = 6–12	nCD spikes from 0.03 to 6.28, while PSNR improves from 34.9 to 35.2 dB
Linear Drift	~42%	nCD degrades approximately linearly with increasing SH degree	SH = 16–24	Meets billboard-onset threshold at high degrees; layered peeling occurs
Safe Regime	~5%	nCD remains strictly within $2\times\text{nCD}_{\text{best}}$ across all evaluated SH degrees	No failure observed	Highly dense spatial texture ($k$ large) pushes $\omega_{\text{fake}} > 24$, preventing collapse

Key Findings¶

The Billboard Paradox in Synthetic Stress Tests: In synthetic settings with controlled texture, approximately 89% of evaluated shapes experience billboard degradation at high SH degrees. The model produces sharper rendering (PSNR increases by 0.3–0.5 dB) while the underlying 3D point cloud collapses into an off-surface hollow billboard sheet (nCD error degrades by over two orders of magnitude).
Inherent Shielding in Real-World Scenes: Across all 10 real-world datasets, zero opaque billboard failures were detected (mean nCD at SH=24 is only 1.15× that at SH=3). Real-world scenes naturally exhibit abundant high-frequency spatial textures (large $k$), driving the parallax-induced frequency $\omega_{\text{fake}} \approx k\Delta z$ well beyond tested capacities ($L=24 \ll \omega_{\text{fake}}$) and safely locking optimization within the identifiability window.
The Double-Edged Nature of High Angular Capacity: While higher SH orders improve fidelity on non-Lambertian specular highlights, they simultaneously erode the photometric energy barrier that penalizes incorrect geometry.

Highlights & Insights¶

Formulating Geometric Identifiability as Frequency Projection: By introducing the first-hit abstraction and pointwise projection theorem, the paper bypasses non-convex optimization dynamics to derive clean analytical bounds on geometric identifiability.
Parallax as a Spatial-to-Angular Frequency Multiplier: Demonstrating that geometric error scales spatial frequency into the angular domain ($\omega_{\text{fake}} \approx k\Delta z$) provides a rigorous theoretical explanation for why texture-rich regions reconstruct robustly while textureless surfaces suffer from geometric floaters.
Theoretical Justification for Capacity Scheduling: The proofs mathematically substantiate why progressive SH scheduling (starting from diffuse $L=0$ before enabling higher orders) and geometric regularizers (such as 2DGS, SuGaR, or normal constraints) are fundamentally necessary to prevent degenerate shortcut solutions.

Limitations & Future Work¶

Opaque First-Hit Assumption Boundary: While well-validated for mostly opaque scenes, the first-hit model cannot directly represent semi-transparent participating media, volumetric smoke, or subsurface scattering.
Extreme Non-Lambertian Reflectance Limits: Assumption 4 excludes mirror-like reflections or severe inter-reflections capable of cancelling diffuse patterns; in such settings, specular residuals inherently contain high angular frequencies, challenging the spatial/angular decoupling.
Continuous Camera Sampling Idealization: The theoretical derivation assumes continuous ray coverage across viewing arcs; discrete camera distributions under sparse-view capture require further formal analysis regarding angular aliasing.

vs Classical Plenoptic Sampling Theory: While pioneering works by Chai et al. and Durand et al. examined plenoptic bandwidth and sampling density in image-based rendering, this work repurposes frequency transport to establish geometric identifiability bounds for 3D radiance fields.
vs Shape-Radiance Ambiguity (Ref-NeRF / NeRF++): Prior neural radiance field studies qualitatively noted ambiguities between specular highlights and concave geometry; this paper provides the first quantitative frequency-scaling law ($\omega_{\text{fake}} \approx k\Delta z$) governing this trade-off in 3DGS.
vs Geometry-Aware 3DGS (2DGS / SuGaR / PGSR)：These methods incorporate planar constraints or depth/normal priors to encourage surface alignment; this theoretical work proves why such geometric constraints are indispensable when angular capacity is large, explaining how they restrict the admissible geometry to eliminate the billboard-favorable regime.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Establishes the first rigorous mathematical framework and spectral projection theory for 3DGS geometric identifiability and billboard failure.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 100 controlled synthetic stress tests and 10 real-world 3D scanning benchmarks, achieving full theoretical-empirical consistency.
Writing Quality: ⭐⭐⭐⭐⭐ Flawless mathematical rigor, clear physical intuition, elegant prose, and coherent notation.
Value: ⭐⭐⭐⭐⭐ Provides an indispensable theoretical foundation for downstream mesh extraction and physical interaction built on top of 3DGS, providing concrete guidelines for capacity scheduling and regularization design.