Confidence-Based Mesh Extraction from 3D Gaussians

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/r4dl/CoMe
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Mesh Extraction, Surface Reconstruction, Confidence Framework, Variance Losses

TL;DR

To prevent 3D Gaussian Splatting from faking view-dependent reflections with corrupted geometry, this paper introduces a self-supervised confidence framework that dynamically balances photometric and geometric losses, combines it with color and normal blending variance penalties, and decouples the D-SSIM luminance term, achieving state-of-the-art unbounded mesh extraction without heavy external priors.

Background & Motivation

Accurate reconstruction of high-quality surface meshes from multi-view captures remains a foundational problem in computer vision. Recently, 3D Gaussian Splatting (3DGS) has revolutionized novel view synthesis through its explicit volumetric representation and highly efficient software rasterization, sparking a surge of interest in extracting continuous triangular surface meshes directly from optimized Gaussians. Traditional pipelines optimize Gaussian primitives under photometric reconstruction losses combined with geometric priors, subsequently executing post-hoc surface extraction techniques like Marching Tetrahedra. However, a fundamental bottleneck of 3DGS is the intrinsic coupling of geometry and appearance: each primitive possesses spatial covariance and spherical harmonics (SH) color simultaneously, which inevitably creates ambiguity during gradient backpropagation.

In complex real-world scenes featuring specular highlights, sharp view-dependent reflections, or volatile illumination, low-degree spherical harmonics inherently lack the capacity to express high-frequency radiance variations. Consequently, gradient-based optimization frequently finds an undesirable shortcut: it distorts underlying geometry to minimize photometric loss. Specifically, it populates semi-transparent surfaces with high-opacity Gaussians tucked immediately behind them that appear only under certain viewing angles, thereby destroying surface smoothness and triggering destructive over-densification. Existing solutions attempt to mitigate these artifacts by enforcing expensive multi-view stereo constraints, performing continuous bidirectional mesh-Gaussian consistency checks during training, or relying on large pre-trained monocular normal/depth foundation models. Unfortunately, these external scaffolds severely compromise the native computational simplicity and real-time efficiency of 3DGS.

This dilemma poses an essential question: can we achieve high-fidelity surface extraction purely within an efficient, self-supervised 3DGS framework without external network priors or multi-view geometric overhead? Drawing inspiration from Bayesian networks and feed-forward reconstruction architectures, this work introduces a self-supervised uncertainty mechanism directly into explicit radiance fields. Core idea: equip each Gaussian with learnable self-supervised confidence values to dynamically balance photometric error against geometric supervision, penalize ray-blended color and normal variances to anchor primitives to physical surfaces, and decouple the D-SSIM luminance term for robust appearance modeling.

Method

Overall Architecture

The proposed method (termed CoMe) builds upon explicit 3D Gaussian Splatting and tetrahedral surface extraction (Marching Tetrahedra). Taking multi-view posed images as input, the pipeline produces high-precision unbounded surface meshes. Throughout training, three core mechanisms operate collaboratively: first, for each view, a scalar confidence map is rasterized along with color and normal buffers, driving a self-supervised confidence loss that dynamically scales photometric gradients and recalibrates the densification split threshold; second, ray-marching accumulation is regularized via per-primitive color and normal variance losses that penalize inconsistent primitives along view rays; third, image-sensor-processing (ISP) variations are resolved using an improved appearance embedding that decouples D-SSIM to supervise luminance modifications independently of contrast and structure.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view Posed Images"] --> B["Decoupled Appearance Modeling<br/>Applies corrections only to L1 and D-SSIM luminance"]
    B --> C["Forward Splatting & Confidence Accumulation<br/>Rasterize color, normal, and scalar confidence maps"]
    C --> D["Self-Supervised Confidence Framework & Densification<br/>Dynamically balances photometric vs. geometric losses"]
    C --> E["Blending Variance Penalties<br/>Minimizes color and normal variance along rays"]
    D --> F["Joint Radiance & Geometry Field Optimization<br/>Incorporates SOF geometric losses"]
    E --> F
    F --> G["Marching Tetrahedra Surface Extraction<br/>Binary search extraction of unbounded meshes"]

Key Designs

1. Self-Supervised Confidence Framework and Steered Densification: Dynamically balancing ambiguous gradients

In regions with specular highlights, fine foliage, or sparse camera coverage, massive photometric reprojection errors generate disproportionate positional gradients, causing standard 3DGS to clone and split primitives uncontrollably into pathological over-densification. To resolve this, each Gaussian is augmented with a learnable scalar parameter \(\gamma_i\) (initialized to 0), transformed via exponential activation into an initial confidence value of \(\tilde{\gamma}_i = \exp(\gamma_i) = 1\). During forward rendering along ray \(r\), per-primitive confidence values are accumulated via alpha-blending identically to color, yielding the rendered confidence map \(\hat{C}(r) = \sum_{i=0}^{N-1} w_i(r) \tilde{\gamma}_i\). The photometric reconstruction is then governed by a self-supervised confidence loss:

\[\mathcal{L}_{\text{conf}} = \mathcal{L}_{\text{rgb}} \cdot \hat{C} - \beta \cdot \log \hat{C}\]

where \(\beta > 0\) (defaulting to 0.075) controls the penalty trade-off. Since \(\partial \mathcal{L}_{\text{conf}} / \partial \mathcal{L}_{\text{rgb}} = \hat{C}\), photometric error updates are dynamically scaled down in ambiguous or unresolvable regions where the model predicts low confidence (\(\hat{C} < 1\)). Crucially, geometric regularization terms (such as depth-normal consistency) remain unscaled by \(\hat{C}\), allowing geometry to assert dominance precisely where visual appearance is uncertain. Furthermore, standard gradient-based densification splits Gaussians whose positional gradient exceeds a threshold \(\tau_{\text{grad}}\). To prevent over-densification in low-confidence zones, the split threshold is dynamically adapted:

\[\bar{\tau}_{\text{grad}} = \frac{\tau_{\text{grad}}}{\min(\tilde{\gamma}_i, 1)}\]

By clamping the denominator to a maximum of 1, unconfident Gaussians face a substantially heightened split threshold, effectively suppressing recursive over-cloning and yielding a cleaner, more compact set of primitives.

2. Variance Reducing Losses for Blending: Preventing geometry from faking view-dependent radiance

Even when gradient magnitudes are stabilized, standard alpha-blending introduces an inherent loophole: the optimization evaluates only the aggregated final ray color, leaving individual primitives along the ray unconstrained. Consequently, optimization often places dense, opaque Gaussians behind transparent surfaces, faking high-frequency specular effects via viewpoint-dependent occlusion. To penalize this deceptive behavior, the paper introduces a weighted per-primitive color variance loss against the ground truth pixel color \(I_{\text{gt}}\):

\[\mathcal{L}_{\text{color-var}} = \sum_{i=0}^{N-1} w_i(r) \left\| \text{sh}(\boldsymbol{\theta}_i, d) - I_{\text{gt}} \right\|_2^2\]

Under the condition that the rendered color approximates \(I_{\text{gt}}\), minimizing this term is equivalent to minimizing the variance of primitive colors contributing to the ray, thereby preventing Gaussians along the same line of sight from adopting conflicting appearances. In a parallel manner, rendered pixel normals produced by alpha-blending individual Gaussian normals \(\mathbf{n}_i\) often hide severe local orientation scattering. The paper introduces an elegant normal variance loss:

\[\mathcal{L}_{\text{normal-var}} = \sum_{i=0}^{N-1} w_i(r) \left\| \mathbf{n}_i - N \right\|_2^2 = 1 - \|N\|_2^2\]

where \(N = \sum_i w_i(r) \mathbf{n}_i\) is the blended pixel normal. Utilizing the property that unit normals satisfy \(\|\mathbf{n}_i\|_2 = 1\), the loss reduces to \(1 - \|N\|_2^2\) with zero runtime overhead. When primitives along a ray are consistently aligned to the true physical surface, \(\|N\|_2 \approx 1\) and the penalty vanishes; whenever normal vectors scatter, the magnitude drops, enforcing surface smoothness and spatial coherence.

3. Decoupled Appearance Modeling: Isolating D-SSIM luminance to protect geometric gradients

Real-world captures frequently suffer from shifting exposures and varying illumination caused by camera auto-exposure and image sensor processing (ISP). While previous methods adopt VastGaussian-style CNN modules to predict an appearance-compensated image \(\hat{I}_{\text{app}} = \hat{I} \odot \sigma(\mathbf{M}_i)\), they restrict \(\hat{I}_{\text{app}}\) strictly to the L1 loss and evaluate D-SSIM using the raw render \(\hat{I}\), assuming SSIM is purely structural. However, analyzing the three-factor decomposition of SSIM, \(l(I, \hat{I}) \cdot c(I, \hat{I}) \cdot s(I, \hat{I})\), reveals that the luminance factor \(l(\cdot)\) is acutely sensitive to illumination changes and can be up to an order of magnitude larger than contrast or structure errors. Feeding uncompensated images into D-SSIM forces the 3D Gaussians to absorb exposure fluctuations into spatial geometry. Conversely, using \(\hat{I}_{\text{app}}\) across all terms introduces CNN upsampling blur that corrupts high-frequency geometric gradients. The authors resolve this with a decoupled D-SSIM formulation:

\[\mathcal{L}_{\text{D-SSIM}}^{\text{dec}} = l(I_{\text{gt}}, \hat{I}_{\text{app}}) \cdot c(I_{\text{gt}}, \hat{I}) \cdot s(I_{\text{gt}}, \hat{I})\]

Only the luminance term receives the appearance-corrected image \(\hat{I}_{\text{app}}\), while contrast \(c(\cdot)\) and structure \(s(\cdot)\) remain directly supervised by the raw Gaussian render \(\hat{I}\). This eliminates geometric distortion induced by ambient lighting shifts while preserving sharp, uncorrupted structural supervision.

Loss & Training

The overall training objective combines the confidence-weighted photometric loss, geometric priors, and both blending variance regularizers:

\[\mathcal{L} = \mathcal{L}_{\text{conf}} + \mathcal{L}_{\text{geom}} + \lambda_{\text{color-var}} \mathcal{L}_{\text{color-var}} + \lambda_{\text{normal-var}} \mathcal{L}_{\text{normal-var}}\]

where \(\mathcal{L}_{\text{geom}}\) includes depth continuity and depth-normal consistency from SOF. Loss weights are fixed across all benchmarks to \(\lambda_{\text{color-var}} = 5 \times 10^{-1}\) and \(\lambda_{\text{normal-var}} = 5 \times 10^{-3}\). The confidence parameters \(\gamma_i\) are trained with a learning rate of \(2.5 \times 10^{-4}\), and the trade-off hyperparameter is set to \(\beta = 0.075\) (with \(\beta = 0.05\) as a lightweight variant). \(\mathcal{L}_{\text{conf}}\) activates at iteration 500 alongside densification. Unbounded meshes are extracted post-hoc using the accelerated Marching Tetrahedra binary search implementation from SOF.

Key Experimental Results

Main Results

The method is evaluated on the standard unbounded benchmark Tanks & Temples as well as the dense real-world indoor benchmark ScanNet++, evaluating geometry quality using the F1-score (higher is better) and reporting training runtimes measured on a single NVIDIA RTX 4090 GPU.

Dataset / Scene	PGSR (Bounded)	QGS (Bounded)	GOF (Unbounded)	SOF (Unbounded)	RaDe-GS (Unbounded)	MILo (Unbounded)	Ours (Full)	Ours (\(\beta=0.05\))
Tanks & Temples (Avg F1)	0.496	0.474	0.453	0.474	0.461	0.485	0.521	0.505
- Barn	0.548	0.536	0.484	0.535	0.529	0.541	0.534	0.519
- Caterpillar	0.437	0.374	0.402	0.408	0.464	0.389	0.472	0.451
- Courthouse	0.238	0.183	0.288	0.297	0.316	0.322	0.333	0.315
- Ignatius	0.728	0.733	0.674	0.736	0.536	0.757	0.782	0.752
- Meetingroom	0.367	0.374	0.275	0.309	0.343	0.281	0.372	0.358
- Truck	0.658	0.645	0.596	0.558	0.576	0.617	0.634	0.637
Optimization Runtime	28 min	41 min	40 min	17 min	8 min	60 min	18 min	16 min
ScanNet++ (Avg F1)	0.631	0.573	0.623	0.615	0.613	0.624	0.668	0.667

Ablation Study

Stepwise ablation on Tanks & Temples and ScanNet++ demonstrating the cumulative gain of each proposed contribution:

Config	Tanks & Temples F1	ScanNet++ F1	Note
SOF [49] (Baseline)	0.474	0.615	Base 3DGS unbounded mesh extraction pipeline
+ Improved Appearance	0.493	0.625	Resolves illumination drift (+0.019 / +0.010)
+ Self-Supervised Confidence (\(\mathcal{L}_{\text{conf}}\))	0.509	0.655	Balances gradients and prevents over-densification (+0.030 on indoor)
+ Color Variance (\(\mathcal{L}_{\text{color-var}}\))	0.519	0.658	Constrains view-dependent multi-primitive stacking (+0.010 on outdoor)
+ Normal Variance (\(\mathcal{L}_{\text{normal-var}}\)) (Full)	0.521	0.668	Regulates planar structures and normal consistency (+0.010 on indoor)

Comparison of appearance modeling strategies on Tanks & Temples: - Standard VastGaussian: F1 = 0.475 - PGSR Appearance Modeling: F1 = 0.478 - Hierarchical 3DGS (H3DGS): F1 = 0.484 - VastGaussian Improved (All terms use \(\hat{I}_{\text{app}}\)): F1 = 0.490 - Ours (D-SSIM Luminance Decoupled): F1 = 0.493

Key Findings

• Self-supervised confidence drives massive indoor gains: Adding \(\mathcal{L}_{\text{conf}}\) increases the ScanNet++ F1-score from 0.625 to 0.655. In untextured or dimly lit indoor scenes, unconfident regions are prevented from driving excessive gradient updates, enabling geometric regularizers to establish coherent planar structures.
• Variance losses address distinct scene modalities: Color variance loss provides substantial improvements in outdoor scenes with specular reflections (Tanks & Temples F1 improves by 0.010), while normal variance loss yields larger improvements on planar indoor geometry (ScanNet++ F1 improves by 0.010).
• Hyperparameter \(\beta\) provides intuitive uncertainty regulation: When \(\beta=0\), the model lacks incentives to increase confidence, leading to severe under-reconstruction. When \(\beta=0.2\), excessive penalty triggers explosive primitive counts. The choice \(\beta=0.075\) delivers optimal performance across diverse datasets with fewer total primitives than the baseline without confidence, while \(\beta=0.05\) functions as a memory-efficient compact configuration.

Highlights & Insights

• Repurposing learned confidence as a densification damper: Rather than using confidence merely as a passive visualization tool or loss weight, this work integrates per-primitive confidence directly into the gradient split threshold denominator \(\bar{\tau}_{\text{grad}}\). This ensures that uncertain or occluded regions are prevented from runaway splitting, tackling the root cause of 3DGS over-densification.
• Surgical decomposition of SSIM for appearance robustness: Recognizing that the luminance term \(l(\cdot)\) is predominantly non-structural and corrupted by camera exposure differences, decoupling it allows the network to compensate for lighting variations while feeding uncorrupted, sharp gradients into contrast and structure terms.
• Closed-form normal variance via unit vector geometry: By leveraging the mathematical property that individual normal vectors have unit length, the blended normal variance simplifies analytically to \(1 - \|N\|_2^2\), providing an elegant, zero-overhead constraint that enforces clean surface orientations.

Limitations & Future Work

• Moderate capture density requirement: The method assumes reasonably dense multi-view coverage. In severely sparse-view capture scenarios, confidence attenuation suppresses over-densification but cannot hallucinate missing surfaces, occasionally leaving holes in unobserved areas. Integrating lightweight monocular priors in low-confidence zones remains a promising avenue.
• Background geometry under-densification: Far-field background structures suffer from insufficient capture resolution and low sampling density, leading to sparser reconstruction. Multi-view diffusion priors could provide complementary detail for distant regions.
• Broader 3DGS applications: The self-supervised confidence loss demonstrates potential as a general, heuristic-free primitive pruning and densification controller for broader downstream tasks such as 3D inpainting and super-resolution.

Related Work & Insights

• vs SOF / GOF: While SOF and GOF established fast tetrahedral extraction for 3DGS, their geometry frequently degrades under severe reflections and lighting shifts. CoMe acts as a direct upgrade to SOF, maintaining its 18-minute runtime while completely removing surface artifacts and boosting average F1 by nearly 0.05.
• vs MILo: MILo enforces bidirectional Gaussian-to-mesh consistency during training, which inflates optimization runtime to 60 minutes per scene and tends to over-smooth fine surface detail. CoMe avoids online mesh iteration entirely, running 3.3× faster with crisper geometric boundaries.
• vs PGSR / QGS: Bounded multi-view baselines rely heavily on epipolar geometry and multi-view stereo constraints that fail on unbounded backgrounds and over-smooth thin structures. CoMe operates without multi-view matching overhead, delivering superior completeness and fidelity on unbounded scenes.

Rating

• Novelty: ⭐⭐⭐⭐☆ Introduces primitive-level self-supervised confidence to 3DGS mesh extraction, coupled with D-SSIM luminance decoupling and elegant blending variance losses.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks across Tanks & Temples, ScanNet++, and Mip-NeRF 360 against leading bounded and unbounded methods with clean component ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear theoretical rationale, rigorous analysis of photometric-geometric trade-offs, and insightful mathematical simplifications.
• Value: ⭐⭐⭐⭐⭐ Establishes a new SOTA for unbounded mesh reconstruction while strictly preserving the high efficiency of 3DGS without external model dependencies.

Related Papers

• [ECCV 2026] Sparse-View Surface Reconstruction using Gaussian Splatting through High-Confidence Depth Propagation with Normal Priors
• [CVPR 2026] Confidence-Guided Multi-Scale Aggregation for Sparse-View High-Resolution 3D Gaussian Splatting
• [CVPR 2026] BEA-GS: BEyond RAdiance Supervision in 3DGS for Precise Object Extraction
• [ECCV 2026] Gaussians on Fire: High-Frequency Reconstruction of Flames
• [ECCV 2026] SplatPainter: Interactive Authoring of 3D Gaussians from 2D Edits via Test-Time Training