Do Flat Minima Improve Sparse Novel View Synthesis?¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://bbangsik13.github.io/FASR
Area: 3D Vision
Keywords: Sparse Novel View Synthesis, 3D Gaussian Splatting, Flat Minima, Loss Sharpness, Structure-Aware Regularization

TL;DR¶

This paper fundamentally investigates the relationship between loss sharpness and generalization in sparse novel view synthesis, revealing that pursuit of uniform flat minima causes over-smoothing in fine details, and introduces Structure-Aware Sharpness Regularization (FASR) to adaptively balance detail preservation and floater suppression based on local image structure.

Background & Motivation¶

Novel view synthesis methods based on 3D Gaussian Splatting (3DGS) and Neural Radiance Fields (NeRFs) have demonstrated remarkable photorealistic rendering under dense viewpoint supervision. However, in sparse-view regimes, these scene representations severely overfit the few available training views. In novel, unobserved viewpoints, this overfitting manifests as distorted geometry, blurred textures, and persistent detached artifacts commonly referred to as floaters. Existing remedies predominantly rely on injecting external priors—such as monocular depth predictors, correspondence matchers, or heuristic ensemble regularizations. Despite their empirical success, these approaches overlook the foundational optimization dynamics and loss landscape characteristics of the underlying 3D representation.

In machine learning theory, seeking flatter minima is widely acknowledged to improve generalization bounds, as epitomized by Sharpness-Aware Minimization (SAM). Nevertheless, directly transferring flat-minima optimization to sparse novel view synthesis exhibits an unexpected failure mode: minimizing global loss sharpness does not monotonically translate into superior novel view generalization. The root cause lies in the spatial heterogeneity of geometric sensitivity during 3D reconstruction. Parameters characterizing high-detail regions (e.g., sharp silhouettes and high-frequency textures) inherently require a sharp loss landscape; even minute primitive displacements induce drastic pixel rendering deviations necessary to resolve exact spatial boundaries. In contrast, low-detail smooth regions and erroneous spatial floaters are over-sensitive to small viewpoint variations, creating sharp, spurious local minima that demand flattening to ensure cross-view consistency. A uniform flatness constraint invariably penalizes high-detail regions excessively while failing to regularize under-constrained flat regions adequately.

To resolve this dilemma, this work proposes Structure-Aware Sharpness Regularization (FASR), which optimizes 3D Gaussians without needing bulky pre-trained foundation models. The core idea is to compute a 2D geometric tolerance map from local image structures and adaptively scale both the worst-case perturbation radius and the regularization weight—restraining perturbations in high-detail areas to preserve indispensable loss sharpness, while expanding perturbations in low-detail zones to penalize floaters and encourage flatter minima for generalization.

Method¶

Overall Architecture¶

The proposed FASR optimization framework operates as a plug-and-play optimizer on the geometric attributes of 3D Gaussians (mean positions \(\mu_i\), rotation quaternions \(q_i\), and scaling vectors \(s_i\)). The pipeline begins by analyzing training view gradients to construct a 2D Geometric Tolerance Map, measuring the Euclidean distance from every pixel to its nearest high-gradient structural boundary. During each training iteration, every Gaussian primitive queries its local tolerance at its projected 2D center and converts this metric into a 3D perturbation radius via perspective geometry. A Structure-Aware Perturbation step then computes worst-case geometric shifts \(\hat{\mathcal{G}}\) along local ascending gradients. Finally, a Structure-Aware Regularization Weighting scheme balances the empirical training loss against the worst-case perturbed loss, enforcing flatness in smooth regions while retaining sharp minima around intricate structures.

The end-to-end optimization workflow is illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Sparse Multi-View Input Images"] --> B["Geometric Tolerance Map Construction<br/>Gradient Detection and Distance Transform"]
    B --> C["Gaussian Projection and 3D Tolerance Mapping<br/>Pixel Distance Scaled via Depth and Focal Length"]
    C --> D["Structure-Aware Sharpness Perturbation<br/>Adaptive Worst-Case Neighborhood Ascent"]
    D --> E["Worst-Case Loss and Dynamic Weighting<br/>Detail Sharpness Retention and Floater Flattening"]
    E --> F["Gaussian Parameter Gradient Update<br/>Mitigate Floaters while Preserving Crisp Edges"]

Key Designs¶

1. Geometric Tolerance Map Construction: Quantifying Rendering Sensitivity to Structural Perturbations

To quantify how local image structures respond to geometric variations, the method establishes a 2D geometric tolerance map \(\Gamma^v(p)\) for each training view \(v\). Let \(H^v\) denote the set of high-detail pixels identified by prominent spatial gradients. For any pixel coordinate \(p\), the geometric tolerance represents its shortest Euclidean distance to the nearest high-detail boundary:

\[\Gamma^v(p) = \min_{p' \in H^v} \|p - p'\|_2\]

Consequently, pixels in vicinity of intricate edges receive small tolerance values approaching zero, indicating that even infinitesimal coordinate shifts induce substantial rendering deviations. Conversely, pixels situated within expansive low-detail regions yield large tolerance values, where moderate primitive displacements remain visually imperceptible. This distance serves as a direct indicator of geometric perturbation tolerance.

2. Structure-Aware Sharpness: Perspective-Calibrated Adaptive Perturbation Neighborhoods

Standard SAM enforces a uniform perturbation radius \(\rho\) across all network parameters, which overshoots local maxima in sensitive edge regions while under-perturbing smooth areas. The method decouples perturbation estimation to each individual Gaussian's geometric attributes \(\theta_i \in \{\mu_i, q_i, s_i\}\). Projecting the 3D mean \(\mu_i\) onto the image plane yields projected coordinates \(\mu'_i\), from which the local tolerance \(\gamma_i = \Gamma^v(\mu'_i)\) is extracted. By applying perspective projection principles, a 2D image shift \(\gamma_i\) corresponds to a 3D physical displacement scaled by \(d_i / f\), where \(d_i\) represents Gaussian depth along the optical axis and \(f\) is camera focal length. The structure-aware worst-case perturbation is formulated as:

\[\hat{\theta}_i = \theta_i + \gamma_i \frac{d_i}{f} \rho_\theta \frac{\nabla_{\theta_i} \mathcal{L}}{\|\nabla_{\theta_i} \mathcal{L}\|_2}\]

This perspective-aware scaling guarantees restrained, stable perturbation around fragile fine-grained boundaries while promoting exploratory ascent across low-frequency backgrounds and detached floaters.

3. Structure-Aware Regularization Weighting: Spatially Modulated Sharpness Penalization

Once the perturbed scene configuration \(\hat{\mathcal{G}}\) and its corresponding worst-case loss \(\hat{\mathcal{L}}\) are evaluated, the optimization objective must balance empirical fitting against sharpness minimization. Imposing uniform flatness penalization inevitably degrades the sharpness essential for crisp detail reconstruction. The framework introduces a primitive-level dynamic weighting factor \(\bar{\gamma}_i = 0.95 \cdot \gamma_i / \gamma_{\max}\), where \(\gamma_{\max}\) denotes the maximum image tolerance and 0.95 is an empirical ceiling. The composite loss is defined as:

\[\mathcal{L}_{\text{total}} = \mathcal{L} + \frac{\bar{\gamma}_i}{1 - \bar{\gamma}_i} (\hat{\mathcal{L}} - \mathcal{L}) = \frac{1 - 2\bar{\gamma}_i}{1 - \bar{\gamma}_i} \mathcal{L} + \frac{\bar{\gamma}_i}{1 - \bar{\gamma}_i} \hat{\mathcal{L}}\]

For Gaussians delineating sharp object contours, \(\bar{\gamma}_i \to 0\), causing the sharpness penalty to vanish so the optimization focuses purely on empirical reconstruction accuracy. For Gaussians situated in uniform backgrounds or spurious floaters, \(\bar{\gamma}_i \to 0.95\), giving prominence to the flat-minima penalty and suppressing sharp, non-generalizable local minima.

Loss & Training¶

The training algorithm executes two forward passes per iteration: first evaluating the empirical photometric loss \(\mathcal{L}\) and computing parameter gradients, then displacing Gaussian attributes within their structure-adaptive radii to compute the worst-case loss \(\hat{\mathcal{L}}\). The composite gradient is subsequently propagated back to update the base parameters \(\mathcal{G}\) using standard Adam optimization. This procedure requires no auxiliary neural networks or pre-trained geometric priors.

Key Experimental Results¶

Main Results¶

Extensive evaluations were conducted on standard sparse-view benchmarks: LLFF under an ultra-sparse 3-view regime and MipNeRF-360 under a 12-view split (averaged across multiple runs):

Dataset / Baseline Method	Configuration	PSNR ↑	SSIM ↑	LPIPS ↓
LLFF (3 views)	3DGS [ACM TOG'23]	19.810 ± .339	.6790 ± .0078	.2145 ± .0065
	3DGS + Ours	20.783 ± .300	.7197 ± .0032	.1965 ± .0034
	CoR-GS [ECCV'24]	20.185 ± .142	.7015 ± .0040	.2029 ± .0035
	CoR-GS + Ours	20.862 ± .154	.7283 ± .0042	.1932 ± .0030
	DropGaussian [CVPR'25]	20.461 ± .212	.7070 ± .0045	.2064 ± .0047
	DropGaussian + Ours	20.853 ± .227	.7295 ± .0045	.1969 ± .0040
	NexusGS [CVPR'25]	21.048 ± .049	.7382 ± .0008	.1776 ± .0009
	NexusGS + Ours	21.348 ± .078	.7511 ± .0012	.1714 ± .0011
	SE-GS [ICCV'25]	20.725 ± .217	.7203 ± .0049	.1861 ± .0058
	SE-GS + Ours	21.141 ± .223	.7403 ± .0042	.1803 ± .0038
MipNeRF-360 (12 views)	3DGS [ACM TOG'23]	18.903 ± .179	.5499 ± .0036	.3734 ± .0042
	3DGS + Ours	19.303 ± .185	.5622 ± .0051	.3552 ± .0047
	CoR-GS [ECCV'24]	19.515 ± .243	.5733 ± .0049	.3741 ± .0066
	CoR-GS + Ours	19.805 ± .233	.5833 ± .0058	.3681 ± .0069
	DropGaussian [CVPR'25]	19.514 ± .199	.5722 ± .0042	.3657 ± .0036
	DropGaussian + Ours	19.625 ± .254	.5750 ± .0053	.3627 ± .0051
	NexusGS [CVPR'25]	18.506 ± .098	.5222 ± .0031	.3587 ± .0021
	NexusGS + Ours	18.736 ± .103	.5316 ± .0034	.3522 ± .0024
	SE-GS [ICCV'25]	19.931 ± .288	.5930 ± .0063	.3702 ± .0054
	SE-GS + Ours	20.135 ± .232	.5960 ± .0059	.3644 ± .0052

Ablation Study¶

Component-wise ablation on the LLFF dataset confirms the individual and joint efficacy of Structure-Aware Sharpness (SAS) and Structure-Aware Regularization Weighting (SAR):

SAM Baseline	SAS (Adaptive Perturbation)	SAR (Adaptive Weighting)	PSNR ↑	SSIM ↑	LPIPS ↓	Note
✗	✗	✗	19.810 ± .339	.6790 ± .0078	.2145 ± .0065	Vanilla 3DGS baseline with severe floaters and overfitting
✓	✗	✗	20.198 ± .218	.6958 ± .0034	.2095 ± .0036	Naive SAM: uniform flattening causes noticeable blurring
✓	✗	✓	20.570 ± .161	.6980 ± .0037	.2142 ± .0042	SAR only: partially protects edge sharpness
✓	✓	✗	20.560 ± .259	.7116 ± .0038	.2023 ± .0032	SAS only: improves local maximum estimation fidelity
✓	✓	✓	20.783 ± .300	.7197 ± .0032	.1965 ± .0034	Full Model (Ours Full): achieves optimal clarity and metrics

Key Findings¶

Flatness and generalization are non-monotonically correlated: Visualizing the loss landscape on the synthetic Lego scene demonstrates that while naive SAM achieves the lowest Hessian maximal eigenvalue (\(\lambda_{\max} = 1.88 \times 10^{-3}\), vs \(2.81 \times 10^{-3}\) for 3DGS), its novel view test error (.03269) is higher than that of our method (.03078 with \(\lambda_{\max} = 2.77 \times 10^{-3}\)). This proves that enforcing excessive flatness across all parameters sacrifices necessary high-frequency detail.
Sparse under-constrained regions benefit most: Stratifying test errors by camera covisibility levels reveals that our performance advantage is pronounced in regions seen by fewer views. In regions with covisibility of 1 view, the error drop reaches \(-0.0142 \pm .0074\), compared to \(-0.0066 \pm .0043\) in regions covered by 3 views, verifying that flatness regularization effectively remedies under-constrained degrees of freedom.
Universal compatibility across representations and optimizers: Applying the principle to Random Weight Perturbation (RWP) yields solid gains on LLFF (PSNR increased from 20.079 to 20.650). In dynamic streaming 3D Gaussians (Yun et al.), it reduces temporal jittering (mTV decreased from .1109 to .0989). Furthermore, applying the strategy to FreeNeRF in implicit volume rendering raises PSNR from 19.523 to 19.584.

Highlights & Insights¶

Rethinking the "flatter is better" paradigm in 3D vision: The work uncovers the spatial heterogeneity of 3D radiance field optimization—demonstrating that high-detail regions require sharp minima for sub-pixel boundary positioning, whereas low-detail background areas benefit from flat minima to eliminate under-constrained floater artifacts.
Perspective-consistent 2D-to-3D perturbation translation: Without introducing auxiliary neural estimators, the framework leverages geometric tolerance scaled by \(d_i / f\) to establish an exact, physically meaningful perturbation magnitude in camera space.
Orthogonal plug-and-play optimizer: The approach functions entirely as an optimization regularizer without changing 3D scene architectures or relying on heavyweight pre-trained foundation models, making it compatible with current and future sparse reconstruction frameworks.

Limitations & Future Work¶

Increased training computational budget: Computing worst-case parameters requires an extra back-propagation step and forward render per iteration, roughly doubling per-iteration training time compared to standard 3DGS.
Sensitivity to input gradient quality: The geometric tolerance map relies on accurate edge extraction. In situations with severe motion blur, low light, or sensor noise, tolerance estimates may degrade.
Future Directions: Developing fast gradient approximations or leveraging historical momentum to estimate Hessian curvature without double forward-backward passes would substantially accelerate training; extending structure-aware flatness to feed-forward generalizable reconstruction models also represents a promising avenue.

vs SAM (Sharpness-Aware Minimization): SAM was formulated for general neural network classification with uniform parameter perturbations, producing blurred renderings when applied naively to 3DGS. FASR introduces structure-aware sharpness (SAS) and regularization weighting (SAR) to adaptively balance detail preservation and floater elimination.
vs Sparfels: Sparfels targets worst-case robustness under unposed sparse views by approximating the objective with an upper bound on color variance along rays with frozen Gaussian means. In contrast, FASR directly optimizes the worst-case loss with respect to all 3D Gaussian geometric parameters.
vs Prior-based Sparse 3DGS (NexusGS, CoR-GS): Most state-of-the-art sparse 3DGS methods incorporate external foundation model priors (such as depth or dense correspondence). FASR operates purely on the loss landscape, providing orthogonal, complementary performance boosts when combined with these prior-guided methods.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Pioneering exploration of loss landscape sharpness in sparse novel view synthesis with an elegant structure-aware formulation]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive evaluations spanning static and dynamic scenes, explicit 3DGS and implicit NeRF, diverse optimizers, and extensive landscape visualizations]
Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary clarity, rigorous mathematical formulations, and compelling ablation analyses]
Value: ⭐⭐⭐⭐⭐ [Offers a fresh optimization perspective for 3D reconstruction with broad applicability across the field]