Improving Sparse-View 3DGS Generalization via Flat Minima Optimization

Conference: ECCV 2026
arXiv: 2607.00885
Code: None
Area: 3D Vision
Keywords: 3D Gaussian Splatting, sparse-view novel view synthesis, flat minima optimization, scale-adaptive perturbation, parameter reinitialization

TL;DR

This work addresses overfitting in sparse-view 3DGS from the perspective of flat minima optimization: through scale-adaptive position perturbation (SAP) + random application + magnitude scheduling + periodic reinitialization of non-position parameters, it drives 3DGS to converge to flatter regions of the loss landscape without changing the network architecture, achieving SOTA or near-SOTA rendering quality under sparse-view settings on LLFF and Mip-NeRF360.

Background & Motivation

3D Gaussian Splatting (3DGS) has become a mainstream method for novel view synthesis, balancing training speed and real-time rendering quality. However, when input views are extremely sparse (e.g., only 3 views), 3DGS severely overfits to the training views: the optimized Gaussian parameters become highly sensitive to tiny positional shifts, and rendering quality drops sharply once the viewpoint deviates from the training poses.

This phenomenon is highly consistent with the classic observation in neural network training that sharp minima lead to poor generalization. In deep learning, flat minima optimization methods, such as SAM and RWP, inject parameter perturbations to force the model to converge to flat regions of the loss landscape and have been shown to significantly improve generalization. The core insight of this work is: the 3DGS training process can be reinterpreted as a supervised learning system—camera poses are inputs, rendered images are outputs, and Gaussian parameters are learnable weights. Under sparse views, overfitting is therefore essentially convergence to sharp minima that are highly sensitive to positional perturbations, exactly the problem flat minima optimization is designed to solve.

However, directly transferring flat minima optimization to 3DGS would ignore its unique geometric structure: Gaussians are anisotropic 3D primitives, and different Gaussians vary greatly in size and shape. A uniform isotropic perturbation is either too strong for small Gaussians, damaging details, or too weak for large Gaussians, providing insufficient regularization. Therefore, this work proposes a flat minima optimization framework tailored to the geometric properties of 3DGS. Core Idea: generate anisotropic perturbations according to each Gaussian's own spatial scale and shape, together with random application and magnitude scheduling, and periodically reset non-position parameters to stabilize training, so that 3DGS converges under sparse views to minima that are flat while preserving details.

Method

Overall Architecture

The full method overlays two mechanisms, perturbation and reinitialization, on the standard 3DGS training pipeline, without introducing additional network modules or external priors. The inputs are an SfM-initialized Gaussian point cloud and sparse training views. In each iteration, position noise is first sampled for each Gaussian from a scale-adaptive covariance and randomly applied with probability p rather than perturbing all Gaussians in every iteration; the perturbation magnitude then grows linearly from 0 to the target value over training. Forward rendering is then performed on the perturbed parameters, the standard L1+SSIM loss is computed, and the gradients from this loss update the original unperturbed parameters. At fixed iteration intervals, the scale, rotation, and higher-order SH coefficients are temporarily restored to their SfM initial values and frozen for a short window. Opacity is reset according to the standard 3DGS mechanism, while positions are unaffected.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["SfM initial point cloud"] --> B["Scale-adaptive perturbation<br/>Sample position noise according to Gaussian covariance"]
    B --> C["Random application<br/>Each Gaussian is perturbed with probability p"]
    C --> D["Magnitude scheduling<br/>Perturbation strength grows linearly with t/T"]
    D --> E["3DGS rendering + compute L1+SSIM loss"]
    E --> F["Gradient update of original parameters<br/>(perturbation only used for loss evaluation)"]
    F --> G["Periodic reinitialization<br/>Restore scale/rotation/SH coefficients to initial values"]
    G -.->|Triggered every 1000 iterations| B
    F --> H["Output: Gaussian field at a flat minimum"]

Key Designs

1. Scale-Adaptive Perturbation (SAP): making noise respect each Gaussian's shape

Under sparse views, the essence of 3DGS overfitting is that Gaussian positions are extremely sensitive to tiny shifts, exactly a symptom of sharp minima. Intuitively, position perturbations should be injected to force optimization to find flat regions that are insensitive to positional changes. The difficulty is that Gaussian sizes in 3DGS vary dramatically: a Gaussian covering a large wall and a small Gaussian describing an object boundary react very differently to noise of the same magnitude.

SAP solves this by making the perturbation magnitude proportional to the spatial covariance of the Gaussian. Specifically, for the i-th Gaussian, given its 3D covariance matrix \(\mathbf{\Sigma}_i = \mathbf{R}_i \mathbf{S}_i \mathbf{S}_i^\top \mathbf{R}_i^\top\), displacement noise \(\delta_i \sim \mathcal{N}(0, \gamma^2 \mathbf{\Sigma}_i)\) is sampled from it, where \(\mathbf{S}_i = \text{diag}(s_i)\) denotes the three-axis scale of this Gaussian. In this way, noise is largest along the Gaussian's most extended axis and smallest along its shortest axis. Large Gaussians receive strong perturbations to improve robustness, while small Gaussians receive weak perturbations to preserve details. The perturbed position is \(\mathbf{x}_i' = \mathbf{x}_i + \delta_i\), and other parameters, including scale, rotation, color, and opacity, remain unchanged.

Unlike 3DGS-MCMC, which adds noise during parameter updates so that noise accumulates into the trajectory, this work perturbs only during loss evaluation; the parameters themselves remain clean, effectively smoothing the local loss landscape around the original parameters. This is more precise than directly adding isotropic noise to all parameters. In the ablation study, isotropic variants scaled by the maximum axis, average axis, or a fixed value all achieve lower PSNR than the anisotropic version (20.88 vs 20.54~20.73), verifying the necessity of respecting geometric shape.

2. Random application: perturb with probability p rather than perturbing all Gaussians to avoid over-smoothing

RWP-style methods commonly render both perturbed and unperturbed models in each iteration and mix their losses, but this doubles rendering cost in 3DGS. More importantly, the authors observe that if all Gaussians are perturbed in every iteration, the model tends to over-smooth high-frequency details, because the perturbation signal always dominates the gradient direction and fine geometric structures are washed out.

This work implements a similar effect in a lighter and more effective way: in each iteration, SAP perturbation is independently applied to each Gaussian with probability p (default 0.3), while the remaining Gaussians keep their original positions. Formally, \(\mathbf{x}_i' = \mathbf{x}_i + \delta_i\) with probability p, and \(\mathbf{x}_i' = \mathbf{x}_i\) otherwise. This brings three benefits: (1) no double rendering is needed, so training overhead is almost unchanged; (2) the sparsity of perturbation prevents the model from over-relying on perturbation signals and helps preserve fine structures; (3) it is essentially equivalent to implicit interpolation between perturbed and unperturbed losses, implemented through random sampling rather than explicit weighting. Ablations confirm that replacing random application with explicit mixed loss, i.e., weighted loss after two renderings, worsens LPIPS from 0.184 to 0.212, indicating that random sampling is indeed more favorable for preserving perceptual quality.

3. Perturbation magnitude scheduling: linear growth from zero to prevent early training instability

At the early stage of training, Gaussians have just been initialized from the SfM point cloud and have not yet captured the coarse scene structure. Applying full-strength perturbations at this point is equivalent to adding strong noise on top of unconverged parameters, which can easily cause optimization divergence or convergence to poor solutions. The authors address this by multiplying the SAP perturbation magnitude by a linear scheduling factor \(\alpha(t) = t / T\), so that the perturbation strength grows linearly from 0 to the target value over training iterations. Formally, \(\mathbf{x}_i' = \mathbf{x}_i + \alpha(t) \cdot \delta_i\) with probability p.

Ablation results show that removing scheduling, i.e., using full-strength perturbation from beginning to end, reduces PSNR from 20.88 to 20.69 and leads to clear instability in the training curve. This supports the intuition of first building the coarse structure and then applying strong regularization: scheduling lets the model learn the approximate geometric layout of the scene early on, and then gradually introduces perturbations after Gaussian positions become relatively stable to flatten the loss landscape.

4. Periodic Gaussian reinitialization: freezing non-position degrees of freedom to assist regularization

Position perturbation is the main driver for promoting flat minima in this work, but regularizing only the position dimension is still insufficient, because overfitting in scale, rotation, and higher-order SH coefficients can also harm generalization. The authors design a lightweight complementary mechanism: every 1000 iterations, scale, rotation, and higher-order SH coefficients are temporarily restored to their SfM initialization states and frozen for W=100 iterations. Opacity is reset in every iteration according to the standard 3DGS reset mechanism, while positions and the total number of Gaussians are unaffected.

The intuition is that periodically zeroing non-position parameters forces optimization to relearn these degrees of freedom within each cycle, thereby reducing their overfitting to training views. From the training/test PSNR curves (Fig.5), without reinitialization, training PSNR keeps increasing while test PSNR plateaus, which is a typical overfitting signal. With reinitialization, each reset causes a brief drop in metrics but they recover quickly, and the final test PSNR continues to improve and surpasses the baseline without reinitialization. In the ablation, removing reinitialization reduces PSNR by 0.3 (20.88→20.58). Although the magnitude is smaller than removing SAP, it is valuable as a zero-cost supplementary regularization mechanism.

Loss & Training

The loss function completely follows standard 3DGS: \(\mathcal{L} = (1 - \lambda)\mathcal{L}_1 + \lambda\mathcal{L}_{\text{SSIM}}\), where \(\lambda = 0.2\). The key difference is that the loss is always computed on the perturbed parameters \(\hat{\theta}_t\), while the gradient update is applied to the original unperturbed parameters \(\theta_t\): \(\theta_{t+1} \leftarrow \theta_t - \eta \cdot \nabla\mathcal{L}(\hat{\theta}_t)\). This is equivalent to sampling neighboring points around \(\theta_t\) to smooth the loss landscape, rather than letting noise accumulate into the parameter trajectory.

Key hyperparameters: perturbation probability \(p_{\max}=0.3\), perturbation coefficient \(\gamma=2\), reinitialization interval of 1000 iterations, and freeze window W=100 iterations. Perturbed positions are clamped to ensure that displacement does not exceed the Gaussian's own scale. Training is conducted on NVIDIA RTX TITAN / A6000, and the remaining hyperparameters are consistent with DropGaussian.

Key Experimental Results

Main Results

LLFF dataset (3/6/9 views):

Method	3-view PSNR	3-view SSIM	3-view LPIPS	6-view PSNR	6-view SSIM	6-view LPIPS	9-view PSNR	9-view SSIM	9-view LPIPS
3DGS	19.22	0.649	0.229	23.80	0.814	0.125	25.44	0.860	0.096
DNGaussian	19.12	0.591	0.294	22.18	0.755	0.198	23.17	0.788	0.180
FSGS	20.43	0.682	0.248	24.09	0.823	0.145	25.31	0.860	0.122
CoR-GS	20.45	0.712	0.196	24.49	0.837	0.115	26.06	0.874	0.089
DropGaussian	20.76	0.713	0.200	24.74	0.837	0.117	26.21	0.874	0.088
Ours	20.88	0.731	0.184	24.76	0.840	0.114	26.23	0.875	0.088

Mip-NeRF360 dataset (12/24 views):

Method	12-view PSNR	12-view SSIM	12-view LPIPS	24-view PSNR	24-view SSIM	24-view LPIPS
3DGS	18.52	0.523	0.415	22.80	0.708	0.276
FSGS	18.80	0.531	0.418	23.70	0.745	0.230
CoR-GS	19.52	0.558	0.418	23.39	0.727	0.271
DropGaussian	19.74	0.577	0.364	24.13	0.762	0.225
Ours	19.50	0.584	0.348	24.19	0.771	0.219

Under the LLFF 3-view setting, the proposed method is best on all three metrics. Under the Mip-NeRF360 12-view setting, SSIM and LPIPS are best, while PSNR is slightly lower than DropGaussian (19.50 vs 19.74). Under the 24-view setting, all three metrics are best.

Ablation Study

Ablation under the LLFF 3-view setting (default configuration marked with †):

Ablation Dimension	Config	PSNR	SSIM	LPIPS
Noise Distribution	Anisotropic†	20.88	0.731	0.184
	Isotropic (maximum axis)	20.73	0.725	0.188
	Isotropic (average axis)	20.67	0.724	0.185
	Isotropic (fixed value)	20.54	0.714	0.188
Perturbed Parameter	Position†	20.88	0.731	0.184
	Rotation	20.23	0.710	0.195
	Scale	20.22	0.707	0.195
	Opacity	20.21	0.709	0.194
	Position+scale	20.33	0.719	0.199
Training Strategy	Full method†	20.88	0.731	0.184
	Random application→mixed loss	20.84	0.714	0.212
	w/o magnitude scheduling	20.69	0.721	0.191
	w/o reinitialization	20.58	0.713	0.197

Plug-and-play compatibility (cross-paradigm validation):

Dataset	Baseline Method	Baseline PSNR/SSIM/LPIPS	+Ours PSNR/SSIM/LPIPS
LLFF 3-view	FSGS	20.43/0.682/0.248	21.03/0.725/0.196
LLFF 3-view	DropGaussian	20.76/0.713/0.200	21.05/0.734/0.192
Mip-NeRF360 12-view	Difix3D+	19.25/0.554/0.375	19.63/0.575/0.386
Mip-NeRF360 12-view	AnySplat+3DGS 3k	16.97/0.338/0.432	17.36/0.384/0.427

Key Findings

• Position perturbation is the most effective perturbation dimension: perturbing positions brings about 0.6 PSNR improvement relative to perturbing other parameters, while perturbing both position and scale performs worse than perturbing position alone (20.33 vs 20.88). This indicates that introducing flatness in position space is most critical, and multidimensional perturbations may introduce conflicting regularization signals.
• The anisotropic noise design is the single largest contributor among all ablations: the improvement from fixed isotropic noise (20.54) to anisotropic noise (20.88) reaches 0.34 PSNR, showing that respecting Gaussian shape is not a minor tuning choice but a core design.
• Perturbation robustness analysis directly verifies the flat minima hypothesis: after training, different magnitudes of SAP perturbation are applied to Gaussian positions. The test PSNR of the proposed method drops much less than that of 3DGS and DropGaussian, and degradation on training views is also smaller, indicating that the method indeed converges to flatter minima.
• Strong plug-and-play capability: the method can be seamlessly integrated into three types of sparse-view pipelines, including optimization-based methods (FSGS, DropGaussian), diffusion-enhanced methods (Difix3D+), and methods that fine-tune after feed-forward initialization (AnySplat). It consistently brings improvements, indicating that flat minima regularization is complementary to existing methods.
• Gains diminish as the number of views increases: under the Mip-NeRF360 36-view setting, the proposed method (25.01 PSNR) is slightly lower than 3DGS (25.15), indicating that when supervision is sufficiently rich, the marginal benefit of flat minima regularization decreases and 3DGS itself can already converge to a good solution.

Highlights & Insights

• This work transfers classic generalization theory, namely flat minima, to 3D representation learning and makes nontrivial adaptations to geometric properties. Directly using isotropic perturbations reduces performance, as shown in the table, which proves that this is not a superficial transplant. Anisotropic covariance sampling is the key, and its essence is whitened perturbation in the local coordinate system of each Gaussian.
• The design of perturbing only loss evaluation rather than parameter updates is very clean. Compared with SAM, which requires two forward and backward passes for min-max optimization, this work needs only one forward pass, and the training overhead is almost unchanged. This technique is worth extending to other scenarios requiring implicit parameter regularization: as long as a parameter perturbation space can be defined, smoothing can be achieved through single-forward perturbation-loss-backpropagation.
• Random application in place of explicit loss mixing is an elegant engineering equivalence. RWP-style methods explicitly mix \(\mathcal{L}(\theta) + \mathcal{L}(\theta+\epsilon)\), while this work uses Bernoulli sampling \(p\) to implement implicit mixing in each iteration. The mathematical expectation is the same, but double rendering is avoided. Similar ideas can transfer to training other generative models that require perturbation regularization.
• Periodic reinitialization as free regularization warrants attention. It introduces no extra loss term, does not change the optimizer, and does not increase memory usage. It only performs parameter rollback plus brief freezing at specific iterations, yet stably improves PSNR by 0.3. This strategy of periodically resetting part of the parameters may apply to other overparameterized scenarios, such as NeRF MLP weights or even selected layers in large language model (LLM) fine-tuning.
• Perturbation robustness analysis (Fig.4) is compelling as direct evidence for flat minima. Instead of only reporting final metrics, it applies perturbations with different strengths after training and observes the performance degradation curves. This is a general test for minimum flatness and can serve as a standard experiment for evaluating any optimization method targeting generalization.

Limitations & Future Work

• Only positions are perturbed, while flatness in other parameter dimensions remains insufficiently explored. The ablation shows that perturbing position+scale reduces performance, but this may be because the regularization signals from the two perturbation sources interfere with each other, rather than because scale flatness itself is useless. Designing decoupled perturbation strategies, such as alternating perturbations or independent schedules, may unlock larger gains.
• Hyperparameters, including p, gamma, reinitialization interval, and freeze window, rely on manual settings, and different scenes may require tuning. The authors do not provide an automated tuning strategy or sensitivity analysis, such as performance curves for p in 0.1~0.5 and gamma in 1~4. Additional tuning cost may be needed in practical deployment.
• Under 36 views, the method is instead worse than 3DGS, indicating that it may become redundant regularization when supervision is sufficient. The model can already converge to a sufficiently flat solution, and extra perturbations instead introduce noise. A natural improvement is to design an adaptive switch: when the training/test PSNR gap falls below a threshold, perturbation is automatically reduced or disabled.
• The optimal period and window length for reinitialization are not deeply analyzed; the current 1000/100 iterations are empirical values. The optimal values may differ across datasets and sparsity levels. One could dynamically adjust them according to training curves, such as triggering reinitialization when overfitting is detected.
• Validation is limited to standard benchmarks, lacking tests on extremely sparse settings (1-2 views), dynamic scenes, or different 3DGS variants such as 2DGS and Scaffold-GS. The plug-and-play experiments cover three paradigms but do not evaluate more backbones.
• There is no direct comparison with SAM-like methods. Although SAM is impractical in 3DGS due to the high cost of two backward passes, as the benchmark method for flat minima optimization, the lack of comparison prevents readers from assessing the relative merits of the proposed random perturbation scheme and adversarial perturbation schemes in 3DGS, even if the proposed method is more efficient.

Related Work & Insights

• vs SAM / Sharpness-Aware Minimization: SAM searches for flat minima through min-max adversarial perturbations and requires two gradient computations per update. This work achieves a similar effect with random perturbation and a single forward pass, resulting in lower training cost. The adversarial direction in SAM may be more precise, but covariance sampling under the geometric constraints of 3DGS is already sufficiently effective, demonstrating the advantage of using domain knowledge to guide perturbation design.
• vs DropGaussian / DropoutGS: These methods perform structural regularization by randomly dropping Gaussians, essentially preventing the model from relying on a small number of primitives. This work instead performs perturbation regularization in parameter space, making the model insensitive to parameter shifts. The two are complementary: plug-and-play experiments show that adding the proposed method on top of DropGaussian still improves performance (20.76→21.05 PSNR), confirming the independence of the two regularization mechanisms.
• vs depth-prior methods (DNGaussian, FSGS): These methods introduce external priors such as monocular depth estimation to constrain geometry. They are effective but depend on additional networks. This work introduces no external priors and improves purely through the optimization strategy. It surpasses FSGS on 3-view LLFF (20.88 vs 20.43) and can be stacked as a plug-in on FSGS to further increase performance to 21.03.
• vs 3DGS-MCMC: MCMC-style methods inject noise during parameter updates, as in Langevin dynamics, and the noise accumulates into the trajectory. This work perturbs only during loss evaluation, keeping the parameter trajectory clean. The different injection positions of noise lead to different optimization dynamics: the former explores the posterior distribution, while the latter smooths the loss landscape.
• Insight: The core paradigm of this work, adapting a classic generalization theory from one field to the geometric properties of a new representation form, is highly transferable. For example, explicit 3D representations such as triplane / tensor decomposition also have geometrically meaningful parameters, such as spatial grids, and similar scale-adaptive perturbations can be designed. For 4DGS in dynamic scenes, perturbations along the temporal dimension can similarly draw on the covariance-scaling idea in this work.

Rating

• Novelty: ⭐⭐⭐⭐ Introducing flat minima optimization to 3DGS is not an entirely new concept, but the combined scheme of SAP, random application, and reinitialization designed for Gaussian anisotropic geometry provides substantive innovation rather than a simple application of existing methods.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ The main experiments cover two standard benchmarks with multiple view settings. The ablations cover three dimensions: noise distribution, perturbed parameters, and training strategy. The paper also includes plug-and-play compatibility validation, perturbation robustness analysis, training curve analysis, robustness tests for SfM quality, experiments with increased view counts, and additional benchmarks in the appendix.
• Writing Quality: ⭐⭐⭐⭐ The methodology is clear, the motivation chain is complete, and the argument from the flat minima perspective is persuasive. The perturbation robustness curve in Fig.4 is a highlight, but direct comparison with SAM-like methods is missing.
• Value: ⭐⭐⭐⭐ The lightweight design with zero architecture changes, zero external priors, and plug-and-play use provides high practical value. The flat minima perspective offers a new interpretive framework for 3DGS optimization and may inspire more systematic research on generalization theory in 3D representation learning.

Related Papers

• [CVPR 2026] GenSplat: Bridging the Generalization Gap in 3DGS Language Comprehension
• [CVPR 2026] Faster-GS: Analyzing and Improving Gaussian Splatting Optimization
• [ICLR 2026] A Step to Decouple Optimization in 3DGS
• [CVPR 2026] Intrinsic Geometry-Appearance Consistency Optimization for Sparse-View Gaussian Splatting
• [CVPR 2026] SparseOIT: Improving Order-Independent Transparency 3DGS via Active Set Method