StereoGS: Sparse-View 3D Gaussian Splatting via Stereo Priors

Conference: ECCV 2026
arXiv: 2606.30545
Code: https://stringerywh00.github.io/StereoGS_project_page/ (project page)
Area: 3D Vision
Keywords: Sparse-view novel view synthesis, 3D Gaussian Splatting, stereo priors, depth regularization, geometric consistency

TL;DR

StereoGS addresses the inherent scale ambiguity and cross-view inconsistency in monocular depth priors of sparse-view 3DGS by constructing virtual stereo camera pairs, introducing stereo depth regularization, and applying gradient-aware opacity decay, achieving state-of-the-art novel view synthesis on four datasets: LLFF, DTU, Mip-NeRF360, and Blender.

Background & Motivation

3D Gaussian Splatting (3DGS) has achieved remarkable success in novel view synthesis owing to its real-time rendering capability. However, when the input views are sparse, the overlapping areas between adjacent frames are extremely small, leading to severely insufficient geometric constraints. Under such conditions, 3DGS suffers from severe overfitting—the scene structure cannot be reliably reconstructed, while floaters and geometric deformations are prevalent. To mitigate this, several existing works have attempted to introduce monocular depth priors (e.g., DNGaussian, FSGS) to impose geometric constraints on Gaussian optimization. Although this design alleviates the issues to some extent, monocular depth fundamentally suffers from two insurmountable drawbacks: first, scale ambiguity—monocular estimation only provides relative depth without an absolute metric scale, and using it for regularization conflicts with photometric consistency, misleading the optimization; second, cross-view inconsistency—depth maps are inferred independently for each frame, meaning the same spatial point is often assigned different depth values across different views. The resulting spatial misalignment when back-projecting into 3D space causes unstable Gaussian optimization and severe artifacts.

Some recent works (Binocular3DGS, NexusGS, MVPGS) attempt to introduce cross-view consistency. Binocular3DGS uses rendered depth maps to warp views and compute photometric loss, but this indirect pixel-level constraint lacks direct gradient guidance for 3D Gaussian primitives, particularly failing in textureless regions. NexusGS only utilizes epipolar geometry during the initialization phase, lacking continuous binocular consistency constraints during optimization, which still leads to gradual geometric degradation. The more fundamental contradiction lies in: existing methods either invest resources in initialization (MVS point clouds) or add constraints during optimization, but few methods simultaneously achieve "robust initialization + continuous geometric alignment + adaptive pruning of noisy points."

This paper proposes StereoGS, whose core idea is to directly embed stereo vision priors into the 3DGS optimization loop: a virtual right-view camera is constructed next to each training view to render the corresponding right-view image, and then the real left view and rendered right view are formed into a stereo pair. This pair is fed into a foundation stereo model to obtain absolute-scale stereo depth as a supervision signal. Concurrently, a gradient-aware opacity decay strategy based on relative gradient magnitude is designed to adaptively preserve Gaussians that contribute significantly to rendering while pruning redundant floaters. Core Idea: Introduce online stereo depth regularization throughout the 3DGS optimization process, providing absolute-scale and binocularly-consistent geometric supervision via virtual stereo pairs + a foundation stereo model, complemented by gradient-aware opacity decay and zero-shot MVS dense initialization, thoroughly eliminating geometric ambiguity under sparse views from a training-time strategy perspective without increasing inference overhead.

Method

Overall Architecture

StereoGS overlays three orthogonal components on top of standard 3DGS: first, it uses a multi-view depth estimator to construct a dense, geometrically consistent initial point cloud (replacing sparse SfM); second, it applies absolute scale constraints on Gaussian geometry using a stereo depth regularization term during optimization; third, it replaces periodic opacity reset with gradient-aware opacity decay to adaptively prune redundant Gaussians. During inference, all three modules are discarded, rendering it completely equivalent to original 3DGS with zero extra overhead.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Sparse Training Views (3/6/9 views)"] --> B["Consistency-Aware Dense Initialization<br/>MVSAnywhere + Cross-View Reprojection Filtering"]
    B --> C["3D Gaussian Optimization Loop"]
    C --> D["Stereo Depth Regularization<br/>Virtual Right View + FoundationStereo"]
    C --> E["Gradient-Aware Opacity Decay<br/>Relative Gradient β = g / ḡ"]
    D --> F["Geometrically Consistent 3D Gaussians"]
    E --> F
    F -->|"Standard splatting rendering"| G["High-Quality Novel View Synthesis"]

Key Designs

1. Stereo Depth Regularization: Replacing Monocular Constraints with Absolute-Scale Binocular Priors

Monocular depth estimation can only provide relative depth, lacking a unified metric scale, and monocular maps of different views are inferred independently, which often causes inconsistent depth values for the same scene point across frames. StereoGS solves this by dynamically synthesizing a virtual right camera for each training view (left camera) during optimization: it renders a right-view image \(\hat{I}_r\) from the current 3D Gaussians, and forms a stereo pair using the real left-view image \(I_l\) and the rendered right-view image. This pair is fed into a foundation stereo model, FoundationStereo, to obtain the left-view disparity \(\hat{D}_l\). The real left-view image is intentionally used instead of the rendered one because the estimated disparity serves as an absolute depth reference; using a clean, real image avoids rendering noise and ensures the reliability of the prior.

To filter out unreliable disparities in occluded, textureless, or anomalous regions, the method introduces a three-layer validity mask: a left-right consistency occlusion mask \(M_{\text{occ}}\) (pixels where the difference after left-right disparity warping exceeds a threshold \(\tau=2.0\)), a background mask \(M_{\text{bg}}\) (pixels with dark backgrounds or a zero alpha channel in synthesized images), and a disparity anomaly mask \(M_{\text{anomaly}}\) (non-positive, Inf, or NaN values). The union of these three forms the final validity mask \(M_{\text{valid}} = 1 - (M_{\text{bg}} \lor M_{\text{occ}} \lor M_{\text{anomaly}})\). In valid regions, the disparity is converted to depth \(Z_{\text{stereo}} = fd/\hat{D}_l\) (\(f\) is the focal length, \(d\) is the baseline), and supervision is applied to the rendered depth \(\hat{Z}\) using an L1 loss in the inverse depth space:

\[\mathcal{L}_{\text{depth}} = \left\| M_{\text{valid}} \odot \left(\frac{1}{\hat{Z}} - \frac{1}{Z_{\text{stereo}}}\right) \right\|_1\]

The inverse depth space offers two advantages: better numerical stability and stronger geometric constraints in near-field and foreground regions. This regularization term directly back-propagates gradients to Gaussian primitives, providing absolute-scale and binocularly-consistent geometric supervision, fundamentally overcoming the scale ambiguity of monocular priors.

2. Gradient-Aware Opacity Decay: Adaptive Pruning of Redundant Gaussians

Standard 3DGS uses periodic opacity reset to clear noisy Gaussians. While effective for dense views, under sparse views, this uniformly suppresses both stable surface Gaussians and floaters, destroying the established geometric structure. Binocular3DGS instead uses a fixed decay rate, treating all Gaussians equally, which similarly fails to distinguish "rendering-contributing structures" from "useless floaters."

The insight of StereoGS is that the gradient magnitude of a Gaussian's opacity, \(g = |\nabla_\alpha \mathcal{L}|\), naturally reflects its importance in reducing rendering error—Gaussians with large gradients are being actively moved by optimization, indicating a high contribution to the current rendering, and should be preserved; Gaussians with extremely small gradients are barely needed and are more likely to be useless floaters. Since absolute gradient values are extremely small (on the order of \(10^{-6}\)), directly using absolute values deteriorates into constant decay. Inspired by the concept of "relative advantage" in GRPO, the method computes the relative gradient \(\beta = g / \bar{g}\) for each Gaussian (where \(\bar{g}\) is the average gradient of all Gaussians in the current iteration), and derives a dynamic decay factor through an exponential soft-thresholding function:

\[\gamma = 1 - (1 - \gamma_{\text{base}}) \exp(-s \cdot \beta)\]

The final opacity is updated as \(\hat{\alpha} = \gamma \alpha\). When \(\beta < 1\) (below-average contribution), \(\gamma\) approaches \(\gamma_{\text{base}}\), applying a strong penalty; when \(\beta\) is large, \(\gamma\) approaches 1, with almost no decay. The hyperparameters are set to \(\gamma_{\text{base}} = 0.99\) and \(s = 0.5\). Ablation studies show that step and linear functions perform worse than the constant strategy because they over-preserve Gaussians with \(\beta > 1\); the exponential function transitions smoothly around \(\beta = 1\), making it the optimal choice, achieving 21.91 PSNR on LLFF 3-view, which is significantly better than the 21.45 of the constant strategy.

3. Consistency-Aware Dense Initialization: Replacing Sparse SfM with Zero-Shot MVS Point Clouds

Under sparse views, point clouds generated by SfM are extremely sparse and noisy, leading to only 18.96 and 17.66 PSNR on LLFF and DTU respectively at initialization. The method adopts a zero-shot multi-view depth estimator, MVSAnywhere, to estimate depth maps for each training view—each time taking the target view and the remaining views as source frames to input into the model, yielding a set of multi-view consistent depth maps. Subsequently, following the classic MVS geometric filtering strategy, outliers are removed via cross-view reprojection errors, and the filtered depth maps are back-projected and fused into a dense point cloud. Compared to PDCNet+ and MVSFormer, MVSAnywhere exhibits stronger zero-shot generalization capability, generating denser and more structurally complete point clouds. It achieves 19.75 PSNR on LLFF 3-view initialization alone, whereas under the same conditions, MVSFormer achieves only 21.08, and PDCNet+ achieves only 20.10 (both tested with all components). Note that this only improves initialization quality; without subsequent optimization constraints, high-quality initialization still gradually degrades during Gaussian optimization. Thus, dense initialization must work in tandem with stereo depth regularization to realize its full effectiveness.

Loss & Training

The total loss is \(\mathcal{L} = \mathcal{L}_{\text{color}} + \mathcal{L}_{\text{depth}}\), where the color loss is inherited from standard 3DGS:

\[\mathcal{L}_{\text{color}} = (1-\lambda)\mathcal{L}_1 + \lambda \mathcal{L}_{\text{D-SSIM}}\]

Stereo depth regularization is activated only after a certain number of iterations (on step 20,000 for LLFF/DTU/Mip-NeRF360, and step 4,000 for Blender), with total training steps of 30,000 and 7,000, respectively. Densification is performed every 100 steps starting from step 1,000. All experiments were conducted on a single RTX 4090, with a training time of approximately 43 minutes (LLFF 3-view) and a peak VRAM usage of 3.6 GB. In comparison, FSGS takes 26.5 minutes but is limited to 3-view, and MVPGS takes only 6.4 minutes but generates up to 1.52 million Gaussians. Although StereoGS trains more slowly, it only generates 126k Gaussians, leading to high inference efficiency.

Key Experimental Results

Main Results

Dataset	Views	Metric	StereoGS	StereoGS*	Prev. SOTA
LLFF	3-view	PSNR↑	21.91	22.05	21.44 (Binocular3DGS)
LLFF	3-view	SSIM↑	0.773	0.783	0.751
LLFF	3-view	LPIPS↓	0.157	0.147	0.168
LLFF	6-view	PSNR↑	24.92	25.40	24.87 (Binocular3DGS)
DTU	3-view	PSNR↑	21.46	22.00	20.71 (Binocular3DGS)
DTU	3-view	SSIM↑	0.879	0.890	0.877 (MVPGS)
Mip-NeRF360	12-view	PSNR↑	20.25	20.51	20.09 (D2GS)
Blender	8-view	PSNR↑	24.83	25.04	24.71 (Binocular3DGS)

(Ours* denotes the additional overlay of the DropGaussian strategy with a dropout rate of 0.3, which is complementary to the DropGaussian concept but originates from a different perspective.)

Ablation Study

Init (CAD)	Stereo Reg (SDR)	Grad Decay (GAOD)	LLFF PSNR	DTU PSNR
-	-	-	16.02	10.99
✓	-	-	19.75	14.10
-	✓	-	17.32	12.46
-	-	✓	18.18	15.05
✓	✓	-	19.79	15.57
✓	-	✓	21.18	19.76
✓	✓	✓	21.91	21.46

Key Findings

• Dense initialization contributes the most fundamental improvement: By only adding CAD, LLFF rises from 16.02 to 19.75 (+3.73), and DTU rises from 10.99 to 14.10 (+3.11), indicating that sparse SfM point clouds are one of the primary bottlenecks in sparse views.
• The combination of gradient-aware opacity decay (GAOD) and dense initialization yields the strongest effect: Achieving 21.18 on LLFF and 19.76 on DTU, which vastly outperforms having only initialization and regularization (19.79 / 15.57). This shows that GAOD can "lock in" the advantages of high-quality initialization, preventing geometric degradation during optimization.
• The validity mask (\(M_{\text{valid}}\)) is indispensable: Removing it drops LLFF from 21.91 to 21.30, and DTU from 21.46 to 20.82, because noisy disparities in occluded/textureless regions directly mislead optimization.
• Different stereo models are all effective, but FoundationStereo is optimal: LiteAnyStereo / S2M2 / FoundationStereo achieve 21.28 / 21.53 / 21.91 on LLFF 3-view, respectively, showing that the stereo regularization framework is generalizable and not bound to a specific model.

Highlights & Insights

• "Online prior" design via virtual stereo pairs: Instead of pre-computing stereo depth, it dynamically synthesizes right-view images at each iteration based on the current rendering state. This synchronizes the stereo supervision signal with the evolution of the Gaussians, while keeping the real left-view image to maintain depth prior reliability. This hybrid pair construction ("half real, half rendered") elegantly balances prior quality with optimization loop closure.
• Generality of relative gradient normalization: Normalizing absolute gradients into relative gradients \(\beta = g/\bar{g}\) draws inspiration from the concept of "within-group relative advantage" in GRPO. This approach can be extended to any scenario requiring dynamic evaluation of Gaussian/particle importance, offering strong transfer value.
• Orthogonal components: Ablations show that the contributions of the three modules (CAD/SDR/GAOD) are relatively independent and can be selectively combined based on computational resources. For example, one can use only CAD+GAOD to save training costs when ultimate quality is not required.

Limitations & Future Work

• In extremely textureless regions (e.g., large lawns, walls), the matching accuracy of the stereo model drops, causing the stereo depth prior to fail and rendering quality to degrade (as explicitly noted in the paper's failure cases).
• Reflective objects pose a challenge to MVSAnywhere's multi-view depth estimation, as view-dependent reflections are misidentified as geometrically inconsistent during reprojection filtering and are pruned.
• Online stereo depth regularization introduces approximately 43 minutes of training time (vs. 4 minutes for standard 3DGS), which is a burden for scenarios requiring fast iteration. Future work could explore pre-computing stereo priors offline via distillation.
• The method depends on tuning baseline parameters for FoundationStereo (where \(d=4.0\) is optimal); different scene scales may require parameter retuning.

Related Work & Insights

• vs. DNGaussian / FSGS: Both employ monocular depth regularization, which fails to resolve scale ambiguity and cross-view inconsistency. StereoGS replaces monocular priors with stereo priors, fundamentally providing absolute scale constraints—representing an upgrade in prior type rather than mere engineering tricks.
• vs. Binocular3DGS: While also introducing binocular information, Binocular3DGS uses RGB warping to calculate photometric loss, which lacks direct geometric gradients. StereoGS supervises directly in the depth space, which is more robust to textureless regions. Furthermore, Binocular3DGS uses a fixed decay rate, whereas StereoGS adaptively scales via relative gradients, retaining more effective surface Gaussians.
• vs. MVPGS / NexusGS: Both introduce cross-view consistency during the initialization phase (via MVS point clouds or epipolar geometry), but lack continuous geometric constraints during optimization, leading to gradual degradation of the initial quality. StereoGS's stereo regularization continuously enforces constraints throughout the optimization, fixing this flaw of "having a good start but failing to maintain it."

Rating

• Novelty: ⭐⭐⭐⭐ The direction of replacing monocular priors with online virtual stereo pairs + foundation stereo models is very clear, and the design of relative gradient decay is a highlight.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across four datasets, multi-view settings, and detailed ablations (masks/stereo models/decay functions/hyperparameters).
• Writing Quality: ⭐⭐⭐⭐ Clear structure, complete motivation chain, and honest representation of failure cases.
• Value: ⭐⭐⭐⭐ It achieves SOTA on the highly practical sparse-view 3DGS task with zero inference overhead, making it deployment-friendly.

Related Papers

• [CVPR 2025] S2Gaussian: Sparse-View Super-Resolution 3D Gaussian Splatting
• [AAAI 2026] SparseSurf: Sparse-View 3D Gaussian Splatting for Surface Reconstruction
• [CVPR 2025] DropGaussian: Structural Regularization for Sparse-view Gaussian Splatting
• [ECCV 2026] Capacity-Controlled Multi-View Stylization of 3D Gaussian Splatting
• [ECCV 2024] CoR-GS: Sparse-View 3D Gaussian Splatting via Co-Regularization