MVGS: Multi-view Regulated Gaussian Splatting for Novel View Synthesis

Conference: ECCV2026
Paper: ECCV Paper
Project: MVGS Project Page
Authors: Xiaobiao Du, Yida Wang, Xin Yu
Area: 3D Vision
Keywords: Gaussian Splatting, multi-view consistency, novel view synthesis, coarse-to-fine training, adaptive densification

TL;DR

MVGS replaces single-view iterative fitting in Gaussian Splatting with joint multi-view updates, cross-intrinsic guidance, and two densification strategies, improving 3DGS PSNR on Mip-NeRF 360 from 28.69 to 29.61 while increasing training time from 0.3 to 2.4 hours.

Background & Motivation

3D Gaussian Splatting (3DGS) represents a scene with explicit Gaussian primitives carrying position, shape, color, and opacity, then projects and blends them into an image. Compared with neural radiance fields that require extensive sampling and network queries, this representation supports both high image quality and real-time rendering. Rendering efficiency, however, does not mean training already exploits multi-view information effectively: conventional 3DGS usually samples only one training image per iteration. A Gaussian position or shape that fits the current photograph may need an opposing adjustment in another photograph, making local fitting unstable.

The paper focuses on how a shared set of Gaussians receives supervision from different viewpoints, rather than introducing another Gaussian shape. Including several photographs in one update allows jointly visible structures to receive constraints from several directions at once, but simply increasing the view count introduces difficulties. Full-resolution images are expensive, and images with little overlap or large viewpoint differences can demand more detail than the existing primitives can represent. Consequently, replacing single-view training with a larger batch does not fully address initialization quality or local representation capacity.

MVGS addresses these issues in one optimization procedure: it first accommodates more views at low resolution, then progressively refines the scene while locating densification targets through camera discrepancies and cross-view errors. Multi-view consistency here primarily comes from joint image supervision of shared Gaussian parameters, rather than an additional matching network or pretrained feature teacher. Core Idea: let several views jointly determine Gaussian updates, improve initialization through coarse-to-fine training, and allocate additional primitives to regions with large viewpoint discrepancies or persistent errors across views.

Method

Overall Architecture

The inputs are multi-view training images, camera parameters, and an initialization point cloud obtained through SfM or a comparable source. The Gaussian state contains 3D positions, covariance-defined rotations and scales, appearance parameters, and opacity; training continually optimizes this shared scene representation. The output is an optimized Gaussian set that can render a novel camera view through the corresponding baseline renderer. MVGS is a general training scheme for per-scene optimization, not a generalizable reconstruction network that predicts an entire scene in one forward pass.

The procedure starts with the scale schedule of cross-intrinsic guidance, performs multi-view regulated learning within each scale, and supplements representation capacity through multi-view augmented densification and cross-ray densification. The former changes the densification threshold according to discrepancies between sampled cameras, whereas the latter maps high-error regions from several views back into shared 3D space. These densification operations support training; they are not additional image-analysis branches required during novel-view inference. The diagram shows training dependencies rather than prescribing a strictly sequential invocation frequency for the two densification strategies.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Multi-view images and cameras<br/>SfM-initialized Gaussians"] --> CIG["Cross-intrinsic Guidance"]
    CIG --> MVRL["Multi-view Regulated Learning"]
    MVRL --> MVAD["Multi-view Augmented<br/>Densification"]
    MVAD --> CRD["Cross-ray Densification"]
    CRD --> State["Updated Gaussians"]
    State -->|Continue training| MVRL
    State -->|Training complete| Render["Render a novel camera view"]
    Target["Ground-truth images<br/>at the current scale"] -.->|Image supervision and error maps| MVRL
    MVRL -.->|Multi-view error maps| CRD

Key Designs

1. Cross-intrinsic Guidance: constrain the scene with more low-resolution views before refining texture

Cross-intrinsic Guidance (CIG) uses a four-level image pyramid with downsampling factors \(s\in\{8,4,2,1\}\). Each level changes both the supervision image size and the camera focal length and principal point by the same scale factor, keeping images and projections in a consistent pixel coordinate system. It does not re-estimate the physical camera calibration; it constructs the effective intrinsics corresponding to each resolution. At low resolution, each image is cheaper to process, allowing more views in an update and letting more observation directions influence the coarse scene structure. As resolution increases, training inherits the Gaussian state from the previous level rather than reinitializing at full resolution.

The first three scales receive only a few thousand iterations each, without requiring the low-resolution images to be fitted to convergence. These images lack sufficient detail, so prolonged optimization cannot supply the supervision needed for the final textures. The last scale applies no downsampling and concentrates on refining details within the existing structure. The authors consider a maximum downsampling factor of 8 sufficient for multi-view training; excessive downsampling instead blurs supervision and makes subsequent detail recovery harder. The paper represents the schedule with scale-specific view counts, but the available full text does not enumerate a complete, directly reproducible set of view counts and iteration budgets, so no configuration is invented here.

2. Multi-view Regulated Learning: give shared primitives gradients from several directions in one update

Multi-view Regulated Learning (MVRL) renders each of the \(M\) currently sampled views and computes its image reconstruction loss. Every loss back-propagates to the same Gaussian parameters, with a primitive receiving the corresponding contribution only when it influences that view's rendering. Section 3.1 explicitly specifies summing gradients rather than averaging them; following Equation (4) and its accompanying explanation, the core update quantity is:

\[ \nabla_{\theta}\mathcal{L}_{\mathrm{MV}} =\sum_{m=1}^{M}\nabla_{\theta}\mathcal{L}_{m}. \]

Here, \(\theta\) denotes the Gaussian parameters being optimized, and \(\mathcal{L}_{m}\) is the loss for view \(m\). If different views request a similar positional adjustment, their contributions reinforce the shared update signal. If their requested directions conflict, vector addition can cancel part of the conflicting components. This is joint optimization of shared parameters, not a procedure that explicitly recovers geometric ground truth and forces every view into exact agreement. The paper's multi-view coherent constraint should not be mistaken for an additional cross-view matching loss with a separately specified formulation.

Summation also changes update magnitude, so the method affects not just direction but the effective update strength of Gaussians jointly observed in several views. The paper interprets this as an adaptive increase in effective learning rate; it does not explicitly maintain a learning-rate table indexed by visible-view count. This distinction makes multi-view information and larger updates factors that need to be separated. For reproduction or further study, comparing summation and averaging at a fixed effective learning rate would help identify the source of improvement more clearly than merely increasing view count.

The theoretical discussion concerns gradient variance and directional consistency, but lower relative noise should not be rewritten as an unconditional reduction in the absolute variance of summed gradients. Under independent noise and aligned mean signals, noise standard deviation grows as \(\sqrt{M}\) while the signal can grow as \(M\), improving the signal-to-noise ratio. Page 8 also gives a summed-gradient variance relationship that grows with view count and correlation. Figure 3's discussion of relative variance therefore needs a normalization convention; this note does not extend it into a convergence guarantee for arbitrarily correlated gradients.

3. Multi-view Augmented Densification: lower the refinement threshold when viewpoints differ substantially

Multi-view Augmented Densification (MVAD) addresses sampled training views with substantially different coverage. A limited set of Gaussians may fit one direction adequately yet lack the capacity to satisfy fine-detail requirements from several sides at once. MVAD first normalizes sampled camera translations to a unit sphere, then uses pairwise relative translation distances and rotation-matrix similarities to measure viewpoint discrepancies. Normalization prevents the same distance threshold from depending directly on a scene's world-coordinate scale. For \(M\) views, the number of camera pairs considered is \((M^2-M)/2\).

When the discrepancy reaches a predefined threshold, the original densification threshold is halved, allowing more primitives to qualify for refinement under the existing densification criteria. This does not unconditionally double every Gaussian; it changes the trigger threshold for density control. The goal is to increase capacity so that joint supervision does not merely force too few primitives to compromise repeatedly between viewpoints. The available full text does not clearly specify the complete rule for combining translation and rotation into one discrepancy measure or aggregating triggers across camera pairs. This note therefore retains the confirmed camera normalization and threshold-halving mechanism without reconstructing the damaged formula as an exact implementation.

4. Cross-ray Densification: localize difficult regions from several error maps in 3D space

Cross-ray Densification (CRD) further addresses where additional Gaussians should be allocated. It generates loss maps for several training views and uses a sliding window of size \((h,w)\) to locate image regions with high average loss. Window averaging focuses on a difficult region rather than letting one isolated high-error pixel determine the 3D densification target. For each selected window, one camera ray is cast from each of its four vertices, mapping the 2D image region to a spatial extent. The paper then describes constructing cuboids from intersecting planes formed by rays from different viewpoints to approximate overlapping high-error regions in 3D.

Gaussians in these spatial regions are selectively densified, giving local structures that affect several photographs additional representational freedom. This complements MVAD: MVAD determines whether to densify more aggressively, while CRD provides error-driven spatial localization. CRD uses rays to select densification regions during training; it does not replace the entire 3DGS renderer with a ray-tracing renderer. Ray intersection is not a rigorously verified surface reconstruction either, and the available text does not provide complete geometric tolerances or cuboid construction details. The mechanism is therefore best understood as multi-view-error-guided capacity allocation, rather than exact 3D correspondence recovery.

A Worked Example

Consider a calibrated static indoor scene in which several photographs show the same table from different directions; this illustrates the mechanism rather than introducing an additional experiment. Training first uses more views at \(s=8\) to establish coarse structure, then refines the same Gaussians through \(s=4\), \(s=2\), and \(s=1\). Within a scale, rendering errors from the table's front and side jointly affect its edge primitives in one MVRL update, instead of correcting one another across separate iterations. If the sampled cameras exceed the discrepancy threshold, MVAD lowers the densification threshold so complex boundaries can more readily receive additional primitives. If table-edge windows retain high average error across views, CRD projects them into space and prioritizes the jointly implicated regions for densification. After training, only the learned Gaussian scene is needed; a novel camera can be rendered without ground-truth images, error windows, or joint multi-view back-propagation.

Loss & Training

Per-view supervision follows the 3DGS image loss, combining L1 error and D-SSIM with a weighting coefficient before jointly optimizing the selected views. The central changes are multi-view gradient aggregation, resolution scheduling, and density control, not additional supervision targets generated by a pretrained model. Scale transitions inherit the coarse-stage Gaussian state, while view count and image resolution jointly determine the work performed in each update. Comparing loss curves at an equal iteration count therefore does not establish equal-wall-clock convergence speed. The available full text does not completely specify scale-wise view counts, window size, camera discrepancy thresholds, or exact training budgets, so these omissions are not turned into default hyperparameters here.

Key Experimental Results

Main Results

The following paired baseline comparisons are selected from Table 1 on page 11 and evaluate novel view synthesis on real-world scenes. PSNR is measured in dB; higher PSNR and SSIM and lower LPIPS are better. Exact splits and resolutions are not fully specified in the available main text.

Dataset	Method	PSNR ↑	SSIM ↑	LPIPS ↓
Mip-NeRF 360	3DGS	28.69	0.870	0.182
Mip-NeRF 360	3DGS + MVGS	29.61	0.873	0.173
Tanks&Temples	Scaffold-GS	23.96	0.853	0.177
Tanks&Temples	Scaffold-GS + MVGS	25.54	0.902	0.093
Deep Blending	3D-HGS	29.76	0.905	0.242
Deep Blending	3D-HGS + MVGS	29.74	0.906	0.236

3DGS gains 0.92 dB on Mip-NeRF 360, and Scaffold-GS gains 1.58 dB on Tanks&Temples. However, 3D-HGS PSNR on Deep Blending falls from 29.76 to 29.74, a decrease of 0.02 dB, even though the other two metrics improve. The experiments therefore support an overall improvement, but not a claim that every metric improves for every baseline.

Ablation Study

The following results come from Table 3 on page 14, using the 3DGS baseline on Mip-NeRF 360 and cumulatively adding components in the table's order. These are not independent leave-one-component-out ablations, so adjacent gains should not be treated as order-independent contributions.

Config	PSNR ↑	SSIM ↑	LPIPS ↓
Baseline	28.69	0.870	0.182
+ Multi-view Regulated Learning	29.26	0.871	0.179
+ Cross-ray Densification	29.37	0.872	0.178
+ Multi-view Augmented Densification	29.52	0.872	0.175
+ Cross-intrinsic Guidance (full model)	29.61	0.873	0.173

Adding MVRL first provides 0.57 dB; subsequently adding CRD, MVAD, and CIG in this order provides another 0.11, 0.15, and 0.09 dB, respectively. This supports joint updates as a major source of improvement while showing that joint gradients alone do not reach full-model performance.

The third table is selected from Table 2 on page 11 and compares representation size and training time on Mip-NeRF 360. Counts retain the original table's Num. convention, in millions; they should not be treated as exact GPU memory, file size, or rendering speed measurements.

Method	Num. (millions)	Training time (hours)	PSNR ↑
3DGS	1.1	0.3	28.69
3DGS + MVGS	0.8	2.4	29.61
Scaffold-GS	0.4	0.2	28.84
Scaffold-GS + MVGS	0.3	1.5	29.82
3D-HGS	0.7	0.4	29.66
3D-HGS + MVGS	0.5	2.6	30.21

Key Findings

• Calculated from Table 2's reported values, 3DGS training takes 8 times as long, while the representation count falls from 1.1 to 0.8 million; more aggressive local densification does not necessarily produce more final primitives.
• Figure 9 on page 15 reports degradation beyond a certain view count, which the authors attribute to overfitting a local viewing region; exact optimal values are not guessed from the extracted plot text.
• Figure 7 on page 13 uses iteration count on its horizontal axis, supporting improved optimization trajectories without overriding the wall-clock cost explicitly reported in Table 2.

Highlights & Insights

• Optimization itself is a reusable improvement target. MVGS is not tied to one Gaussian shape: its training logic works with 3DGS, Scaffold-GS, and 3D-HGS, demonstrating room for gains beyond representation design.
• Low resolution carries more geometric constraints, while high resolution refines detail. CIG is more than a conventional image pyramid because it spends the computational savings from reduced resolution on additional viewpoints.
• Error maps can allocate 3D representation capacity. CRD turns difficult 2D reconstructions into 3D densification cues; this idea is transferable, although precise localization still depends on cameras and geometric implementation.

Limitations & Future Work

• Training cost is a clear practical trade-off. All three baselines take substantially longer to train, so general applicability should not imply a low-cost plug-in; equal-time quality comparisons would be particularly valuable.
• Joint information and effective step size are entangled. Summation increases updates, and the presented ablations do not isolate summation versus averaging at a fixed effective learning rate; a controlled comparison would clarify the gains.
• Theoretical claims need to separate absolute variance from relative noise. Figure 3's relative-variance discussion does not establish universally lower absolute variance after summation, and correlation assumptions do not replace empirical gradient analysis in real scenes.
• Reproduction details and evidence coverage are limited. Some equations are damaged in the available extraction, and MVAD discrepancies, CRD geometry, and training settings are incompletely specified; the introduction mentions dynamic and large-scale applications, but the verifiable main tables primarily cover static real-world scenes, so no additional task results are invented here.

Related Work & Insights

• vs 3DGS / Scaffold-GS / 3D-HGS: These baselines provide different explicit scene representations, whereas MVGS changes how they receive supervision and expand capacity; it is a complementary optimization scheme, not a claimed replacement for all representation design.
• vs MVSplat / LatentSplat: The paper groups these as approaches that obtain multi-view information through pretrained networks, whereas MVGS constrains shared parameters through per-scene optimization without such a feature teacher.
• vs Pixel-GS / AbsGS: These methods focus on gradient-driven density control, while MVGS combines densification with camera discrepancies and overlapping multi-view errors; whether the selection rules are complementary requires combination experiments.
• Research direction: Comparing random view selection, geometric coverage, and gradient-complementary selection at equal training time could test whether additional viewpoints contribute genuinely new information; this is a reader-proposed direction, not a result established in the paper.

Rating

• Novelty: 4/5. Joint multi-view optimization is intuitive, but its combination with scale scheduling and spatial densification forms a concrete reusable method.
• Experimental Thoroughness: 4/5. Multiple baselines, datasets, cumulative ablations, and cost measurements are provided, but equal-time and effective-learning-rate-controlled comparisons are missing.
• Writing Quality: 3/5. The method is coherent, while the noise convention and several implementation details need clarification; damaged equation extraction is a separate limitation of the currently readable source.
• Value: 4/5. Useful when additional offline training cost is acceptable for better novel-view quality, but not a direct substitute for fast training.