FillGS: Filling Observation Gaps in 4D Gaussian Splatting via Viewpoint-Time Selection and Generative Refinement

Conference: ECCV 2026
Paper: ECCV Official
Project: https://otonari726.github.io/fillgs/
Area: 3D Vision
Keywords: 4D Gaussian Splatting, dynamic novel view synthesis, active view selection, video diffusion model, spatiotemporal sparse observation

TL;DR

To tackle motion blur and severe artifacts caused by spatiotemporal observation gaps in sparse-view 4D Gaussian Splatting, FillGS actively selects highly informative virtual spatiotemporal viewpoints using rendering sensitivity and motion-aware observation density, refines rendered views via a video diffusion model, and robustly fine-tunes the 4DGS field with matching-confidence and co-visibility masks.

Background & Motivation

4D Gaussian Splatting (4DGS) has rapidly emerged as a leading framework for real-time dynamic novel view synthesis, leveraging explicit point-based primitives and fast differential splatting to achieve both photorealistic visual fidelity and interactive rendering speed. Despite its great promise in virtual reality, 3D content creation, and autonomous interaction, 4DGS remains critically dependent on dense, synchronized multi-camera setups. In real-world captures with limited viewpoint diversity or sparse camera arrangements, camera baseline coverage is substantially constrained, leaving large spatiotemporal regions under-observed. In scenes featuring fast or complex object motions, moving elements appear at specific spatial locations only fleetingly; consequently, the associated 4D Gaussians receive minimal photometric and geometric supervision, which precipitates severe floaters, structural collapse, and blurred motion boundaries when novel or extrapolated views are synthesized.

To compensate for missing observations, recent efforts leverage generative diffusion models to post-process and refine rendered views from unobserved camera angles. However, the efficacy of generative enhancement hinges heavily on the choice of virtual viewpoints. Existing paradigms typically rely on hand-crafted heuristics, such as uniform random sampling or naive geometric interpolation between existing training cameras. While spatial-coverage heuristics can effectively guide virtual viewpoints in static 3D scenes, they fail fundamentally in dynamic 4D scenes: observation sparsity is not a static spatial property, but rather an evolving spatiotemporal continuum modulated by object motion over time. View selection strategies that focus solely on static spatial spread frequently bypass the specific spatiotemporal coordinates where fast-moving Gaussians are critically starved of constraints, wasting generative inference budgets without resolving underlying reconstruction ambiguities.

Furthermore, generated images cannot be treated as ground-truth observations without discernment. Generative models inevitably introduce geometric inconsistencies, structural hallucinations, or texture drift. Forcing 4DGS optimization to treat these synthetic views as ground truth in regions already well-constrained by real cameras degrades baseline fidelity, while failing to excise generative hallucinations corrupts the underlying physical radiation field. Core Idea: Formulate dynamic 4DGS observation gap filling as an active viewpoint-time selection and trustworthy generative guidance problem, actively targeting under-constrained spatiotemporal regions via rendering perturbation sensitivity and motion-aware observation density, and filtering out generative hallucinations through dense feature matching confidence and dynamic co-visibility masks for robust fine-tuning.

Method

Overall Architecture

The FillGS pipeline operates across four integrated stages: first, an initial 4DGS representation is trained on sparse multi-view video inputs; second, an active viewpoint-time selection mechanism identifies exploration starting points using perturbation sensitivity and scores local/global candidate camera trajectories using motion-aware observation density; third, rendered sequences from selected viewpoints are constructed into spatiotemporal patches and enhanced by a video diffusion model; finally, the 4DGS model is robustly fine-tuned using a dual-mask modulation scheme that balances geometric matching consistency with motion-aware observation need.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Sparse Multi-view Dynamic Video"] --> B["Stage 1: 4DGS Baseline Model Training"]
    B --> C["Stage 2: Sensitivity-Based Starting Point Selection<br/>Combining reconstruction error and finite-difference sensitivity"]
    C --> D["Stage 3: Local & Global Virtual Candidate Generation<br/>Pose perturbations and bounding box trajectory interpolation"]
    D --> E["Stage 4: Motion-Aware Observation Density Scoring<br/>Evaluating Gaussian velocities and temporal observation counts"]
    E --> F["Stage 5: Spatiotemporal Video Diffusion Enhancement<br/>Spatiotemporal patch completion and consistency refinement"]
    F --> G["Stage 6: Dual-Mask Robust Fine-Tuning<br/>Joint modulation via matching confidence and co-visibility weights"]
    G --> H["Output: High-Fidelity 4DGS Radiance Field"]

Key Designs

1. Sensitivity-Based Starting Point Selection: Proxying epistemic uncertainty via finite-difference perturbations

To efficiently guide virtual camera exploration, the framework must first locate which existing spatiotemporal observation points are surrounded by unstable geometry. Because ground-truth imagery is unavailable at unobserved coordinates, directly computing unseen reconstruction error is impossible. FillGS exploits the insight that 4D Gaussians in poorly constrained regions exhibit high parametric volatility: minor perturbations in camera pose or timestamp drastically shift Gaussian visibility and alpha-blending weights, leading to abrupt visual variance in rendered outputs. This local output variation acts as an effective finite-difference surrogate for epistemic reconstruction uncertainty.

For every observed viewpoint \(v\), an exploration priority score \(\mathcal{E}(v)\) is calculated by multiplying the normalized reconstruction error by the normalized perturbation variance: $$ \mathcal{E}(v) = \mathcal{L}(v) \cdot \mathcal{V}(v) $$ where \(\mathcal{L}(v)\) represents the per-view mean squared reconstruction error normalized to average 1 across the dataset. The local variation \(\mathcal{V}(v)\) is computed across a set of small perturbations \(\mathcal{D}\): $$ \mathcal{V}(v) = \sum_{\delta \in \mathcal{D}} w_\delta \left| \hat{I}\theta(v \oplus \delta) - \hat{I}\theta(v \ominus \delta) \right|_2^2 $$ The perturbation set \(\mathcal{D}\) includes 3 translational shifts, 3 rotational shifts, and 1 temporal shift. This metric approximates the local Lipschitz constant of the splatting renderer, reliably identifying observed viewpoints whose spatiotemporal neighborhoods suffer from extreme instability to anchor downstream exploration.

2. Local and Global Dual-Candidate Generation: Balancing near-field stability with blind-spot traversal

Relying solely on local infinitesimal perturbations traps camera exploration in local basins without reaching wide camera baselines, whereas purely global random view generation frequently samples unviable viewing angles outside the scene manifold. FillGS harmonizes these approaches with a dual local-global candidate generation mechanism.

Locally, the method applies 6 translational and 4 rotational shifts around the starting camera, combined with 2 interpolated poses toward the two closest training cameras at that timestamp, producing 12 local candidate views. Globally, it computes a 3D bounding box spanning the starting camera and its nearest observed neighbor, expands the volume by the maximum reachable local displacement step, and uniformly samples 12 spatial candidate coordinates. The viewing direction for each sampled coordinate is interpolated from neighboring training views, and a smooth camera trajectory is constructed connecting the selected global viewpoint back to its nearest training camera. This ensures that candidate sets comprehensively cover both near-geometry gaps and wide-baseline extrapolations.

3. Motion-Aware Observation Density Scoring: Quantifying information gain via primitive velocity and frequency

Static view selection heuristics typically evaluate Gaussian coverage via total observation count. However, in dynamic scenes, static background Gaussians are consistently visible across all video frames and naturally accumulate large observation counts, while fast-moving foreground objects pass through spatial locations briefly, yielding sparse temporal observations. Relying solely on raw visibility counts inevitably biases selection toward static scene regions, completely ignoring dynamic motion gaps.

FillGS resolves this tension by tracking both the cumulative temporal observation count \(C_{g_i}(t)\) and the velocity magnitude \(s_{g_i}(t)\) for each 4D Gaussian primitive \(g_i\) at time \(t\), defining the motion-aware observation deficiency \(D_{g_i}(t)\): $$ D_{g_i}(t) = \frac{s_{g_i}(t)}{1 + C_{g_i}(t)} $$ Gaussians characterized by high velocity and minimal historical observations receive elevated deficiency values, indicating acute under-constraint. To score each candidate viewpoint, the deficiency values \(D_{g_i}(t)\) are rendered into the candidate camera frustum in place of color, and the mean pixel intensity across the rendered buffer serves as the candidate's score. The candidate viewpoint capturing the highest density of rapid, under-observed Gaussians is selected, thereby maximizing the active information gain for dynamic scene completion.

4. Dual-Mask Robust Fine-Tuning: Safeguarding physical fidelity against generative hallucinations

While video diffusion models synthesize realistic visual details, their outputs frequently exhibit geometric warps, hallucinated semantics, or slight identity drifts. Directly injecting raw generated frames into 4DGS optimization destabilizes convergence and corrupts valid geometry. A robust supervision mechanism must prevent updates to already well-reconstructed regions and filter out unfaithful synthetic artifacts.

FillGS enforces these constraints through two complementary spatial masks: - Matching-Confidence Consistency Mask: Utilizes the dense feature matching network RoMa v2 to compute correspondence confidence maps between the generated virtual image and its two nearest training views, taking the pixel-wise maximum. Pixels failing geometric matching are deemed generative hallucinations and down-weighted to zero. - Motion-Aware Co-Visibility Mask: Inverts normalized Gaussian observation deficiency \(D_{g_i}(t)\) into an adaptive co-visibility weighting field. Regions with dense physical observation receive near-zero loss weights, preserving existing ground-truth fidelity, whereas sparsely observed, fast-moving regions receive high weights to absorb generative completion. Together, these masks restrict gradients strictly to plausible, under-constrained spatiotemporal regions.

Loss & Training

Using E-D3DGS as the underlying 4DGS representation, the model is trained for 30,000 iterations. At every 6,000-iteration milestone, active viewpoint selection generates 20 virtual camera trajectories, which are processed through the video diffusion model and introduced into the training pool. The robust fine-tuning objective combines modulated L1 and D-SSIM losses: $$ \mathcal{L}{\text{ft}} = \sum(p) \right] $$ This loss ensures that pseudo-supervision guides the dynamic Gaussians without distorting verified multi-view structure.}} M_{\text{conf}}(p) \cdot M_{\text{covis}}(p) \left[ (1 - \lambda_{\text{ssim}}) |I_{\text{gen}}(p) - \hat{I}(p)| + \lambda_{\text{ssim}} \mathcal{L}_{\text{D-SSIM}

Key Experimental Results

Main Results

Evaluation is conducted under a challenging sparse-camera configuration (training on only 2 cameras, evaluating on both spatial interpolation and extrapolation trajectories) across 6 scenes of the Neural 3D Video benchmark and 5 scenes of the Technicolor dataset. Metrics include PSNR, SSIM, LPIPS, FID, and DINOv2 visual feature cosine similarity.

Table 1: Quantitative comparison on Neural 3D Video Dataset (2-camera sparse split)

Viewpoint Selection Strategy / Method	Interp PSNR↑	Interp SSIM↑	Interp LPIPS↓	Interp FID↓	Extrap PSNR↑	Extrap SSIM↑	Extrap LPIPS↓	Extrap FID↓
E-D3DGS Baseline (No Generation)	20.42	0.775	0.293	129.33	14.68	0.723	0.363	160.01
Interpolation-based Selection (GS-GS)	20.75	0.777	0.293	120.08	14.75	0.716	0.370	157.52
FisherRF-based Selection (Fisher Info)	21.01	0.774	0.303	135.36	15.59	0.716	0.373	180.27
Coverage-based (3D) Selection (ExploreGS)	19.91	0.754	0.317	148.68	15.30	0.700	0.388	175.53
Coverage-based (4D) Selection (Per-frame)	19.09	0.737	0.325	140.19	15.51	0.681	0.407	182.99
FillGS (Ours)	21.73	0.792	0.256	83.81	16.24	0.740	0.312	137.32

Table 2: Quantitative comparison on Technicolor Dataset (2-camera sparse split)

Viewpoint Selection Strategy / Method	Interp PSNR↑	Interp SSIM↑	Interp LPIPS↓	Interp FID↓	Extrap PSNR↑	Extrap SSIM↑	Extrap LPIPS↓	Extrap FID↓
E-D3DGS Baseline	19.63	0.606	0.378	137.86	16.10	0.690	0.341	131.48
Interpolation-based Selection	19.90	0.612	0.376	115.24	16.88	0.701	0.338	123.77
FisherRF-based Selection	20.08	0.614	0.421	152.56	18.32	0.646	0.444	163.51
Coverage-based (3D) Selection	20.12	0.622	0.370	106.63	17.61	0.710	0.332	110.54
Coverage-based (4D) Selection	20.03	0.619	0.373	110.07	17.68	0.717	0.329	106.47
FillGS (Ours)	22.55	0.748	0.250	73.44	19.88	0.720	0.315	106.45

Ablation Study

Table 3: Ablation study of viewpoint selection components (Neural 3D Video Dataset)

Configuration	Interp PSNR↑	Interp SSIM↑	Interp LPIPS↓	Interp FID↓	Extrap PSNR↑	Extrap LPIPS↓	Extrap FID↓
Full Model (FillGS)	21.73	0.792	0.256	83.81	16.24	0.312	137.32
w/o starting-viewpoint selection	21.06	0.773	0.286	126.78	15.75	0.393	144.98
w/o sensitivity-based selection	21.24	0.763	0.279	100.77	16.01	0.379	146.25
w/o error-based selection	21.52	0.766	0.275	100.14	16.22	0.377	142.81
w/o candidate-viewpoint selection	21.13	0.778	0.282	113.72	15.07	0.391	147.03
w/o observation-count-based selection	20.55	0.768	0.294	135.20	15.66	0.346	150.63
w/o speed-based selection	21.06	0.761	0.283	106.35	15.79	0.350	146.81
w/o global candidate view creation	21.39	0.778	0.283	116.88	15.85	0.353	150.18
Random selection	20.59	0.762	0.295	130.43	15.07	0.384	172.37
Negative selection	20.31	0.765	0.294	129.30	15.06	0.379	175.95

Table 4: Ablation study of fine-tuning strategies (Neural 3D Video Dataset)

Fine-tuning Configuration	Interp PSNR↑	Interp SSIM↑	Interp LPIPS↓	Interp FID↓	Extrap PSNR↑	Extrap LPIPS↓	Extrap FID↓
Full Model (Dual-Mask Modulation)	21.73	0.792	0.256	83.81	16.24	0.312	137.32
w/o weight (Uniform loss on all pixels)	20.05	0.716	0.334	128.57	14.70	0.409	167.40
w/o consistency mask	21.26	0.778	0.283	107.96	15.81	0.355	149.50
w/o co-visibility mask	21.96	0.780	0.270	98.43	15.93	0.359	153.95
Comparison: 3DGS-Enhancer weighting	21.41	0.763	0.281	100.11	15.48	0.374	156.29
Comparison: ExploreGS weighting	20.97	0.770	0.287	109.74	15.14	0.353	151.38
Comparison: UA-4DGS weighting	19.96	0.745	0.322	135.11	14.69	0.419	201.89

Key Findings

1. Dramatic Perceptual & Structural Gains: On Neural 3D Video, FillGS lifts interpolation PSNR by +1.31 dB and drives FID down from 129.33 to 83.81 (a >35% perceptual improvement). On Technicolor, it boosts interpolation PSNR from 19.63 dB to 22.55 dB (+2.92 dB) with an FID of 73.44, resolving fine silhouette tears and dynamic smearing.
2. Failure of Naive Spatial Coverage in 4D: Coverage-based strategies (ExploreGS 3D/4D) perform worse than basic interpolation (e.g. 19.09 dB vs 20.75 dB) because they excessively push cameras toward static spatial boundaries while missing internal motion gaps. Removing the speed term \(s_{g_i}(t)\) from the deficiency metric drops interpolation PSNR by 0.67 dB, underscoring the necessity of motion-aware weighting.
3. Consistency Mask as Primary Line of Defense: Fine-tuning without weights degrades extrapolation PSNR to 14.70 dB (inferior even to the baseline without generation). Removing the matching-confidence consistency mask inflates FID from 83.81 to 107.96, proving that feature matching filtering is critical to prevent diffusion hallucinations from corrupting 3D geometry.

Highlights & Insights

• Perturbation Sensitivity as Training-Free Uncertainty Surrogate: By measuring output variance across finite-difference pose and time perturbations, the model captures local Lipschitz stability and epistemic uncertainty without requiring complex Bayesian ensembling or MC-dropout overhead.
• Velocity-Scaled Observation Deficiency Metric: Formulating observation sparsity as primitive speed divided by cumulative temporal observation frequency provides an elegant, physics-grounded mechanism to direct camera attention to fleeting dynamic events.
• Decoupled Reliability and Necessity Masks: Employing RoMa v2 matching confidence to quantify whether synthetic pixels are physically plausible, combined with Gaussian deficiency to determine whether those pixels are actually needed, establishes a principled framework for generative model distillation.

Limitations & Future Work

• Representational Capacity of Underlying 4DGS: In scenes with severe topological changes or chaotic non-rigid motion, the base 4DGS deformation field may lack sufficient expressiveness, leaving certain regions unrecoverable regardless of generated image quality.
• Diffusion Model Effective Horizon: Generative enhancement degrades when virtual cameras diverge too far beyond the diffusion model's pre-trained distribution. Sampling range must remain bounded to avoid generating unusable outputs.
• Future Directions: Integrating active selection directly with feed-forward 4D Large Reconstruction Models (4D-LRMs) and incorporating the diffusion model's own internal denoising variance into the active sampling objective.

Related Work & Insights

• vs ExploreGS [ICCV 2025]: ExploreGS focuses exclusively on static 3D scenes via global ray-angle coverage maps, which over-allocates virtual cameras to extrapolative margins in dynamic scenes. FillGS incorporates Gaussian speed and temporal counts to resolve time-varying observation voids.
• vs GS-GS [CVPR 2025] / UA-4DGS [NeurIPS 2024]: GS-GS limits virtual viewpoints to naive linear interpolation between observed cameras, failing to reach wide unobserved angles. UA-4DGS relies on static per-Gaussian uncertainty. FillGS conducts joint local-global active search coupled with RoMa v2 dense correspondence filtering.

Rating

• Novelty: ⭐⭐⭐⭐☆ [Pioneering active spatiotemporal view selection for generative-guided dynamic 4DGS]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [New sparse 2-camera benchmark splits for interpolation and extrapolation with comprehensive metric coverage]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulations, coherent motivation, and elegant design justifications]
• Value: ⭐⭐⭐⭐☆ [Offers a solid, practical blueprint for synthesizing dynamic 4D scenes under sparse camera constraints]