---

title: >- [Paper Note] Unified Panoramic–Gaussian Representation for Monocular 4D Scene Synthesis description: >- [ECCV2026][video_generation][Dynamic Gaussian Splatting] Unified Panoramic-Gaussian representation coupling spherical panoramic exploration and dynamic Gaussian splatting via fixed-geometry progressive expansion and masked distillation for monocular 4D scene extrapolation. tags: - ECCV2026 - video_generation - 3d_vision - Dynamic Gaussian Splatting - Panoramic Representation - Novel View Synthesis date: 2026-09-19 content_hash: 8267c4fc322f1a45

Unified Panoramic–Gaussian Representation for Monocular 4D Scene Synthesis

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/LogosRoboticsGroup/PanoGaussian
Area: Video Generation / 3D Vision
Keywords: Monocular 4D Scene Synthesis, Dynamic Gaussian Splatting, Panoramic Representation, Video Diffusion Priors, Unseen Region Extrapolation

TL;DR

Addressing the limitation of monocular 4D reconstruction being confined to observed view interpolation and generative priors suffering severe geometric distortion under large camera shifts, PanoGaussian presents a unified Panoramic-Gaussian representation that couples panoramic trajectory alignment, progressive view expansion with frozen geometry, and masked 4DGS distillation to synthesize physically coherent 4D scenes across unseen regions.

Background & Motivation

Reconstructing dynamic 4D scenes from monocular casual videos is foundational for immersive media and virtual reality. However, prevailing 4D reconstruction techniques—spanning dynamic Neural Radiance Fields and 4D Gaussian Splatting—are fundamentally formulated as view interpolation tasks. Consequently, these frameworks are restricted to the observed camera trajectory; navigating the camera into unseen regions beyond the original trajectory leads to missing surfaces, empty white voids, and severe structural collapse due to absent visual and geometric priors.

To hallucinate unseen areas, conditioning camera-guided video diffusion models as dynamic priors offers an intuitive path for scene extension. Yet, pure video generative models lack explicit 3D geometric awareness and are predominantly optimized under small, stochastic camera perturbations. When extrapolated across substantial camera angles, they inevitably produce catastrophic geometric tearing and perspective distortion. While panoramic coordinate systems naturally afford a continuous 360° field of view and global spatial context, projecting dynamic scenes into panoramic space severely distorts scale and non-rigid object shapes during motion, making direct panoramic representations unsuitable for physically grounded 4D dynamics.

The core tension lies in the fact that extrapolating unseen regions requires panoramic global exploration, whereas maintaining dynamic object fidelity strictly demands the rigid geometric constraints of an explicit 3D representation. Core idea: unify panoramic trajectory exploration with explicit dynamic Gaussian Splatting, using aligned panoramic protocols to guide video diffusion models in progressive unseen-region extrapolation, and distilling hallucinated content back into 4D Gaussians via masked pixel alignment to eliminate cascading errors and achieve geometry-consistent 4D scene synthesis.

Method

Overall Architecture

Taking a monocular dynamic video as input, PanoGaussian outputs an explicit dynamic 4D Gaussian field encompassing complete geometry and texture for both observed and unobserved views. The overarching framework operates across three tightly coupled stages: first, camera poses and stable depth maps extracted from a baseline 4D Gaussian backbone are unprojected into a world-coordinate point cloud, establishing an aligned origin in spherical panoramic coordinates; second, multiple tangent directions are constructed on the panoramic sphere to guide progressive trajectory expansion, where source frames are warped under frozen 4DGS geometry and inpainted by a panoramic-pretrained video diffusion model; finally, the refined novel-view videos supervise the dynamic Gaussians via masked Mean Squared Error (MSE), adaptively initializing new Gaussians and closing the loop between generative exploration and physical consistency.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Monocular Video & 4DGS Backbone"] --> B["Panoramic Trajectory Alignment<br/>Unproject point cloud to spherical domain"]
    B --> C["Fixed-Geometry Progressive View Expansion<br/>Iterative tangential warping & diffusion inpainting"]
    C --> D["Masked Dynamic Gaussian Distillation<br/>Region-wise MSE loss for 4DGS refinement"]
    D --> E["Geometry-Consistent 4D Scene Rendering"]

Key Designs

1. Panoramic Trajectory Alignment: Bridging the Generative Training-Inference Gap

Prior camera-conditioned video diffusion priors are typically trained on small, random trajectories, yet extreme view synthesis during inference requires large-angle camera shifts. This domain discrepancy causes severe generation degradation and structural drift. PanoGaussian establishes a unified panoramic trajectory protocol: during pretraining on OpenVid dynamic videos based on TrajectoryCrafter, camera trajectories are sampled along fixed panoramic elevations with random directions to apply double reprojection, conditioning the diffusion prior to internalize panoramic spatial continuity and parallax; during inference, per-frame depth \(D_t\) and camera poses \([R_t, K_t]\) are obtained from the initial 4DGS backbone (MoSca) to unproject pixels into 3D point clouds \(P_t = \Phi^{-1}([I_t, D_t], R_t, K_t)\). The point cloud centroid \(P^c = (x_c, y_c, z_c)\) serves as the panoramic sphere origin \((\phi, \theta)\), aligning the principal camera direction with the reference axis. Aligning training and inference under an identical geometric convention eliminates domain drift and ensures stable multi-view synthesis.

2. Fixed-Geometry Progressive View Expansion: Structurally Halting Error Propagation

Directly generating extreme novel viewpoints in a single step causes content fragmentation and visual hallucinations, whereas autoregressive multi-step generation compounds artifacts over time. To overcome this dilemma, PanoGaussian defines \(M=8\) uniformly distributed tangent directions on the panoramic sphere \(d_m = (\cos(2\pi m / M), \sin(2\pi m / M))\) with an angular step \(\alpha=15^\circ\) across \(E=6\) expansion steps. The accumulated rotation vector is computed as:

\[\mathbf{s}_{m,e} = \mathbf{n}\cos(e\alpha) + (\mathbf{a}_m \times \mathbf{n})\sin(e\alpha)\]

Crucially, at each expansion step \(e\), warping always samples directly from the original observed video \(V_{\text{src,inf}}\) using fixed 4DGS depth and camera poses, rather than re-warping previously hallucinated frames. Unobserved regions are identified via a binary mask \(M_e\) indicating missing ray projections. The warped intermediate video \(V_{e,\text{warped}}\) and mask \(M_e\) are fed into the video diffusion model to inpaint missing regions and recover fine visual details into \(V_{e,\text{refined}}\). Because the anchor geometry remains grounded in verified observations, hallucinated content is strictly confined to newly revealed areas, completely preventing cascading error accumulation.

3. Masked Dynamic Gaussian Distillation: Dual-Representation Closed Loop

Although panoramic video inpainting recovers unseen areas, non-rigid object motions in panoramic space suffer from local stretching and geometric distortion that violate physical dynamics. To anchor generated content in an explicit 3D structure, PanoGaussian distills refined videos into the 4D Gaussian field. Along each expanded trajectory \(P_{e,t}\), the Gaussian model renders corresponding video frames \(V_{e,\text{render}}\). Distillation is supervised strictly over newly generated regions using a masked MSE loss:

\[\mathcal{L}_{\text{refine}} = \text{MSE}(M_e \cdot V_{e,\text{refined}},\, M_e \cdot V_{e,\text{render}})\]

During optimization, new Gaussian primitives are spawned and densified in inpainted regions and jointly updated with the original video under MoSca's physical motion scaffold regularizations, requiring no noisy single-frame external depth estimation. Every 2,000 Gaussian iterations, the refined 3D geometry updates the panoramic trajectory depth guides, creating a progressive synergy where panoramic diffusion exploration and explicit Gaussian physical priors iteratively enhance one another.

Loss & Training

The video diffusion prior is trained on OpenVid-1M (\(384 \times 672\) resolution, 49 frames) by fine-tuning only the cross-attention and patch embedding weights in the Ref-DiT blocks using AdamW (\(2 \times 10^{-6}\) learning rate, 25,000 steps). During test-time scene optimization, MoSca serves as the 4DGS backbone, trained for 10,000 iterations with refinement loss weight \(\lambda_{\text{refine}} = 0.1\). Progressive panoramic expansion is invoked every 2,000 iterations. Per-scene optimization requires approximately 2.0 hours (1.4 hours for 4DGS fitting and 0.6 hours for diffusion generation), rendering at 32.42 FPS during inference.

Key Experimental Results

Main Results

The framework is evaluated on the challenging DyCheck iPhone dataset, which features long monocular sequences with significant camera motion. Beyond standard full-image metrics (PSNR, SSIM, LPIPS) and covisible metrics (mPSNR, mSSIM, mLPIPS), the authors introduce unseen-region metrics (uPSNR, uSSIM, uLPIPS) computed exclusively on regions outside training camera covisibility to assess pure extrapolation capability.

Table 1: Comparison with reconstruction-based and generation-based methods on the DyCheck iPhone dataset (Core data from original Table 1):

Method	Type	PSNR (dB) ↑	SSIM ↑	LPIPS ↓	mPSNR (dB) ↑	uPSNR (dB) ↑	uSSIM ↑	uLPIPS ↓
4D G.S.	Reconstruction	15.71	0.450	0.398	16.54	12.82	0.302	0.452
Shape-of-Motion	Reconstruction	16.79	0.510	0.391	17.32	14.84	0.321	0.448
MoSca	Reconstruction	17.31	0.573	0.354	19.32	14.23	0.295	0.461
Cat4D	Diffusion	15.91	0.427	0.398	17.39	14.06	0.351	0.428
TrajectoryCrafter	Diffusion	13.58	0.373	0.483	15.48	10.88	0.315	0.441
CogNVS	Diffusion	16.94	0.449	0.598	18.63	14.97	0.371	0.487
4DGT	Feed-forward	15.04	0.446	0.423	16.12	12.11	0.301	0.450
PanoGaussian (Ours)	Panoramic-Gaussian	18.71	0.598	0.323	19.85	16.72	0.495	0.385

On the standard Nvidia Dynamic dataset where trajectories exhibit minimal extrapolation (original Table 2), PanoGaussian achieves 26.75 dB PSNR and 0.069 LPIPS, matching MoSca (26.72 dB / 0.070) and outperforming 4D G.S. (21.45 dB / 0.199), confirming that unseen-region extrapolation does not sacrifice interpolation fidelity.

Ablation Study

Ablation studies on DyCheck isolate the impact of individual architectural designs (original Table 3), alongside an error accumulation analysis across expansion steps (original Table 4):

Table 3: Quantitative ablation on different components of the system on DyCheck dataset:

Config	PSNR (dB) ↑	SSIM ↑	LPIPS ↓	uPSNR (dB) ↑	uSSIM ↑	uLPIPS ↓	Note
Full Model (PanoGaussian)	18.71	0.598	0.323	16.72	0.495	0.385	Best across all metrics
Planar with 4DGS	16.47	0.550	0.377	14.25	0.302	0.445	Drops 2.47 dB in uPSNR without panoramic coords
Pano w/o 4DGS refinement	17.12	0.555	0.358	15.08	0.421	0.418	Lacks 3D distillation; suffers dynamic distortions
Without Panoramic Training	18.46	0.584	0.327	16.32	0.476	0.397	Domain drift causes artifacts under large angles
Without Progressive Exp.	17.51	0.576	0.351	15.22	0.446	0.417	Single-step generation yields fragmented outputs
Without Gaussian Splatting	14.67	0.392	0.445	12.61	0.314	0.482	Severe temporal inconsistency without 3D geometry

Table 4: Reconstruction quality across progressive expansion steps on DyCheck:

Expansion Step	6	12	18	24
Full PSNR / uPSNR (dB)	18.71 / 16.72	18.72 / 16.74	18.69 / 16.68	18.68 / 16.66

Key Findings

• Panoramic-Gaussian synergy is paramount: Completely stripping the 3D Gaussian representation plummets PSNR by 4.04 dB and uPSNR by 4.11 dB, proving that generative models alone cannot maintain physical structures. Conversely, replacing panoramic coordinates with planar expansion causes a 2.47 dB drop in uPSNR, demonstrating the indispensability of panoramic global context.
• Error cascading is successfully suppressed: As shown in Table 4, increasing expansion steps from 6 to 24 results in virtually stable metrics (uPSNR fluctuations \(<0.08\) dB), validating the design of warping from frozen observation geometry and isolating losses to masked unknown areas.

Highlights & Insights

• Unified Panoramic-3DGS duality: Instead of forcing 3DGS to extrapolate blindly or relying on video diffusion models without explicit 3D anchors, PanoGaussian leverages spherical coordinates as an optimal intermediary that naturally couples 360° global field of view with 3D Gaussian physical constraints.
• Masked fixed-geometry warping paradigm: Traditional autoregressive video generation accumulates visual drift when re-warping previously generated frames. Restricting inpainting strictly to masked novel regions while anchoring geometry to the original video frames provides a generalizable blueprint for long-range scene extrapolation.
• Efficient distillation overhead: By combining fast 3DGS rasterization with masked distillation, per-scene optimization finishes in ~2.0 hours (versus 4.2 hours for Shape of Motion), presenting practical feasibility for high-fidelity 4D content authoring.

Limitations & Future Work

• Performance drop on outward-facing scenes: The authors note that the framework performs best on inward-facing, object-centric setups where inter-view overlap anchors 3D geometry. In outward-facing open captures, distant backgrounds lack sufficient multi-view parallax constraints.
• Dependence on video generative prior quality: The visual quality and dynamic realism of unseen regions are bounded by the underlying diffusion model, which can occasionally inject semantic hallucinations or subtle texture flickering into the Gaussian field.
• Test-time optimization latency: Optimizing for ~2.0 hours per scene precludes interactive real-time roaming creation. Developing feed-forward 4D Gaussian transformers for panoramic extrapolation remains an open avenue.

Related Work & Insights

• vs MoSca / Shape of Motion (4D Reconstruction): While state-of-the-art dynamic reconstructors achieve sharp interpolation, they fail completely in unobserved camera directions, rendering blank voids. PanoGaussian breaks the interpolation boundary by infusing generative priors.
• vs TrajectoryCrafter / CogNVS (Video Generation): These generative models can hallucinate novel viewpoints along specified trajectories but lack multi-view spatial consistency, leading to severe geometric deformation on complex objects. PanoGaussian distills generative frames back into 4D Gaussians, enforcing rigorous physical and multi-view coherence.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Pioneering unification of panoramic trajectory protocols with dynamic Gaussian Splatting for monocular 4D scene extrapolation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Thoroughly validated on DyCheck, Nvidia, Kubric-4D, and in-the-wild videos, featuring dedicated unseen-region metrics and step-wise error analysis.
• Writing Quality: ⭐⭐⭐⭐⭐ Cohesive narrative, clear mathematical formulations, and clean conceptual flow between architecture and key designs.
• Value: ⭐⭐⭐⭐⭐ Establishes a foundational paradigm for extending monocular dynamic reconstructions to complete 360° immersive 4D virtual environments.

Related Papers

• [ECCV 2026] Schroedinger’s Cat: Probabilistic Representation and Prediction of Potential Scene Kinematics
• [ECCV 2026] CustomX: Unified Character, Action, and Scene Customization in Video World Models
• [AAAI 2026] SphereDiff: Tuning-free Omnidirectional Panoramic Image and Video Generation via Spherical Latent Representation
• [CVPR 2026] CineScene: Implicit 3D as Effective Scene Representation for Cinematic Video Generation
• [ICLR 2026] NeRV-Diffusion: Diffuse Implicit Neural Representation for Video Synthesis