PASTEL: Panoramic Alignment for Monocular 4D Scene Reconstruction¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/LogosRoboticsGroup/PASTEL
Area: 3D Vision
Keywords: Monocular 4D Reconstruction, Panoramic Alignment, Generative Priors, Trajectory Planning, Gaussian Splatting

TL;DR¶

PASTEL projects monocular video into a 2D spherical panoramic space to transform unconstrained 6-DoF trajectory exploration into a structured 2D directional search, combining static-dynamic point aggregation with strategic confidence masking to reliably extrapolate unobservable 4D dynamic scenes beyond camera boundaries.

Background & Motivation¶

Reconstructing geometry-consistent and motion-coherent 4D scenes from casually captured monocular video is fundamental for immersive virtual reality and embodied AI applications. While recent 4D Gaussian Splatting (4DGS) and dynamic neural radiance field approaches have achieved remarkable novel view synthesis under multi-view camera setups, they degrade substantially on monocular video inputs due to restricted fields of view and heavy self-occlusions. Pure reconstruction frameworks rely strictly on observed pixel-level supervision and cannot extrapolate scene geometry and texture into invisible regions beyond visible camera frustums, causing severe missing content and boundary collapse when rendering viewpoints outside the original trajectory.

An intuitive approach to extrapolate these invisible regions is incorporating camera-controlled video diffusion models as generative priors. However, the stochastic nature of diffusion denoising inherently causes severe high-frequency texture flicker, semantic drift, and color inconsistencies across synthetic viewpoints. Directly injecting unconstrained generative outputs into 4D reconstruction introduces pervasive artifacts and geometric ghosts. Crucially, the farther an extrapolated camera trajectory deviates from the captured video, the more unobserved content the diffusion prior must hallucinate, compounding geometric and photometric corruption. As a result, effective camera trajectories must strictly balance two competing objectives: maximizing spatial exploration beyond observable boundaries while minimizing viewpoint deviation and overlap.

Searching for an optimal sequence of $L$ camera poses across $M$ trajectories in unconstrained 3D Cartesian space requires navigating an intractable parameter space of $M \times L \times 4 \times 4$ values, which inevitably gets trapped in disjointed or suboptimal viewpoints. Core idea: reformulate scene exploration by anchoring it within a spherical 2D panoramic space, reducing the 6-DoF Cartesian search to a continuous 2D directional expansion with explicit visibility boundaries, leveraging static-dynamic warped guidance for diffusion priors, and strategically supervising 4D reconstruction through dual invisible and unreliable confidence masks.

Method¶

Overall Architecture¶

PASTEL transforms the unconstrained exploration of 3D camera cones into deterministic trajectory planning and distillation anchored in a spherical panoramic domain. The overall pipeline consists of four sequential stages: monocular point cloud back-projection and panoramic coordinate transformation, adaptive outward trajectory planning with safe depth margins, static-dynamic decoupled view expansion for generative guidance, and strategic pseudo-supervision using confidence masks. Given a monocular video with estimated depth, poses, and dynamic flow masks, PASTEL produces an unbounded, globally consistent 4D Motion Scaffold representation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Monocular Video Input<br/>RGB frames + Depth + Camera Poses + Dynamic Masks"] --> B["Panoramic Scene Alignment & Spherical Center Filtering<br/>Back-project to 3D point cloud, filter percentiles, and project to 2D panorama"]
    B --> C["Adaptive Outward Trajectory Identification<br/>2D panoramic ray-casting + Depth constraint R + SLERP interpolation"]
    C --> D["Static-Dynamic Decoupled View Expansion<br/>Temporal static cloud aggregation and dynamic per-frame warping"]
    D --> E["Video Diffusion Prior Refinement<br/>Constrained denoising guided by warped videos to synthesize candidate views"]
    E --> F["Strategic Confidence Supervision<br/>Extract invisible masks and low-SSIM unreliable masks for targeted 4DGS loss"]
    F --> G["High-Fidelity Unbounded 4D Scene Output<br/>Consistent 3D/4D Gaussian Splatting rendering"]

Key Designs¶

1. Panoramic Scene Alignment & Spherical Center Filtering: Dimensionality Reduction from Cartesian to 2D Spherical Domain

Conventional trajectory search operates in unconstrained 3D Cartesian coordinates, optimizing high-dimensional 6-DoF extrinsics where visibility boundaries are fragmented. PASTEL eliminates this complexity by back-projecting all $T$ frames of monocular video into 3D world space using precomputed metric depth and camera poses. To prevent foreground moving objects from distorting the reference center, the algorithm filters out the bottom 30% of depth points before calculating the mean coordinate of the remaining points to establish a robust spherical center $P_c = (x_c, y_c, z_c)$.

Every 3D point $p_t^j = (x_t^j, y_t^j, z_t^j)$ is subsequently mapped to polar coordinates on the unit sphere centered at $P_c$: $$\theta_t^j = \operatorname{atan2}(y_t^j - y_c, x_t^j - x_c), \quad \phi_t^j = \arcsin\left(\frac{z_t^j - z_c}{\|p_t^j - P_c\|_2}\right)$$ which are scaled to pixel coordinates $(u_t^j, v_t^j)$ on a $w_p \times h_p$ equirectangular panoramic image. Each video frame's field of view forms a coherent connected point set $\mathcal{M}_t$ in the panoramic domain. This projection reduces 6-DoF camera placement into a structured 2D directional search where coverage boundaries become globally quantifiable and visually continuous.

2. Adaptive Outward Trajectory Identification: Directional Ray-Casting with Safe Depth Constraints

Within the 2D panorama, PASTEL defines $M$ uniformly spaced directional unit vectors $d_m = (\cos(2\pi m / M), \sin(2\pi m / M))$ to guarantee omnidirectional expansion with minimal trajectory overlap. To initialize an extrapolation trajectory in direction $d_m$, the algorithm identifies a starting camera $c_m$ positioned at the outer boundary of current visual coverage. To prevent trajectories from initiating inside foreground dynamic objects—which would induce severe occlusion or projection singularities—PASTEL enforces a valid search radius constraint $R = \operatorname{median}(\|p - P_c\|_2) \cdot \tan(\alpha)$ based on median scene depth and angular expansion step $\alpha$.

Within the valid spatial region $\Omega_R$, the starting camera is chosen by maximizing the projection inner product along direction $d_m$: $$c_m = \arg\max_{t} \max_{j \in \Omega_R} \left[ \left(u_t^j - \frac{w_p}{2}\right) d_{m,u} + \left(v_t^j - \frac{h_p}{2}\right) d_{m,v} \right]$$ The camera position is then advanced along $d_m$ in the panoramic space and mapped back into 3D Cartesian space, with its orientation oriented directly toward spherical center $P_c$. Smooth camera trajectories of $L$ frames are constructed by interpolating between the novel camera $(\tilde{R}_{t,m}, \tilde{\mathbf{t}}_{t,m})$ and the captured camera $(R_t, \mathbf{t}_t)$ using Spherical Linear Interpolation (SLERP) for rotations and linear interpolation for translations.

3. Static-Dynamic Decoupled View Expansion & Strategic Confidence Supervision: Isolating Diffusion Hallucinations

Even with smooth trajectories, feeding blank or sparse views to diffusion models causes temporal background flickering. PASTEL introduces a decoupled projection mechanism: optical flow dynamic masks separate the static point cloud $\mathcal{P}_t^{\text{static}}$ from dynamic foreground elements. Static points across all timestamps are merged into an aggregated global point cloud $\bigcup_{t=1}^T \mathcal{P}_t^{\text{static}}$, which is rendered into the novel trajectory views as a globally consistent background image $I_t^{\text{static}}$. Dynamic points are projected per-frame to form $I_t^{\text{moving}}$, yielding a dense warped video sequence $I_t^{\text{warped}} = I_t^{\text{static}} \cup I_t^{\text{moving}}$. This provides a rigid geometric conditioning anchor for the video generative model (TrajectoryCrafter) to synthesize refined candidate frames $\tilde{V}'$.

To prevent generative hallucinations from degrading high-quality reconstructed regions, PASTEL formulates a dual-mask strategic supervision mechanism: - Invisible Mask $M_t^{\text{invisible}}$: Identifies unobserved regions where warped pixel values are invalid (-1); - Unreliable Mask $M_t^{\text{unreliable}}$: Detects degraded or occluded regions where the Structural Similarity Index between Gaussian Splatting rendering and geometric warping falls below threshold $\epsilon$: $\operatorname{SSIM}(I_t^{\text{render}}, I_t^{\text{warped}}) < \epsilon$ (empirically set to $\epsilon = 0.3$).

The refinement loss is strictly confined to the union mask $M_t^{\text{refine}} = M_t^{\text{invisible}} \cup M_t^{\text{unreliable}}$: $$\mathcal{L}_{\text{refine}} = \mathcal{L}_{\text{rgb}}(M_t^{\text{refine}} \cdot I_t^{\text{render}}, M_t^{\text{refine}} \cdot I_t^{\text{refined}})$$ This ensures the diffusion prior acts exclusively as an inpainting agent in genuine blind spots without corrupting native monocular video observations.

Loss & Training¶

PASTEL uses MoSca (Motion Scaffolds) as its 4D representation and is optimized for 6000 steps. The primary reconstruction loss is computed over captured video views using standard photometric and geometric regularization, while $\mathcal{L}_{\text{refine}}$ supervises synthesized camera trajectories. Preprocessing (UniDepth depth estimation, BootsTAPIR point tracking, RAFT optical flow) requires approximately 5 minutes. Trajectory expansion and TrajectoryCrafter generation take 10 minutes, and 4DGS optimization takes about 1.5 hours, achieving a rendering speed of 38 FPS.

Key Experimental Results¶

Main Results¶

Quantitative evaluations are conducted on the DyCheck IPhone benchmark, which features challenging monocular camera motions and provides covisibility masks. In addition to standard covisibility masked metrics (mPSNR, mSSIM, mLPIPS), evaluation is performed over full images (PSNR, SSIM, LPIPS) to measure scene completion beyond visible frustums.

Results from Table 1 of the paper are detailed below (Rec. denotes reconstruction time in hours; FPS denotes rendering speed):

Method	Paradigm	PSNR↑	SSIM↑	LPIPS↓	mPSNR↑	mSSIM↑	mLPIPS↓	Rec. (h)	FPS
4D GS	Pure Reconstruction	15.71	0.450	0.398	16.54	0.594	0.347	1.2	44
Shape-of-Motion	Pure Reconstruction	16.79	0.510	0.391	17.32	0.598	0.296	2.0	40
MoSca	Pure Reconstruction	17.33	0.572	0.355	19.32	0.706	0.264	1.3	38
USPLAT4D	Pure Reconstruction	17.63	0.583	0.341	19.63	0.716	0.250	-	-
Cat4D	Generative Prior	15.91	0.427	0.398	17.39	0.607	0.341	-	-
TrajectoryCrafter	Pure Generative	12.71	0.324	0.519	14.24	0.417	0.519	-	-
CogNVS	Generative Prior	15.42	0.362	0.689	16.94	0.449	0.598	-	-
PASTEL (Ours)	Panoramic Alignment + Prior	18.54	0.582	0.335	19.75	0.739	0.247	1.5	38

On the Kubric-4D benchmark featuring large camera displacements (Table 2 of the paper), PASTEL achieves a state-of-the-art full-image PSNR of 22.75 dB (surpassing CogNVS at 22.63 dB and TrajectoryCrafter at 20.93 dB) and reduces full-image LPIPS to 0.212 (compared to CogNVS at 0.232), demonstrating reliable view expansion under extreme camera motions.

Ablation Study¶

Table 3 of the paper systematically isolates the contribution of each core component on the DyCheck dataset:

Config	PSNR↑	SSIM↑	LPIPS↓	mPSNR↑	mSSIM↑	mLPIPS↓	Note
Full model	18.54	0.582	0.335	19.75	0.739	0.247	Best performance across all metrics
w/o panorama (linear)	15.73	0.497	0.370	19.42	0.728	0.249	Linear trajectory extension; full PSNR drops 2.81 dB
Cartesian-TopK	17.31	0.548	0.352	19.55	0.733	0.251	Top-K coverage from 100 random 3D paths still lags by 1.23 dB
w/o trajectory alignment	17.87	0.572	0.342	19.57	0.736	0.247	Abrupt viewpoint jumps degrade diffusion conditioning
w/o view expansion	17.44	0.564	0.357	19.45	0.733	0.250	Missing multi-frame static background aggregation
w/o strategic supervision	14.09	0.306	0.548	14.80	0.506	0.445	Unconstrained diffusion prior collapses full PSNR by 4.45 dB

Key Findings¶

Strategic supervision prevents catastrophic model degradation: Eliminating the confidence masks causes a catastrophic collapse across all metrics (PSNR plummets by 4.45 dB to 14.09 dB, and LPIPS worsens to 0.548). Without masking, diffusion noise and dataset color biases overwrite genuine high-fidelity observations.
Spherical panorama fundamentally outperforms Cartesian trajectory heuristics: Selecting the best coverage trajectories from 100 random 3D paths (Cartesian-TopK) still trails the panoramic formulation by 1.23 dB in full-image PSNR, while naive linear extrapolation drops by 2.81 dB. The continuous 2D panoramic space guarantees uniform outward exploration without erratic cross-view occlusions.

Highlights & Insights¶

Dimensionality Reduction for Trajectory Planning: Replacing unconstrained $M \times L \times 4 \times 4$ Cartesian extrinsic optimization with 2D panoramic ray-casting elegantly converts an ill-posed 3D search into an interpretable boundary extrapolation problem.
Decoupled Point-Cloud Warping as Diffusion Anchors: Merging static background point clouds across all timestamps creates a wide-baseline physical anchor for generative models, drastically suppressing generative hallucination.
Dual Structural Confidence Masking: Combining SSIM structural deviation with geometric visibility masks establishes a principled firewall that confines generative priors strictly to unobserved and degraded regions.

Limitations & Future Work¶

Reliance on Upstream Estimators: Reconstructing 3D points relies on monocular depth (UniDepth) and pose estimation (BootsTAPIR). Severe depth scale drift or tracking loss in low-texture scenes can distort panoramic reprojection.
Extrapolation Depth Bound: Because novel cameras focus toward the scene's spherical center, PASTEL specializes in outward orbiting expansions. Penetrating deeply into fully enclosed interior spaces remains challenging.
Multi-Stage Offline Latency: Generating warped diffusion guidance and optimizing 4D Gaussians takes ~1.5 hours per scene, preventing real-time interactive generation. Future work could explore end-to-end feed-forward 4D distillation.

vs MoSca [16] / USPLAT4D [8]: Top-tier monocular 4DGS methods that excel inside the training field of view but leave severe empty voids outside covisibility cones. PASTEL augments MoSca with panoramic prior guidance, improving full-image PSNR by 1.21 dB over MoSca and 0.91 dB over USPLAT4D.
vs Cat4D [31] / CogNVS [4] / TrajectoryCrafter [35]: Generative novel view synthesis models produce plausible hallucinations but lack rigorous 4D physical and multi-view geometric consistency. PASTEL constrains the diffusion model within warped geometric boundaries and confidence masks, combining generative creativity with geometric stability.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Elegant reformulation of monocular 4D camera expansion into a 2D spherical panoramic space]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Thorough benchmarking across DyCheck, Kubric-4D, and challenging dynamic occlusions with comprehensive ablations]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulation, cohesive narrative, and well-designed comparative figures]
Value: ⭐⭐⭐⭐⭐ [Pioneering paradigm for integrating video generative priors into dynamic 3D/4D scene reconstruction]