content_hash: 5275e1ddb872bc5d

Racing in Volume with Flow Ensembles¶

Conference: ECCV 2026
Paper: ECCV Official Page
Code: https://humansensinglab.github.io/monaco4d
Area: 3D Vision
Keywords: Dynamic Scene Reconstruction, 3D Gaussian Splatting, Flow Ensemble, Kalman Filter, Extreme Motion Streaming

TL;DR¶

Addressing catastrophic tracking breakdowns in streaming 4D Gaussian reconstruction under extreme outdoor vehicle displacements, FastFlowGS fuses multi-scale correspondence ensembles via geometric uncertainty and a Kalman temporal prior, achieving the first robust streaming reconstruction on the newly released Monaco4D benchmark.

Background & Motivation¶

Streaming 4D reconstruction and novel view synthesis hold immense promise for virtual reality and immersive sports broadcasting. However, existing streaming paradigms are largely confined to controlled indoor studio setups observing human-speed motions, or forward-facing autonomous driving trajectories recorded from vehicle-mounted cameras. In high-speed outdoor motorsports such as Formula 1, race cars scream past trackside cameras at over 300 km/h, their carbon-fiber bodywork exhibiting complex specular reflections while navigating abrupt illumination transitions inside tunnels. When captured laterally at 30 Hz from fixed trackside spectator cameras, cars displace by 200 to 400 pixels between consecutive frames—an order of magnitude beyond what conventional optical flow and point tracking algorithms can reliably track.

Existing dynamic 3D representations suffer from fundamental structural failures under these extreme regimes. Canonical deformation models and deformable radiance fields presuppose small inter-frame motions or camera-dominated parallax, suffering severe tearing and smearing under large non-rigid shifts. More critically, state-of-the-art streaming Gaussian baselines (e.g., 3DGStream, HiCoM, QUEEN, and TrackerSplat) uniformly commit to a single pixel-space correspondence mechanism. When this single mechanism fails due to specular glare, textureless bodywork, or massive inter-frame displacement, positional errors propagate directly into subsequent frames. The displaced Gaussians instantly escape the foreground dynamic masks, cutting off optimization gradients and collapsing the entire reconstruction into empty primitives within 2 to 5 frames. Furthermore, available benchmarks fail to evaluate this domain: autonomous driving datasets lack external spectator parallax, while indoor captures feature slow dynamics and synthetic illumination.

This work builds on the critical insight that while no single tracking level is trustworthy under extreme displacements, cross-scale disagreements precisely pinpoint regional tracking failures. Core idea: FastFlowGS ensembled sparse feature matches, semi-dense point tracks, and dense optical flow via cross-level agreement scoring, lifted them into 3D through uncertainty-aware multi-view triangulation, and solved a closed-form information-form Kalman update to seed rapid per-frame optimization, paired with the photorealistic Unreal Engine 5 benchmark Monaco4D.

Method¶

Overall Architecture¶

FastFlowGS decomposes the dynamic environment into three independently handled layers composited during rendering: a static background, a dynamic foreground, and a sky dome. At time step \(t=0\), the background is reconstructed using Hierarchical 3DGS and its geometry is permanently frozen; subsequent frames merely finetune spherical harmonics coefficients to accommodate ambient lighting drifts. The sky is modeled via a low-resolution cubemap skybox. The dynamic foreground Gaussians are tracked, re-initialized, and rapidly optimized on a per-frame streaming basis.

To update dynamic Gaussians at frame \(t\), the system computes 2D motion fields across multiple spatial scales (sparse keypoint matches, semi-dense point tracks, and dense optical flow) and densifies non-dense signals via Gaussian-weighted Voronoi interpolation. Next, a cross-level agreement filter penalizes pixels exhibiting inter-tracker discrepancy or excessive distance from tracker observations. Using Gaussian-to-pixel rasterization alpha weights, these motion vectors are associated with individual 3D primitives. The method then performs uncertainty-aware weighted Direct Linear Transformation (DLT) triangulation independently for each tracker level, deriving both 3D position estimates and closed-form covariance matrices. Finally, a variational Kalman update merges multi-scale measurements with a constant-velocity random walk temporal prior in information form, yielding optimal position initializations and guiding pre-optimization uncertainty-driven densification before a few gradient iterations.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-View Video Streams<br/>Trackside / Onboard / Drone Views"] --> B["Scene Layer Factorization<br/>Static Background + Skybox + Dynamic Foreground"]
    B --> C["Multi-Scale Motion Field Estimation & Voronoi Densification<br/>Sparse Matches / Semi-Dense Tracks / Dense Flow"]
    C --> D["Cross-Level Motion Agreement Filtering<br/>Downweighting Discrepancies & Distant Pixels"]
    D --> E["Gaussian-to-Pixel Association<br/>Alpha Blending Weights & Flow Amplitudes"]
    E --> F["Uncertainty-Aware Multi-View Triangulation<br/>Weighted DLT Ray Intersection & Covariance Estimation"]
    F --> G["Variational Temporal Motion Fusion<br/>Information-Form Kalman Precision Update"]
    G --> H["Uncertainty-Driven Densification & Rapid Optimization<br/>Pre-Optimization Splitting & Hybrid Composition Rendering"]

Key Designs¶

1. Scene Layer Factorization and Multi-Scale Motion Field Estimation: Overcoming VRAM Ceilings and Motion Ambiguities

Full-resolution end-to-end streaming optimization of expansive outdoor racing scenes exceeds modern GPU memory limits. FastFlowGS decouples the scene into a static background (pre-optimized once via Hierarchical 3DGS with only spherical harmonics updated to capture tunnel lighting changes), a lightweight skybox, and dynamic car foregrounds. To cope with rapid 200–400 px lateral motion, the method abandons single-tracker reliance and constructs a multi-scale motion ensemble \(\mathcal{L}\) comprising sparse feature matches (LightGlue), semi-dense point tracks (CoTracker3 / Bootstap), and dense optical flow (RAFT). Sparse and semi-dense observations are propagated across the image via Gaussian-weighted Voronoi interpolation across \(K=8\) nearest centers, producing full-resolution motion fields alongside a spatial proximity confidence term \(C^{\text{vor}}\) that discounts pixels isolated from genuine observations.

2. Cross-Level Motion Agreement Scoring: Unsupervised Rejection of Tracking Breakdowns

Because trackers possess orthogonal failure modes—sparse features vanish over glossy livery while dense flow blurs around occlusions—blindly trusting any single level introduces cascading drift. Rather than imposing rigid heuristics, FastFlowGS uses inter-tracker consensus as an unsupervised proxy for geometric validity. For any tracker \(i \in \mathcal{L}\) with flow field \(\mathbf{F}_i\), the per-pixel confidence is formulated as:

\[C_i(p) = C_i^{\text{vor}}(p) \cdot \exp\left( -\frac{1}{2 \bar{F}^2} \sum_{k \in \mathcal{L}} \|\mathbf{F}_i(p) - \mathbf{F}_k(p)\|^2 \right)\]

where \(\bar{F}\) denotes the mean flow magnitude across trackers (clamped from below to avoid zero-division in static regions). When sparse, semi-dense, and dense vectors disagree in direction or magnitude, the exponential penalty sharply suppress their confidence weights, preventing corrupted 2D motion cues from contaminating 3D spatial updates.

3. Uncertainty-Aware Multi-View Triangulation: Closed-Form Covariance Derivation

To translate 2D flow vectors onto individual 3D primitives, the framework records per-pixel top-\(K\) Gaussian indices and alpha blending weights \(\alpha_{v,p,i}\), weighting each Gaussian's motion observation by its render prominence, tracker confidence, and flow magnitude. Lifted unit ray directions \(\hat{\mathbf{d}}_{v,i,l}\) and camera centers \(\mathbf{o}_v\) form a weighted Direct Linear Transformation (DLT) linear system, solved in closed form for each level \(l\) as \(\hat{\boldsymbol{\mu}}_{i,l}^{3d} = (\mathbf{A}^\top \mathbf{W} \mathbf{A})^{-1}\mathbf{A}^\top \mathbf{W}\mathbf{b}\). Rays forming an angle \(< 25^\circ\) with the motion vector are pruned to prevent axial ambiguity. Crucially, FastFlowGS derives an explicit measurement covariance matrix from the reprojection residuals:

\[\hat{\boldsymbol{\Sigma}}_{i,l} = \hat{\sigma}_{i,l}^2 (\mathbf{A}^\top \mathbf{W} \mathbf{A})^{-1}, \quad \hat{\sigma}_{i,l}^2 = \frac{\|\mathbf{W}[\mathbf{A}\hat{\boldsymbol{\mu}}_{i,l}^{3d} - \mathbf{b}]\|_2^2}{N_i - 3}\]

Large eigenvalues immediately flag degenerate multi-view geometries (near-parallel baseline rays, scarce camera coverage, or noisy flow), directly parameterizing measurement uncertainty in the fusion step.

4. Variational Temporal Motion Fusion and Uncertainty-Driven Densification: Information-Filter Initializations

Standard streaming Gaussians rely on iterative photometric gradient descent, which fails completely when large shifts throw primitives out of their gradient attraction basin. FastFlowGS unifies temporal trajectory continuity and multi-level triangulations within a global variational energy minimization objective, leading to an analytical information-form Kalman update:

\[\boldsymbol{\mu}_{i,t}^{3d} = \boldsymbol{\Sigma}_{i,t} \left[ \mathbf{P}_{i,t|t-1}^{-1} \boldsymbol{\mu}_{i,t-1}^{3d} + \sum_{l \in \mathcal{L}} \hat{\boldsymbol{\Sigma}}_{i,l}^{-1} \hat{\boldsymbol{\mu}}_{i,l}^{3d} \right], \quad \boldsymbol{\Sigma}_{i,t}^{-1} = \mathbf{P}_{i,t|t-1}^{-1} + \sum_{l \in \mathcal{L}} \hat{\boldsymbol{\Sigma}}_{i,l}^{-1}\]

The random walk process covariance \(\mathbf{Q}_t\) dynamically adapts to the median 3D displacement of successfully triangulated Gaussians: fast frames expand process noise to let reliable measurements dominate, whereas slow frames tighten the prior. Completely unobserved Gaussians seamlessly drift along their temporal prior with growing uncertainty. In contrast to conventional schemes that wait for multi-step rendering losses to trigger densification, FastFlowGS identifies Gaussians with large covariance \(\boldsymbol{\Sigma}_{i,t}\) before optimization starts, immediately splitting or cloning them to front-load capacity where multi-view geometry is weakest.

Key Experimental Results¶

Main Results¶

Evaluations are conducted on the public indoor CMU-Panoptic benchmark (standard frame rate and 5× temporal subsampling to simulate violent motions) and the newly curated Monaco4D outdoor benchmark. Under Monaco4D's full-quality protocol (8M static background Gaussians), all prior streaming baselines collapse by frame 2–5, leaving 3DGS-Base as the primary point of comparison. Under the reduced-memory protocol (1M background Gaussians), FastFlowGS is compared against all streaming baselines.

Dataset / Setting	Method	VMAF ↑	PSNR ↑	M-PSNR ↑	Time (s/frame) ↓	VE ↑
CMU-Panoptic (Native Rate)	D-3DGS	45.45	25.981	24.070	120.54	0.38
CMU-Panoptic (Native Rate)	3DGStream	48.39	27.189	21.563	6.75	7.17
CMU-Panoptic (Native Rate)	HiCoM	53.93	27.033	23.784	5.89	9.16
CMU-Panoptic (Native Rate)	QUEEN	49.21	27.046	25.006	8.81	5.59
CMU-Panoptic (Native Rate)	TrackerSplat	46.24	24.719	22.536	14.14	3.27
CMU-Panoptic (Native Rate)	Ours (FastFlowGS)	60.71	27.893	26.141	4.00	15.18
CMU-Panoptic (Native Rate)	Ours-faster	55.79	27.262	25.722	3.00	27.90
CMU-Panoptic (5× Skip)	HiCoM	36.66	21.105	13.160	5.89	6.22
CMU-Panoptic (5× Skip)	QUEEN	51.16	26.448	25.390	8.81	5.81
CMU-Panoptic (5× Skip)	Ours (FastFlowGS)	50.66	27.571	27.190	4.00	12.67
Monaco4D (Fairmont Full)	3DGS-Base	41.87	21.060	16.400	11.82	3.54
Monaco4D (Fairmont Full)	Ours (FastFlowGS)	43.93	21.530	18.880	8.47	5.19
Monaco4D (Main Straight)	3DGS-Base	55.89	21.430	15.990	8.51	6.57
Monaco4D (Main Straight)	Ours (FastFlowGS)	56.24	22.220	18.960	7.11	7.91

Under the reduced-memory protocol on Monaco4D, baselines HiCoM, ReCon-GS, and TrackerSplat suffer complete tracking failures, dropping below 3 VMAF across multiple tracks. In contrast, FastFlowGS consistently leads in VMAF and VMAF Efficiency (VE) across all tracks: Fairmont (VMAF 46.89, VE 1.88), Main Straight (VMAF 50.10, VE 2.39), Rascasse (VMAF 53.84, VE 2.34), Tunnel (VMAF 48.90, VE 2.57), Uphill (VMAF 55.84, VE 2.07), and Pool (VMAF 51.87, VE 2.36).

Ablation Study¶

Component ablations evaluate the individual impact of sparse matches, dense flow, the cross-level disagreement filter, and the Kalman temporal prior on CMU-Panoptic (Football) and Monaco4D (Fairmont hairpin, 5 cars). #iters indicates the number of per-frame optimization steps needed to cross target PSNR thresholds (≥ 24 dB on CMU, ≥ 18.5 dB on Monaco4D).

Configuration (Sparse / Dense / Disagree / Kalman)	CMU VMAF ↑	CMU M-PSNR ↑	CMU #iters ↓	Monaco VMAF ↑	Monaco M-PSNR ↑	Monaco #iters ↓	Note
Only Sparse (✓ / ✗ / ✗ / ✗)	46.14	25.31	350	39.33	18.65	1000	Fails on specular and textureless car livery
Only Dense (✗ / ✓ / ✗ / ✗)	45.32	25.29	320	40.76	18.64	940	Robust outdoors but prone to motion blur
Sparse + Dense (✓ / ✓ / ✗ / ✗)	46.49	25.37	300	40.84	18.71	840	Multi-scale complementary coverage
+ Disagreement Filter (✓ / ✓ / ✓ / ✗)	46.82	25.43	260	41.17	18.76	760	Accelerates convergence, cuts 80 iters on Monaco
Full Model (FastFlowGS) (✓ / ✓ / ✓ / ✓)	49.37	25.58	200	43.93	18.88	700	Temporal prior unlocks maximal fidelity and speed

Key Findings¶

Initialization quality dictates streaming viability: Streaming failures under extreme displacements stem from primitives falling outside the dynamic mask and severing gradient flow, rather than optimization iteration budgets. 3DGS-Base smears vehicle surfaces, scoring only 16.40 dB M-PSNR on Fairmont, whereas FastFlowGS reaches 18.88 dB (+15.1% dynamic PSNR improvement).
Kalman temporal prior delivers the largest singular boost: Adding the Kalman prior provides the biggest quality jump across both datasets (+2.76 VMAF on Monaco4D, +2.55 on CMU) while trimming required convergence iterations from 760 to 700 on Monaco4D and 260 to 200 on CMU. The prior confines the parameter search space to smooth physical trajectories.
Disagreement scoring acts as an essential outdoor noise gate: Outdoor specularities and tunnel transitions generate erroneous optical flow vectors. Filtering inter-level disagreements reduces Monaco4D convergence iterations from 840 to 760, verifying the necessity of discarding unreliable correspondences prior to optimization.

Highlights & Insights¶

Multi-scale consensus as unsupervised reliability verification: Instead of committing to a single point tracker or optical flow estimator, the cross-level agreement metric turns inter-tracker inconsistency into an explicit confidence penalty, cleanly removing tracking errors without requiring ground truth.
Direct linkage from DLT residuals to information Kalman updates: Deriving closed-form 3D covariance matrices from multi-view ray intersections allows measurements to weight themselves naturally in an information filter. Confident rays drive accurate updates, while occluded regions rely safely on temporal propagation.
Uncertainty-driven pre-optimization densification: Replacing traditional post-hoc gradient-based densification with an uncertainty-guided heuristic allows the model to split high-covariance Gaussians before backpropagation begins, dramatically accelerating single-frame convergence.

Limitations & Future Work¶

Omission of 3D rotations and scales in motion models: The current formulation updates Gaussian centroids \(\boldsymbol{\mu}\) but leaves orientation quaternions and scale vectors untouched. Rapid yaw rotations through sharp hairpins require standard gradient descent to reconstruct geometry from scratch.
Overconfident covariance under shared pixel evidence: The variational framework assumes statistical independence between tracking levels, despite sparse and dense trackers sharing underlying RGB pixels. The resulting aggregated precision \(\boldsymbol{\Sigma}_{i,t}^{-1}\) can become overconfident, reducing the optimizer's error-correcting flexibility.
Heavy offline point tracking preprocessing: While online Gaussian optimization requires only 3–8 seconds per frame, extracting multi-view CoTracker point tracks requires approximately 727 seconds per frame offline, remaining a bottleneck for real-time live broadcasting pipelines.

vs TrackerSplat: TrackerSplat relies exclusively on 2D point trackers to position Gaussians before per-frame fine-tuning. FastFlowGS enriches this concept into a multi-scale ensemble of sparse matches, semi-dense tracks, and optical flow, fortified by DLT covariance modeling and Kalman variational updates to avoid catastrophic single-tracker loss.
vs HiCoM / 3DGStream / QUEEN: These streaming methods encode deformations via spatial hash grids or quantized features. Because grid cell resolutions impose an upper bound on representable motion, they derail under large 200–400 px displacements. FastFlowGS operates on continuous 3D ray geometry, bypassing spatial discretization ceilings.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Pioneering streaming 4D Gaussian reconstruction under extreme large-displacement outdoor sports dynamics, supported by the comprehensive Monaco4D benchmark.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluations across CMU-Panoptic and Monaco4D (six track sequences, full and reduced memory protocols, ablation breakdowns, and iteration budgets).
Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear mathematical formulation, compelling problem motivation, and insightful analyses of baseline failure modes.
Value: ⭐⭐⭐⭐⭐ Sets a new standard for high-speed dynamic view synthesis and real-time immersive sports broadcasting.