REON-NVS: Real-Time Online Novel-View Synthesis from Sparse-View Videos

Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: Online Novel-View Synthesis, Dynamic Scene Reconstruction, State Space Model, Latent Pose Estimation, Sparse-View Video

TL;DR

Addressing the computational bottleneck, reliance on iterative camera pose optimization, and requirement for dense camera arrays in dynamic streaming NVS, REON-NVS introduces the first fully feedforward online framework that couples self-supervised latent pose estimation and mapping with a Bi-SSM-based temporal scene reconstructor, rendering photorealistic novel views at 32 FPS on a single A100 GPU without accumulated delay.

Background & Motivation

Reconstructing dynamic 3D scenes from multi-view video streams and synthesizing free-viewpoint video in real time is a fundamental pillar for next-generation immersive media, including virtual reality (VR), augmented reality (AR), and interactive telepresence. While Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have revolutionized static and offline dynamic scene rendering, mainstream dynamic reconstruction approaches predominantly operate in an offline, batch-optimization paradigm. They require capturing the complete video sequence beforehand and spending dozens of minutes to hours optimizing time-conditioned deformation fields or 4D representations, completely precluding real-time applications such as live streaming broadcasts or interactive virtual meetings where video frames arrive sequentially.

To enable streamable rendering, recent online dynamic frameworks—most notably 3DGStream, HiCoM, and Instant Gaussian Stream (IGS)—have introduced incremental updates to adjust 3D Gaussian parameters frame by frame. However, these methods encounter two severe obstacles in realistic sparse-view streaming. First, they rely heavily on dense camera setups (often requiring dozens of viewpoints) to anchor 3D Gaussian geometry, collapsing or yielding severe floaters when restricted to only two or three viewpoints. Second, because each arriving frame requires iterative gradient-based optimization steps, their processing speeds hover between 0.1 and 5.5 FPS—falling far short of standard 30 FPS broadcast requirements and causing compounding delays over time. Furthermore, their dependence on external Structure-from-Motion (SfM) pipelines like COLMAP introduces substantial preprocessing latency, rendering true end-to-end online deployment impossible.

The paper's angle of attack is that if explicit 3D geometry fitting and iterative pose regression are the root causes of streaming latency, one can bypass both via an end-to-end geometry-free feedforward architecture. The core idea is to eliminate iterative pose estimation and explicit geometric optimization entirely, introducing an end-to-end feedforward online NVS framework that unifies self-supervised latent pose estimation, scale-anchored pose mapping, and Bi-SSM temporal state propagation to achieve unposed, delay-free 32 FPS rendering from sparse streaming videos.

Method

Overall Architecture

The inference pipeline of REON-NVS operates in a strictly online, forward-sequential manner. At each time step \(t\), the system receives sparse multi-view video frames \(\{I_t^k\}_{k=1}^n\) (\(n=2\) in default experiments) and a user-specified physical target camera pose \(p_t^T\). The framework proceeds through two major stages: first, a lightweight feedforward pose estimator predicts 9D relative latent camera poses against an initial canonical reference frame, and a pose mapper translates the physical target pose into this latent space. Second, the scene reconstructor tokenizes the input frames and their associated Plücker ray embeddings, passes them through a Bi-SSM History Fusion Module that recurses hidden states over time to yield a coarse prediction \(I_t^H\), and finally applies a hybrid Bi-SSM/Transformer View Refinement Module to restore sharp local textures, directly outputting the final target frame \(I_t^T\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Streaming Sparse Frames & Target Pose<br/>{I_t^k} and physical pose p_t^T"] --> B["Self-Supervised Latent Pose Estimation<br/>predicts 9D latent poses relative to canonical I_r"]
    B --> C["Scale-Anchored Pose Mapping<br/>resolves scale ambiguity & maps p_t^T to latent space"]
    C --> D["Bi-SSM History Fusion<br/>recursively propagates hidden states to synthesize I_t^H"]
    D --> E["Hybrid View Refinement<br/>aligns current inputs to restore high-frequency details"]
    E --> F["Output Refined Target Novel-View Frame I_t^T<br/>fully feedforward streaming at 32 FPS"]

Key Designs

1. Self-Supervised Latent Pose Estimation: Eliminating Iterative SfM and Pose-Supervision Instability

Conventional online pipelines depend on COLMAP or per-frame numerical optimization to obtain camera poses, which introduces significant latency and fails under camera shake. Moreover, directly training neural networks to regress physical camera poses in SE(3) is inherently unstable due to non-Euclidean manifold constraints and noisy ground-truth labels. Drawing inspiration from RayZer, REON-NVS establishes the first-frame image \(I_0^1\) as a canonical reference frame \(I_r\) with an identity rotation and zero translation. For any incoming frame \(I_t^k\) at time \(t\), a lightweight feedforward pose predictor \(\mathcal{P}\) directly infers a 9D latent pose relative to \(I_r\): $$ \tilde{p}_t^k = \mathcal{P}(I_t^k, I_r) $$ comprising 3D translation and a 6D continuous rotation representation. Rather than supervising \(\mathcal{P}\) with ground-truth metric poses, the network is trained end-to-end using RGB reconstruction loss on synthesized target views. This self-supervision enables the model to organically learn a smooth, well-conditioned latent pose manifold, sidestepping label noise while operating with minimal parameters and sub-millisecond execution.

2. Scale-Anchored Pose Mapping: Bridging Latent Representations with Physical Navigation

While self-supervised latent poses enable fast feedforward synthesis, their coordinates lack metric physical meaning, whereas user interaction requires specifying novel viewpoints using metric Euclidean camera poses \(p_t^T\). Furthermore, sparse multi-view geometry inherently suffers from scale ambiguity—a large-scale distant scene cannot be distinguished from a scaled-down nearby scene based on 2D observations alone. To establish bidirectional control, REON-NVS introduces an MLP-based pose mapper \(\mathcal{M}\) conditioned on a calibrated reference pair \((\tilde{p}', p')\): $$ \tilde{p}_t^T = \mathcal{M}(p_t^T, (\tilde{p}', p')) $$ Here, the reference pair corresponds to the initial frame of the second viewpoint \(I_0^2\) relative to \(I_r\), where \(\tilde{p}' = \mathcal{P}(I_0^2, I_r)\) is extracted from the latent estimator and \(p'\) is obtained via a standard feedforward physical pose estimator (e.g., Depth Anything 3). Because \(p_t^T\) and \(p'\) share the exact same physical metric scale, conditioning the MLP on this pair grounds the transformation, allowing arbitrary smooth camera trajectories to be mapped accurately into target latent poses.

3. Bi-SSM History Fusion: \(O(1)\) Temporal State Propagation for Long-Range Memory

In sparse-view streaming, regions that are currently occluded or out-of-view from the input cameras have frequently been observed in preceding frames. However, standard Transformer temporal attention suffers from quadratic computational complexity \(\mathcal{O}(T^2)\) with respect to sequence length, forcing prior methods to use short sliding windows that sever inter-window consistency. REON-NVS addresses this by deploying a stack of \(N\) Bi-SSM blocks built upon Mamba2. At each time step \(t\), the input frames and their Plücker ray embeddings \([I_t^k, \tilde{P}_t^k]\) together with the target ray embedding \(\tilde{P}_t^T\) are tokenized into \(kn + n\) non-overlapping patch tokens.

The module employs bidirectional scanning where input tokens and target tokens are flipped in the backward branch, ensuring the target view token integrates information from all observation views. Crucially, the hidden state of each SSM block is passed sequentially across consecutive video timesteps: $$ I_t^H, H_t = \mathcal{H}\left({[I_t^k, \tilde{P}t^k]}, \tilde{P}_t^T, H\right) $$ where the initial forward hidden state at step \(t\) is initialized from the final forward token state of step \(t-1\), \(h_{1,f}^t = h_{kn+n,f}^{t-1}\). This constant-memory \(\mathcal{O}(1)\) recurrence enables REON-NVS to retain and retrieve visual information observed hundreds of frames earlier without incurring any computational overhead during streaming.

4. Hybrid View Refinement: Compensating SSM Recurrent Smoothing to Recover Crisp Edges

Although state-space models excel at temporal aggregation, recurrent state compression acts as an implicit low-pass filter over extended sequences, leading to slight blurring of high-frequency textures and edge boundaries in the coarse output \(I_t^H\). To counter this degradation, REON-NVS incorporates a View Refinement Module \(\mathcal{R}\): $$ I_t^T = \mathcal{R}\left({[I_t^k, \tilde{P}_t^k]}, [I_t^H, \tilde{P}_t^T]\right) $$ The refinement module deliberately omits temporal hidden-state propagation, focusing strictly on aligning the current multi-view observations \(\{I_t^k\}\) with the coarse prediction \(I_t^H\). Architecturally, it interleaves Bi-SSM blocks with standard Transformer self-attention blocks. Because a Bi-SSM block consumes approximately half the parameter budget of an equivalent Transformer block, this hybrid design enables building a significantly deeper network that progressively refines intermediate representations while preserving real-time throughput, generating crisp, photorealistic outputs \(I_t^T\).

Loss & Training

The framework is optimized in two progressive phases: 1. Phase 1 (Self-Supervised Pretraining): Jointly trains the pose estimator and scene reconstructor on the large-scale static RealEstate10K dataset (~80K video sequences) for 100K iterations on 8-frame clips. In this phase, the target latent pose \(\tilde{p}_t^T\) is obtained by passing the ground-truth target image into the pose estimator. 2. Phase 2 (Long-Sequence Fine-Tuning & Pose Mapper Training): Extends training sequences to 64 frames for 20K iterations on a mixture of RealEstate10K and dynamic multi-view sequences (the proposed dataset and SelfCap), bolstering dynamic motion handling. Finally, the reconstructor is frozen, and the pose mapper \(\mathcal{M}\) is trained for 20K iterations using a batch size of 64.

The full loss supervises both the coarse prediction \(I_t^H\) and the refined output \(I_t^T\): $$ \mathcal{L} = \mathcal{L}{\text{photo}}(I_t^T, I_t^{GT}) + \lambda}} \mathcal{L{\text{flicker}}(I_t^T, I_t^{GT}) $$ where \(\mathcal{L}_{\text{photo}} = \mathcal{L}_{\text{MSE}} + \lambda_{\text{per}} \mathcal{L}_{\text{per}}\) blends pixel-level MSE and VGG perceptual loss. To penalize frame-to-frame temporal inconsistency, the flicker loss measures the difference between consecutive feature representations in VGG feature space: $$ \mathcal{L})] \right|_1 $$ effectively penalizing structural temporal jittering and flickering.}} = \left| [\phi(I_t^T) - \phi(I_{t-1}^T)] - [\phi(I_t^{GT}) - \phi(I_{t-1}^{GT

Key Experimental Results

Main Results

REON-NVS is evaluated across two distinct streaming protocols: the static multi-camera Neural 3D Video dataset (6 indoor scenes, 2 input views, \(256 \times 256\) resolution) and the newly proposed Handheld Moving-Camera Dynamic Scene Benchmark (30 complex real-world scenes). All evaluations are benchmarked on a single NVIDIA A100 GPU.

Table 1: Quantitative comparison on Neural 3D Video (Paper Table 1)
Note: IGS trains on 4 scenes of Neural 3D Video; hence comparisons against IGS are conducted on the remaining two scenes (denoted as †).

Method	Pose Req.	Per-scene Init.	FPS (A100) ↑	PSNR (dB) ↑	SSIM ↑	LPIPS ↓	Flicker ↓
3DGS (per-frame)	✓	✓	0.1	19.22	0.532	0.394	85.10
3DGStream	✓	✓	0.6	20.38	0.580	0.536	14.88
HiCoM	✓	✓	0.7	19.63	0.583	0.420	13.96
DepthSplat (feedforward)	✓	✗	9.7	19.81	0.692	0.324	19.26
RayZer (unposed feedforward)	✗	✗	19.0	21.97	0.596	0.287	24.16
REON-NVS (Ours)	✗	✗	32.0	23.63	0.715	0.199	8.97
IGS†	✓	✓	5.5	21.23	0.653	0.425	6.40
REON-NVS (Ours)†	✗	✗	32.0	24.62	0.737	0.191	8.76

Table 2: Quantitative comparison on Moving-Camera Dynamic Scene Benchmark (Paper Table 2)
Evaluates resilience to moving camera rigs, rapid viewpoint shifts, and emerging dynamic backgrounds.

Method	Pose Req.	Per-scene Init.	FPS (A100) ↑	PSNR (dB) ↑	SSIM ↑	LPIPS ↓	Flicker ↓
3DGStream	✓	✓	0.6	19.52	0.491	0.501	50.24
HiCoM	✓	✓	0.7	18.19	0.441	0.599	62.88
IGS	✓	✓	5.5	19.96	0.466	0.508	62.12
DepthSplat	✓	✗	9.7	19.72	0.561	0.386	49.51
RayZer	✗	✗	19.0	21.57	0.503	0.434	52.66
REON-NVS (Ours)	✗	✗	32.0	23.26	0.617	0.274	45.45

Ablation Study

Table 3: Ablation study on the flicker loss \(\mathcal{L}_{\text{flicker}}\) across datasets (Paper Table 3)

Dataset	Variant	PSNR ↑	SSIM ↑	LPIPS ↓	Flicker ↓
Neural 3D Video	w/o \(\mathcal{L}_{\text{flicker}}\)	23.41	0.706	0.211	10.27
Neural 3D Video	w/ \(\mathcal{L}_{\text{flicker}}\) (Full)	23.63	0.715	0.199	8.97
Moving-Camera Dataset	w/o \(\mathcal{L}_{\text{flicker}}\)	23.09	0.601	0.295	46.30
Moving-Camera Dataset	w/ \(\mathcal{L}_{\text{flicker}}\) (Full)	23.26	0.617	0.274	45.45
RealEstate10K	w/o \(\mathcal{L}_{\text{flicker}}\)	26.67	0.874	0.119	20.96
RealEstate10K	w/ \(\mathcal{L}_{\text{flicker}}\) (Full)	27.22	0.882	0.112	19.91

Table 4: Reconstruction performance of history fusion under various SSM architectures (Paper Table 4, RealEstate10K)

Bidirectional Scanning	Cross-Block Direction Alternation	Token Merge Operation	PSNR (dB) ↑
✗ (Vanilla Mamba2 unidirectional)	✗	-	19.79
✓	✗	Concat	20.82
✓	✗	Add	21.60
✓	✓	Concat	21.08
✓	✓	Add (Ours)	22.04

Key Findings

1. Unprecedented Real-Time Throughput: Operating at 32 FPS on an A100 GPU, REON-NVS is the only framework that surmounts the 30 FPS broadcast threshold while performing full pose estimation and novel view rendering in a single forward pass. Prior streamable methods like 3DGStream (0.6 FPS) and HiCoM (0.7 FPS) lag significantly behind real time even when discounting their minutes-long initialization phases.
2. Robust Long-Term Historical Recall (Figure 6 & 7): In targeted retrieval experiments tracking viewpoints that revisit regions unobserved by current camera views for over 190 frames, disabling SSM temporal hidden-state propagation causes PSNR to degrade sharply from 25 dB down to 15 dB. In contrast, REON-NVS maintains a steady ~24-25 dB PSNR curve across hundreds of frames at zero extra computational cost.
3. Superior Generalization to Moving Camera Rigs: On the moving-camera benchmark, 3DGS-based streamable baselines struggle because initialized 3D Gaussians cannot accommodate newly appearing objects or shifting backgrounds, plateauing below 20 dB PSNR. REON-NVS achieves 23.26 dB PSNR (+3.3 dB over the best streamable baseline), confirming the resilience of geometry-free feedforward modeling.

Highlights & Insights

• Decoupling Self-Supervised Pose Manifolds from Metric Control: Utilizing self-supervised reconstruction loss to discover an intrinsic latent pose manifold avoids training instability on SE(3) manifolds, while an auxiliary scale-anchored MLP mapper bridges metric target poses to this manifold seamlessly.
• Pioneering SSM Recurrence in 4D Neural Video Representation: Demonstrates that Mamba2-based state-space hidden states can store 3D spatial-temporal visual context with \(O(1)\) inference cost, surmounting the sequence length limitations that force Transformers into rigid temporal sliding windows.
• Complementary Dual-Stage History/Refinement Design: Recognizes the fundamental trade-off between recurrent temporal smoothing and fine edge fidelity, solving it cleanly by dedicating the Bi-SSM module to temporal consistency and a hybrid Transformer/SSM module to current-frame textural sharpening.

Limitations & Future Work

• Artifacts under Extreme Viewpoint Extrapolation: In scenarios where the target camera view deviates substantially from the visual frustums of the two input cameras and has never been observed in historical memory, the geometry-free network may generate stretched 2D hallucinations due to the lack of an explicit 3D bounding prior.
• Finite Information Capacity of SSM Hidden States: Over thousands of streaming frames, the fixed-capacity hidden-state vectors inevitably experience gradual information decay for long-past static details, suggesting the future potential of coupling SSMs with external sparse keyframe key-value memory retrieval.
• Stereo Video Constraint: The present implementation is optimized for dual-camera streaming rigs; extending the feedforward pipeline to support monocular dynamic video or dynamic camera arrays with arbitrary view counts remains open for future exploration.

Related Work & Insights

• vs 3DGStream / HiCoM / IGS: Prior streamable 3DGS systems perform online gradient descent on Gaussian attributes for every single arriving frame, rendering them slow (<6 FPS) and vulnerable to overfitting on sparse views. REON-NVS executes purely in a feedforward pass at 32 FPS without per-scene initialization.
• vs RayZer: While RayZer also employs self-supervised latent poses, it operates on static single images and lacks temporal recurrence, suffering from pronounced frame-to-frame flickering (flicker metric up to 24-52). REON-NVS incorporates Bi-SSM state propagation and flicker regularization, boosting PSNR by +1.6~1.7 dB and cutting flicker by 15%~60%.
• vs DepthSplat: DepthSplat enforces explicit 3D Gaussian depth unprojection using ground-truth camera poses; REON-NVS demonstrates that for dynamic sparse streaming, geometry-free Plücker ray processing yields higher photorealism (23.63 dB vs. 19.81 dB) while running three times faster.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Introduces the first fully feedforward, unposed online dynamic NVS pipeline leveraging Bi-SSM state-space recurrence for temporal streaming memory.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Introduces a 150-scene real-world handheld dynamic dataset and provides comprehensive comparisons on both fixed and moving camera benchmarks alongside detailed ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Methodological progression is logically rigorous, offering compelling visual analysis and candid discussion of streaming trade-offs.
• Value: ⭐⭐⭐⭐⭐ Represents an impactful paradigm shift toward real-time, low-latency free-viewpoint video, AR telepresence, and robotics perception.

Related Papers

• [ECCV 2026] Nexels: Neurally-Textured Surfels for Real-Time Novel View Synthesis with Sparse Primitives
• [ECCV 2026] Reflection-aware generative novel view synthesis
• [ECCV 2026] RefracGS: Novel View Synthesis Through Refractive Water Surfaces with 3D Gaussian Ray Tracing
• [ECCV 2026] Do Flat Minima Improve Sparse Novel View Synthesis?
• [ECCV 2026] Predictive Photometric Uncertainty in Gaussian Splatting for Novel View Synthesis