Node-RF: Learning Generalized Continuous Space-Time Scene Dynamics with Neural ODE-based NeRFs

Conference: CVPR 2025
arXiv: 2603.12078
Code: Will be open-sourced
Area: 3D Vision
Keywords: Neural ODE, Dynamic NeRF, Continuous Time Modeling, Temporal Extrapolation, Trajectory Generalization

TL;DR

This paper proposes Node-RF, which tightly couples Neural ODEs with dynamic NeRFs by modeling continuous-time scene dynamics through the ODE evolution of latent vectors. This enables long-term temporal extrapolation beyond the training sequence and generalization across trajectories without requiring optical flow or depth supervision.

Background & Motivation

Background: Dynamic NeRFs (D-NeRF, HyperNeRF, etc.) model 4D scenes via deformation fields or frame-conditioned latent codes, but are inherently discrete and frame-specific.

Limitations of Prior Work: (a) They only support interpolation near training timestamps and lack a mechanism for long-range extrapolation; (b) deformation fields or latent codes are sequence-specific and cannot generalize to motion patterns under different initial conditions.

Key Challenge: Existing methods "memorize" observed frames rather than "understanding" the underlying dynamics—necessitating a transition from discrete frame representations to continuous-time dynamic modeling.

Goal: (1) Continuous-time scene extrapolation—predicting arbitrary time points beyond the training sequence; (2) trajectory generalization—learning from multiple sequences sharing the same dynamics and generalizing to unseen initial conditions.

Key Insight: Neural ODEs naturally model continuous-time evolution, and their differential equation formulation allows for smooth, consistent long-term predictions.

Core Idea: Leveraging Neural ODEs to drive the continuous temporal evolution of NeRF's latent states, thereby achieving extrapolation and generalization.

Method

Overall Architecture

The input consists of multi-view dynamic image sequences. The model learns an ODE model \(f_\theta\) that evolves a latent vector \(z_t\) over time, and a NeRF renderer \(F_\Theta\) decodes \(z_t\) into a 3D scene at each time point. Training is end-to-end using only photometric loss. Two modes are supported: single-sequence continuous dynamics (extrapolation) and multi-sequence generalization learning.

Key Designs

1. ODE-Driven Temporal Evolution:

   • Function: Modeling the continuous-time evolution of latent scene states using Neural ODEs.
   • Mechanism: \(z_{t_0},...,z_{t_N} = \text{ODESolve}(f_\theta, z_{t_0}, (t_0,...,t_N))\), where \(f_\theta\) is a learnable differential equation. For single sequences, an ODE-RNN VAE is used to warm-start and learn the initial state from the first two frames.
   • Design Motivation: ODEs provide continuous and smooth temporal evolution unrestricted by discrete frames, in principle supporting query and extrapolation at any time step.

2. NeRF Spatial Decoding:

   • Function: Decoding the latent vector obtained from ODE evolution into a 3D scene.
   • Mechanism: \(F_\Theta(\mathbf{x}, \mathbf{d}, z_{t_i}) = (\mathbf{c}, \sigma)\), utilizing a standard NeRF volume rendering pipeline conditioned on the time-varying latent vector.
   • Design Motivation: Decoupling temporal modeling (ODE) from spatial modeling (NeRF) to allow each to focus on its respective task.

3. Multi-Sequence Generalization Learning:

   • Function: Learning a generalizable dynamics model from multiple sequences that share the same dynamics but have different initial conditions.
   • Mechanism: Learning a canonical latent code \(z_{can}\). The initial conditions (pose \(p_0^c\), velocity \(v_0^c\)) are encoded using an MLP and concatenated with \(z_{can}\) to serve as the input to the ODE, predicting evolution under different initial conditions. An additional pose/velocity decoder provides auxiliary supervision.
   • Design Motivation: A shared dynamics model can extract abstract motion laws from multiple trajectories.

Loss & Training

The total loss is formulated as \(\mathcal{L} = \mathcal{L}_{NeRF} + 10^{-2}(\mathcal{L}_p + \mathcal{L}_v) + 10^{-22}\mathcal{L}_{lipschitz}\). Lipschitz regularization constrains the smoothness of NeRF layer weights, promoting a more structured latent space. The model uses a 512-dimensional latent code and dopri5/Euler ODE solvers. In the single-sequence mode, the first two frames are encoded using an ODE-RNN VAE to obtain the initial state \(z_{t_0}\), which then evolves freely via the ODE. In the multi-sequence mode, initial poses and velocities are encoded by an MLP and concatenated with the canonical latent code. The pose/velocity decoder provides auxiliary supervision to help the latent space encode physical quantities. Gradient clipping is applied to the ODE solver during training to prevent explosion.

Key Experimental Results

Main Results (4× Long-Range Extrapolation, Bouncing Balls)

Method	X-CLIP Sim↑	LLaVA-Video Sim↑	Motion Smoothness↑	Object Consistency↑
D-NeRF	0.1691	0.7807	0.9947	0.9735
4D-GS	0.1484	0.7230	0.9954	0.9259
HexPlane	0.1732	0.6673	0.9962	0.7741
TiNeuVox	0.1773	0.7883	0.9947	0.9643
MotionGS	0.1760	0.7693	0.9947	0.9756
Node-RF	0.1775	0.7937	0.9965	0.9778

Pendulum Dataset

Method	Interpolation SSIM↑	Interpolation PSNR↑	Extrapolation SSIM↑	Extrapolation PSNR↑
SimVP	—	—	0.617	15.804
D-NeRF	0.437	13.906	0.426	13.295
4D-GS	0.455	13.391	0.463	12.940
Node-RF	0.531	17.057	0.469	15.920

Multi-Sequence Generalization (IoU)

Method	Oscillating Ball	Bifurcating Hill
Vid-ODE	—	0.000
SimVP	—	0.295
D-NeRF(c)	0.0008	0.003
Node-RF	0.3327	0.485

Ablation Study

Setting	SSIM↑	LPIPS↓	PSNR↑	IoU↑	Note
\(\mathcal{L}_{NeRF}\) only	0.630	0.492	28.66	0.273	Generalizes even with only photometric loss
\(+\mathcal{L}_p +\mathcal{L}_v\)	0.661	0.440	29.08	0.325	Auxiliary supervision improves dynamics
\(+\mathcal{L}_{lipschitz}\) (Full)	0.662	0.436	29.09	0.333	Regularization improves latent space structure

Key Findings

• Node-RF achieves extrapolation up to 4× the training sequence length, whereas methods like D-NeRF degrade rapidly beyond training timestamps, exhibiting severe geometric collapse and object disappearance.
• Multi-sequence training enables the model to generalize to completely unseen initial conditions—validated on systems from bouncing balls to pendulums, demonstrating that it learns abstract physical laws rather than merely memorizing training trajectories.
• The learned latent embeddings can be utilized for dynamical system analysis—enabling the identification of critical points without needing an explicit model.
• Lipschitz regularization is crucial for extrapolation stability: it constrains the sensitivity of NeRF outputs to minor shifts in the latent code, thereby preventing numerical instability during long-range extrapolation.
• The dopri5 solver offers more stable extrapolation compared to Euler's method, but at a higher computational expense; practical applications require a trade-off between accuracy and speed.

Highlights & Insights

• The coupling of ODE and NeRF is elegant: the ODE handles "temporal reasoning" while NeRF manages "spatial rendering," each acting according to its strength yet fully trainable end-to-end.
• Trajectory generalization is a genuine contribution: instead of just reconstructing observed sequences, the model predicts unseen trajectories after learning abstract dynamic laws.
• Latent embeddings can be used for system analysis—going beyond rendering itself, demonstrating potential for scientific discovery.

Limitations & Future Work

• Validated only on synthetic or simple scenes (bouncing balls, pendulums, steak); its applicability to complex real-world scenes remains unknown.
• Multi-sequence generalization requires the assumption of identical dynamics—can it function when different types of motion are mixed?
• Based on NeRF (not 3DGS), which limits rendering speed and makes real-time interaction difficult to achieve.
• Requires multi-view inputs; performance under monocular scenarios has not been validated.
• The computational cost of the ODE solver scales linearly with the number of sequences, and scalability to 100+ sequences has not been verified.
• Long training time: Bouncing Balls takes 18h, Pendulum takes 24h, and multi-sequence generalization experiments (Oscillating Ball/Bifurcating Hill) each require approximately 72h.
• Sensitive to nODE depth: 3 layers are optimal, while performance drops with 5 and 7 layers, with 7 layers completely underfitting on Bifurcating Hill.

Related Work & Insights

• vs D-NeRF: Discrete deformation fields vs. continuous ODE dynamics, where D-NeRF lacks extrapolation capabilities.
• vs DONE: Both utilize Neural ODEs, but DONE requires intermediate mesh reconstruction steps, whereas Node-RF is end-to-end and implicit.
• vs MonoNeRF: Requires additional supervision such as optical flow, depth, or masks, whereas Node-RF relies solely on photometric supervision.

Rating

• Novelty: ⭐⭐⭐⭐ The idea of coupling ODE and NeRF is insightful, and trajectory generalization is a new direction.
• Experimental Thoroughness: ⭐⭐⭐ Only synthetic scenes are used; validation on real-world data is lacking.
• Writing Quality: ⭐⭐⭐⭐ Clear, with well-defined problems.
• Value: ⭐⭐⭐⭐ It opens up a continuous dynamics direction for dynamic scene modeling.

Related Papers

• [CVPR 2025] NeRFPrior: Learning Neural Radiance Field as a Prior for Indoor Scene Reconstruction
• [CVPR 2025] Joint Optimization of Neural Radiance Fields and Continuous Camera Motion from a Monocular Video
• [CVPR 2025] Decompositional Neural Scene Reconstruction with Generative Diffusion Prior
• [CVPR 2025] SLAM3R: Real-Time Dense Scene Reconstruction from Monocular RGB Videos
• [CVPR 2025] Continuous 3D Perception Model with Persistent State