Skip to content

OmniNWM: Omniscient Driving Navigation World Models

Conference: ECCV 2026
arXiv: 2510.18313
Code: https://github.com/Arlo0o/OmniNWM
Area: Autonomous Driving
Keywords: Driving World Models, Panoramic Multimodal Generation, Normalized Plücker Ray Maps, Intrinsic Occupancy Rewards, Closed-Loop Simulation

TL;DR

OmniNWM integrates the three elements of driving world models—state, action, and reward—into a unified probabilistic framework. It jointly generates pixel-aligned panoramic RGB, semantics, depth, and 3D occupancy. By decoupling motion control from camera rig geometry through normalized panoramic Plücker ray maps, it achieves zero-shot cross-camera transfer. Furthermore, it directly derives dense rewards from the generated occupancy volume, enabling closed-loop evaluation of planning agents within the generated world.

Background & Motivation

An ideal driving world model should approximate the joint multimodal posterior of the real environment: predicting future states, evaluating actions, and providing rewards, all integrated within a single probabilistic framework. However, practical driving world models often address only one of these aspects. Most existing methods merely generate single-modality RGB videos, treating multimodal sensor simulation as a mixture of independent conditional generation tasks (training separate models and then splicing them together). This falls into the "conditional independence fallacy"—where modalities are not pixel-aligned, generated depths and semantics do not correspond to the RGB frames, let alone lifting them into a consistent 3D occupancy. More problematic is long-term rollout degradation: models are trained on ground-truth historical frames but queried autoregressively on their own error-prone generated frames during inference. This exposure bias leads to exponential accumulation of errors, causing the entire video to collapse after a few dozen frames.

The second bottleneck lies in motion control. Existing methods use sparse representations such as waypoints or raw camera poses, which entangle "how the ego-vehicle moves" with "how cameras are mounted." Consequently, the latent space tends to overfit to specific camera extrinsic calibrations. When transferring to a different dataset or a different camera rig (e.g., from 6 cameras to 3), the control signals fail. The authors refer to this as geometric covariate shift: the trajectory distribution supports of the source and target domains do not overlap, pushing the KL divergence to infinity, which is catastrophic for zero-shot transfer. The third gap is rewards: a truly functional world model must provide physically grounded reward signals for closed-loop policy evaluation. However, the few works attempting rewards rely on external, black-box learned reward models, which themselves suffer from distribution shifts, failing to close the loop between generation and planning.

Therefore, the key insight of this paper is that instead of addressing these three tasks in isolation, they should be optimized jointly on a shared latent manifold. Core Idea: Unify the "state-action-reward" triad of driving world models into a joint probabilistic framework. A shared latent manifold jointly generates pixel-aligned panoramic RGB, semantics, and depth, which are then lifted into 3D occupancy (state). Normalized panoramic Plücker ray maps decouple motion from camera geometry (action). Finally, dense rewards are intrinsically derived from the generated occupancy to close the planning loop (reward). These three components reinforce rather than run in isolation.

Method

Overall Architecture

OmniNWM is designed as a differentiable simulation environment \(\mathcal{E}\), operating as a unified probabilistic simulation loop. For the state pathway, a Panoramic Diffusion Transformer (PDiT) performs joint denoising on a shared latent manifold to simultaneously produce panoramic RGB, semantic, and metric depth videos. These are then projected via lightweight geometric mapping into a 3D semantic occupancy \(V_{occ}\). Since the three modalities share the same denoising trajectory and backpropagate gradients mutually, they are pixel-aligned by design. For the reward and planning pathway, the generated occupancy directly derives dense rewards and acts as a multimodal context fed into the built-in OmniNWM-VLA agent to infer and plan future trajectories. For the action pathway, the planned trajectory is encoded into a normalized panoramic Plücker ray map to serve as geometric constraints that guide the next generation step, closing the loop: the agent perceives state \(S_t\), outputs action \(a_t\), which is projected onto the normalized Plücker manifold to condition the next state generation \(S_{t+1}\sim\mathcal{E}(S_t,a_t)\). This process is stabilized by structured panoramic forcing to prevent long-term rollout collapse.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Reference Panorama + Input Trajectory"] --> B["Joint Multimodal Generation PDiT<br/>RGB/Semantics/Depth Shared Latent Manifold"]
    B --> C["Geometric Lifting<br/>2D Observation → 3D Semantic Occupancy"]
    C --> D["Intrinsic Occupancy Reward<br/>Dense Penalties: Collision/Drivable/Traffic"]
    C --> E["OmniNWM-VLA<br/>Tri-MMI Fusion of Three Modalities → Planned Trajectory"]
    E -->|Planned Trajectory| F["Normalized Panoramic Plücker Ray Map<br/>Decouple Motion & Camera Geometry"]
    F -->|Conditioning Next Frame| B
    D -.->|Evaluate Policy| E
    B -.->|Structured Panoramic Forcing Stabilizes Rollout| B

Key Designs

1. PDiT Joint Multimodal Generation: Growing Aligned RGB/Semantics/Depth At Once on a Shared Latent Manifold

Addressing the "conditional independence fallacy" caused by modular pipelines—where separately trained modalities fail to align in pixel space, inflating occupancy lifting and reconstruction errors. OmniNWM shifts from minimizing isolated reconstruction losses to jointly optimizing multimodal projections on a shared latent manifold via a Panoramic Diffusion Transformer. Specially, high-dimensional inputs are first compressed into compact spatial-temporal latents using a pre-trained 3D VAE. (To preserve semantic topology in continuous projections, semantic maps are colored with a fixed palette before encoding and discretized back to classes using nearest-neighbor search after decoding). The latents of RGB, colored semantics, and metric depth are concatenated along the channel dimension to form a unified latent \(z_{joint}\). The PDiT approximates the conditional reverse diffusion \(p_\theta(z^{joint}_{t-1}\mid z^{joint}_t, C)\) on this joint latent. Crucially, this shared denoising trajectory ensures pixel-level alignment: semantic and depth gradients are backpropagated into the joint latent alongside RGB gradients, synchronizing the outputs. Lifting 2D observations to 3D occupancy thus relies on an intrinsically consistent foundation. Ablation studies confirm that joint semantic and depth supervision yields gains of 3.0 and 2.7 mIoU, respectively.

2. Normalized Panoramic Plücker Ray Maps: Decoupling Motion from Camera Rig Geometry

This design tackles geometric covariate shift. Traditional control methods using waypoints or raw extrinsic parameters overfit the latent space to specific rig calibrations, collapsing under different rigs. OmniNWM employs a parameter-free encoder to map input trajectories to high-dimensional ray maps, which are downsampled and patchified as Plücker embedding tokens, then concatenated with diffusion latent tokens for 3D full attention. The parameter-free formulation injects control signals as precise, pixel-aligned geometric constraints instead of learned semantic priors. The true ingenuity lies in the "normalization": raw Plücker embeddings remain sensitive to the absolute scale and pose of the capture rig. Thus, the authors rigidly transform all camera rays to a unified reference frame (e.g., initial front camera) before calculating Plücker coordinates. Using source intrinsic parameters, pixels are back-projected into unit-normalized world direction vectors \(\hat{d}^{(k)}_{u,v}=R_k K_k^{-1}[u,v,1]^T/\|R_k K_k^{-1}[u,v,1]^T\|_2\). Then, both camera centers and direction vectors are rotated to align with the reference frame, prompting a recalculation of the rays anchored to this normalized system:

\[\hat{p}^{(k)}_{u,v}=\left(o^{(k\to 0)}\times\hat{d}^{(k\to 0)}_{u,v},\ \hat{d}^{(k\to 0)}_{u,v}\right)\]

This representation is intrinsically invariant to the capture rig. It applies a projection operator \(\Phi:\mathbb{R}^6\to\mathcal{M}_{motion}\) to filter out rig-specific geometries, mapping different sensor configurations onto a single normalized motion manifold. This compresses the source-to-target KL divergence from "arbitrarily large" to only irreducible aleatoric uncertainty \(\epsilon\). The impact is immediate: normalization reduces rotation and translation errors by 90.6% and 87.9%, respectively, and zero-shot translation error on nuPlan plummets from >7m in baselines directly to 1.65m. Additionally, this invariant representation unifies multi-view trajectories into a shared 3D Plücker space, significantly expanding the diversity of the training trajectory distribution (Fig. 4c). The authors also provide a game-theoretic explanation: natural driving data is not a set of independent trajectories but a multi-agent system at instantaneous Nash equilibrium. By explicitly controlling only the ego-vehicle and treating surrounding vehicles as reactive latent variables, the model samples from the learned equilibrium distribution—the truck "yielding" during an ego cut-in is not scripted, but the most likely response conditioned on aggressive ego actions on the equilibrium manifold.

3. Structured Panoramic Forcing: Thickening the Training Manifold with Structured Noise to Suppress Long-Term Exposure Bias

Long-term rollout collapse stems from exposure bias: training is conditioned on ground-truth history, while inference rolls out on error-prone predictions, turning distribution drift into a snowballing failure. While standard scheduled sampling injects i.i.d. Gaussian noise, errors in panoramic videos are structurally coupled—either temporally correlated (accumulated motion drift) or spatially correlated (cross-view inconsistency). Hence, this work injects decoupled hierarchical noise into the latent representation \(z^{(t,v)}\):

\[\tilde{z}^{(t,v)}=z^{(t,v)}+\underbrace{\alpha^{(t)}\cdot\epsilon_{temp}}_{\text{Temporal Drift}}+\underbrace{\beta^{(v)}\cdot\epsilon_{spat}}_{\text{Spatial Inconsistency}}\]

where \(\epsilon_{temp}\) mimics temporal trajectory drift, and \(\epsilon_{spat}\) simulates geometric misalignment across camera views. This "stochastic manifold thickening" relies on a clear geometric intuition: conventional models learn a transition operator only on a thin ground-truth manifold \(\mathcal{M}_{GT}\). Once inference drifts off the manifold, it enters undefined regions, causing exponential error growth. Training on perturbed states expands the effective support into a tubular neighborhood \(\mathcal{M}_\epsilon=\{x:d(x,\mathcal{M}_{GT})<\epsilon\}\), forcing the model to learn a contractive mapping that pulls drifted states back onto the manifold. This transition operator approximates local contraction (with the empirical Lipschitz constant dropping from ≈1.08 without forcing to ≈0.94; a value less than 1 indicates that the ground-truth manifold acts as an attractor, damping perturbations instead of amplifying them). It also unlocks flexible inference: both frame-level autoregression (many-to-one, for high-precision planning) and clip-level autoregression (one/many-to-many, for long rollouts, sacrificing temporal granularity for efficiency) run stably. This design is the structural prerequisite for the model's long-term capabilities—FVD on 201 frames stabilizes at 25.22 compared to 386.72 under pure autoregression.

4. Intrinsic Occupancy Rewards + OmniNWM-VLA Closed-Loop: Growing Rewards Directly from Generated 3D Occupancy

Since external black-box reward models suffer from distribution shift, this work renders rewards "intrinsic." By querying the generated 3D occupancy volume directly, it derives a dense potential field to evaluate the safety and execution of the ego trajectory:

\[\hat{R}=1+(R_{col}+R_{bd}+R_{vel})/N_{reward}\]

The three terms represent dense penalties calculated from the voxel grid: collision penalty \(R_{col}=-\alpha_{col}\cdot I_{col}\cdot|v|\) (overlaps between the ego-vehicle volume and obstacle voxels, scaled by speed to reflect kinetic risk, \(\alpha_{col}=0.5\)); drivable area constraint \(R_{bd}=-\alpha_{bd}\cdot I_{non\text{-}drivable}\) (penalizes deviation from drivable surfaces, \(\alpha_{bd}=0.3\)); and traffic efficiency \(R_{vel}=-\alpha_{vel}\cdot\tanh(|v-v_{target}|)\cdot I_v\) (encourages maintaining target speed, \(\alpha_{vel}=0.2\)). Because these rewards are physically consistent, any planning agent can be evaluated closed-loop within the generated world. To handle PDiT's high-dimensional panoramic contexts, the authors introduce OmniNWM-VLA based on Qwen-2.5-VL. Unlike planners that rely only on sparse objects or layouts, it employs a plug-and-play Tri-modal Mamba Interpreter (Tri-MMI) as a state projector \(\phi:\{RGB,Depth,Sem\}\to\mathbb{R}^d\). This framework uses linear-complexity selective state space modeling to fuse photometric, geometric, and semantic contexts (using RGB context to "gate" geometric and semantic features), before tokenizing and feeding them into the VLM—enabling joint semantic-geometric reasoning like "yielding to the truck on the left." Its action head regresses dense trajectory tuples \((x,y,\theta)\) with yaw angles \(\theta\) at 12Hz closed-loop (rather than conventional 2Hz sparse waypoints), significantly narrowing the sim-to-real gap and simulating reactive maneuvers like cutting in.

Loss & Training

The three main pillars target distinct objectives. The PDiT backbone uses Rectified Flow Matching to regress the velocity field that transports probability density from a Gaussian prior to the data distribution:

\[\mathcal{L}_{PDiT}=\mathbb{E}_{t,X_0,X_1}[\|f_\theta(X_t,t,C)-(X_1-X_0)\|^2]\]

This learns straight trajectories in latent space, ensuring stable long-term sampling. OmniNWM-VLA is optimized as a conditional sequence model using causal language modeling, maximizing the log-likelihood of the next token in the trajectory sequence (where tokens explicitly contain yaw angles to remain compatible with Plücker representations). The occupancy generator is trained via a composite loss: depth BCE, voxel-wise semantic cross-entropy, Scene-Class Affinity Loss (to optimize scene structural completeness), and class-balanced cross-entropy (to mitigate small-object sparsity). Training proceeds in three progressive stages: (1) single-view control (17 frames, 224x400, 10k iterations); (2) multi-view multimodal joint training (6-view panorama, 3k iterations); (3) variable length and resolution fine-tuning (17/33 frames, up to 448x800, 3k iterations) on 48 A800 GPUs with AdamW (lr 1e-4, wd 0.01). Closed-loop alignment uses a "decouple-then-couple" strategy: first pre-train PDiT and geometric lifting to convergence, then freeze the world model and fine-tune VLA, and finally joint fine-tune all components to optimize the closed-loop performance.

Key Experimental Results

Main Results

On nuScenes, OmniNWM achieves SOTA performance across generation quality, camera control precision, occupancy prediction, depth estimation, and closed-loop planning. Notably, it delivers top-tier generation quality without relying on heavy voxel/point cloud inputs, using only normalized panoramic ray maps.

Task Dataset Metric OmniNWM Prev. SOTA Note
RGB Video Gen nuScenes FID ↓ / FVD ↓ 5.45 / 23.63 6.45 / 25.55 (UniScene / DiST-4D) Optimal without heavy voxel conditions
Camera Control nuScenes TransErr ↓ 1.18 m 7.56 m (UniScene) Normalization cuts translation error by 87.9%
Semantic Occ nuScenes-Occupancy mIoU ↑ 19.8 16.4 (Hi-SOP) Camera-only outperforms LiDAR methods (L-CONet 15.8)
Panoramic Depth nuScenes Abs.Rel ↓ / \(\delta<1.25\) ↑ 0.23 / 0.81 0.26 / 0.73 (M2Depth) Generative approach crushes DiST-4D (0.39)
Zero-Shot Transfer nuPlan FVD ↓ / TransErr ↓ 79.24 / 1.65 118.60 / 7.12 (DiST-4D) Cross-rig without fine-tuning
Closed-Loop Planning nuScenes (150 scenes) SPR ↑ 87.3% Impromptu-VLA / Qwen-2.5-VL with lower performance Same text prompt, fair 12Hz comparison

Ablation Study

Configuration Key Metrics Note
Full Model FVD201 = 25.22 Structured panoramic forcing + joint generation
Pure Autoregressive (No structured forcing) FVD201 = 386.72 Complete collapse over long horizons
Standard scheduled sampling (No structured noise) FVD201 = 178.65 Improvement but still severely degraded, as structurally coupled errors are unmodeled
Without ray map normalization RotErr 1.71 rad / TransErr 9.75 m Overfitted to specific rig, geometric overfitting
With normalization RotErr 0.16 rad / TransErr 1.18 m Rotation/translation error reduced by 90.6% / 87.9%
Occupancy input only RGB IoU/mIoU 28.9 / 17.1 Baseline
+ Semantics 31.5 / 16.8
+ Semantics + Depth 33.3 / 19.8 Joint semantics and depth add +3.0 / +2.7 mIoU

Key Findings

  • Structured panoramic forcing is a structural prerequisite for long-term stability: its modeling of errors (decoupling temporal drift and spatial inconsistency) matters more than the noise magnitude itself. Given the same noise injection, unstructured Gaussian noise still degrades to an FVD of 178.65 at 201 frames, while structured forcing maintains 25.22.
  • Ray map normalization is crucial for zero-shot cross-rig generalization: removing it causes immediate overfitting to nuScenes intrinsic parameters, sliding into >7m translation errors on nuPlan. Restoring it squeezes translation error down to 1.65m, validating the approach of decoupling motion and camera geometry.
  • Generative occupancy surpasses LiDAR: treating occupancy as a deterministically mapped output of a pixel-aligned joint manifold (rather than an independent random variable) preserves higher geometric consistency than direct regression from RGB or sparse LiDAR. Moreover, the Pearson correlation between generated occupancy rewards and ground-truth occupancy / LiDAR+Box rewards reaches 0.96 and 0.94, mitigating circular reasoning concerns in closed-loop evaluation.
  • Policies are highly sensitive to reward hyperparameters: an excessively high collision penalty (0.8) causes the agent to "freeze" frequently in dense traffic, dropping SPR to 74.6%. Overemphasizing velocity (0.6) spikes the collision rate to 14.5%. \((\alpha_{col},\alpha_{bd},\alpha_{vel})=(0.5,0.3,0.2)\) provides the optimal balance, achieving 87.3% SPR and a 3.4% collision rate.

Highlights & Insights

  • Endogenizing the "reward" is the most elegant design: instead of relying on external black-box models, rewards are directly queried and derived from the self-generated 3D occupancy. This inherently aligns with the generated state and beautifully closes the "generation \(\leftrightarrow\) planning" loop. Quantitative validation via high Pearson correlation coefficients (0.96 / 0.94) proves that the generated occupancy is reliable, addressing circular arguments when using self-generated environments for self-evaluation.
  • Normalized Plücker ray maps are the key trick for transferability: translating control signals from "learned semantic priors" to "parameter-free, pixel-aligned geometric constraints" and aligning them to a normalized reference system successfully cancels out geometric covariate shift. This "normalize-then-control" mechanism can be adapted to any camera-controllable generation task requiring generalization across diverse sensor configurations.
  • "Stochastic manifold thickening" provides a geometric explanation for exposure bias: thickening the training manifold from a 1D curve into a tubular neighborhood forces the model to learn contractive mappings. Proving this via an empirical Lipschitz constant <1 shows that the ground-truth manifold acts as an attractor—this framing is far more insightful than simply stating "noise improves robustness" and illuminates why structured noise outperforms Gaussian noise.
  • Explaining "emergent reactive interactions" through Nash equilibrium is highly compelling: controlling only the ego-vehicle and treating surrounding vehicles as reactive latents allows realistic interactions (e.g., a truck yielding to an ego cut-in) to emerge as maximum likelihood responses on the equilibrium manifold, rather than being hardcoded. This offers a principled guideline for whether world models should support arbitrary "God-mode" editing (arbitrary editing pushes states off the physically consistent equilibrium manifold).

Limitations & Future Work

  • Training and evaluation primarily rely on nuScenes / nuScenes-Occupancy. Although zero-shot transfer to nuPlan and self-collected datasets performs well, depth/semantic ground truths depend on LiDAR projections, MVS, and multi-dataset segmentation models, whose annotation quality directly limits the fidelity of occupancy and subsequent rewards.
  • The weights and formulations of the intrinsic rewards (collision, drivable area, velocity efficiency) are empirically set. Policies are highly sensitive to these parameters (as shown in ablations where extreme weights lead to freezing or reckless driving). Whether these weights generalize across different cities or driving cultures remains to be verified.
  • The Nash equilibrium assumption treats surrounding vehicles entirely as reactive latents. This prevents the model from actively simulating adversarial or irrational behaviors that deviate from equilibrium, which are exactly the long-tail scenarios crucial for safety stress testing. Decoupling ego control from the rest preserves consistency but limits the flexibility of interactive editing.
  • The computational overhead is high (requiring 48 A800 GPUs), and the "decouple-then-couple" multi-stage training adds implementation and tuning complexity. Active real-time performance at 12Hz closed-loop simulation was not quantified.
  • vs UniScene / DiST-4D (Occupancy/Geometry Conditioned Driving World Models): These approaches rely on heavy voxel grids or point clouds to enforce 3D structure. OmniNWM does the opposite: it generates consistent occupancy (mIoU 19.8 vs 16 range) and higher-quality videos (FVD 23.63 vs 25.55) through a shared latent manifold and lightweight geometric lifting without heavy inputs, proving "joint optimization for alignment" outperforms "stacking input conditions."
  • vs CameraCtrl / MotionCtrl (Camera-Controllable Video Generation): Designed for monocular sequences using raw extrinsics or learned control modules, these models overfit to specific rigs and do not generalize across datasets. OmniNWM utilizes normalized panoramic Plücker ray maps to enforce multi-view consistency and decouple motion from sensor calibration, cutting translation error to 1.18m vs 9.48m (CameraCtrl*) and enabling zero-shot transfer across 3/6-camera configurations.
  • vs Drive-WM (World Models with External Rewards): Drive-WM relies on external black-box reward models susceptible to distribution shift, complicating closed-loop setups. OmniNWM intrinsically derives dense rewards from its generated occupancy, offering physical consistency and enabling closed-loop evaluation for arbitrary planners.
  • vs Impromptu-VLA / DriveVLM (VLA Planners): Most VLA planners output sparse waypoints at 2Hz, which aligns poorly with high-fidelity world model steps and lacks geometric precision. OmniNWM-VLA employs Tri-MMI to merge three modalities with linear complexity, runs in a 12Hz closed loop, and explicitly regresses yaw angles to interface with Plücker control, outperforming both with a closed-loop SPR of 87.3%.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ The first driving world model to integrate states, actions, and rewards within a unified probabilistic framework. Both normalized Plücker ray maps and intrinsic occupancy rewards represent highly original contributions.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across six dimensions (generation, control, occupancy, depth, closed-loop, zero-shot transfer). Detailed ablation studies dissect key components, and Pearson correlation coefficients effectively resolve circular reasoning concerns.
  • Writing Quality: ⭐⭐⭐⭐ Clear organization with a balanced combination of theory (KL bounds, Lipschitz constants, Nash equilibrium) and engineering. However, the theoretical derivations are highly formal and packed with dense equations, presenting a minor barrier to engineering-focused readers.
  • Value: ⭐⭐⭐⭐⭐ Bridges the gap between high-fidelity video synthesis and safety-critical planning. It serves as a robust closed-loop foundation for next-generation driving simulation with high utility and extensibility.