OmniNWM: Omniscient Driving Navigation World Models

Conference: ECCV 2026
arXiv: 2510.18313
Code: https://github.com/Arlo0o/OmniNWM
Area: Autonomous Driving
Keywords: Driving World Models, Panoramic Multimodal Generation, Normalized Plücker Ray Graphs, Intrinsic Occupancy Reward, Closed-Loop Simulation

TL;DR

OmniNWM bridges the "state-action-reward" triad of driving world models within a unified probabilistic framework: it jointly generates pixel-aligned panoramic RGB / semantic / depth / 3D occupancy, decouples ego-motion from camera assembly geometry using normalized panoramic Plücker ray graphs to achieve zero-shot cross-camera transfer, and directly derives dense rewards from the generated occupancy volume, enabling closed-loop evaluation of planning agents in the generated world.

Background & Motivation

An ideal driving world model should approximate the joint multimodal posterior of the real environment: predicting future states, evaluating actions, and providing rewards under a single probabilistic framework. However, existing driving world models often achieve only one of these aspects. Most methods only generate single-modal RGB videos and treat multimodal sensor simulation as a collection of independent conditional generation tasks (training separate models and stitching them together). This falls into the "conditional independence fallacy"—where modes are not pixel-aligned, resulting in depth and semantics that do not match the RGB, let alone lifting them into a consistent 3D occupancy. More critically, they suffer from autoregressive degradation: models are trained on ground-truth history but roll out using their own error-prone past predictions. This exposure bias causes errors to accumulate exponentially over time, causing the video quality to collapse after dozens of frames.

The second bottleneck lies in action control. Existing methods use sparse representations such as waypoints or raw camera poses to control motion. These representations entangle "how the ego-vehicle moves" with "how the cameras are mounted", causing the latent space to overfit to a specific extrinsic calibration. When applied to a different dataset or a different camera rig (e.g., from 6 cameras to 3), the control signal fails—a phenomenon the authors term "geometric covariate shift". In this case, the support sets of the trajectory distributions in the source and target domains are completely disjoint, driving the KL divergence to infinity and making zero-shot transfer impossible. The third gap is rewards: a truly useful world model must provide physically grounded reward signals for closed-loop policy evaluation. However, the few works that incorporate rewards rely on external, black-box learned reward models, which themselves suffer from distribution shift and fail to close the loop between generation and planning.

Thus, the entry point of this paper is: instead of optimizing these three components separately, they should be jointly optimized on a shared latent manifold. Core Idea: Unify the "state-action-reward" triad of driving world models into a joint probabilistic framework—jointly generating pixel-aligned panoramic RGB, semantics, and depth on a shared latent manifold, which are then lifted into 3D occupancy (state); decoupling motion from camera geometry via normalized panoramic Plücker ray graphs (action); and intrinsically deriving dense rewards from the generated occupancy to close the planning loop (reward), with the three components supporting rather than isolating each other.

Method

Overall Architecture

OmniNWM is designed as a differentiable simulation environment \(\mathcal{E}\), operating as a unified probabilistic simulation loop. For the state pathway, a Panoramic Diffusion Transformer (PDiT) jointly denoises tokens on a shared latent manifold, synchronously generating panoramic RGB, semantic, and metric depth videos. These are then projected into a 3D semantic occupancy grid \(V_{occ}\) via a lightweight geometric mapping. Since the three modalities share the same denoising trajectory and backpropagate gradients to each other, they are naturally pixel-aligned. For the reward and planning pathway, the generated occupancy grid directly derives dense rewards and simultaneously serves as multimodal context for the built-in OmniNWM-VLA agent to reason and plan future trajectories. For the action pathway, the planned trajectory is encoded into a normalized panoramic Plücker ray graph, serving as a geometric constraint to guide the next generation step, thus closing the loop: the agent observes state \(S_t\), outputs action \(a_t\), which is projected onto the normalized Plücker manifold to condition the next state generation \(S_{t+1}\sim\mathcal{E}(S_t,a_t)\). The entire workflow is stabilized by structured panoramic forcing, preventing long-tail autoregressive drift during long-horizon rollouts.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Reference Panorama + Input Trajectory"] --> B["Joint Multimodal Generation PDiT<br/>RGB/Semantics/Depth Shared Latent Manifold"]
    B --> C["Geometric Lifting<br/>2D Observations → 3D Semantic Occupancy"]
    C --> D["Intrinsic Occupancy Reward<br/>Collision/Feasible Area/Traffic Flow Dense Penalties"]
    C --> E["OmniNWM-VLA<br/>Tri-MMI Fusion of Three Modalities → Planned Trajectory"]
    E -->|Planned Trajectory| F["Normalized Panoramic Plücker Ray Graph<br/>Decoupling Motion and Camera Geometry"]
    F -->|Condition Next Frame| B
    D -.->|Evaluate Policy| E
    B -.->|Structured Panoramic Forcing Stabilizes Long-Horizon| B

Key Designs

1. PDiT Joint Multimodal Generation: Simultaneously obtaining aligned RGB / Semantics / Depth on a shared latent manifold

This addresses the "conditional independence fallacy" caused by modular stitching: training modalities independently leads to pixel-level misalignment between depth, semantics, and RGB, which amplifies errors during subsequent 3D lifting. OmniNWM avoids minimizing separate reconstruction losses and instead uses a Panoramic Diffusion Transformer to project multiple modalities onto a shared latent manifold for joint optimization. Specifically, a pre-trained 3D VAE compresses high-dimensional inputs into compact spatio-temporal latents. (To preserve semantic topology in continuous projections, semantic maps are colored with a fixed palette before encoding and discretized back to categories via nearest-neighbor after decoding). The latents of RGB, colored semantics, and metric depth are concatenated along the channel dimension into a unified latent \(z_{joint}\), allowing PDiT to approximate the conditional reverse diffusion \(p_\theta(z^{joint}_{t-1}\mid z^{joint}_t, C)\) on this joint latent. Crucially, this shared denoising trajectory enforces pixel-level alignment: gradients from depth and semantics flow back to the joint latent just like RGB gradients, yielding naturally synchronized outputs. Consequently, lifting 2D observations to 3D occupancy relies on an intrinsically consistent foundation. Ablation studies confirm that joint semantic and depth supervision brings gains of 3.0 and 2.7 mIoU respectively.

2. Normalized Panoramic Plücker Ray Graphs: Decoupling ego-motion entirely from camera rig geometry

This design directly tackles geometric covariate shift. Controlling motion via waypoints or raw extrinsic parameters overfits the latent space to a specific calibration, causing the model to fail when the rig changes. OmniNWM maps the input trajectory to a high-dimensional ray graph using a parameter-free encoder, downsamples and patchifies it into Plücker embedding tokens, and fuses them with diffusion latent tokens via 3D full attention. The parameter-free design injects control signals as precise, pixel-aligned geometric constraints rather than learned semantic priors. The key innovation is "normalization": since raw Plücker embeddings remain sensitive to the absolute scale and pose of the capture rig, the authors rigidly transform all camera rays to a unified reference frame (e.g., the initial front camera). Specifically, pixels are back-projected to unit-normalized world direction vectors using the source intrinsics: \(\hat{d}^{(k)}_{u,v}=R_k K_k^{-1}[u,v,1]^T/\|R_k K_k^{-1}[u,v,1]^T\|_2\). Then, both the camera center and direction vectors are rotated to align with the reference frame. Finally, Plücker coordinates are recomputed anchoring this normalized system:

\[\hat{p}^{(k)}_{u,v}=\left(o^{(k\to 0)}\times\hat{d}^{(k\to 0)}_{u,v},\ \hat{d}^{(k\to 0)}_{u,v}\right)\]

This representation is inherently invariant to the capture rig. It applies a projection operator \(\Phi:\mathbb{R}^6\to\mathcal{M}_{motion}\) to filter out rig-specific geometry, projecting varied sensor configurations onto the same normalized motion manifold, thereby compressing the KL divergence between the source and target domains to only the irreducible aleatoric uncertainty \(\epsilon\). The results are immediate: normalization reduces rotation and translation errors by 90.6% and 87.9%, respectively, and drops zero-shot translation error on nuPlan from >7m in baselines to 1.65m. Additionally, this invariant representation unifies multi-view trajectories into a shared 3D Plücker space, expanding the diversity of the training trajectory distribution (Fig. 4c). The authors also provide a game-theoretic explanation: natural driving data is not a collection of independent trajectories, but a multi-agent system at a transient Nash equilibrium. Thus, by explicitly controlling only the ego-vehicle and treating surrounding vehicles as reactive latent variables, the model can sample from the learned equilibrium distribution—e.g., when the ego-vehicle cuts in, the truck "yields", which is not hardcoded but is the highest-likelihood response conditioned on the aggressive ego-action on the equilibrium manifold.

3. Structured Panoramic Forcing: Thickening the training manifold via structured noise to compress long-horizon exposure bias

The root of long-horizon collapse is exposure bias: training conditions the model on ground-truth history, but inference relies on its own error-prone predictions, leading to compounding errors as distribution shift occurs. Standard scheduled sampling adds i.i.d. Gaussian noise, but panoramic video errors exhibit structured coupling—either correlated over time (cumulative motion drift) or across viewpoints (spatial inconsistency). Therefore, this work injects decoupled, hierarchical noise into the latent representation \(z^{(t,v)}\):

\[\tilde{z}^{(t,v)}=z^{(t,v)}+\underbrace{\alpha^{(t)}\cdot\epsilon_{temp}}_{\text{Temporal Drift}}+\underbrace{\beta^{(v)}\cdot\epsilon_{spat}}_{\text{Spatial Inconsistency}}\]

Here, \(\epsilon_{temp}\) mimics temporal trajectory drift, while \(\epsilon_{spat}\) models geometric misalignments across camera views. The intuition behind this "stochastic manifold thickening" is that conventional models only learn a transition operator on the infinitely thin ground-truth manifold \(\mathcal{M}_{GT}\). Once inference drifts off the manifold, the model enters an undefined region, leading to exponential error growth. Training on perturbed states expands the effective support to a tubular neighborhood \(\mathcal{M}_\epsilon=\{x:d(x,\mathcal{M}_{GT})<\epsilon\}\), forcing the model to learn a contraction mapping that pulls drifted states back to the manifold. This makes the transition operator locally contractive (the empirical Lipschitz constant drops from ≈1.08 without forcing to ≈0.94; a value less than 1 implies the ground-truth manifold acts as an attractor, damping rather than amplifying perturbations). This also unlocks flexible inference: both frame-level autoregression (many-to-one, for high-precision trajectory planning) and clip-level autoregression (one/many-to-many, for long horizons, trading temporal granularity for efficiency) run stably. This design serves as the structural prerequisite for long-horizon capabilities, keeping the FVD at 201 frames stable at 25.22 compared to 386.72 under pure autoregression.

4. Intrinsic Occupancy Reward + OmniNWM-VLA Closed-Loop: Growing rewards directly from the generated 3D occupancy

To avoid the distribution shift issues of external black-box reward models, this work implements an "intrinsic" reward. By directly querying the generated 3D occupancy grid, it derives a dense potential field to evaluate the safety and legality of the ego trajectory:

\[\hat{R}=1+(R_{col}+R_{bd}+R_{vel})/N_{reward}\]

The three terms are dense penalties computed from voxel grids: collision penalty \(R_{col}=-\alpha_{col}\cdot I_{col}\cdot|v|\) (assessing overlap between the ego volume and obstacle voxels, scaled by velocity to reflect kinetic risk, with \(\alpha_{col}=0.5\)); feasible drivable area constraint \(R_{bd}=-\alpha_{bd}\cdot I_{non\text{-}drivable}\) (penalizing deviations from drivable surfaces, with \(\alpha_{bd}=0.3\)); and traffic flow efficiency \(R_{vel}=-\alpha_{vel}\cdot\tanh(|v-v_{target}|)\cdot I_v\) (encouraging maintenance of target speed, with \(\alpha_{vel}=0.2\)). Because this reward is physically consistent, it enables closed-loop evaluation of any planning agent in the generated world. To fully consume the high-dimensional panoramic contexts of PDiT, the authors introduce OmniNWM-VLA based on Qwen-2.5-VL. Unlike prior planners that only look at sparse objects or layouts, it uses a plug-and-play Tri-Modal Mamba Interpreter (Tri-MMI) as a state projector \(\phi:\{RGB,Depth,Sem\}\to\mathbb{R}^d\). It leverages selective state space modeling with linear complexity to fuse photometric, geometric, and semantic contexts (using RGB context to "gate" geometric and semantic features) before tokenizing and feeding them into the VLM—allowing semantic-geometric reasoning like "yielding to the truck on the left". Its action head regresses dense trajectory tuples \((x,y,\theta)\) containing the heading angle \(\theta\) and runs closed-loop at 12Hz (compared to 2Hz sparse waypoints in prior works), significantly reducing the sim-to-real gap and accommodating reactive maneuvers like cutting in.

Loss & Training

Each of the three pillars has its own objective. The PDiT backbone uses Rectified Flow Matching to regress the velocity field that transports probability density from a Gaussian prior to the data distribution: \(\mathcal{L}_{PDiT}=\mathbb{E}_{t,X_0,X_1}[\|f_\theta(X_t,t,C)-(X_1-X_0)\|^2]\), learning straight trajectories in the latent space to ensure sampler stability over long horizons. OmniNWM-VLA is treated as a conditional sequence model optimized via causal language modeling, maximizing the log-likelihood of the next token in the trajectory sequence (with heading angles explicitly included in tokens to ensure compatibility with the Plücker representation). The occupancy generator is trained with a compound loss: depth BCE + voxel-level semantic cross-entropy + Scene-Class Affinity Loss (optimizing structural scene integrity) + class-balanced cross-entropy (alleviating small object sparsity). Training proceeds in three progressive stages: ① single-view control (17 frames, 224×400, 10k iterations) \(\rightarrow\) ② multi-view multimodal joint training (6-view panorama, 3k iterations) \(\rightarrow\) ③ variable-length variable-resolution fine-tuning (17/33 frames, up to 448×800, 3k iterations), using 48 A800 GPUs and AdamW (lr 1e-4, wd 0.01). Closed-loop alignment adopts a "decouple-then-couple" strategy: first pre-training PDiT and geometric lifting to convergence, then freezing the world model to fine-tune the VLA, and finally joint fine-tuning for overall closed-loop optimization.

Key Experimental Results

Main Results

On nuScenes, OmniNWM achieves SOTA performance across multiple benchmarks: generation quality, camera control accuracy, occupancy prediction, depth, and closed-loop planning. Notably, it achieves top generation quality without relying on heavy voxel/point cloud inputs, using only normalized panoramic ray graphs.

Task	Dataset	Metric	OmniNWM	Prev. SOTA	Description
RGB Video Generation	nuScenes	FID ↓ / FVD ↓	5.45 / 23.63	6.45 / 25.55 (UniScene / DiST-4D)	Optimal without heavy voxel conditions
Camera Control	nuScenes	TransErr ↓	1.18 m	7.56 m (UniScene)	Normalization reduces translation error by 87.9%
Semantic Occupancy	nuScenes-Occupancy	mIoU ↑	19.8	16.4 (Hi-SOP)	Camera-only outperforms LiDAR methods (L-CONet 15.8)
Panoramic Depth	nuScenes	Abs.Rel ↓ / δ<1.25 ↑	0.23 / 0.81	0.26 / 0.73 (M2Depth)	Generative approach dominates DiST-4D (0.39)
Zero-Shot Transfer	nuPlan	FVD ↓ / TransErr ↓	79.24 / 1.65	118.60 / 7.12 (DiST-4D)	Cross-rig evaluation without fine-tuning
Closed-Loop Planning	nuScenes (150 scenes)	SPR ↑	87.3%	Impromptu-VLA / Qwen-2.5-VL Lower	Fair comparison with identical text prompts at 12Hz

Ablation Study

Configuration	Key Metrics	Description
Full Model	FVD201 = 25.22	Structured panoramic forcing + Joint generation
Pure Autoregressive (w/o structured forcing)	FVD201 = 386.72	Complete collapse over long horizons
Standard Scheduled Sampling (w/o structured noise)	FVD201 = 178.65	Improvement shown, but severe degradation persists due to unmodeled structured error coupling
w/o Ray Graph Normalization	RotErr 1.71 rad / TransErr 9.75 m	Overfitting to a specific rig, geometric overfitting
With Normalization	RotErr 0.16 rad / TransErr 1.18 m	Rotation/translation errors reduced by 90.6% / 87.9%
Occupancy input RGB only	IoU/mIoU 28.9 / 17.1	Baseline
+ Semantics	31.5 / 16.8
+ Semantics + Depth	33.3 / 19.8	Joint semantics and depth yield +3.0 / +2.7 mIoU gains respectively

Key Findings

• Structured panoramic forcing serves as the "structural prerequisite" for long-horizon stability: its modeling of error (decoupling temporal drift and spatial inconsistency) matters more than the noise magnitude itself. With the same noise level, un-structured Gaussian noise still drops to an FVD of 178.65 at 201 frames, while structured noise stabilizes at 25.22.
• Normalized ray graphs are critical for zero-shot cross-rig generalization: removing them causes the model to instantly overfit to the nuScenes intrinsics, resulting in a translation error of >7m on nuPlan. Restoring them drops the translation error to 1.65m, validating the approach of decoupling motion from camera geometry.
• Generative occupancy outperforms LiDAR: treating occupancy as a deterministic derivation from a pixel-aligned joint manifold (rather than an independent random variable) yields higher geometric consistency than direct regression from RGB or sparse LiDAR. Moreover, the Pearson correlation between the generated occupancy reward and ground-truth occupancy / LiDAR+Box reaches 0.96 / 0.94, alleviating concerns of circular reasoning in closed-loop evaluation.
• Reward hyperparameters are highly sensitive to policy behavior: an excessive collision penalty (0.8) causes the agent to "freeze" frequently in dense traffic, dropping the SPR to 74.6%, while an over-emphasis on velocity (0.6) spikes the collision rate to 14.5%. \((\alpha_{col},\alpha_{bd},\alpha_{vel})=(0.5,0.3,0.2)\) provides the optimal balance, yielding 87.3% SPR and a 3.4% collision rate.

Highlights & Insights

• Endogenizing the "reward" is a brilliant design: rewards are no longer provided by external black-box models but are queried directly from the generated 3D occupancy grid. This inherently aligns the reward with the generated state, closing the loop between generation and planning. Additionally, the Pearson correlation coefficients of 0.96/0.94 quantitatively prove the reliability of generative occupancy, addressing criticisms of circular reasoning when "evaluating oneself with self-generated content."
• Normalized Plücker ray graphs are the key trick for transferability: replacing learned semantic priors with parameter-free, pixel-aligned geometric constraints and unifying them under a canonical reference frame mathematically flattens geometric covariate shift. This "normalize-then-control" approach can be extended to any camera-controllable generation task requiring generalization across sensor configurations.
• "Stochastic manifold thickening" provides a geometric explanation for exposure bias: thickening the training manifold from a thin curve into a tubular neighborhood forces the model to learn a local contraction mapping. Proving that the empirical Lipschitz constant is <1 validates that the ground-truth manifold acts as an attractor—a formulation far deeper than simply stating "noise improves robustness," explaining why structured noise outperforms Gaussian noise.
• Interpreting "emergent reactive interactions" through Nash equilibrium is highly insightful: controlling only the ego-vehicle and treating surrounding vehicles as reactive latents allows realistic yielding behaviors to emerge as the highest-likelihood response on the equilibrium manifold rather than through hardcoded scripts. This offers a principled response to whether world models should support arbitrary "God-mode" editing (which can push the state off the physically consistent equilibrium manifold).

Limitations & Future Work

• Training and major evaluations are centered on nuScenes / nuScenes-Occupancy. Although zero-shot transfer works well on nuPlan and in-house datasets, depth and semantic ground truths still rely on LiDAR projection, MVS, and multi-dataset segmentation networks. Label quality and coverage directly impact the accuracy of final occupancy estimation and rewards.
• The three penalty terms of the intrinsic reward (collision, boundaries, velocity) and their weights are empirically set and highly sensitive to policy behaviors (as shown in ablations where extreme settings lead to frozen or aggressive driving). It remains uncertain if these weights transfer seamlessly to other cities or road cultures without recalibration.
• The equilibrium manifold assumption treats surrounding vehicles as purely reactive latents, which limits the model's ability to simulate adversarial or irrational traffic participants that deviate from equilibrium. Yet, these are precisely the edge cases safety evaluations aim to stress-test. The decision to control only the ego-vehicle trades editing flexibility for consistency.
• The computational threshold is high (48 A800 GPUs), and the multi-stage "decouple-then-couple" training scheme introduces significant training and tuning overheads. Although the 12Hz closed-loop frequency reduces the sim-to-real gap, the real-world deployment latency is not quantitatively addressed.

Related Work & Insights

• vs UniScene / DiST-4D (Occupancy/Geometry Conditioned Driving World Models): These baselines rely on heavy volumetric conditions, such as voxel grids or point clouds, to maintain 3D structures. OmniNWM does the opposite: it generates consistent occupancy (mIoU 19.8 vs 16 range) with higher generation quality (FVD 23.63 vs 25.55) through joint generation on a shared latent manifold and lightweight lifting, proving that joint optimization is more effective than stacking input conditions.
• vs CameraCtrl / MotionCtrl (Camera-Controllable Video Generation): Designed for monocular sequences using raw extrinsics or learned controllers, these methods overfit to specific rigs and fail to generalize across datasets. OmniNWM uses normalized panoramic Plücker ray graphs to enforce multi-view consistency and decouple motion from calibration, reducing translation error to 1.18m (vs 9.48m for CameraCtrl*) and enabling zero-shot transfer across 3-view and 6-view rigs.
• vs Drive-WM (World Models with External Rewards): Drive-WM relies on external black-box reward models susceptible to distribution shift, hindering closed-loop evaluation. OmniNWM derives dense rewards directly from the generated occupancy, providing physically consistent and closed-loop evaluation for any planning agent.
• vs Impromptu-VLA / DriveVLM and other VLA Planners: Most VLA planners output sparse waypoints at 2Hz, which are incompatible with high-fidelity world model simulations and lack geometric precision. OmniNWM-VLA integrates a Tri-MMI interpreter to fuse three modalities with linear complexity, runs closed-loop at 12Hz, and explicitly regresses heading angles to interface with Plücker control, achieving an SPR of 87.3% (outperforming both baselines).

Rating

• Novelty: ⭐⭐⭐⭐⭐ The first driving world model to integrate state, action, and reward into a unified probabilistic framework. Both normalized Plücker ray graphs and intrinsic occupancy rewards represent highly original contributions.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across generation, control, occupancy, depth, closed-loop planning, and zero-shot transfer. Ablations validate all three primary designs, and corporate correlation metrics dispel doubts about self-evaluation.
• Writing Quality: ⭐⭐⭐⭐ Well-structured, balancing theoretical concepts (KL bounds, Lipschitz, Nash equilibrium) with solid engineering implementation. However, equations and notations are dense, and some formal derivations present a steep learning curve for engineering readers.
• Value: ⭐⭐⭐⭐⭐ Bridges high-fidelity video synthesis and safety-critical planning, providing a solid closed-loop foundation for next-generation autonomous driving simulation with excellent scalability.

Related Papers

• [CVPR 2026] WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World
• [CVPR 2026] MAD: Motion Appearance Decoupling for Efficient Driving World Models
• [CVPR 2026] DriveVLN: Towards Mapless Vision-and-Language Navigation in Autonomous Driving
• [ECCV 2024] Neural Volumetric World Models for Autonomous Driving
• [ICCV 2025] LookOut: Real-World Humanoid Egocentric Navigation