Skip to content

NavWM: A Unified Navigation World Model for Foresight-Driven Planning

Conference: ECCV 2026
arXiv: 2606.24101
Code: None
Area: Robotics / Embodied AI
Keywords: World Models, Visual Navigation, Multimodal Trajectory Prediction, Visual Foresight Planning, Latent World Reasoning

TL;DR

NavWM proposes a unified navigation world model that jointly learns latent world reasoning (depth/semantic scene abstraction), anchor-based multimodal trajectory prediction, and Flow Matching conditional visual generation on a shared bidirectional Mamba backbone. This allows the world model itself to act as a closed-loop "visual foresight" planner—simulating future observations for each candidate trajectory and selecting the optimal path—achieving state-of-the-art (SOTA) performance in offline generation quality (PSNR 14.17 \(\rightarrow\) 17.34) and zero-shot navigation success rate (44%) across five robotic datasets.

Background & Motivation

Visual navigation is a core capability of embodied intelligence, requiring agents to navigate safely, avoid obstacles, and reach targets in unstructured environments solely using camera observations. Traditional navigation policies (GNM, ViNT, NoMaD) learn a direct mapping from visual observations to actions. However, such purely reactive policies lack explicit foresight capabilities, leading to myopic decision-making and mode collapse, which limits their generalization in unseen environments.

Recent works have begun to introduce world models into navigation, but existing paradigms face three key challenges. First, modularly decoupled training: Navigation policies and world models are typically trained separately (e.g., NWM uses DiT as a world model to guide NoMaD), failing to leverage shared spatiotemporal dynamics, which leads to suboptimal representations. Second, a lack of explicit scene abstraction: Even though some methods attempt to unify the policy and the world model (e.g., UniWM), their latent representations lack structured geometric or semantic cues. Consequently, the model can only implicitly infer spatial regularities from raw features, rendering long-horizon prediction and planning unstable. Third, monomodal collapse in the action space: Real-world navigation often has multiple feasible trajectories leading to the same target. However, most existing methods model action prediction as a single trajectory, compressing the multimodal action space into a single mode and making the agent prone to local optima or stagnation.

Core Idea: Perception (scene understanding), generation (future frame synthesis), and control (action prediction) inherently share the need to model the underlying environmental structure and spatiotemporal dynamics. NavWM simultaneously learns these three tasks using a unified bidirectional Mamba backbone. It achieves structured scene understanding by injecting geometric/semantic priors via Latent World Tokens, expands the action space with anchor-based multimodal trajectory proposals, and ultimately enables the generative world model to act as a closed-loop "visual foresight" planner: simulating future observations for multiple candidate trajectories and selecting the optimal path after evaluating target alignment.

Method

Overall Architecture

NavWM utilizes a State Space Model (bidirectional Mamba) as the unified backbone. The inputs are the historical frame sequence \(\mathcal{O}_c=\{o_{t-M},\dots,o_{t-1}\}\), the current observation \(o_t\), and the goal image \(o_g\). The outputs are \(K\) candidate trajectories \(\mathcal{T}=\{\hat{a}_{t:t+N-1}^{(k)}\}_{k=1}^K\) and the corresponding predicted future observations \(\mathcal{O}=\{\hat{o}_{t+1:t+N}^{(k)}\}_{k=1}^K\). The action \(a=(\mu,\phi)\) contains a 2D translation \(\mu\in\mathbb{R}^2\) and rotation \(\phi\in\mathbb{R}\) in the egocentric coordinate system.

Process: Input images are first projected into a compact latent space via a pretrained VAE encoder, then concatenated before a set of learnable Latent World Tokens. They are fed into the bidirectional Mamba for spatiotemporal encoding, and history tokens are then compressed into a fixed-length representation via an attention pooling layer to improve long-horizon reasoning efficiency. The scene representation \(\mathcal{H}\) output by Mamba is split into three dedicated heads: the latent world reasoning head (predicting depth/semantics), the multimodal trajectory proposal head (generating \(K\) candidate trajectories), and the CDiT world modeling head (generating future frames conditioned on trajectories and scene features). During inference, the world model simulates future observations for each candidate trajectory and evaluates target alignment via visual foresight to select the optimal path for closed-loop navigation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Historical Frames + Current Observation + Goal Image"] --> B["VAE Encoding → Latent Space"]
    B --> C["Bidirectional Mamba + Latent World Tokens<br/>Spatiotemporal Encoding → Scene Representation H"]
    C --> D["Latent World Reasoning<br/>Depth/Semantic Scene Abstraction"]
    C --> E["Anchor-based Multimodal Trajectory Proposal<br/>Intent Prediction + Control Regression → K Trajectories"]
    C --> F["CDiT World Modeling<br/>Flow Matching + Optical Flow Conditioning"]
    E --> F
    F --> G["Visual Foresight Closed-loop Planning<br/>Evaluate K Trajectories Future Frames → Select Best One"]

Key Designs

1. Latent World Reasoning: Internalizing Scene Geometry and Semantics with Latent World Tokens

Long-horizon foresight planning in navigation requires the agent to possess an intrinsic understanding of 3D geometry and semantics; otherwise, future frame synthesis lacks a physical foundation. NavWM introduces a set of learnable Latent World Tokens that actively absorb structured environmental priors during Mamba’s spatiotemporal encoding process. The supervision signal is distilled from foundation vision models: Depth Anything V2 is used to generate dense depth pseudo-labels \(\mathcal{D}_{gt}=\{d_n\}\), and SAM is used to generate semantic feature pseudo-labels \(\mathcal{S}_{gt}=\{s_n\}\) (covering the prediction horizon \(n\in\{t+1,\dots,t+N\}\)). The spatiotemporally encoded Latent World Tokens are upsampled via a CNN decoder to output depth predictions \(\hat{\mathcal{D}}\) and semantic predictions \(\hat{\mathcal{S}}\).

To handle the massive variance in depth scales between indoor and outdoor navigation environments, depth prediction employs a scale-invariant loss:

\[\mathcal{L}_{si}=\frac{1}{P}\sum_{p=1}^{P}g_p^2-\frac{\lambda}{P^2}\left(\sum_{p=1}^{P}g_p\right)^2,\quad g_p=\log\hat{d}_p-\log d_p\]

Semantic alignment uses the standard MSE loss. The final reasoning loss is \(\mathcal{L}_{reason}=\lambda_{depth}\mathcal{L}_{si}+\lambda_{sem}\mathcal{L}_{mse}\). The critical value of this auxiliary task is that it does not serve depth/semantic prediction for its own sake, but rather forces the Mamba backbone to internalize the geometric and semantic structure of the scene during the encoding phase. This ensures that subsequent action prediction and visual generation both benefit from the same "physically plausible" scene representation.

2. Anchor-based Multimodal Trajectory Proposal: Decoupling Intent Uncertainty and Control Uncertainty

Reliable foresight planning requires exploring multiple feasible future trajectories rather than sticking to a single one—where the latter easily falls into local optima. Drawing inspiration from trajectory prediction, NavWM decomposes future uncertainty into two complementary sources: intent uncertainty (high-level navigation target of where to go) and control uncertainty (low-level motion of how to travel), corresponding to the two stages of anchor target prediction and conditional trajectory generation.

In the first stage, an anchor codebook \(\mathcal{A}\in\mathbb{R}^{K\times 2}\) is constructed: K-Means clustering is first performed on the ground-truth target coordinates of the training set, followed by Farthest Point Sampling (FPS) to select the \(K\) most representative target locations. Given the scene representation \(\mathcal{H}\), the anchor coordinates are concatenated with the scene features and sent to two lightweight MLP heads: the intent head predicts anchor probabilities \(\hat{\boldsymbol{\pi}}\in\mathbb{R}^K\), and the control head regresses spatial offsets \(\Delta\hat{\mathbf{A}}\in\mathbb{R}^{K\times 2}\) for each anchor, producing modified targets \(\widetilde{\mathcal{A}}=\mathcal{A}+\Delta\hat{\mathbf{A}}\). During training, the anchor \(k^*\) closest to the ground truth is identified to optimize the intent classification loss and the offset regression Huber loss:

\[\mathcal{L}_{anchor}=-\log(\hat{\boldsymbol{\pi}}_{k^*})+\mathcal{L}_{huber}(\Delta\hat{x}_{k^*},\Delta x_{gt})+\mathcal{L}_{huber}(\Delta\hat{y}_{k^*},\Delta y_{gt})\]

In the second stage, the complete future waypoint sequence \(\hat{\mathcal{T}}_k\) is directly regressed for each modified target (assuming conditional independence across steps to avoid the high computational cost of step-by-step autoregressive generation). Supervision is only applied to the matched anchor \(k^*\): \(\mathcal{L}_{traj}=\mathcal{L}_{huber}(\hat{\mathcal{T}}_{k^*},\mathcal{T}_{gt})\). The final action loss is \(\mathcal{L}_{action}=\mathcal{L}_{anchor}+\lambda_{traj}\mathcal{L}_{traj}\). The elegance of this anchor mechanism lies in using discrete anchors to cover intent multimodality while maintaining control precision via continuous offsets. Decoupling and resolving them separately yields both higher diversity and greater accuracy than pure diffusion or GMM approaches.

3. CDiT World Modeling and Flow Matching: Action-Conditioned Visual Foresight

To enable the world model to "imagine" the future based on candidate trajectories, NavWM adopts a Conditional Diffusion Transformer (CDiT) as the visual generation backbone. The key improvement of CDiT over standard DiT lies in how conditions are injected: predicted actions are first converted into a dense optical flow field \(\mathcal{F}\) (explicitly encoding camera motion), which is spatially pooled and fused with the diffusion timestep embedding to modulate intermediate activations via Adaptive Layer Normalization (AdaLN). Meanwhile, scene features \(\mathcal{H}\) produced by the unified backbone are injected via cross-attention to provide contextual guidance.

Generative training employs continuous-time Flow Matching, modeling future frame synthesis as a deterministic mapping from a Gaussian prior to the target latent distribution. Given the ground-truth latent \(z_1\) and noise \(z_0\sim\mathcal{N}(0,\mathbf{I})\), a linear probability path \(z_\tau=\tau z_1+(1-\tau)z_0\) (\(\tau\sim\mathcal{U}(0,1)\)) is defined, and the network \(v_\theta\) regresses the constant vector field \(u_\tau=z_1-z_0\) that drives noise to data:

\[\mathcal{L}_{visual}=\mathbb{E}_{\tau\sim\mathcal{U}(0,1),z_0\sim\mathcal{N},z_1\sim q}\left[\|v_\theta(z_\tau,\tau,\mathcal{F},\mathcal{H})-(z_1-z_0)\|_2^2\right]\]

The advantage of Flow Matching over DDPM lies in straighter sampling trajectories and fewer inference steps, which is critical for real-time inference in navigation. The combination of CDiT, optical flow conditioning, and cross-attention scene injection ensures that generated frames outperform standard DiT backbones in both visual quality and geometric consistency.

4. Two-Stage Joint Training and Uncertainty-Weighted Multi-Task Loss

Although perception, control, and generation tasks share an underlying representation, their gradient magnitudes and convergence rates vary significantly. Direct joint training can easily lead to one task dominating the optimization. NavWM employs a two-stage strategy: In the first stage, all components are jointly trained using teacher forcing—the CDiT world modeling head receives ground-truth trajectories as conditions, allowing the perception, action, and generation branches to learn mutually consistent spatiotemporal representations. In the second stage, the Mamba backbone and the trajectory proposal head are frozen, and only CDiT is fine-tuned. The conditions are switched from field-based trajectories to trajectories predicted by the model itself—this step aims to eliminate exposure bias between training and inference, adapting the world model to imperfect action prediction inputs.

The multi-task loss is automatically balanced via uncertainty weighting: \(\mathcal{L}=\sum_{k\in\mathcal{T}}\left(\frac{1}{2}\exp(-s_k)\mathcal{L}_k+\frac{1}{2}s_k\right)\), where \(\mathcal{T}=\{\text{visual},\text{action},\text{reason}\}\) and \(s_k\) is a learnable log-variance for each task. The intuition is that tasks with larger prediction errors are automatically assigned a smaller weight (\(\exp(-s_k)\) decreases), while the regularization term \(\frac{1}{2}s_k\) prevents the weights from degenerating to zero, adaptively balancing the three objectives without manual parameter tuning.

A Complete Example: Single-Step Closed-Loop from Observation to Decision

Suppose the agent is currently in an indoor corridor, observing a T-junction 3 meters ahead, and the goal image indicates a left turn into a certain room. The single-step inference process is as follows: The VAE encodes the past 4 historical frames, the current frame, and the goal image into latent tokens (approx. 1024 dimensions). The bidirectional Mamba receives these tokens along with 16 Latent World Tokens for spatiotemporal encoding, which are compressed via attention pooling into the scene representation \(\mathcal{H}\). The latent world reasoning head decodes a depth map of the area ahead from the world tokens (showing the wall distance at 2.1m and a left opening 1.8m wide) and a semantic segmentation (the left opening is marked as "walkable"). Concurrently, in the anchor-based trajectory proposal head: the intent head scores \(K=7\) anchors, with the highest probabilities going to "go straight 1.5m", "turn left 45 degrees and advance 2m", and "turn left 30 degrees and advance 1.8m"; the control head regresses minor offset corrections for each anchor. Conditioned on these three trajectories and with \(\mathcal{H}\) as the cross-attention context, the CDiT generates the corresponding future 4 frames of observations using Flow Matching (4-step sampling). The visual foresight planner compares the probability and position of the target room's door appearing in the three groups of future frames, finding that the doorway is clearly visible in the third frame of the "turn left 45 degrees" sequence and matches the goal position—selecting this trajectory to execute the first-step action, entering the next round of the closed loop.

Loss & Training

The total loss consists of three components: \(\mathcal{L}_{reason}\) (depth scale-invariant loss + semantic MSE), \(\mathcal{L}_{action}\) (anchor classification loss + offset regression Huber loss + trajectory regression Huber loss), and \(\mathcal{L}_{visual}\) (Flow Matching vector field regression MSE). These are automatically balanced via uncertainty weighting.

Training configuration: Resolution \(256\times256\), history window of 4 frames, prediction horizon of 4 frames, parameter size of 1.5B. AdamW optimizer is used, with a learning rate warming up from \(5\times10^{-5}\) to \(1\times10^{-4}\) (500 steps) and then cosine decaying to \(1\times10^{-5}\). 8 \(\times\) A100 GPUs, global batch size of 64. The first stage lasts for 100k steps (teacher forcing + 15% goal masking probability), and the second stage lasts for 50k steps (fine-tuning only CDiT using predicted trajectories).

Key Experimental Results

Main Results

Offline Evaluation (Table 1): Evaluated on four datasets—Go Stanford, SCAND, RECON, and HuRoN. For navigation metrics, NavWM substantially outperforms the prior SOTA, UniWM (ATE 0.302, RPE 0.116) and NWM (ATE 0.642, RPE 0.211), with an ATE of 0.207 and RPE of 0.066. For visual generation, NavWM's PSNR reaches 17.340 (UniWM 14.172, NWM 14.343), LPIPS decreases to 0.243 (UniWM 0.282, NWM 0.295), and SSIM improves to 0.507 (UniWM 0.435).

Method ATE↓ RPE↓ AOE↓ MAOE↓ PSNR↑ SSIM↑ LPIPS↓ DreamSim↓
GNM 1.519 0.622 18.165 23.961 - - - -
ViNT 1.424 0.617 17.074 23.831 - - - -
NoMaD 1.070 0.376 12.463 17.645 - - - -
NWM 0.642 0.211 10.496 16.534 14.343 0.422 0.295 0.091
UniWM 0.302 0.116 9.468 13.221 14.172 0.435 0.282 0.102
NavWM 0.207 0.066 8.152 12.855 17.340 0.507 0.243 0.084

Navigation Rollout Evaluation (Table 2): For the Image Goal Navigation task, the average trajectory segment length is 43 steps, and success is defined as arriving within a 0.5m radius of the target. NavWM achieves a success rate of 72% in seen environments (UniWM 66%, NWM 58%) and 44% in zero-shot unseen environments (TartanDrive) (UniWM 36%, NWM 23%), with both ATE and RPE being optimal.

Method Seen SR↑ Seen ATE↓ Unseen SR↑ Unseen ATE↓
ViNT 0.24 1.948 0.11 2.423
NoMaD 0.29 1.711 0.16 2.242
NWM 0.58 1.141 0.23 1.712
UniWM 0.66 0.645 0.36 1.122
NavWM 0.72 0.381 0.44 0.765

Ablation Study

Module Ablation (Table 3): Core modules are progressively added on RECON+HuRoN. With only latent world reasoning (WR), the ATE is 0.513; adding action prediction (WR+AP) reduces ATE to 0.338; further adding the world model (WR+AP+WM) drops ATE to 0.254. For visual quality, the world model alone (WR+WM) achieves a PSNR of 16.627; adding action prediction (WR+AP+WM) increases the PSNR to 17.622 and reduces LPIPS from 0.293 to 0.240. Significant positive synergetic effects are observed among the three modules, proving the necessity of joint training.

WR AP WM ATE↓ RPE↓ PSNR↑ LPIPS↓
✓ 0.513 0.187 - -
✓ ✓ 0.338 0.137 14.286 0.249
✓ ✓ - - 16.627 0.293
✓ ✓ ✓ 0.254 0.113 17.622 0.240

Multimodal Trajectory Comparison (Table 4): Compare the anchor method vs GMM vs diffusion strategies (NoMaD), with APD (Average Pairwise Distance) introduced to evaluate trajectory diversity. Without world model planning, the accuracies of the three methods are close (ATE 0.85-0.88), but the anchor method has the highest APD. After integrating world model planning, the ATE of the anchor method drops from 0.85 to 0.36 (a 58% reduction), far exceeding GMM (0.88 \(\rightarrow\) 0.79, 10% reduction) and diffusion (0.86 \(\rightarrow\) 0.58, 33% reduction), with APD simultaneously increasing to 1.49 (diffusion 0.59, GMM 0.31). Ablations on candidate count \(K\) show that NavWM’s performance plateaus around \(K\approx 7\) (diffusion around 5, GMM even earlier), indicating that the anchor method provides more diverse exploration spaces while maintaining high fidelity.

Key Findings

  • The three-way synergy of joint training is the core source of improvement: WR+AP reduces the navigation ATE from 0.513 to 0.338 (a 34% drop), and further adding WM brings it down to 0.254 (an additional 25% drop)—the three tasks mutually reinforce each other because their shared spatiotemporal dynamic representation serves all of them simultaneously.
  • The "planning gain" of anchor-based multimodal trajectories far outperforms baselines: For the same world model planning, the anchor method improves the ATE by 58%, while diffusion only yields 33%, and GMM yields 10%. The key is not just having the highest absolute trajectory diversity, but that anchor trajectories achieve the best balance between diversity and physical feasibility, providing a better action space for the high-level planner.
  • Two-stage training is crucial for generation quality: Fine-tuning CDiT with predicted trajectories in the second stage significantly improves image reconstruction quality (see Figure 4 in the original paper), demonstrating that eliminating exposure bias makes the world model more robust under actual inference conditions.
  • Excellent zero-shot generalization: On the completely unseen TartanDrive dataset, the SR reaches 44%, far exceeding UniWM (36%) and NWM (23%). The internalization of structured scene priors by latent world reasoning is likely the unlock for this generalization.

Highlights & Insights

  • Paradigm shift of "world model as the planner": Traditional approaches use world models to assist navigation policies. Conversely, NavWM makes the world model the main entity of closed-loop planning—generating future frames corresponding to multiple trajectories and directly comparing visual quality to make decisions, elevating "generation" from an auxiliary tool to the core of decision-making.
  • Intent-control decoupled anchor design: Explicitly decomposing trajectory uncertainty into intent (where to go) and control (how to travel), using a discrete anchor codebook to cover intent multimodality and continuous offset regression to maintain control precision. This combination of "discrete anchors + continuous corrections" is not entirely new to trajectory prediction, but deeply coupling it with visual foresight planning to let the world model benefit from multimodality is an exquisite system design.
  • Flow Matching replacing DDPM for real-time inference: Navigation scenarios are highly sensitive to inference latency. Flow Matching's straight sampling paths drastically reduce inference steps while maintaining generation quality, making it a highly pragmatic technical choice.
  • Latent World Tokens as shared representation anchors: Instead of acting as an independent module, a set of learnable tokens actively absorbs geometric/semantic priors during Mamba encoding. This internalized, structurally knowledgeable representation then simultaneously serves action prediction and visual generation—representing a gentle yet powerful representation learning method that can be transferred to any embodied task requiring multi-task shared scene understanding.

Limitations & Future Work

  • Dependencies on the quality ceiling of foundation vision model distillation: The supervision of latent world reasoning originates from Depth Anything V2 and SAM. Thus, the prediction errors of these models will propagate directly to NavWM's scene understanding, and performance is unknown in extreme lighting, transparent objects, specular reflections, or other scenes where foundation models fail.
  • Static nature of offline anchor codebook clustering: The anchor codebook is constructed once before training via K-Means+FPS and cannot be adaptively updated during training. If the test environment's target distribution differs significantly from the training set (e.g., extremely long corridors vs small rooms), the fixed codebook may provide insufficient coverage.
  • Sole support for image-goal navigation: The current framework is conditioned on a goal image and does not support language instructions or coordinate-based goals. Extending this to multimodal goal conditions would require redesigning the conditional injection mechanism.
  • Deployment challenges of a 1.5B parameter model: The training configuration using 8 \(\times\) A100 GPUs and the 1.5B parameter size pose challenges for real-world robotic edge-side deployment. The authors do not discuss model compression or distillation schemes.
  • Unquantified cumulative drift in closed-loop rollouts: Although it is claimed that visual foresight enables real-time error correction, there is a lack of systematic analysis of cumulative drift over long horizons (>100 steps). Qualitative trajectory plots in Figure 3 hint that drift is still occurring.
  • vs NWM (Navigation World Models): NWM trains a DiT world model and a navigation policy in a decoupled manner, where the world model operates strictly as an "independent planning" guide for NoMaD. NavWM's unified framework avoids the representation gap, and its CDiT+Flow Matching is more efficient and generates higher quality frames than a standard DiT.
  • vs UniWM: UniWM is the first attempt to unify a navigation policy and a world model into an autoregressive multimodal model (based on Anole-7B), but its latent representation lacks explicit scene abstraction. NavWM's Latent World Token mechanism fills this gap, leading to substantial improvements in both accuracy and generalization.
  • vs Diffusion Policy / NoMaD: Diffusion policies express action multimodality via a denoising process but fundamentally still model a single distribution, leaving them vulnerable to mode collapse. NavWM's anchor mechanism explicitly maintains discrete multimodal hypotheses, providing better diversity and interpretability.
  • Transferable Ideas: The paradigm of using Latent World Tokens combined with foundation model distillation can be extended to other embodied tasks requiring structured scene understanding (e.g., manipulation, grasping). The "generate-and-verify" framework consisting of anchor-based multimodal prediction + visual foresight evaluation can be transferred to any planning task that requires exploring multiple paths before making a decision.

Rating

  • Novelty: ⭐⭐⭐⭐ While a unified framework learning perception/control/generation jointly is not an entirely new concept (with UniWM pioneering it), the combined design of structured scene distillation via Latent World Tokens + anchor-based multimodal trajectories + visual foresight closed-loop planning offers substantial innovation.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Five datasets, dual evaluation dimensions (offline + online), module-level ablations, cross-method multimodal comparisons, sensitivity analysis of the candidate count, and zero-shot generalization testing—the experimental pipeline is complete and the conclusions are highly compelling.
  • Writing Quality: ⭐⭐⭐⭐ The motivation pipeline is very clear (three pain points \(\rightarrow\) three designs \(\rightarrow\) three synergies), and the methods section coordinates well with formulas and text, although some training details of the 1.5B parameter model (e.g., which layers are frozen in the second stage) are a bit briefly described.
  • Value: ⭐⭐⭐⭐ Establishes a complete paradigm of "unified backbone + structured priors + multimodal actions + visual foresight planning" for navigation world models. The planning gain analysis of multimodal trajectories (Table 4) provides direct reference value for related fields.