Pondering the Way: Spatial-perceiving World Action Model for Embodied Navigation

Conference: ECCV2026
arXiv: 2606.29908
Code: TBD
Area: Robotics / Embodied AI
Keywords: Visual Navigation, World Models, Joint Generation, Diffusion Transformer, Action Planning

TL;DR

SWAM proposes to combine trajectory planning and future observation generation in visual navigation, generating intermediate RGB-D frame sequences and corresponding action trajectories directly from start/goal RGB images in a single diffusion inference. This eliminates the traditional two-stage "sample-then-verify" paradigm and comprehensively outperforms baselines in terms of accuracy, efficiency, and zero-shot generalization.

Background & Motivation

Embodied navigation requires an agent to plan a reachable path from current observations given a goal image. Traditional methods mainly fall into two categories: direct policy methods (such as NoMaD) that predict actions end-to-end, which are fast to infer but lack explicit imagination of the future and struggle to recover from errors; and two-stage pipelines based on world models, which first sample a large number of candidate action sequences from an external policy, rollout the corresponding visual trajectories using an action-conditioned video prediction model (such as NWM), and then select the one that best matches the goal. Although the latter "verification-centric" paradigm provides simulation-based evidence for path selection, it fundamentally decouples the goal intention from trajectory generation. Decision quality is bottlenecked by the coverage of candidate samples, and long-range planning requires exhaustive rollouts of candidates, causing immense computational overhead (NWM+NoMaD taking 245 seconds/sample for 16 candidates). Furthermore, these methods lack explicit spatial constraints, often leading to generated observation sequences with abrupt perspective changes or geometrically infeasible trajectories.

SWAM, proposed in this paper, completely transforms this design paradigm. The core insight is: since the world model possesses generative capabilities, why relegate it to a passive verifier? If the world model is allowed to simultaneously generate both "how to go" and "what will be seen" from the very beginning, action and observation would constrain each other during the generation process. This naturally guarantees goal alignment, temporal consistency, and spatial feasibility—all achieved in a single inference without candidate rollout and iterative evaluation. Core Idea: SWAM is a joint observation-action generation framework based on a Diffusion Transformer. Given start and target RGB images, a single forward pass simultaneously generates intermediate RGB-D observation sequences and corresponding planar action trajectories. By injecting spatial priors via depth pseudo-labels during training, it only requires monocular RGB at inference, fundamentally replacing the "sample-and-verify" two-stage pipeline.

Method

Overall Architecture

SWAM is built on the Diffusion Transformer (DiT) of the pretrained video generation model CogVideoX, extending it into a navigation planner that jointly generates RGB-D observation frames and action sequences. Unlike traditional two-stage schemes (NWM+NoMaD) that first sample candidate actions and then rollout visual sequences separately, SWAM encodes all information into a single token sequence, denoising vision and action simultaneously during the diffusion process to produce the full planning result in one inference. The framework pipeline is shown below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Start RGB + Goal RGB"] --> B["CogVideoX 3D-VAE<br/>Encoded as RGB/Depth Latents"]
    B --> C["Conditional Latents<br/>Start RGB-D + Goal RGB-D"]
    C --> D["DiT Joint Denoising<br/>Unified Token Sequence"]
    D --> E["Denoising Completed"]
    E --> F1["VAE Decoder →<br/>RGB-D Frame Sequence"]
    E --> F2["VGAR Cross-Attention<br/>Refine Action Token"]
    F2 --> G["MLP Projection →<br/>Planar Action Trajectory"]

Key Designs

1. Joint Observation-Action Diffusion Framework: Unifying actions and vision into a single token sequence for joint diffusion

Traditional video diffusion models only generate image frame sequences. The innovation of SWAM lies in encoding actions as tokens and incorporating them into the diffusion sequence. Specifically, at each sampling step \(n\), the RGB image and depth map are encoded by a frozen CogVideoX 3D-VAE into latent variables \(z_n^i, z_n^d\) (spatial size \(h \times w\), dimension \(d\)), while the action \(a_n = (\Delta x_n, \Delta y_n)\) is projected into an action token \(x^a\) of the same dimension via a learnable MLP. In this way, the intermediate RGB latent sequence, depth latent sequence, and action token sequence are concatenated into a unified token sequence \(X_0\) as the diffusion target. The RGB-D latents of the start and goal frames serve as conditional information, remaining clean and not participating in the diffusion process. During denoising, the DiT simultaneously predicts the noise residuals for all tokens, allowing action generation and visual generation to share the same latent space representation. This ensures that actions and observations are mutually aligned during generation, avoiding geometric inconsistencies such as "the path looks straight in the images, but the action sequence turns."

2. Visual-Guided Action Refinement (VGAR): Secondary correction of action tokens using generated RGB-D evidence

After diffusion denoising, the visual and action tokens are finalized in the last layer of DiT. However, directly applying a linear projection to the action tokens may lose local geometric details. The VGAR module introduces a lightweight gated residual cross-attention before action decoding: the action tokens act as queries to attend to the key-value pairs of the visual tokens, computing the cross-attention output \(C\). A gating mechanism \(G = \sigma(\text{MLP}([X_a; C]))\) controls the injection of visual information: \(\Delta X_a = G \odot W(C)\), and the final refined action token is \(X_a' = X_a + \Delta X_a\). This gating design is highly elegant: if the current action token itself is already accurate (e.g., in a straight-line scenario), the gating value automatically decreases to reduce over-reliance on visual inference; if the action token is ambiguous (e.g., a turning decision at an intersection), the spatial cues in the visual frames take a dominant role. VGAR does not modify the diffusion backbone, enabling end-to-end training without disturbing the distribution of pretrained features.

3. Trajectory Scale Regularization Loss (TSR): Direct supervision of cumulative displacement to eliminate long-range drift

When the diffusion loss \(\mathcal{L}_{\text{DDPM}}\) acts individually on the action token at each step, minor errors at each timestep accumulate through trajectory integration, leading to severe drift of the end position in long-distance planning. The TSR loss directly supervises the deviation of cumulative displacement from the true target position:

\[\mathcal{L}_{\text{TSR}} = \frac{\|\hat{p}_N - p_G\|_2}{N}, \quad \hat{p}_N = \sum_{n=1}^N \hat{a}_n\]

where \(\hat{p}_N\) is the predicted endpoint of the trajectory, \(p_G\) is the true goal location, and division by \(N\) normalizes the loss across trajectories of different lengths. The significance of this loss is that: among multiple predicted sequences with similar local denoising errors, TSR favors trajectories that have minor local deviations but correct global displacements. Consequently, it transforms "long-range drift"—a problem easily ignored by diffusion models—into a directly optimizable signal. The combined loss is \(\mathcal{L} = \mathcal{L}_{\text{DDPM}} + \lambda_{\text{TSR}} \mathcal{L}_{\text{TSR}}\).

4. Depth-Aware Spatial Prior: Injecting geometric constraints via depth pseudo-labels during training, with monocular RGB only at inference

Generative models based purely on RGB struggle to perceive scene geometry (e.g., locations of obstacles, ground terrain), which can cause generated trajectories to traverse untraversable areas. SWAM solves this by introducing DepthAnything V3 during the training phase to predict depth pseudo-labels for each frame, encoding the depth maps and RGB images in parallel as latents into the diffusion sequence. Consequently, the model learns the prior of "what objects look like in 3D space" during training, while during inference, it only requires monocular RGB—the depth branch is used only during training, and during inference, depth frames are generated by the model itself. Ablation experiments demonstrate that simply adding the depth branch reduces the ATE from 2.09 to 1.70 and RPE from 0.70 to 0.47 on RECON, indicating that geometric awareness brings substantial improvements to navigation planning.

Loss & Training

The loss function is a weighted combination of the diffusion loss and the TSR loss: \(\mathcal{L} = \mathcal{L}_{\text{DDPM}} + \lambda_{\text{TSR}} \mathcal{L}_{\text{TSR}}\). During training, the segment length \(N \in \{9, 17, 33, 65\}\) is randomly sampled to support variable-length predictions, using RoPE interpolation for variable-length positional encoding. The model is initialized from the pretrained CogVideoX weights, and the newly added modules (depth branch, action head, VGAR) are all initialized with zero, ensuring that the initial state is fully consistent with the original pretrained model behavior. Optimization setup: Adam (lr=\(1 \times 10^{-4}\)), 1000 warm-up steps, maximum gradient clipping norm of 1.0, bf16 mixed-precision + gradient checkpointing.

Key Experimental Results

Main Results

Dataset	Metric	NWM+NoMaD (×16)	CogVideoX (Joint)	SWAM (Ours)	Gain
RECON	ATE↓	1.53	2.09	0.93	-39.2%
RECON	RPE↓	0.49	0.73	0.43	-12.2%
SCAND	ATE↓	2.18	2.25	1.15	-47.2%
SCAND	RPE↓	0.46	0.67	0.34	-26.1%
TartanDrive	ATE↓	6.23	4.90	1.55	-75.1%
TartanDrive	RPE↓	1.30	1.07	0.68	-47.7%
Inference Time (s/sample)	-	245.98	14.12	16.91	-93.1% vs ×16

SWAM substantially outperforms the baselines on all datasets in both ATE and RPE. Notably, even with 16 sampled candidates and 245 seconds of computation, NWM+NoMaD's trajectory accuracy remains far inferior to SWAM's single-inference result (16.91 seconds). The 75.1% reduction in ATE on TartanDrive (off-road driving) is particularly remarkable, indicating that the depth prior is highly effective in complex terrains.

Zero-Shot Generalization

Setting	Model	ATE↓	RPE↓
Trained on HuRoN	NWM+NoMaD (×16)	3.73	0.95
Zero-Shot (Not trained on HuRoN)	SWAM	2.94	0.85

In zero-shot inference on the unseen HuRoN dataset, SWAM's ATE is actually lower than that of NWM+NoMaD trained on HuRoN (2.94 vs 3.73, a 21.2% reduction), demonstrating exceptional cross-scene generalization capabilities.

Ablation Study

Configuration	RECON ATE↓	SCAND ATE↓	TartanDrive ATE↓
Baseline (CogVideoX joint)	2.09	2.25	4.90
+ Depth	1.70	2.27	4.61
+ TSR	2.06	1.63	2.63
+ Depth + TSR	1.01	1.12	1.94
+ Depth + TSR + VGAR (Full Model)	0.94	1.15	1.55

Ablation conclusions: TSR contributes the most to long-range scenarios in TartanDrive (4.90 → 2.63), the depth branch is most critical in structured environments (RECON), and VGAR provides consistent marginal improvements across all scenarios. The three components are mutually complementary rather than redundant.

Key Findings

• TSR loss is the key to eliminating long-range drift, showing exceptionally prominent effects in off-road scenarios like TartanDrive that exhibit a strong forward-motion bias—TSR alone reduces ATE from 4.90 to 2.63.
• Geometric constraints from the depth prior are more effective in structured environments (RECON/SCAND) than in unstructured environments (TartanDrive), as depth estimation is inherently noisier in wild terrains.
• SWAM's primary failure modes concentrate on "ambiguous vegetation" (misidentifying traversable grass as obstacles) and "sharp turns" (where planar displacement action representation fails to capture quick heading changes). The authors plan to introduce semantic traversability reasoning and richer action representations in the future.

Highlights & Insights

• Joint Generation Paradigm: Transforming the decoupled "sample-verify-select" pipeline into a single joint diffusion inference represents a paradigm shift in the navigation world model domain—not an incremental improvement, but a fundamental change in how actions and observations interact.
• Elegant and Simple TSR Loss: Without modifying the training process of the diffusion backbone, simply adding a global displacement constraint to the loss function resolves the cumulative drift issue of diffusion models in long-range planning. This concept can be transferred to other sequence-generation tasks.
• Zero-Initialized New Modules: All newly added modules are zero-initialized to ensure that at the start of training, the model's behavior is fully aligned with the pre-trained state. This progressive learning of new modalities is a practical transfer learning technique.
• Gated Design in VGAR: The gating mechanism automatically adjusts the injection strength of visual evidence, preventing excessive interference when the action tokens are already accurate, embodying an elegant "visual intervention only when needed" philosophy.

Limitations & Future Work

• Misjudgments in ambiguous vegetation scenarios expose the weakness of lacking semantic traversability reasoning—relying solely on depth cannot distinguish "soft" vegetation from "hard" obstacles, requiring the integration of semantic segmentation or language guidance.
• The planar displacement (\((\Delta x, \Delta y)\)) action representation has insufficient information in sharp turn scenarios, as it cannot encode heading changes; it needs to be expanded to a representation that includes heading angles.
• The current framework generates fixed-length sequences (8 steps). Although variable-length training strategies provide preliminary support for longer planning, strictly speaking, it is not yet a millisecond-level online closed-loop planner. There is still an engineering gap from video generation to real-time navigation control.

Related Work & Insights

• vs NWM (Navigation World Model): NWM is an action-conditioned video prediction world model following a two-stage "generate candidate actions \(\rightarrow\) rollout videos \(\rightarrow\) rank and select" path. SWAM converts it into joint generation, achieving an order of magnitude faster inference speed at comparable accuracy.
• vs CogVideoX (Joint Baseline): The authors specifically built a joint RGB-action generation baseline backed by CogVideoX for a fair comparison. SWAM comprehensively outperforms it after adding depth, TSR, and VGAR, demonstrating that task-specific designs for navigation, rather than general video generation capacity, are the key to improvement.
• vs NoMaD / GNM: Direct policy methods (like NoMaD) output actions end-to-end and infer extremely fast (~0.2s) but lack the ability to simulate future observations. SWAM trades slightly more inference time for a substantial increase in trajectory accuracy and zero-shot generalization capabilities.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Transforming the paradigm of navigation world models from "verification-centric" to "joint generation", the combination of three designs (joint diffusion + VGAR gating + TSR loss) is complete and necessary.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ 4 datasets + zero-shot generalization + detailed ablation analysis + failure mode analysis. Baselines include multiple sampling settings of NWM+NoMaD and a fair unified CogVideoX backend baseline.
• Writing Quality: ⭐⭐⭐⭐ Clear motivation, standardized methodology, rich figures and text. However, the code is not yet open-sourced, and the failure mode analysis in the appendix could be further deepened.
• Value: ⭐⭐⭐⭐⭐ Represents a clear SOTA in navigation world models. The joint generation paradigm has potential for cross-task transfer (e.g., robotic arm manipulation), and the TSR loss can be transferred to other sequence generation tasks.

Related Papers

• [ECCV 2026] NavWM: A Unified Navigation World Model for Foresight-Driven Planning
• [CVPR 2026] Motus: A Unified Latent Action World Model
• [ICLR 2026] Embodied Navigation Foundation Model
• [ICML 2026] Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model
• [ECCV 2026] A scalar per patch from pre-trained ViTs enables fast moving navigation in the real world