RAE-NWM: Navigation World Model in Dense Visual Representation Space

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/20robo/raenwm
Area: Robotics & Embodied AI
Keywords: navigation world models, dense visual representations, flow matching, time-driven gating, image-goal navigation

TL;DR

RAE-NWM moves action-conditioned navigation prediction into frozen DINOv2 space, using CDiT-DH and generation-time-dependent action gating to improve long-horizon structural stability and reach 78.95% success in Habitat, with remaining trade-offs in path efficiency and texture-sensitive environments.

Background & Motivation

When a robot navigates with a world model, predicted video is evidence for evaluating actions rather than the final product: whether doorways, wall boundaries, and the goal move appropriately after a forward motion or turn matters more than the appearance of an individual frame. Methods such as NWM generally simulate future observations in a VAE latent space and use those predictions for planning. A VAE is optimized for image reconstruction, so the information retained through compression is not necessarily the information most useful for recursive motion prediction. Good short-term reconstruction can still give way to structural drift after repeated predictions, leaving the planner to choose actions based on an inaccurate future.

This paper improves that baseline through representation choice. Spatial DINOv2 patch features contain useful semantic and geometric information, but they are high-dimensional, and semantic recognition alone does not establish their suitability for dynamics. The authors first freeze different encoders and train only linear dynamics probes to predict future features from current features and relative actions, measuring predictability through the global coefficient of determination \(R^2\). DINOv2 outperforms VAE, MAE, SigLIP, and ResNet50 representations; its advantage becomes substantially smaller when token positions are shuffled. This provides empirical evidence tied to spatial structure, not a claim that DINOv2 already contains an accurate physical model.

The remaining challenge is to generate stable futures in this high-dimensional space while applying action constraints appropriately at different stages of generation. Replacing the encoder alone leaves both difficult high-dimensional optimization and interference between control signals and detail synthesis. Core Idea: retain dense spatial representations, learn generative action-conditioned dynamics within them, and support continuous rollouts with a decoupled prediction head and time-driven gating; planning compares features directly rather than reintroducing pixel reconstruction errors into the state loop.

Method

Overall Architecture

The inputs are historical first-person RGB frames, relative planar motion, and a prediction interval; the output is the DINOv2 patch representation of a future observation. The action contains two translation components and a yaw angle, while the interval specifies how far ahead that motion extends. The model constructs a dense state representation, uses time-gated action conditioning to control CDiT-DH feature generation, and performs representation-space rollout and planning.

Both the encoder and the pretrained RAE decoder remain frozen, with learning confined to the dynamics network between them. The decoder serves visualization and pixel-level metrics only: updating the history with generated states and scoring candidate actions do not require it. Predicting future images and controlling the robot therefore share a predictor without sharing a mandatory frame-by-frame decoding path.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Historical RGB frames"] --> B["Dense State Representation"]
    U["Action and prediction interval<br/>Generation time"] --> C["Time-Gated Action Conditioning"]
    B --> D["CDiT-DH Feature Generation"]
    C --> D
    N["Gaussian noise and<br/>ODE integration"] --> D
    T["Velocity from true future features<br/>and noise"] -.->|Training supervision only| D
    D --> E["Representation-Space Rollout and Planning"]
    E -->|Update history window| D
    E --> F["Score candidate trajectories<br/>Execute action"]
    E --> G["Frozen RAE decoder<br/>Visualization and pixel evaluation only"]

The training supervision in the diagram is not an inference input: true future frames are unavailable at test time, so predicted tokens must update the context. Generation time must also be distinguished from robot motion time. The former indicates the stage of denoising, while the prediction interval describes the latter.

Key Designs

1. Dense State Representation: retain spatially localized semantic and geometric cues for prediction

The frozen DINOv2 encoder discards its global classification token and retains \(16\times16=256\) patch tokens, each with 768 dimensions. Features from historical frames form the context. Here, "uncompressed" means retaining the visual encoder's dense patch representation without additional VAE-style latent compression; it does not mean lossless per-pixel storage or necessarily more tokens than a VAE. The authors' configuration uses fewer dynamics tokens than NWM, but a high feature dimension per token.

The linear probe tests whether this choice is worthwhile: it applies only learnable linear transformations to the current state and action to predict the future state, preventing a powerful generator from obscuring differences between representations. \(R^2\) measures prediction residuals relative to total target variation, with larger values indicating that the linear model explains more of the future variation. The spatial-shuffling control further suggests that the advantage is not simply knowing that an image depicts a corridor. Nevertheless, this remains evidence of predictability on the data, not a substitute for closed-loop success or proof that real dynamics are linear in every environment.

2. Time-Gated Action Conditioning: adjust motion-signal strength across generation stages

Relative actions, prediction intervals, and continuous generation time are separately encoded with Gaussian Fourier embeddings. Action and interval embeddings are concatenated and passed through an MLP to describe the intended motion and prediction horizon. A separate generation-time branch passes through a linear layer, SiLU, and Sigmoid to produce an element-wise gate; the gated dynamics feature is then added to the time embedding to form the global condition. This condition influences generation through adaptive layer normalization (AdaLN), whereas historical visual context enters through cross-attention, giving the two information sources distinct interfaces.

\[ \mathbf{c}=\mathbf{t}_{\mathrm{emb}}+g(\mathbf{t}_{\mathrm{emb}})\odot\mathbf{c}_{\mathrm{dyn}}. \]

Intuitively, high-noise stages should establish the global layout change induced by the robot's action, while low-noise stages refine local structure. Fixed-strength additive conditioning may be too weak or continue interfering with details. Gate values are constrained between 0 and 1 but learned rather than prescribed to change monotonically. Stronger early conditioning and weaker late conditioning motivate the design; the main paper does not provide enough gate-curve detail to treat that pattern as a demonstrated universal rule.

3. CDiT-DH Feature Generation: separate deep conditional modeling from high-dimensional velocity prediction

CDiT receives noisy future tokens with two-dimensional sine-cosine positional embeddings, uses self-attention to model spatial dependencies, and applies cross-attention with historical frame tokens as keys and values. The global condition from the preceding design enters each block through AdaLN, informing the network both about previous observations and the hypothesized action. The deep backbone produces contextualized features rather than directly performing the final velocity regression in the high-dimensional token space.

Final prediction is delegated to a shallow-and-wide Decoupled Diffusion Transformer (DDT) head. This head re-embeds the noisy input, modulates it with backbone features through spatial AdaLN, and incorporates generation time to output the velocity field. The rationale is not that additional modules invariably improve performance: conventional backbones struggle to optimize high-dimensional DINOv2 prediction directly. Removing DDT produces long-horizon LPIPS worse even than the SD-VAE-encoder ablation. The paper adapts the RAE/DDT architectural principle to navigation dynamics conditioned on historical observations and actions.

4. Representation-Space Rollout and Planning: reuse predicted states for both continuation and action evaluation

Each prediction starts from Gaussian noise and uses an ordinary differential equation (ODE) solver, guided by historical context and the action condition, to produce a clean future representation. That representation enters a sliding history window before the next prediction. There is no repeated image-decoding and feature-re-encoding loop, avoiding an additional source of reconstruction loss at every step. The frozen RAE decoder is called only when a predicted video needs to be displayed. Measuring errors directly in feature space also helps check whether apparent improvements come only from the decoder.

Planning uses the Cross-Entropy Method (CEM): each iteration samples 120 candidate action sequences and performs an 8-step representation-space rollout for each, evaluating candidates by the distance between predicted features and goal-image features before selecting a trajectory for execution. The distance is the mean cosine distance between corresponding normalized DINOv2 patch tokens, with smaller values indicating closer features. It is neither metric distance to the goal nor an explicit collision probability, so feature-based goal agreement still requires closed-loop navigation validation. The main paper does not specify all CEM iteration and candidate-refitting hyperparameters, and these should not be guessed into a complete implementation.

A Worked Example

Consider a robot in a corridor with a goal image taken beyond a turn ahead. Its recent RGB frames become 256 tokens per frame, each with 768 dimensions. One CEM iteration proposes 120 future action sequences, including a candidate that first moves forward and then turns. This is an illustrative walkthrough of the data flow, not an additional experimental case from the paper.

For the candidate's 1st prediction interval, the model receives the current history, relative translation and yaw, and prediction interval, then integrates from noise to generate the next state. The generation-time gate controls how the action condition affects denoising, while the DDT head predicts the high-dimensional velocity. Once the resulting tokens enter the history window, the 2nd prediction uses context that includes a generated state. The same process continues for 8 steps, potentially accumulating errors, which directly connects representation stability to planning quality.

After the other candidates undergo the same process, their predictions are scored against the goal image's DINO features and an action is selected for execution. In closed-loop operation, the next environment step supplies a fresh real observation and triggers replanning, rather than requiring unquestioning execution of the entire imagined future. Action selection does not need an RGB video for human viewing; decoding is a side branch.

Loss & Training

Flow matching linearly interpolates between true future features and standard Gaussian noise, training the network to predict velocity along this path. Using the readable notation in the source, the clean target is at \(t=0\) and pure noise at \(t=1\):

\[ \mathbf{z}^{(t)}=(1-t)\mathbf{z}^{(0)}+t\mathbf{z}^{(1)},\qquad \mathbf{u}^{(t)}=\mathbf{z}^{(1)}-\mathbf{z}^{(0)}. \]

Training matches the conditional velocity prediction to this target velocity; inference integrates backward from \(t=1\) to \(t=0\). Equation (6) is damaged in the cached text, so a reconstructed full expectation-based loss is not presented as the authors' exact formula. The deterministic DINO-Reg control instead uses direct mean squared error (MSE) regression on future DINO tokens, separating the effects of representation choice and the generative transition objective.

For real-world data, one model is trained on the combined training splits of SACSoN/HuRoN, RECON, and SCAND and evaluated on their held-out trajectories. The Habitat model is trained separately on 1,000 trajectories collected in Matterport3D and evaluated on unseen trajectories. Splitting is at the trajectory level, which should not be reinterpreted as demonstrated zero-shot generalization to unseen scenes. Both encoder and decoder are frozen; only the intervening transition model is optimized. Detailed training hyperparameters are referred to Supplementary Section A, which is absent from the supplied cache, so training steps, learning rates, and ODE step counts are not filled in.

Key Experimental Results

Main Results

Table 1 reproduces selected results from the paper's Table 2 for trajectory prediction/planning up to 2 seconds on held-out real-world trajectories. ATE measures absolute deviation of the predicted trajectory, whereas RPE measures relative motion error; both are lower-is-better. The original table does not label units in its header, so values are retained without assigning meters or angular units. DINO-Reg uses the same representation space and navigation protocol as the proposed model, but is not a full reproduction of DINO-WM.

Model	SACSoN ATE	SACSoN RPE	RECON ATE	RECON RPE	SCAND ATE	SCAND RPE
GNM	3.71	1.00	1.87	0.73	2.12	0.61
NoMaD	3.73	0.96	1.95	0.53	2.24	0.49
NWM	4.12	0.96	1.13	0.35	1.28	0.33
DINO-Reg	3.14	0.75	1.46	0.36	1.44	0.33
RAE-NWM	2.91	0.70	1.36	0.37	1.14	0.28

On SACSoN, ATE falls from NWM's 4.12 to 2.91, a reduction of approximately 29.4%. On RECON, however, NWM's 1.13/0.35 is better than the proposed model's 1.36/0.37. Gains on SACSoN must not be generalized into consistent superiority across all datasets.

Table 2 presents the paper's Table 3 for Habitat/Matterport3D image-goal navigation. Episodes average 8 meters, with success defined as stopping within 1 meter of the target. SR is the proportion of successful episodes; SPL weights success by path length, penalizing both failures and detours. Both are higher-is-better.

Method	SR (%)	SPL (%)
NoMaD	16.67	15.13
OmniVLA	36.67	34.78
NWM	43.33	38.66
One-Step WM	72.67	69.10
RAE-NWM	78.95	63.58

Relative to One-Step WM, SR is 6.28 percentage points higher, but SPL is 5.52 percentage points lower. The authors suggest that one-step generation can evaluate longer planning trajectories more efficiently under a fixed compute budget, whereas iterative generation favors success rate here. This is an explanation, not a causal conclusion with every planning-budget variable isolated.

Open-loop evaluation also distinguishes two settings. The paper's Table 1 directly predicts the SACSoN observation at 16 seconds, with LPIPS/FID of 0.349/15.90 for RAE-NWM versus 0.470/33.06 for NWM. Figure 6 instead rolls out sequentially at 4 FPS to 16 seconds. The former bypasses intermediate states to examine distant action controllability and must not be conflated with accumulated rollout error. LPIPS measures perceptual distance, while FID compares feature distributions of generated and real images; both are lower-is-better.

Ablation Study

Table 3 presents the paper's Table 4, comparing dynamics-conditioning strategies on SACSoN with the main model otherwise fixed. ATE/RPE have the same meaning as in Table 1. These results support a contribution from learned gating rather than attributing every improvement to DINOv2 representations.

Conditioning Strategy	ATE	RPE
Simple Addition	3.34	0.79
MLP Fusion	3.54	0.83
Scheduled Gate	3.31	0.78
Learned Gate	2.91	0.70

Compared with Simple Addition, the learned gate reduces ATE by 0.43 and RPE by 0.09. The DINO-Reg control in Table 1 shows that the generative objective lowers ATE on all three datasets, but its RECON RPE of 0.37 is slightly worse than deterministic regression's 0.36. Flow matching therefore does not dominate every metric.

Key Findings

• Long-horizon structural stability is not high-frequency texture fidelity. Static encode-decode FID on RECON is 4.14 for SD-VAE and 5.47 for DINOv2+RAE, showing a texture-reconstruction cost even without dynamics prediction.
• Compute measurements have a specific scope: the generative backbone has 350M parameters versus NWM's 1B, and 256 dynamics tokens versus 784; dynamics-forward latency on one A800 is 19.4 versus 28.8 ms. These are not end-to-end control latencies including multiple CEM candidates and integration steps.
• The source contains an unexplained discrepancy in RECON 16-second rollout LPIPS: Section 5.2 reports 0.458, while Section 6 gives a terminal value of 0.472. This note retains the discrepancy rather than replacing it with a single supposedly exact number.

Highlights & Insights

• Cheap dynamics probes screen representations before training expensive generators. Spatial shuffling separates strong semantics from the usefulness of spatial arrangement, yielding design evidence closer to control requirements than visually appealing reconstructions alone.
• Generation, evaluation, and recurrent state updates operate in the same representation space. This can reduce propagation of decoding error through planning, but independent task metrics remain necessary to check whether that space favors its own predictions.
• Action conditioning need not have fixed strength throughout generation. Gating uses generation time to regulate control information, offering a transferable design for other action-conditioned feature-generation tasks rather than merely a navigation-specific action encoding.

Limitations & Future Work

• The authors explicitly acknowledge loss of high-frequency stochastic textures, also reflected in short-horizon RECON trajectory results. Future representations could preserve local texture cues without sacrificing long-term structure.
• The authors acknowledge the absence of fully scale-matched training. Representation, architecture, and capacity change together, so a 350M model outperforming a 1B baseline does not establish representation choice as the sole source of improvement.
• Accelerating iterative rollouts is an explicit future direction. Higher SR with lower SPL than One-Step WM means a control system cannot optimize success rate alone.
• Note author's observation: DINO distance is used for both planning and part of the evaluation, potentially favoring the proposed representation. LPIPS, FID, and closed-loop navigation mitigate this concern but do not replace broader real-robot closed-loop testing and independent geometric evaluation. The main paper does not report confidence intervals that would support estimating statistical significance.

Related Work & Insights

• vs NWM: Both predict visual futures conditioned on actions, but NWM uses VAE latents, whereas this paper moves transitions and planning into dense DINOv2 space. Its advantage concerns long-term structure and some navigation tasks, not every short-horizon texture-sensitive setting.
• vs DINO-WM / V-JEPA 2: Prior work already explores representation-space prediction and planning. This paper differs through generative action-conditioned rollouts for first-person navigation, not through being the first to build a world model on visual features. DINO-Reg is a deterministic control under this paper's protocol.
• vs RAE / DDT: Frozen visual encoders, reconstruction decoders, and shallow-wide generation heads have prior foundations; this paper combines them with actions, prediction intervals, and historical-frame conditioning. Reuse should examine both whether the representation supports motion prediction and whether the prediction head can handle its dimensionality effectively.

Rating

• Novelty: 4/5. The combination of representation space, generative dynamics, and time-driven gating is well motivated, although its foundational components have precedents.
• Experimental Thoroughness: 4/5. Real-world trajectories, simulated closed-loop control, and multiple ablations are covered, but fully scale-matched comparisons and real-robot closed-loop evidence remain missing.
• Writing Quality: 4/5. The connection between motivation and architecture is clear, although some numerical ambiguities and reliance on supplementary material complicate verification.
• Value: 4/5. Useful guidance for state-representation selection in navigation world models, with deployment still requiring trade-offs among latency, path efficiency, and texture loss.