Skip to content

LatentPilot: Scene-Aware Vision-and-Language Navigation by Dreaming Ahead with Latent Visual Reasoning

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/xxx
Area: Robotics & Embodied AI
Keywords: Vision-and-Language Navigation (VLN), Latent Visual Reasoning, World Model, Privileged Learning, Embodied AI

TL;DR

LatentPilot internalizes action-conditioned future visual dynamics directly into a VLM decision backbone via two-step future-privileged supervision during training, enabling strictly causal lookahead at inference without external world models and achieving new SOTA performance on R2R-CE, RxR-CE, and real robots.

Background & Motivation

Vision-and-language navigation in continuous environments (VLN-CE) requires an embodied agent to follow natural language instructions and carry out sequential actions to reach goal locations in complex 3D scenes. Mainstream VLN models predominantly rely on conditional policy learning, predicting the next navigation action solely based on past trajectories and the current observation. This formulation is fundamentally a static "perceive then act" loop that easily suffers from myopic decision making: the agent cannot anticipate the visual consequences of its prospective actions, frequently leading to tentative dithering, costly collisions, wrong turns in multi-branch junctions, and catastrophic navigation failures.

When humans navigate unfamiliar environments, they rarely act purely on momentary visual observations. Instead, they run rapid mental simulations before committing to an action—anticipating whether turning left reveals a corridor or a doorway, or whether walking forward leads to an open hall or a dead end. Recent attempts equip navigators with external world models or video diffusion generators to explicitly visualize future views and guide path planning. However, externalizing imagination into separate generative models or tree-search sampling introduces multi-second per-step latencies, heavy GPU memory overheads, compounding visual prediction artifacts, and severe policy-model distribution mismatch during execution.

Crucially, offline training trajectories already naturally record the ground-truth visual scenes that the agent encounters after executing its actions. This hindsight information contains rich causal dynamics between actions and environmental changes. This paper asks: why not leverage these recorded future views as training-only privileged supervision to internalize predictive foresight into the decision backbone itself? Core idea: internalize action-conditioned future visual dynamics into an end-to-end multimodal decision backbone via a step-propagated latent vector (Pilot Token), supervised by two-step future-privileged visual targets during training, enabling strictly causal lookahead at inference with zero external generative models and zero runtime latency overhead.

Method

Overall Architecture

LatentPilot is an end-to-end multimodal navigator comprising three primary modules: a SigLIP-initialized vision encoder \(E_\phi\), an LLM decision backbone \(F_\theta\) initialized from 7B LLaVA-Video, and a lightweight linear Pilot projection module \(G_\psi\). At each navigation step \(t\), the backbone receives natural language instruction tokens \(x\), current visual tokens from monocular observation \(o_t\), and the continuous latent state \(z_{t-1}\) propagated from the previous step. It simultaneously predicts the low-level discrete action \(a_t\) and updates the latent vector \(z_t\). The updated \(z_t\) is stored in a lightweight cache and fed into the dedicated Pilot slot at step \(t+1\), linking sequential steps into a continuous internal reasoning process within a single model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Instruction x + Current Observation o_t + Cached Pilot Token z_{t-1}"] --> B["Continuous Latent Propagation<br/>Pilot Token acts as internal working memory input and update"]
    B --> C["Two-Step Future-Privileged Supervision<br/>Train-time t+1 frame as privileged input, t+2 pooled frame as target"]
    C --> D["Flywheel On-Policy Learning<br/>Iterative rollout collection with deviation-aware expert takeover"]
    D --> E["Strictly Causal Cached Inference<br/>Single-forward read-write IPilotCache, zero external generation"]
    E --> F["Output: Execution Action a_t ∈ {FWD, LEFT, RIGHT, STOP}"]

Key Designs

1. Continuous Latent Propagation: Carrying Long-Horizon Trajectory Reasoning via Compact Latents Standard explicit chain-of-thought introduces prohibitive textual token overhead, while conventional recurrent hidden states often struggle to maintain cross-modal causal grounding over extended horizons. LatentPilot establishes a persistent token slot (Pilot slot <|placeholder|>). At step \(t=1\), this slot is populated with a learnable static embedding \(z_0\); at any subsequent step \(t\), it is filled with the continuous vector \(z_{t-1} \in \mathbb{R}^d\) output by the previous step. The unified multimodal sequence is: $\(u_t = [\mathrm{Tok}(x);\, v_t;\, \texttt{PILOT}(z_{t-1})]\)$ where \(v_t = E_\phi(o_t) \in \mathbb{R}^{N_v \times d}\) denotes the visual tokens extracted by \(E_\phi\). The entire sequence is processed by LLaVA-Video with causal attention to generate hidden representations \(H_t \in \mathbb{R}^{N \times d}\). The model outputs action probabilities via a linear action head \(\pi_\theta(a_t \mid x, o_t, z_{t-1}) = \mathrm{Softmax}(W_a h_t^{\mathrm{act}})\), and maps the Pilot-position hidden state \(h_t^{\mathrm{pil}}\) back into the continuous latent space via a single-layer projection \(G_\psi\): $\(z_t = G_\psi(h_t^{\mathrm{pil}})\)$ This design eliminates the need for separate imagination models: \(z_t\) acts simultaneously as the internal latent outcome of current decision reasoning and as the temporal-spatial prior for the next decision, maintaining an unrolled foresight stream across navigation steps.

2. Two-Step Future-Privileged Supervision: Converting Hindsight Consequences into Foresight Representations Without explicit guidance, unconstrained continuous latent vectors are prone to latent collapse, degenerating into uninformative constant representations. LatentPilot adopts Learning Using Privileged Information (LUPI) to ground \(z_t\) in physical environment dynamics by leveraging future trajectory frames. Crucially, it disentangles two consecutive future time steps: step \(t+1\) serves as privileged input, while step \(t+2\) provides the privileged regression target. Specifically, future visual frames are compressed into compact representations via mean pooling over vision tokens: \(\bar{v}_{t+k} = \mathrm{Pool}(E_\phi(o_{t+k})) \in \mathbb{R}^d\). During training, the Pilot slot is teacher-forced with the one-step future latent \(\bar{v}_{t+1}\), and the predicted token \(z_t\) is supervised to reconstruct the two-step future visual latent \(\bar{v}_{t+2}\): $\(\mathcal{L}_{\mathrm{pil}} = \sum_{t=1}^{T-2} \|z_t - \bar{v}_{t+2}\|_2^2\)$ This two-step formulation forces the backbone not merely to align current inputs with immediate actions, but to internalize how the current action will transform the visual world two steps ahead. Because privileged supervision is applied strictly via loss backpropagation during training, inference requires neither \(o_{t+1}\) nor \(o_{t+2}\), successfully turning offline hindsight into causal foresight.

3. Flywheel On-Policy Learning: Correcting Policy Drift with Deviation-Aware Expert Takeover Pure offline imitation learning suffers from severe distribution drift and compounding errors in continuous navigation. LatentPilot employs a flywheel-style closed-loop pipeline named PilotLoop. Each cycle comprises four stages: collecting on-policy trajectories using the current model in the Habitat simulator; triggering an expert policy (shortest-path follower) to take over whenever the agent drifts past a deviation threshold; constructing privileged future target pairs from the aggregated buffer; and jointly fine-tuning the model using action cross-entropy and future latent regression: $\(\min_{\theta, \phi, \psi} \mathbb{E}_{\tau \sim \mathcal{D}} \left[ \mathcal{L}_{\mathrm{act}}(\tau) + \lambda \mathcal{L}_{\mathrm{pil}}(\tau) \right]\)$ with \(\lambda = 0.1\). This DAgger-style expert intervention provides the policy with essential recovery behaviors from off-path states, while ensuring that the learned latent dynamics cover the realistic state distribution explored by the agent.

4. Strictly Causal Cached Inference: Fast Closed-Loop Deployment without Future Leakage During online evaluation and real-world deployment, LatentPilot enforces strict temporal causality. A lightweight read-write cache IPilotCache is maintained. At \(t=1\), the cache initializes with \(z_0\). At step \(t\), the system reads \(z_{t-1}\) from the cache to construct \(u_t\), performs a single standard forward pass through \(F_\theta\) to output \(a_t\) and \(z_t\), and overwrites the cache with \(z_t\). This requires zero generative sampling, zero tree rollouts, and zero extra model invocations. The agent runs at an agile 130 ms per action with 22.8 GB peak memory, delivering predictive foresight comparable or superior to heavy video world models at a fraction of the computational footprint.

Loss & Training

LatentPilot is trained in two phases: pretraining on mixed instruction trajectories from Matterport3D (R2R, RxR, and EnvDrop) and HM3D-synthesized ScaleVLN using shortest-path imitation learning; followed by 8 rounds of PilotLoop flywheel fine-tuning. The overall training objective is: $\(\mathcal{L} = \mathcal{L}_{\mathrm{act}} + \lambda \mathcal{L}_{\mathrm{pil}}\)$ where \(\mathcal{L}_{\mathrm{act}} = -\sum_{t=1}^T \log p_\theta(a_t^{\mathrm{col}} \mid x, o_t, \bar{v}_{t+1})\) optimizes action prediction accuracy and \(\mathcal{L}_{\mathrm{pil}} = \sum_{t=1}^{T-2} \|z_t - \bar{v}_{t+2}\|_2^2\) aligns the Pilot Token with two-step future visual features. Models are optimized using AdamW across 8 NVIDIA A100 GPUs, with rollouts collected in parallel across 16 RTX 4090 GPUs.

Key Experimental Results

Main Results

On the Val-Unseen splits of continuous benchmarks R2R-CE and RxR-CE, LatentPilot with single monocular RGB (SRGB) input substantially outperforms prior monocular VLM navigators and matches or exceeds heavy sensor suites equipped with panoramic cameras, odometry, and depth sensors.

Method Observation (Pano/Odo/D/SRGB) R2R Val-Unseen NE↓ R2R Val-Unseen SR↑ R2R Val-Unseen SPL↑ RxR Val-Unseen NE↓ RxR Val-Unseen SR↑ RxR Val-Unseen SPL↑ RxR Val-Unseen nDTW↑
ETPNav (2023) Pano + Odo + D 4.71 57.0 49.0 5.64 54.7 44.8 61.9
ScaleVLN (2023) Pano + Odo + D 4.80 55.0 51.0 – – – –
NaVid (2024) Monocular RGB 5.47 37.4 35.9 – – – –
NavMorph (2025) Monocular RGB + D 5.75 47.9 33.2 8.85 30.8 22.8 44.2
Uni-NaVid (2025) Monocular RGB 5.58 47.0 42.7 6.24 48.7 40.9 –
NaVILA (2025) Monocular RGB 5.22 54.0 49.0 6.77 49.3 44.0 58.8
StreamVLN (2025) Monocular RGB 4.98 56.9 51.9 6.22 52.9 46.0 61.9
JanusVLN (2025) Monocular RGB 4.78 60.5 56.8 6.06 56.2 47.5 62.1
LatentPilot (Ours) Monocular RGB 4.41 62.0 58.0 5.19 58.2 49.9 67.5

On the physically realistic VLN-PE benchmark featuring the Unitree H1 humanoid robot in NVIDIA Isaac Lab, LatentPilot achieves 56.42% SR, 47.74% SPL, a low fall rate (FR) of 10.65%, and a stuck rate (StR) of only 0.97% on unseen splits, confirming strong physical robustness against real locomotion noise.

Ablation Study

The table below compares different privileged supervision modalities for shaping the Pilot Token (Table 3 in the paper).

Config / Supervision Modality Target Representation Description R2R Val-Unseen SR↑ R2R Val-Unseen SPL↑ R2R Val-Unseen NE↓ R2R Val-Unseen OS↑
Vision (Ours) Mean-pooled future SigLIP feature \(\bar{v}_{t+2}\) 62.0 58.0 4.4 66.3
Text Tokenized textual future descriptions from Qwen2.5-VL 53.2 48.7 5.2 59.6
NaN No privileged supervision; action cross-entropy only 51.7 47.1 5.3 57.0
3D Geometric 3D spatial features extracted by VGGT 50.3 45.6 5.4 55.8

The table below contrasts internalized future foresight against external plug-in video world models in terms of latency and memory footprint (Table 4 in the paper):

World Model Paradigm Mechanism Type Latency T/Act (ms)↓ Peak Memory (GB)↓ R2R Unseen SR↑ R2R Unseen SPL↑
Wan2.1 (1.3B) External video diffusion generation rollout 2040 41.5 53.9 48.6
CogVideoX1.5 (5B) External video diffusion world model 5460 32.4 – –
V-JEPA2 plug-in External visual feature predictor 143 23.6 58.1 52.5
LatentPilot (Ours) Internalized Pilot Token latent lookahead 130 22.8 62.0 58.0

Key Findings

  • Visual feature supervision is superior to text and 3D: Direct regression on pooled future visual embeddings delivers the strongest performance (SR 62.0%). Textual supervision (SR 53.2%) loses fine-grained visual-spatial details, while 3D VGGT embeddings (SR 50.3%) actually underperform the unsupervised baseline NaN (SR 51.7%), demonstrating that cross-domain representation discrepancies can induce negative transfer.
  • Internalized foresight achieves a dual win in speed and accuracy: Compared to external video diffusion world models like Wan2.1 (2040 ms latency, 41.5 GB memory), LatentPilot requires only 130 ms and 22.8 GB (over \(15\times\) faster) while boosting SR by 8.1 percentage points, proving that amortizing dynamics within the policy eliminates compounding simulation errors.
  • Flywheel learning scales steadily without latent collapse: Performance steadily increases from round 1 (SR 54.0%) to round 7 (SR 62.0%) before saturating. 2D PCA projection of collected Pilot Tokens reveals clear, well-separated clusters corresponding to distinct actions (FWD, LEFT, RIGHT), demonstrating that predictive privileged supervision successfully prevents latent collapse.

Highlights & Insights

  • Decoupled Training Privilege and Causal Inference: Converting hindsight trajectory observations into training-time privileged supervision allows the policy to internalize forward visual dynamics without introducing any information leakage during test-time evaluation.
  • Single-Token Recurrent Propagation: By routing continuous predictive states through a single-slot Pilot Token, the system bypasses the high latency of multi-turn text chains-of-thought and the resource burden of auxiliary generative models.
  • Cross-Embodiment Real-World Transfer: Beyond simulation, LatentPilot was successfully deployed on both wheeled (AgileX LiMO Pro) and quadruped (Unitree Go2) platforms, proving that internalized latent foresight effectively handles real-world sensor noise and low-level actuation dynamics.

Limitations & Future Work

  • Limitations: The model relies on a single continuous vector and mean-pooled features, which may lose local details in highly dynamic or cluttered environments; furthermore, the two-step prediction horizon (\(t+1\) and \(t+2\)) is static and cannot adapt to varying robot travel velocities.
  • Future Work: Future investigations could explore multi-token hierarchical latent spaces and adaptive variable-horizon foresight to improve predictive fidelity in complex, dynamically populated scenes.
  • vs NavMorph / Pathdreamer: Earlier approaches rely on external neural world models to synthesize future views for planning, incurring heavy computation and visual artifacts. LatentPilot absorbs predictive visual dynamics directly into the VLM latent space, outputting actions in a single forward pass.
  • vs NaVid / StreamVLN: While existing streaming VLMs focus on sliding-window encoding of past video frames ("what was seen"), LatentPilot explicitly models the causal impact of actions on future observations ("what will be seen"), mitigating myopic path oscillations.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Internalizing offline future frames as privileged supervision via a recurrent Pilot Token is an elegant, highly effective formulation]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated across continuous simulator, Isaac Lab physical humanoid, ablations, and dual real-robot platforms]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, well-structured methodology, precise mathematical formulation, and thorough analysis]
  • Value: ⭐⭐⭐⭐⭐ [Offers an efficient and practical blueprint for integrating world-model capabilities into real-time embodied VLMs]