Language-Conditioned World Modeling for Visual Navigation

Conference: NeurIPS 2026 (task metadata labels it Oral / 0.36%; the official acceptance designation and statistical denominator were not independently verified)
arXiv: 2603.26741v2
Code: https://github.com/UWMILab/LCVN
Area: Robotics & Embodied AI
Keywords: language-conditioned navigation, world models, latent imagination, actor–critic, offline trajectory prediction

TL;DR

The paper introduces a language-conditioned visual navigation dataset and compares two alternative approaches—“diffusion world model + latent-space actor–critic” and “unified autoregressive action/observation prediction”: the former has stronger image structural fidelity in seen environments, whereas the latter predicts offline trajectories better in unseen environments, without validating real closed-loop control.

Background & Motivation

Image-goal navigation models require a destination photograph, whereas natural language can directly specify a destination, intermediate landmarks, and turning instructions. However, “turn left at the gray door after the glass office” does not directly identify a continuous control command: the model must understand landmarks and spatial relations, anticipate what becomes visible after moving, and translate these judgments into planar displacement and yaw. Policy models such as GNM and NoMaD mainly predict actions directly; world models such as NWM explicitly imagine future views, but visual prediction alone does not constitute language-conditioned policy learning.

The paper follows an offline trajectory-prediction protocol inherited from goal-conditioned navigation, rather than giving a robot fresh camera feedback after every step. At test time, only the initial egocentric image and instruction are supplied; the entire subsequent action sequence depends on the model's imagination. This supports reproducible evaluation using existing real trajectories across platforms and exposes error accumulation: an inaccurate imagined view after a turn can remove the visual landmark grounding for later actions, even if the model initially understood the correct turn. The task complements interactive vision-and-language navigation (VLN), rather than replacing its evaluation.

To compare language grounding, dynamics prediction, and policy coupling, the authors provide three English instruction styles and two methodological paradigms. Core idea: replace goal photographs with language and compare “learn imaginable dynamics first, then a latent-space policy” against “share autoregressive representations and jointly predict actions and views” within the same offline trajectory task, examining how semantic goals influence control and long-horizon prediction.

Method

Overall Architecture

LCVN receives an initial RGB image and a persistent language instruction and outputs a continuous navigation action sequence; each action contains two planar displacement components and a yaw angle, with a null action indicating stopping. Training can access complete recorded trajectories, ground-truth actions, and future images; testing cannot access actual intermediate observations, expert future states, or goal photographs.

The two approaches are parallel alternatives, not stages of one combined model. The first trains LCVN-WM, freezes it, and then trains LCVN-AC; inference selects the first action from the initial latent and recursively follows “previous action → imagined next latent state → next action.” The second, LCVN-Uni, encodes instructions, images, and actions into a shared token sequence, jointly generates the next action and observation, and feeds the generated observation into the following step.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    D["Multi-style trajectory annotation"]
    I["Initial image + instruction"]
    W["Language-conditioned latent dynamics"]
    A["Expert-aligned latent policy"]
    U["Unified action–image prediction"]
    O["Offline predicted trajectory"]
    D -.->|Ground-truth trajectory supervision| W
    D -.->|Expert plan and latent-state supervision| A
    D -.->|Joint action and image supervision| U
    I -->|Latent-space approach| W
    W -->|Imagined latent state| A
    A -->|Next action and imagined history| W
    I -->|Unified approach| U
    U -->|Generated observation feedback| U
    A --> O
    U --> O

All solid feedback edges operate inside the models, without observations returning from a real environment; dashed edges denote training-data supervision only. The latent-space approach produces its first action directly from the initial image encoding, without first predicting an additional initial frame.

Key Designs

1. Multi-style trajectory annotation: describe the actual route rather than camera shake

The dataset combines five sources: Go Stanford, ReCon, SCAND, HuRoN, and TartanDrive. The authors normalize per-frame displacements by each source's average step size, remove backward movements and trajectories shorter than three steps, and use Qwen-3.5-9B to segment streams into semantically coherent scenes. Normalization makes actions from different platforms easier to represent in a shared control space, but representation standardization does not establish that the same predicted command is directly executable on every physical robot.

Annotation begins with human-written seed instructions. Examples matched to style, environment, and route complexity are then supplied to Qwen-3.5-9B together with egocentric video and a top-down 2D trajectory map, followed by human review and revision. The map exposes headings and turns, preventing terrain-induced camera shake or local sidesteps from being labeled as global turns; it supports annotation and is not a map supplied to navigation models at test time. Each trajectory receives three English instructions: Concise, Intricate, and Landmark-based, respectively probing minimal directional guidance, extraction of useful information from verbose descriptions, and associations between landmarks and actions.

The final dataset contains 39,016 trajectories and 117,048 instructions, with 28,813, 3,602, 1,500, and 5,101 trajectories in training, seen validation, unseen validation, and test. TartanDrive is excluded from training, exclusively supplies unseen validation, and contributes to the mixed test set. The appendix evaluates quality on 640 sampled trajectories; each trajectory–instruction pair is assessed by three research-group volunteers uninvolved in annotation, averaging 5.2/6.0 across five dimensions. This is an author-organized quality assessment, not independent external certification.

2. Language-conditioned latent dynamics: learn instruction-constrained visual changes from heterogeneously corrupted history

LCVN-WM uses an ImageVAE trained from scratch to compress images into 32×32 latent representations, then predicts the next latent state using a DiT-XL with approximately 1B parameters. Language is not merely added to a global conditioning vector: after freezing the CLIP text encoder, the model inserts multi-head cross-attention between self-attention and the feed-forward layer in every DiT block, allowing visual positions to access instruction embeddings. Semantic cues such as a glass wall on the left or a turn after a door can consequently influence visual evolution rather than only the final action head.

Actions, relative time shifts, and diffusion timesteps follow a separate conditioning pathway. Scalars receive sine–cosine encoding and MLP processing; their embeddings are summed to produce AdaLN scale and shift parameters. Actions specify actual motion direction, and time shifts identify prediction intervals. Language constrains destinations and landmarks but cannot replace action conditioning of dynamics, explaining why missing actions cause substantial degradation in the conditioning ablations.

Diffusion Forcing does not corrupt the entire history window to one shared noise level; it independently samples a noise level for each latent state. Its core transformation is:

\[ \hat{s}_{t-m}^{(\ell_m)}=\sqrt{\alpha_{\ell_m}}\,\hat{s}_{t-m}+\sqrt{1-\alpha_{\ell_m}}\,\epsilon_m. \]

Noise is independent across frames and drawn from a standard Gaussian distribution. Exposure to histories with differing reliability is intended to prepare the model for prediction errors in inference-time context; this training design does not imply that imagined histories receive corrective real-world feedback. History endpoints are indexed somewhat differently in the main text and appendix pseudocode; this note follows the experimental context sizes rather than guessing additional historical frames from that discrepancy.

3. Expert-aligned latent policy: use recorded trajectories to supervise imagined action selection

LCVN-AC learns inside the frozen world model's latent space, rather than through online trial and error in a real environment. A sequence-to-sequence conditional variational autoencoder (CVAE) compresses trajectories into latent plans: during training, an expert encoder reads the complete future trajectory, whereas the learner encoder predicts a plan from only the current latent state and CLIP instruction. The two distributions are aligned with KL divergence directed from the expert to the learner. Future information belongs only to the expert training branch; at test time, the learner must sample its own plan.

The actor chooses an action from the latent state, instruction, and latent plan, and the world model imagines the next latent state. Training rewards measure similarity between that prediction and the expert latent state at the same step—not physical arrival, collision avoidance, or compliance with social norms. The authors use the normalized intrinsic reward:

\[ r_t^{\text{int}}(s^E_{t+1},\hat{s}^{\pi}_{t+1})=\frac{s^E_{t+1}\cdot\hat{s}^{\pi}_{t+1}}{\max(\lVert s^E_{t+1}\rVert,\lVert\hat{s}^{\pi}_{t+1}\rVert)^2}. \]

Besides vector direction, this reward depends on norm mismatch and should therefore not be called ordinary cosine similarity. The critic estimates discounted intrinsic returns, and the actor learns to make imagined trajectories resemble expert demonstrations. An additional instruction-alignment loss raises similarity for matched latent-state/instruction pairs and lowers it for mismatched pairs, discouraging plans that fit geometry while ignoring the destination.

At test time, the initial latent is repeated to fill unavailable history; the learner produces a latent plan and the actor selects the first action, after which only world-model-generated states advance the history. Prediction terminates at a null action or the maximum number of steps. Although the appendix algorithm uses “executed trajectory,” the task definition and limitations explicitly specify offline prediction; that wording is not evidence of closed-loop execution on physical robots.

4. Unified action–image prediction: share autoregressive context between planning and imagination

LCVN-Uni has no separate actor–critic and instead applies LoRA fine-tuning to pretrained GAIR Anole-7B. A VQ tokenizer encodes images, BPE encodes instructions and prompts, and continuous three-dimensional actions are discretized using disjoint bin-token sets. Each training sample contains the current observation, current action, initial observation, and instruction and supervises both the next action and image, rather than alternating between separate action-prediction and image-prediction sample types.

The paper describes predicting both outputs in one unified forward pass, emphasizing avoidance of alternating planning and imagination substeps. The backbone remains causally autoregressive; “joint prediction” should not be read as eliminating autoregressive generation of visual tokens. At test time, its predicted next observation becomes the next input while language and initial-observation conditioning persist. It therefore also generates an entire trajectory within its imagined stream, without reading a fresh camera image at the next step.

The 4096-token window constrains the combination of history and spatial information. Context sizes 1 or 2 use 784 visual tokens per image; context size 4 permits only 625, reducing spatial resolution. The context ablation is consequently not a single-variable test of whether longer histories help or hurt. The semantic prior of a pretrained 7B model also introduces capacity and scale confounds relative to a 1B world model trained from scratch, preventing attribution of all unseen-environment gains to joint architecture alone.

A Worked Example

Suppose the initial view shows a corridor and the instruction requests passing a glass office on the left, turning left at a gray door, and stopping along the left wall. This is an illustrative input for explaining the mechanism, not an additional experiment or measured robot execution.

The latent-space approach generates a plan from the initial latent and instruction and first predicts forward motion; LCVN-WM receives that action and history and imagines the corridor latent after moving. LCVN-AC then updates its action choice using this imagined state, predicts a left turn at the model's estimated door location, and eventually outputs a null action when its stopping condition is met. Training can constrain the relevant latent using a recorded image at the doorway; at test time, an incorrectly imagined door position is not automatically corrected by external observations.

The unified approach places the same initial conditions and current state in a shared token sequence, jointly generates action and next-view tokens at each step, and recursively feeds them back. It may select the correct initial turn but lose the corridor or destination door afterward; the paper's failure case illustrates that correct directional intent does not guarantee later visual grounding. In either approach, “seeing the door” may describe a prediction rather than a sensor-confirmed event.

Loss & Training

The two approaches are trained independently. The appendix algorithm accumulates a diffusion prediction loss for LCVN-WM, learning denoising from corrupted context and the ground-truth next latent state; the cached source does not fully expand its parameterization, so this note does not invent an exact author-specified noise- or velocity-prediction formula. After freezing the world model, LCVN-AC trains the critic with squared-error regression to stop-gradient λ-returns constructed using a target critic. The actor minimizes negative λ-return plus weighted KL plan-consistency and instruction-alignment losses.

LCVN-Uni uses the joint objective:

\[ \mathcal{L}_{\text{joint}}=\mathcal{L}_{\text{plan}}+\lambda\mathcal{L}_{\text{imagine}}. \]

The action loss averages negative log probabilities of ground-truth bin tokens across the three action dimensions and includes text-token cross-entropy. The visual loss is not simply pixel MSE between generated and ground-truth images; it computes the expected squared distance from the true visual embedding to codebook vectors under the predicted token distribution:

\[ \mathcal{L}_{\text{imagine}}=\frac{1}{m}\sum_{j=1}^{m}\sum_{k=1}^{K}\lVert\mathbf{u}_j-\mathbf{c}_k\rVert^2\,P(s_j=\mathbf{c}_k). \]

The former constrains control bins, whereas the latter accounts for visual proximity in the codebook, exposing the shared backbone to both planning and imagination supervision. The appendix specifies the objective but not the numerical joint-loss weight in these passages; that hyperparameter is not guessed here.

LCVN-WM uses AdamW, linear warmup, and a constant learning rate of 1e-4, with 50 deterministic DDIM steps at inference; the NWM comparison uses 250 denoising steps. LCVN-Uni freezes all three tokenizers and updates only rank=16 LoRA adapters in qkv projections, training for 20 epochs with learning rate 2e-4 and global batch size 8. It uses 4 A100 GPUs with 80GB each, per-GPU batch size 1, and gradient accumulation 2. The approaches share current-task training data but not initialization, parameter scale, or inference budgets.

Key Experimental Results

Main Results

Absolute Trajectory Error (ATE) measures global deviation between aligned predicted and reference poses; Relative Pose Error (RPE) measures errors in relative motion between consecutive poses. Success Rate (SR) only tests whether the predicted endpoint is closer to the goal than the agent's average step size:

\[ \text{SR}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\!\left[d\!\left(\hat{p}^{(i)}_T,p^{(i)}_g\right)<\bar{s}\right]. \]

This endpoint threshold does not test collisions, safety, or recovery along the route. Since sources also undergo average-step-size normalization, the reported errors should not automatically be treated as a shared metric control precision across physical platforms. The following selection from main-text Table 1 preserves the original values.

Method	Context	Seen ATE↓	Seen SR↑	Unseen ATE↓	Unseen SR↑	Test ATE↓	Test SR↑
GNM (lang)	4	1.18	0.21	2.72	0.10	1.89	0.16
NoMaD (lang)	4	1.08	0.24	2.61	0.11	1.78	0.18
NWM (lang) + LCVN-AC	4	0.72	0.29	2.21	0.13	1.31	0.22
LCVN-Uni	2	0.36	0.42	1.21	0.27	0.65	0.36
LCVN-Uni	4	0.37	0.41	1.24	0.25	0.65	0.35
LCVN-WM + LCVN-AC	4	0.34	0.43	1.51	0.19	0.76	0.34

Visual evaluation separately uses SSIM, PSNR, LPIPS, and DreamSim. Appendix C.1 explicitly states that long-horizon tests recursively feed generated frames back while conditioning on ground-truth action sequences. “@8” compares the eighth predicted frame with its reference, rather than averaging the first eight steps or measuring navigation success under policy-predicted actions.

In main-text Table 2, LCVN-WM with context 4 obtains single-step SSIM/PSNR/LPIPS/DreamSim of 0.435/20.316/0.189/0.078 and eighth-step values of 0.293/11.527/0.315/0.127. LCVN-Uni with context 2 obtains corresponding single-step values of 0.423/13.325/0.298/0.072 and eighth-step values of 0.218/7.412/0.451/0.119. The latent-space approach is stronger on structural and pixel-related metrics, whereas the unified approach has lower DreamSim; neither wins every visual metric.

Ablation Study

The following selection combines main-text Tables 3 and 4 with appendix Table A3, all evaluated on seen validation. Conditioning rows use LCVN-WM with context 4; instruction-style and unified-strategy rows use LCVN-Uni with context 2. These are different experiment families and should not be treated as one cross-family module-removal experiment.

Experiment family and config	ATE↓	SR↑	SSIM@8↑	DreamSim@8↓
WM: time only	1.82	0.12	0.095	0.798
WM: action + time	0.46	0.35	0.241	0.161
WM: language + action	0.37	0.41	0.275	0.138
WM: language + action + time	0.34	0.43	0.293	0.127
Uni: Concise	0.37	0.41	0.218	0.121
Uni: Intricate	0.39	0.38	0.211	0.123
Uni: Landmark-based	0.32	0.47	0.225	0.116
Uni: Interleave	0.34	0.44	0.201	0.115
Uni: Predict Both	0.36	0.42	0.218	0.119

Removing WM language conditioning lowers seen SR from 0.43 to 0.35, indicating that semantic destinations add information beyond action-conditioned dynamics. Uni's “w/o ins” removes instructions only from the observation-prediction substep, not from the whole system; it is not symmetric with WM training without language.

Instruction-style diagnostics favor landmark-based over intricate instructions, but support only comparisons among these three English annotation styles. Latent-space/pixel-space ablation SR is 0.43/0.36, respectively. Additional Ego4D data also improves unseen visual prediction, but this separate external-data experiment should not be conflated with the main comparison using shared task-training data.

Appendix Table A2 includes both action prediction and imagination latency; NWM and WM timings also include LCVN-AC. These are per-step times, and the efficiency of joint prediction does not mean it is faster than every world-model alternative.

Per-step action prediction + imagination config	Seconds/step	Evidence status
NWM + LCVN-AC	11.2	Reported; 250 denoising steps
LCVN-Uni	20.5	Reported; unified autoregressive approach
LCVN-WM + LCVN-AC	6.4	Reported; 50 DDIM steps
LCVN-WM + distillation	0.6	Reported in appendix; 6 denoising steps, without a corresponding accuracy table
LCVN-WM + 4-bit quantization	0.1	Estimated; explicitly unexplored by the authors, not measured

Key Findings

• LCVN-Uni's unseen SR of 0.27 exceeds WM + AC's 0.19. This is consistent with pretrained priors and shared representations, but the experiments do not separate their causal contributions.
• Language specifies where to go, actions constrain how states change, and time is auxiliary; time alone cannot support useful dynamics and navigation. The authors' interpretation that control grounding is a greater bottleneck than language understanding has conditioning-ablation support, but is not an independent measurement of language understanding.
• Predict Both is 1.3 times faster than Interleave, but lowers SR from 0.44 to 0.42 and increases ATE from 0.34 to 0.36. Some visual metrics improve while others worsen: this is an efficiency–accuracy trade-off.
• Full WM seen SR is 0.43 in main-text Tables 1/3, 0.43 for the latent-space configuration in Table 5, and 0.42 for Concise instructions in Table 4. Values from different experiments are retained as reported rather than silently replaced with one standardized score.

Highlights & Insights

• The task separates correct language interpretation from correct prediction of an entire route. The absence of environment feedback after the initial image exposes loss of visual anchors after turns as a concrete failure mechanism, rather than equating attractive generated views with navigation ability.
• The 2D trajectory map supports annotation, not testing, and addresses geometric correctness of language supervision. This is transferable to cross-platform route-description datasets by distinguishing odometry-supported global turns from local egocentric camera motion.
• World models and policies interact in a shared compact latent space, allowing expert demonstrations to supply imagination rewards directly. The reusable principle is the information division between future-trajectory constraints during training and current-state plan prediction during testing—not the elimination of expert supervision.
• Richer descriptions need not improve control; identifiable landmarks provide higher route-relevant information density. Explicit selection of turning and landmark evidence is a possible extension, not a module already implemented in this paper.

Limitations & Future Work

• No closed-loop robot validation: execution-error recovery, dynamic pedestrian interaction, collision rates, and physical-platform safety are untested. Endpoint proximity under SR remains distinct from route executability.
• Confounded paradigm comparison: Uni uses pretrained 7B + LoRA, whereas WM uses approximately 1B parameters trained from scratch. Shared task data does not equal shared pretrained knowledge, capacity, or compute; matched initialization and scale controls are needed to attribute architectural gains.
• Context changes also change resolution: under the fixed 4096-token limit, context 4 uses 625 visual tokens per image, compared with 784 for context 1/2. Fixing spatial token counts or expanding the window would better isolate history-length effects.
• Limited language and domain coverage: instructions are English and follow three predefined styles; the main unseen domain is TartanDrive off-road driving. Multilingual inputs, noisy speech-like phrasing, ambiguous instructions, and broader indoor/platform transfer remain unvalidated.
• Incomplete latency–accuracy evidence: original WM at 6.4 seconds/step and Uni at 20.5 seconds/step do not demonstrate real-time control. Distillation at 0.6 seconds lacks paired accuracy results, and 4-bit latency of 0.1 seconds is explicitly an unexplored estimate.
• Expert-latent reward bias: rewards favor recorded trajectories even when an instruction permits multiple valid routes. Future work should distinguish deviation from demonstrations from deviation from semantic goals and validate imagination errors using closed-loop observations.

Related Work & Insights

• vs GNM / NoMaD: this paper replaces goal photographs with language and compares world-model approaches against policy-only alternatives. The baselines are adapted using CLIP text encoders and MLPs, so the comparison concerns task-specific language variants rather than all capabilities of the original methods.
• vs NWM: NWM provides action-conditioned visual dynamics; this paper adds per-block language cross-attention, Diffusion Forcing, and dedicated latent-policy learning. NWM (lang) uses AdaLN conditioning, so differences include language injection and world-model training design, not merely text availability.
• vs UniWM: the unified autoregressive approach retains shared multimodal sequences while replacing alternating action/observation prediction with joint training and prediction. Appendix speed gains support reducing alternating calls, but navigation accuracy declines, making this a non-cost-free optimization.
• vs Dreamer / LUMOS / DITTO: LCVN-AC extends latent imagination, latent plans, and expert-consistency rewards to language-conditioned offline navigation. The substantive question is how semantic goals and dynamics errors jointly shape policies, not whether offline reward optimization is equivalent to online reinforcement learning.

Rating

• Novelty: 4/5 — The task, three-style annotation, and systematic paradigm comparison are useful; components largely inherit existing world-model and latent-plan frameworks.
• Experimental Thoroughness: 3/5 — Covers unseen domains, conditioning, styles, and efficiency, but lacks closed-loop execution, matched pretraining controls, and accelerated-model accuracy validation.
• Writing Quality: 4/5 — Main text and appendix explain the mechanisms; wording such as “execution” and “single forward pass” requires protocol-aware interpretation.
• Value: 4/5 — A useful offline benchmark for language–imagination–control relationships, not direct evidence of deployment readiness or safety.