Dual-Anchoring: Addressing State Drift in Vision-Language Navigation

Conference: ECCV2026
Paper: ECCV Paper
PDF: Full Paper
Area: Robotics & Embodied AI
Keywords: vision-language navigation, state drift, instruction progress, landmark memory, retrospective world model

TL;DR

Dual-Anchoring adds training supervision for completed instruction segments and recently passed landmarks to streaming vision-language navigation, raising unseen R2R-CE SR from StreamVLN's 56.9% to 65.6% without retaining the auxiliary reconstruction branch in default deployment.

Background & Motivation

Vision-language navigation requires a robot to understand a natural-language route and continuously produce actions such as moving forward, turning, and stopping from egocentric observations. In continuous environments, the robot must translate linguistic spatial relations into low-level control instead of simply choosing the next node in a navigation graph. Video-LLM navigators such as StreamVLN preserve previous observations through streaming historical context, allowing action prediction to extend beyond the current frame. However, retaining historical inputs does not imply knowing the current task stage: the same corridor opening can require continuing forward or turning, depending on progress.

The paper calls the growing discrepancy between internal state and actual execution state state drift, distinguishing two related sources. Progress drift occurs when an unfinished sub-goal is treated as complete, such as executing a left turn before reaching the end of a corridor. Memory drift occurs when historical representations lose landmark specificity, leaving the robot unsure which room it has just passed and vulnerable to confusing similar locations. With next-action supervision alone, a model can obtain the correct answer from local visual cues without learning to maintain either state consistently. This is the authors' mechanistic hypothesis, supported by navigation improvements and state visualizations, rather than an exhaustive attribution of all navigation failures.

Instead of adding another complex planner at deployment time, the paper directly constrains what information historical representations should retain during training. The language branch restricts completed progress to a contiguous prefix of the original instruction, while the visual branch reconstructs features of the most recently passed landmark from the current model representation. The former supplies task-order constraints, and the latter supplies an object-centric reference to previous observations; both support the existing action-generation backbone. Core Idea: use instruction progress anchoring to constrain the current execution stage and memory landmark anchoring to constrain historical grounding, turning long-horizon state consistency into learnable auxiliary supervision.

Method

Overall Architecture

The model follows StreamVLN's streaming architecture with LLaVA-Video as its backbone, taking a natural-language instruction, current monocular RGB observation, and accumulated historical context as input. Its output remains navigation actions; the additions are Instruction Progress Anchoring (IPA) and Memory Landmark Anchoring (MLA). IPA is a text co-training task, while MLA is an auxiliary feature-reconstruction head; they constrain the same backbone in parallel rather than forming sequential controllers. Dashed edges below indicate training supervision; the action branch is the default deployment path, not a test-time call to Qwen3-VL or SAM.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Instruction and streaming RGB history"] --> Backbone["StreamVLN backbone"]
    Backbone --> IPA["Instruction Progress Anchoring"]
    Backbone --> MLA["Memory Landmark Anchoring"]
    Progress["Offline instruction-prefix labels"] -.-> IPA
    Landmark["Offline historical landmark SAM features"] -.-> MLA
    IPA -.-> Training["Training losses update the backbone"]
    MLA -.-> Training
    Backbone --> Action["Navigation actions<br/>Default deployment path"]
    Action -.-> Training

Supervision is generated from expert trajectories in advance; the navigator does not perform the large-model annotation tasks itself. The paper constructs 3.6M progress descriptions and 937K landmark supervision samples, converting coarse route instructions into time-dependent training targets. Default inference can discard the auxiliary branches; the paper reuses these capabilities as state probes when displaying progress text or landmark features. The method therefore shapes hidden states through explicit objectives rather than running an external memory-retrieval procedure before every action.

Key Designs

1. Instruction Progress Anchoring: turn an ambiguous task stage into a text target constrained by the original instruction

Standard navigation data usually provides complete instructions and trajectories without per-step answers identifying completed sub-goals. The authors overlay frame indices and executed action text in the upper-left corner of trajectory images, helping the offline annotator distinguish forward motion, turns, and movement between adjacent frames. They then sample trajectory steps at intervals, providing the complete instruction and visual history only up to each sampled step rather than using future frames to interpret current progress. Qwen3-VL first compares actions and landmarks against visual evidence, then summarizes completed sub-goals using the instruction's original wording. A language model subsequently refines the output, removes intermediate reasoning, and strictly limits the result to a contiguous prefix of the original instruction. This restriction makes supervision resemble a route checklist rather than a scene description that can freely introduce objects or skip steps. The remaining task is implicit in the excluded instruction suffix; the method should not be read as requiring two independently maintained lists.

During training, the relevant prompt requests completed progress first, and the target sequence places the synthetic progress description before the expert action. The model must therefore interpret task state and predict actions from the same historical context, connecting its linguistic state to the control objective. Unlike a generic action explanation, the supervised quantity is the completion boundary within the entire route, not merely why a left turn is appropriate now. The progress target is also not a scalar completion rate: equally long prefixes can refer to different landmarks and directions, which the original wording preserves. This constraint assumes broadly sequential instruction execution; backtracking, revisits, or nonsequential instructions may be difficult to express as a contiguous prefix. That is an applicability boundary inferred from the method's assumptions, not a separately tested failure finding in the paper.

2. Memory Landmark Anchoring: retrospectively recover the most recently passed landmark from historical model representations

Even correct progress text does not guarantee that the model remembers a door or room's visual details, so the authors add a verifiable objective for visual history. Offline, Qwen3 decomposes instructions into atomic sub-goals containing actions or landmarks, and Qwen3-VL locates each landmark's first appearance in the complete video. Annotations must satisfy strictly increasing appearance times across sub-goals; samples violating that temporal order are filtered out. At each training step, the system selects the most recent landmark frame no later than the current step and extracts high-resolution spatial features using frozen SAM. The complete video is used by the offline landmark localizer; the navigator's reconstruction target always comes from a frame that has already occurred. The stated target is a dense SAM feature map of that landmark frame, without an explicit implementation that crops only one object mask; object-centric supervision should not be equated with single-object segmentation labels.

The reconstructor reads the Video-LLM output sequence containing historical and current visual semantics rather than copying the target landmark image directly. A linear projection and LayerNorm first reduce the feature dimension to lower subsequent attention costs. Learnable queries corresponding to a target spatial grid then retrieve local cues from the compressed sequence through cross-attention. The query outputs are projected into SAM's channel dimension, reshaped into a spatial feature map, and compared with the frozen target using mean squared error. If the backbone has forgotten the historical landmark, reconstruction becomes harder, encouraging gradients that preserve distinguishable historical object information. Unlike world models predicting future images, this model predicts features of past observations rather than relying exclusively on visual states that have not yet occurred. The auxiliary head is neither an explicit map nor an independent action selector; it improves backbone representations through training.

A Worked Example

The instruction in Figure 4 on page 11 asks the agent to leave the bathroom, continue to the end of the hallway, and then turn left. At step 17, the robot encounters an opening that appears suitable for turning but is not the actual end of the hallway. The baseline turns left prematurely, consistent with marking the straight-ahead segment complete and moving to the next sub-goal too early. The dual-anchored model's progress probe indicates that the straight-ahead stage remains unfinished, while its landmark probe reconstructs features of the earlier bathroom. These signals respectively indicate that the turn is not due yet and that the bathroom remains the recent historical reference, helping explain continued forward movement. The auxiliary outputs visualize latent states; production inference need not print progress or display SAM features at every step. This is a qualitative trajectory example, not sufficient evidence by itself of a causal relationship between the internal states and actions.

Loss & Training

Stage 1 pretrains on base navigation data while retaining action, progress-text, and landmark-reconstruction objectives. The navigation data includes 180K trajectories from R2R, RxR, and EnvDrop, plus 155K trajectories from the HM3D subset of ScaleVLN. Progress and action targets use text-generation supervision, while the landmark branch minimizes mean squared error between predicted and frozen SAM features. The weighted objective below summarizes the readable description in Section 3.4; it is not a symbol-by-symbol restoration of the damaged equation extraction.

\[ \mathcal{L}_{\mathrm{Stage1}}=\mathcal{L}_{\mathrm{nav}}+\lambda_{\mathrm{prog}}\mathcal{L}_{\mathrm{prog}}+\lambda_{\mathrm{WM}}\mathcal{L}_{\mathrm{WM}}. \]

Stage 2 uses the Stage 1 policy to collect approximately 240K DAgger correction samples, mitigating the distribution shift between following expert trajectories during training and the model's own trajectories at test time. It also mixes in 400K VideoQA samples and 230K image-text samples to preserve the backbone's general vision-language capabilities. The authors state that both anchoring objectives remain active for navigation-related batches in Stage 2 rather than being abandoned after pretraining. Implementation uses AdamW with a learning rate of 1e-5 and 32 NVIDIA H200 GPUs. The cache does not provide a verifiable complete hyperparameter appendix, so loss weights, query-grid dimensions, and sampling intervals are not supplied here. These training resources and offline annotation costs must be distinguished from the absence of auxiliary-head computation in default inference.

Key Experimental Results

Main Results

All results use validation unseen splits and come from Table 2 on page 10; the comparison selects monocular RGB methods StreamVLN, DualVLN, and the proposed model. NE measures final navigation error to the goal, where lower is better; SR is success rate and SPL is success weighted by path efficiency, both reported as percentages. OSR measures whether the trajectory ever enters the success region rather than whether the final stop is correct; it is omitted below for compactness. Differences in SR and SPL should be read as percentage points, not automatically as relative percentage improvements.

Dataset	Method	NE ↓	SR (%) ↑	SPL (%) ↑
R2R-CE	StreamVLN	4.98	56.9	51.9
R2R-CE	DualVLN	4.05	64.3	58.5
R2R-CE	Dual-Anchoring	4.15	65.6	62.1
RxR-CE	StreamVLN	6.22	52.9	46.0
RxR-CE	DualVLN	4.58	61.4	51.8
RxR-CE	Dual-Anchoring	4.42	61.7	53.3

Against StreamVLN, R2R-CE SR increases by 8.7 percentage points and SPL by 10.2 points; RxR-CE SR increases by 8.8 points. The method does not lead every metric: its R2R-CE NE of 4.15 is still higher than DualVLN's 4.05. The abstract claims a 15.2% relative SR improvement, while the rounded table values of 56.9 and 65.6 yield approximately 15.3%; this note retains the table values without forcing consistency with the abstract.

Ablation Study

Results come from Table 3 on page 12, all evaluated on R2R-CE validation unseen. Standard denotes R2R+RxR+EnvDrop, while All denotes the paper's full-data setting.

Training Setting	IPA	MLA	NE ↓	SR (%) ↑	SPL (%) ↑
Standard	No	No	6.49	40.8	37.4
Standard	Yes	No	6.27	45.4	41.1
Standard	No	Yes	6.01	47.7	43.3
Standard	Yes	Yes	5.73	49.5	44.9
All	No	No	4.98	56.9	51.9
All	Yes	No	4.72	60.8	55.1
All	No	Yes	4.51	62.5	57.3
All	Yes	Yes	4.15	65.6	62.1

Standard controls the total number of training samples: adding IPA replaces half the base navigation samples with progress-augmented data, better isolating the effect of supervision. For All, the authors explicitly do not control total training samples and instead add all progress data; the entire improvement cannot be attributed solely to module structure. The caption says navigation sample counts are identical, while the text distinguishes total-sample controls; these statements must be read together rather than summarized as completely matched data budgets for every ablation. In Standard, MLA alone reaches 47.7% SR, exceeding IPA alone at 45.4%; the combination reaches 49.5%, supporting complementarity without establishing linearly additive gains.

Key Findings

Figure 5 on page 13 divides episodes by geodesic distance into Short [3.85, 7.55), Medium [7.55, 9.81), and Long [9.81, 21.04] meters. The Long group reports relative improvements of 24.7% in SR and 33.2% in SPL; these are relative percentages, not percentage points or absolute success rates. Absolute SR/SPL values for each length group are not inferred from the extracted chart text to avoid guessing bar heights. The separate annotation-quality analysis appears in Table 4 on page 13; the table below retains raw scores rather than only relative improvements.

Annotation Data	Metric	Control	Ours	Control Setting
Progress descriptions	HR (%) ↓	8.13	6.04	No visual kinematics overlay
Progress descriptions	LCS (1–5) ↑	1.71	4.26	No visual kinematics overlay
Landmark frames	LPR (%) ↑	13.9	75.6	Random frame sampling

HR measures whether descriptions contain entities absent from the original instruction, while LCS scores whether completed progress forms a contiguous, non-skipping prefix on a 1–5 scale. LPR uses Qwen3-VL to assess whether a landmark is recognizable in the selected frame; it is not human-annotated localization accuracy, and 75.6% also indicates imperfect supervision. Real-world deployment uses a Unitree Go2 and D435i camera, streaming RGB to an H20 server for action prediction and executing commands through the robot API. Pages 14–15 provide qualitative examples only, without a real-world success-rate table, trial count, or end-to-end latency measurements; they do not quantitatively establish onboard real-time operation.

Highlights & Insights

• The paper separates the presence of historical context from the verifiability of historical state. Increasing input length alone need not prevent landmark forgetting, whereas reconstruction directly constrains retained information.
• Progress supervision reuses instruction prefixes instead of freely generating lengthy reasoning. This gives task order an explicit boundary and limits arbitrary route rewriting by annotators.
• A retrospective world model need not predict every possible future. It supervises representations using visual references that have already occurred, offering another direction for long-horizon control.
• Both anchoring objectives primarily act during training. The transferable idea is to add progress and historical-verification targets to other long-horizon embodied tasks, not to copy an entire navigation controller.

Limitations & Future Work

• The authors do not provide a dedicated limitations section; the following points are primarily reader assessments of the method and evidence, not author-reported failure modes.
• Progress and landmark annotations rely heavily on automated annotators, with model-based quality evaluation as well. Independent human audits should inspect plausible progress labels that conflict with the actual trajectory.
• Strict temporal monotonicity and contiguous prefixes suit sequential routes but may filter valid revisits or detours. State representations that allow backtracking deserve further evaluation.
• Landmark targets are dense features of full frames; object-level interventions and representation analyses are needed to determine whether the model retains instruction-relevant objects rather than mostly background cues.
• Full-data ablations mix supervision-design gains with sample-scale gains. Comparisons matching total tokens, training steps, and compute budgets would strengthen attribution.
• Real-robot examples use a remote server without reporting network latency, disconnection robustness, or comprehensive success statistics. Deployment capability needs standardized repeated trials.
• Several equations are damaged in the text extraction, and the referenced detailed appendix is absent; this note does not invent exact tensor hyperparameters or missing loss weights.

Related Work & Insights

• Compared with StreamVLN: the streaming navigation backbone is retained, while progress-text and landmark-reconstruction supervision are added to the same historical context instead of replacing the control architecture.
• Compared with NavForesee and NavMorph: the paper characterizes these as future-oriented feature-prediction methods, whereas this work emphasizes historical references. The objectives may complement each other, but joint training is not evaluated here.
• Compared with SAM: SAM is a frozen feature teacher rather than an online segmentation-based navigator; its object representations constrain memory in the navigation backbone.
• Research direction: independently perturb progress and memory under repeated rooms, landmark occlusion, and instruction backtracking to test their causal contributions to drift; this is a proposed experiment, not an existing result.

Rating

• Novelty: 4/5. Combining instruction-prefix supervision with historical landmark reconstruction gives navigation state drift explicit learning objectives.
• Experimental Thoroughness: 3/5. Two benchmarks, component ablations, and annotation-quality analysis are included, but full-data budget controls and real-world statistics remain incomplete.
• Writing Quality: 4/5. The two drift modes are clearly explained, although training-time auxiliary outputs and default inference require careful distinction.
• Value: 4/5. The work offers an empirically supported training direction for long-horizon navigation without auxiliary-head computation in default inference.