AgentVLN: Towards Agentic Vision-and-Language Navigation¶
Conference: ECCV2026
Paper: Official page · PDF
Code: https://github.com/Allenxinn/AgentVLN
Area: Robotics / Embodied Intelligence
Keywords: vision-and-language navigation, skill scheduling, cross-space mapping, closed-loop correction, active perception
TL;DR¶
AgentVLN uses Qwen2.5-VL-3B to schedule perception and planning skills through geometrically grounded image prompts, with local correction and on-demand depth queries resolving navigation ambiguities, achieving 67.2% / 69.5% SR on R2R-CE / RxR-CE Val-Unseen.
Background & Motivation¶
Vision-and-language navigation requires a robot to turn an instruction such as walking through a corridor and stopping near a chair into actions in continuous space. A VLM can recognize objects and understand semantics without knowing their metric distance, whether a passage is reachable, or whether its predicted movement respects scene geometry. NaVid and StreamVLN learn navigation policies from video observations; dual-system approaches such as DualVLN and InternVLA-N1 separate high-level decisions from low-level execution, but must still connect the model's two-dimensional observations to three-dimensional trajectories.
The difficulty is not simply remembering more history. When an occlusion hides useful routes, a long movement can compound a trajectory error. When the target is visible but its distance is unclear, selecting a destination from RGB alone can still produce an incorrect stopping location. Additional geometry encoders can help, but increase model and inference costs. AgentVLN instead delegates geometric computation to callable tools and lets the VLM consume results already aligned with image coordinates. This does not remove the depth sensor or mapping system: it reallocates responsibilities between learned semantic reasoning and geometric processing.
Core idea: turn three-dimensional reachability into selectable pixel prompts, execute long movements through skills, and switch to corrective actions or active perception when route evidence or distance information is missing.
Method¶
Overall Architecture¶
The system receives a navigation instruction, current RGB-D observations, camera pose, and historical context. It outputs perception queries, planning-skill calls, or fine-grained actions when necessary. Qwen2.5-VL-3B decides which route to select, when to acquire more evidence, and when to localize the destination; lower-level components handle incremental mapping, waypoint generation, obstacle avoidance, and movement execution.
During ordinary navigation, perception and planning tools supply reachable waypoints projected onto the current image. The VLM selects a prompt consistent with the instruction, and a planning skill executes the associated waypoint. Missing prompts trigger context-driven fine-grained correction. Once the target appears, query-driven perceptual reasoning resolves local ambiguity before the selected target pixel is mapped back into a three-dimensional destination. The diagram shows inference-time routing; dataset construction and training are explained separately below.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
Input["Instruction, RGB-D<br/>pose and history"] --> Brain["Skill scheduling and<br/>cross-space mapping"]
Brain -->|No suitable waypoint| Correct["Context-driven<br/>fine-grained correction"]
Brain -->|Target visible| Query["Query-driven<br/>perceptual reasoning"]
Brain -->|Select reachable waypoint| Execute["Planning skill<br/>closed-loop movement"]
Correct -->|Updated observation| Brain
Query -->|Target pixel to 3D| Execute
Execute -->|Update context if unfinished| Brain
Execute -->|Destination reached| End["Navigation complete"]
Key Designs¶
1. Skill scheduling and cross-space mapping: select visual candidates backed by geometry
The paper formulates high-level decision-making as a Partially Observable Semi-Markov Decision Process, or POSMDP. Partial observability reflects the robot's local sensor coverage; the semi-Markov formulation reflects the variable number of low-level steps consumed by a skill. The principal action space consists of parameterized skill calls rather than velocities or directions issued at every instant. Perception skills return geometric or semantic information and are modeled as not advancing physical time. Planning skills execute closed-loop control over multiple time steps until a termination condition requests another VLM decision. This abstraction does not imply that perception has zero computational latency.
Cross-space mapping makes the tool outputs usable by the VLM. Depth, camera intrinsics, and pose back-project valid pixels into world coordinates to update an occupancy grid, from which the planning layer generates feasible three-dimensional waypoints. The inverse camera-pose transformation and perspective projection then draw these waypoints back into the current RGB image. The VLM selects the prompt that matches the instruction and returns the choice to the planning layer, instead of inventing metric coordinates from appearance. Camera geometry supplies deterministic coordinate conversions, not a new learned loss. The cached equations contain extraction damage, so incomplete expressions are not reproduced here. The contribution is the pixel-aligned interface between semantic selection and physical reachability, rather than forcing all geometric information into latent features.
2. Context-driven fine-grained correction: recover useful observations when candidates disappear
Image prompts depend on the current field of view, so an instruction-consistent waypoint may not always be visible. A wall can hide the next corridor after a turn, or accumulated localization error can leave the robot facing away from a passage. Committing to a distant waypoint in such a state can make matters worse. The context-driven fine-grained strategy, abbreviated CDFG, conditions on history, the current image, and the instruction to issue actions such as Forward, Left, or Right. Looking around, advancing locally, or adjusting orientation can restore useful route evidence.
This is a recovery branch, not a permanent return to VLM control at every movement step. Once a valid path is again projected into the image, the agent resumes efficient waypoint-level execution. Training also injects random trajectory noise to expose the model to states away from the expert route. When no valid waypoint appears in the image, supervision switches to fine-grained action sequences. Recovery therefore has explicit support in the training distribution rather than relying solely on an inference-time request to reconsider. The main text does not provide precise movement increments or complete triggering thresholds, so a strict recovery guarantee cannot be inferred.
3. Query-driven perceptual reasoning: seeing the target does not remove the need to ask about distance
When observed evidence matches the destination description, the agent switches from global path navigation to local target localization. The question changes from which route to follow to which image location represents the requested stopping point. QD-PCoT allows the VLM to request missing information before producing coordinates. For example, it can ask how far away the chair ahead is, invoke a perception skill, and append the returned geometric or semantic information to the context as text. After interactive querying and reasoning, it outputs the destination pixel.
Executing that choice still requires explicit geometry. The system reads depth at the selected pixel, back-projects it into camera coordinates with the camera intrinsics, transforms it into world coordinates using the current pose, and passes the result to a planning skill. Two distinct operations matter: using depth feedback to decide which pixel to select, and converting an already selected pixel into a physical target. Querying handles the former; coordinate transformation handles the latter. On-demand queries reduce the need to load all scene geometry into the VLM context, but they cost computation and depend on reliable depth and pose. No additional model parameters does not mean no additional system overhead.
A Worked Example¶
Consider the illustrative instruction to walk through a corridor and stop beside the chair ahead. This explains the mechanism rather than reproducing a measured trajectory from the paper. Before the chair becomes visible, RGB-D mapping produces projected waypoints. The VLM selects one aligned with the corridor, and a planning skill executes the movement without asking the model to generate every control update.
At a corner, the image may contain no useful waypoint. CDFG uses the recent history to turn or advance locally until the passage comes into view, after which waypoint navigation resumes. When the chair becomes visible, the model switches to local localization. If the distance relationship remains ambiguous, it queries perception before selecting a destination pixel. Depth-based back-projection supplies the physical goal for the final movement. Missing route visibility and uncertain target scale are handled by different mechanisms, not collapsed into an unspecified reflection module.
Loss & Training¶
AgentVLN-Instruct is built in Habitat from R2R and RxR, covering 61 indoor scenes. Target visibility determines the output format at each training step. In global navigation, the mapped waypoint closest to the expert trajectory becomes the label. Off-route states without valid waypoints receive fine-grained action supervision. Local localization uses multi-round reasoning with perception questions and answers, ending in target-pixel prediction. The dataset thus teaches candidate selection, recovery from erroneous states, and information acquisition before coordinate prediction, rather than only imitation of a single expert action sequence.
Training mixes in LLaVA-Video-178K to preserve general multimodal capabilities. The visual encoder and multimodal projection layer are frozen during instruction tuning. The main text does not introduce a separate new loss, or specify the total AgentVLN-Instruct sample count, mixture ratio, and complete tool interfaces; no objective formula or dataset size is invented here.
Optimization uses AdamW with batch size 128, cosine learning-rate scheduling, a peak learning rate of \(2\times10^{-5}\), and a warmup ratio of 0.03. Observations are 640×480 with a 110° field of view, and the history window contains 8 frames. Experiments use 32 NVIDIA A100 GPUs. Lightweight primarily describes the deployed 3B model and the absence of an additional learned 3D module, not a low training-resource requirement.
Key Experimental Results¶
Main Results¶
The following selection comes from Tables 1 and 2, both evaluated on Val-Unseen. NE measures final navigation error in meters and is lower-is-better. SR measures final success; SPL discounts successful episodes that take unnecessarily long paths relative to the shortest route; OS measures whether the trajectory ever enters the success region. nDTW measures normalized dynamic-time-warping similarity to the reference trajectory. Scores other than NE use the paper's percentage scale.
| Dataset | Method | Input | NE ↓ | SR ↑ | SPL ↑ |
|---|---|---|---|---|---|
| R2R-CE | EfficientVLN-4B | RGB | 4.18 | 64.2 | 55.9 |
| R2R-CE | DualVLN-7.1B | RGB | 4.05 | 64.3 | 58.5 |
| R2R-CE | InternVLA-N1-8.3B | RGB-D | 4.83 | 58.2 | 54.0 |
| R2R-CE | AgentVLN-3B | RGB-D | 3.88 | 67.2 | 64.7 |
| RxR-CE | EfficientVLN-4B | RGB | 3.88 | 67.0 | 54.3 |
| RxR-CE | DualVLN-7.1B | RGB | 4.58 | 61.4 | 51.8 |
| RxR-CE | InternVLA-N1-8.3B | RGB-D | 5.91 | 53.5 | 46.1 |
| RxR-CE | AgentVLN-3B | RGB-D | 3.92 | 69.5 | 61.3 |
Relative to EfficientVLN, SR improves by 3.0 / 2.5 percentage points and SPL by 8.8 / 7.0 points on R2R / RxR. These are not matched-input comparisons: AgentVLN explicitly uses depth. Nor does it lead every metric. On RxR, its NE of 3.92 meters is slightly worse than EfficientVLN's 3.88; on R2R, its OS of 73.5 is slightly below EfficientVLN's 73.7. AgentVLN's RxR nDTW is 74.6, compared with DualVLN's 70.0.
Ablation Study¶
Table 3 reports an incremental study on the R2R validation set. The initial VLM-as-Brain addition combines high-level skill scheduling with cross-space mapping, so its entire gain cannot be assigned to either component individually.
| Configuration | NE ↓ | OS ↑ | SR ↑ | SPL ↑ |
|---|---|---|---|---|
| End-to-end baseline | 6.53 | 48.50 | 38.60 | 35.10 |
| + VLM-as-Brain and cross-space mapping | 4.67 | 65.10 | 59.70 | 55.60 |
| + CDFG | 3.90 | 72.10 | 65.60 | 63.40 |
| + QD-PCoT, full model | 3.88 | 73.50 | 67.20 | 64.70 |
Key Findings¶
- Scheduling and mapping provide the largest SR increment, 21.10 percentage points. CDFG adds another 5.90 points and raises SPL from 55.60 to 63.40, supporting the interpretation that recovery reduces wasted movement.
- QD-PCoT adds 1.60 points of SR and 1.30 points of SPL at the final stage. The incremental design does not isolate module interactions, and statistical significance is not supplied here.
- Increasing the history window from 2 to 8 frames raises SR from 56.7 to 67.2; 10 frames gives 67.1. This supports saturation, but the 0.1-point difference alone does not establish severe attention dilution.
- Real-world deployment uses a Unitree Go2, RealSense D455, and RTAB-Map on a Jetson Orin NX. The authors report approximately 3 seconds for a forward pass and autoregressive generation, reduced to approximately 1 second with W4A16. These are high-level decision latencies, not the robot's low-level control period.
Highlights & Insights¶
- Pixel waypoints provide an inspectable interface between semantics and geometry. One can separately examine whether mapping generated a correct candidate and whether the VLM selected it, helping localize failures.
- Correction and disambiguation address distinct information gaps: CDFG changes what is observable by moving, while QD-PCoT supplements geometric evidence by querying. This makes the intervention and its training supervision more specific than generic verbal reflection.
- A reusable lesson is to retain fast physical feedback loops in planning and control while assigning slower semantic decisions to the VLM. Smaller learned models can retain geometric capability, but transfer evaluations must account for tool and sensor costs as well.
Limitations & Future Work¶
- These are primarily reading-based assessments of the main text, which has no dedicated limitations section. Depth errors, pose drift, and calibration errors can directly corrupt projected prompts and target back-projection; systematic robustness tests for these conditions are not provided.
- Real-world evidence consists mainly of indoor/outdoor examples and inference latency, without clear task counts, aggregate success rates, energy measurements, or failure-type distributions. The real-time claim needs to be read in the context of second-scale high-level decisions and continuously running low-level control.
- Complete perception and skill interfaces, visibility-routing thresholds, and the training sample count are insufficiently specified in the main text. Replacing a skill to support a different sensor is an architectural possibility, not a demonstrated comprehensive cross-sensor transfer result.
- Useful follow-ups include separately ablating projected prompts and skill scheduling under matched RGB-D inputs and compute budgets, plus testing sensor degradation, dynamic obstacles, and navigation success before and after quantization.
Related Work & Insights¶
- NaVid / StreamVLN: These emphasize learning actions from video or managing historical context. AgentVLN externalizes reachability computation and asks the VLM to select visual prompts. The distinction is where spatial reasoning occurs, not just parameter count.
- DualVLN / InternVLA-N1: Both separate high-level semantics from low-level execution. AgentVLN emphasizes explicit pixel-waypoint bridging and callable skills. Its smaller parameter count and stronger tabulated success rates should still be considered alongside input modalities and system composition.
- EfficientVLN: The main text contrasts its monocular depth-estimation approach with AgentVLN's explicit depth tools. A useful research question is which spatial judgments should be learned and which should be delegated to verifiable geometry under a fixed sensor budget.
Rating¶
- Novelty: 4/5. The value lies in combining pixel-level geometric interfaces, skill scheduling, and on-demand queries rather than new foundational geometry or planning algorithms.
- Experimental Thoroughness: 3/5. Two unseen-environment benchmarks and incremental ablations are informative, but matched-input comparisons, real-world statistics, and robustness evidence are limited.
- Writing Quality: 4/5. Global navigation and local localization are clearly organized, although claims about eliminating ambiguity and real-time operation are stronger than the presented evidence.
- Value: 4/5. The paper offers an actionable system design for smaller embodied models, especially for researchers studying collaboration between geometric tools and VLMs.