AMCoNav: Asynchronous Multi-module Collaborative Framework for Embodied Visual Navigation

Conference: ECCV2026
Paper: Official paper page
PDF: Full paper
Area: Robotics / Embodied Visual Navigation
Keywords: asynchronous collaboration, multimodal shared context, open-vocabulary navigation, probability fusion, target verification

TL;DR

AMCoNav keeps a lightweight navigation policy running while large models reason on demand in the background, then uses shared context and probability fusion to choose exploration, verification, or direct navigation, achieving 60.0% / 60.0% / 53.3% success on the three HM3D-OVON splits and reducing runtime against a synchronous counterpart without guaranteeing shorter paths.

Background & Motivation

Embodied visual navigation requires an agent to maintain spatial structure from RGB-D observations, interpret category-level or instance-specific goals, and produce timely movement decisions. Modular systems such as MTU3D perform instance representation and navigation scoring efficiently, but lightweight policies can mistake similar objects for the target when observations are ambiguous or instructions require open-vocabulary understanding. Multimodal large models offer room knowledge, object associations, and visual verification, but introduce another question: must the robot stop until they respond?

Serial pipelines can propagate early errors, while parallel decision models can still block if a final arbiter waits for every branch. Asking a large model for a global plan only at the beginning is also insufficient when the actual room layout differs from its initial inference. AMCoNav instead connects signals operating at different timescales: room-level advice may arrive late, whereas observed objects and exploration frontiers must continue to update.

Core idea: turn slow semantic reasoning into asynchronously updated shared state, let the real-time policy continue acting, and use two sources of evidence plus the remaining action budget to decide when to explore, inspect again, or commit to a target.

Method

Overall Architecture

Inputs are RGB images, depth, poses, and a text or image goal; outputs are the next navigation coordinates and the corresponding action sequence. Multimodal Shared Context (MSC) stores global objects, frontiers, observation snapshots, and semantic guidance. The Real-Time Multimodal Decision Module (RMDM) scores objects and frontiers, Zero-shot Large Model Modules (ZLMM) supply on-demand guidance and verification scores, and the Bayesian Probabilistic Decision Fusion Module (BPDFM) selects an action mode.

Here, zero-shot describes how the large-model branch is used, not an absence of training throughout the system. RMDM is a specifically trained navigation network. Except for relevant-object reasoning at task initialization, large-model calls run in separate subthreads. The main loop does not require every response to arrive before proceeding, and missing verification scores have an explicit treatment.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["RGB-D, pose, and goal"] --> MSC["Multimodal Shared Context<br/>MSC"]
    MSC --> RMDM["Real-Time Multimodal Decision<br/>RMDM"]
    MSC --> ZLMM["On-Demand Large-Model Reasoning<br/>ZLMM"]
    ZLMM -->|asynchronous guidance updates| MSC
    RMDM -->|real-time scores| BPDFM["Probability Fusion and Mode Switching<br/>BPDFM"]
    ZLMM -->|available verification scores| BPDFM
    BPDFM --> Action["Explore / Verify / Navigate"]
    Action -->|new observations| MSC

Key Designs

1. Multimodal Shared Context: give delayed results a usable spatial reference

MSC is navigation state, not merely a conversation history. It comprises Semantic Scene Context (SemSC), Spatial Snapshot Context (SpatSC), and Reasoning Semantic Context (ReaSC). SemSC follows MTU3D's Online Query Representation: each local object carries representation features, confidence, a globally positioned 3D box, an instance mask, and an open-vocabulary embedding. IoU-based Hungarian matching merges local objects into global instances. Frontiers mark the boundary between explored and unexplored space; each stores a 3D coordinate and a room type and is updated before a decision. Consequently, large-model verification and real-time scoring can refer to aligned candidates rather than unrelated natural-language answers.

Room recognition makes the latency issue especially clear: the robot may have moved away before the model answers. SemSC therefore maintains a 2D room-type field and a confidence field, placing each result back into the region associated with its observation. The reference point is 1.5 m ahead of the observation direction, and the influence radius is 3 m. Local confidence decays with distance through a Gaussian construction, with different upper bounds inside and outside the field of view. A location accepts a new room label only when its confidence exceeds the stored value, and retains the larger confidence. The real-time policy reads room types at projected frontier locations without waiting for fresh recognition. The cached piecewise expression in Equation (1) is corrupted, so its exact kernel, region conditions, and missing parameters are not reconstructed here.

SpatSC links each observation snapshot to its visible instances and extracts two subsets. For each frontier, it chooses the snapshot with the nearest reference point. For key objects, it selects instances with perception confidence below 0.4 but an RMDM decision score above 0.6, then greedily adds the snapshot covering the most uncovered key objects. This concentrates expensive verification on candidates that look promising but are poorly perceived. The paper calls this a minimal snapshot set, although a greedy coverage procedure does not establish global minimum cardinality.

ReaSC holds a relevant-object list and a sequence of rooms likely to be traversed. Object relevance depends on the task and remains fixed within it. Room guidance is consumed as the agent progresses: the next room is removed when reached, and new observations revise the sequence when the layout diverges from the prediction. The timescale argument is that exploring a room generally takes longer than room-level inference, leaving delayed guidance useful for subsequent movement.

2. Real-Time Multimodal Decision: match object and room semantics separately

RMDM is a four-layer Transformer-style decoder in the main thread. It receives global object queries, frontier queries, and the task goal. Object queries use instance representations and spatial positions derived from 3D boxes; frontier queries carry room types and positions derived from frontier coordinates. Room labels, relevant rooms, relevant objects, and the text or image goal are encoded with CLIP and projected into a shared feature space.

Cross-attention does not mix every semantic input indiscriminately. Frontier queries first attend to room guidance to determine what kinds of space are worth exploring, while object queries attend to relevant-object guidance to exploit contextual associations around the target. Both branches then attend to the task goal, followed by spatial self-attention that models inter-instance geometry and produces per-object and per-frontier scores. The same network can therefore choose a known instance when the target is visible or an exploration direction when it is not. The slow branch supplies semantic conditioning and complementary evidence; the real-time branch retains its own scoring capability.

3. On-Demand Large-Model Reasoning: tie calls to navigation state

ZLMM has four responsibilities rather than asking one model to solve the entire navigation problem on every frame. The Relevant-Object Reasoning Module (RORM) runs once at task initialization, predicts objects likely to be associated with the target, and stores the top five in ReaSC. The Room-Navigation Reasoning Module (RNRM) uses frontier snapshots, supplemented with textual descriptions of out-of-view objects within 2 m, to infer a room sequence toward the goal. It is invoked only when a new room type is detected, allowing guidance to change when the layout provides new evidence.

The Room-Type Perception Module (RTPM) identifies the room ahead and its confidence from the latest image, then updates SemSC. The method text triggers it when average confidence in the visible region falls below 0.4. The Task Decision Module (TDM) reads key-object snapshots, frontier snapshots, and scene descriptions. Scores for frontier snapshots are assigned to their corresponding frontiers. When verifying a suspected target, TDM returns a score and a 2D box; the score is assigned to overlapping object instances. After RMDM makes a decision, the key-object set is rebuilt and TDM is invoked. Processed snapshots are marked to avoid repeated queries.

TDM outputs must align candidate by candidate with RMDM rather than returning an ungrounded instruction such as going to the kitchen. Latency and incomplete snapshot coverage leave some candidates without scores; these receive -1 and are handled separately by fusion. The method text and Table 6 disagree on how the RTPM and navigation thresholds correspond to their parameter identifiers. This note preserves the mechanism stated in the method without treating the ablation identifiers as verified implementation settings.

4. Probability Fusion and Mode Switching: calibrate evidence before considering travel cost

BPDFM uses a calibration dataset and nonlinear fitting to map raw RMDM and TDM scores into success probabilities, then performs Bayesian-style fusion. A missing TDM score is mapped to probability 0.5 as neutral evidence, keeping the fused estimate close to the real-time prior. Operators in the denominator of cached Equation (5) are missing, and the appendix describing the fitting procedure is absent. A guessed reconstruction is therefore not presented as the paper's exact formula; reproducing this component still requires checking the original PDF equation and appendix.

The controller does not always select the highest-scoring candidate. If neither branch has a confident candidate, it explores. When both confirm an object, it navigates toward that object. Disagreement can lead to verification, but only after the step count exceeds 300 or frontier options are exhausted, preventing premature repeated inspection of one suspected target. Verification samples viewpoints around the target, filters positions without sufficient visible free space, and greedily selects three geometrically separated views, visited in ascending distance to the target.

Exploration constructs an undirected weighted graph containing the current position, frontiers, and the target with the highest fused probability. Edge weights represent movement cost. A candidate route must preserve enough budget to reach the current best target after visiting its selected frontiers. The remaining budget is:

\[ D=\max(0,t_{\max}-t_{\mathrm{cur}}). \]

Route evaluation considers the probability of first succeeding at each candidate, cumulative movement cost to that candidate, and a large penalty if every candidate fails. The controller follows the first node of the feasible route with minimum expected cost, falling back to the current best target when no feasible route exists. The paper replaces exhaustive search with state-compressed dynamic programming, but the cache does not provide its states and transitions. Its assertion of polynomial complexity is therefore insufficient to establish polynomial scaling in the number of frontiers. When the remaining budget cannot support another exploration cycle, the agent also commits to the highest-probability object, so final commitment does not always require dual confirmation.

A Worked Example

This is a mechanism illustration, not an additional experiment. Suppose the robot must find a kitchen item. RORM first supplies five relevant objects, while MSC accumulates room and object state during movement. RMDM uses room guidance to choose an unexplored frontier without stopping for another room-level inference. If an observed object has perception confidence below 0.4 but a target score above 0.6, SpatSC includes a snapshot covering it in the material sent to TDM.

Before TDM responds, the candidate follows the missing-evidence branch. Once the response arrives, its 2D box associates the verification judgment with a specific instance. Agreement leads to navigation; disagreement leads to three separated verification views only when the verification condition is satisfied. Weak evidence with sufficient budget instead leads to further expected-cost exploration. This explains both reduced waiting under asynchronous collaboration and the possibility of longer paths under multi-view verification.

Loss & Training

RMDM jointly learns navigation decisions, object classes, relevant objects, relevant rooms, and room types. The task definition accompanying Equation (4) gives the training objective as:

\[ \mathcal L_{\mathrm{total}}=\sum_{h\in\mathcal H}\lambda_h\mathcal L_h, \qquad \mathcal H=\{\mathrm{decision},\mathrm{class},\mathrm{rel\_object},\mathrm{rel\_room},\mathrm{room\_type}\}. \]

All weights are 1. Object class and room type use single-label cross-entropy; decision, relevant-object, and relevant-room heads use binary cross-entropy. Navigation supervision has one positive target: the target instance if already present in the global set, otherwise the frontier closest to the ground-truth target. This is a single positive label trained with BCE, not a decision head that should be described as a categorical softmax.

Base training data include HM3D-OVON, ScanRefer, and Multi3DRefer, but frontier collection uses an optimal policy instead of random sampling to match cost-sensitive planning. Room labels come from unobstructed multi-view images of HM3DSEM regions and large-model inference. Relevant rooms follow the optimal ground-truth trajectory, and relevant objects are provided by a language model.

Training comprises 10 epochs on the full dataset followed by 10 epochs of task-specific fine-tuning. AdamW uses learning rate \(10^{-4}\), \(\beta_1=0.9\), and \(\beta_2=0.98\), requiring approximately 384 A100 GPU hours. Simulation inputs are 360×640 RGB-D images and poses; actions move forward by 0.25 m or turn left/right by 30 degrees. Each simulated step takes 0.5 s, with 18 frames sampled between consecutive targets. GPT-5 is the large model in the main experiments, while module timing uses an A6000; training and timing hardware should not be conflated.

Key Experimental Results

Main Results

The following results are selected from Tables 1–3. Values are on a percentage scale; higher SR and SPL are better. SPL penalizes successful but circuitous navigation using the ratio between shortest and executed path lengths. HM3D-OVON splits are Seen, Seen Synonyms, and Unseen. Differences are percentage points, not relative percentages.

Dataset / Split	Metric	MTU3D	AMCoNav	Difference
HM3D-OVON / Seen	SR / SPL	55.0 / 23.6	60.0 / 15.6	+5.0 / -8.0
HM3D-OVON / Seen Synonyms	SR / SPL	45.0 / 14.7	60.0 / 20.9	+15.0 / +6.2
HM3D-OVON / Unseen	SR / SPL	40.8 / 12.1	53.3 / 14.8	+12.5 / +2.7
SG3D-Nav	s-SR / t-SR / SPL	23.8 / 8.0 / 16.5	24.5 / 8.3 / 11.3	+0.7 / +0.3 / -5.2
GOAT-Bench / Seen	SR / SPL	52.2 / 30.5	55.2 / 25.5	+3.0 / -5.0
GOAT-Bench / Seen Synonyms	SR / SPL	48.4 / 30.3	48.8 / 21.8	+0.4 / -8.5
GOAT-Bench / Unseen	SR / SPL	47.2 / 27.7	47.2 / 21.5	0.0 / -6.2

Claims across methods need qualification. On HM3D-OVON Unseen, MSGNav obtains 48.3 / 27.0 SR / SPL: AMCoNav succeeds more often but has substantially lower SPL. On GOAT-Bench Unseen, MSGNav obtains 52.0 / 29.6, exceeding AMCoNav on both metrics. The abstract's claim of an average 8.7% improvement cannot be directly reproduced from the best previous SR on each split in Table 1. This note therefore uses checkable split-level differences rather than repeating that aggregate claim.

Ablation Study

Table 5 reports SR / SPL on the same three splits. The authors treat MSC and ZLMM together as the global-guidance mechanism; this is not an individual removal study of all four large-model submodules.

Configuration	Seen	Seen Synonyms	Unseen
MTU3D baseline	55.0 / 23.6	45.0 / 14.7	40.8 / 12.1
AMCoNav w/o BPDFM	56.7 / 15.8	53.3 / 17.5	44.2 / 12.5
Full AMCoNav	60.0 / 15.6	60.0 / 20.9	53.3 / 14.8

Adding BPDFM improves SR by 3.3 / 6.7 / 9.1 points over its removal variant, with the largest gain on unseen categories. This difference cannot be attributed entirely to the Bayesian equation, because BPDFM also changes mode switching, verification, and exploration routes.

Key Findings

• Runtime and path efficiency are different quantities. Across 100 random episodes, synchronous TDM averages 174.1 s versus 142.5 s asynchronously, a reported runtime reduction of 18.15%. This comparison simulates a 0.5 s action duration and is not a physical-robot deployment test.
• Table 4 reports 209, 106, and 281 ms for query proposal, RMDM, and BPDFM, respectively; RORM, RNRM, RTPM, and TDM take 1.51, 14.58, 3.36, and 3.99 s. The three real-time components would total 596 ms if executed serially, so these values do not establish an end-to-end pipeline below 0.5 s. The authors also use a minimum inter-decision interval of 30 steps, or 15 s, to explain the asynchronous calling window.
• On GOAT-Bench Unseen, Category, Language, and image-centered goals achieve 68%, 49.2%, and 23.6% success. Image-goal reasoning remains a bottleneck beyond invocation latency alone.
• On Table 7's 50-episode Seen subset, GPT-5 / Doubao 2.0-pro / Doubao 2.0-mini / GPT-4o / Qwen-VL-Pro yield SR of 60.0 / 60.0 / 58.0 / 56.0 / 56.0. GPT-5 has SPL 13.7 here, which must not be mixed with the main table's 15.6. The small sample and absence of confidence intervals do not establish full equivalence across model substitutions.

Highlights & Insights

• An asynchronous interface requires more than starting a thread: object alignment, snapshot coverage, spatial confidence fields, and missing-score handling determine whether late responses can be consumed correctly. Designing these states can have greater system value than shortening one model response in isolation.
• Key-object selection combines unreliable perception with high task relevance, concentrating verification on ambiguities that can change the current decision. The same principle could support active observation after open-vocabulary detection, although thresholds would need recalibration.
• Fused evidence feeds cost-sensitive planning rather than only a stopping rule. This connects where a target may be with where the agent should go first under a finite budget, while showing why success, distance, and waiting time require separate evaluation.

Limitations & Future Work

• The authors acknowledge that ambiguous or incomplete observations can make the real-time branch a bottleneck. Dual confirmation may also let a weak branch block a correct large-model judgment. Confidence-based gating, adaptive verification, and task-specific calibration are future directions, not validated components of the current system.
• Path efficiency has a measurable cost: SPL is below MTU3D on all three GOAT-Bench splits, and SG3D success gains are small. Improvements on HM3D-OVON do not establish substantial gains on every navigation task.
• Evidence comes from simulation. The cache provides no physical-robot validation, long-tail latency distribution, API-failure stress test, or complete token-cost analysis. Expiration of stale responses and retraction of earlier high-confidence but incorrect room labels are also insufficiently developed.
• Reproduction has explicit gaps: Equations (1), (5), and (7) are corrupted in extraction, and the calibration appendix is absent. Table 6's caption, grouped rows, and prose disagree on \(c_2/c_3\). The state-compressed dynamic programming complexity claim lacks an inspectable derivation, so missing formulas, threshold mappings, and runtime bounds are not invented here.

Related Work & Insights

• Compared with MTU3D: AMCoNav reuses object-query representations and base training data while adding asynchronously updated room/object guidance and probability-based decisions. Its gains cannot simply be attributed to a larger perception model, since frontier sampling during training and the decision procedure also change.
• Compared with MSGNav: MSGNav uses a multimodal 3D scene graph and large-model decisions, whereas AMCoNav preserves an independent real-time decision path. MSGNav remains stronger on GOAT-Bench Unseen, showing that a responsiveness trade-off does not imply superiority on all metrics.
• Compared with CLASH / FlexVLN and LM-Nav / BeliefMapNav: As discussed in this paper, the former group may still wait for slow branches, while the latter can struggle to revise global guidance promptly. The transferable contribution is asynchronously revisable guidance with independent action, not multi-model collaboration in the abstract.

Rating

• Novelty: 4/5. Multi-timescale asynchronous state and cost-sensitive fusion form a targeted system design, while the underlying perception representation and multitask training are relatively conventional.
• Experimental Thoroughness: 3/5. Three navigation benchmarks, component ablations, and timing comparisons are useful, but physical deployment, statistical uncertainty, and finer mechanism isolation are missing.
• Writing Quality: 3/5. Architecture and module responsibilities are understandable, but threshold identifiers, aggregate improvement, and complexity statements affect reproducibility; corrupted cached equations separately require source verification.
• Value: 4/5. A useful reference for robot systems that must keep acting while exploiting large-model semantics, subject to path-efficiency costs and the real-time branch's limitations.