MM-Nav: Multi-View VLA Model for Robust Visual Navigation via Multi-Expert Learning¶

Conference: ECCV 2026
Paper: ECCV official page
Project: MM-Nav
Area: Robotics
Keywords: Visual navigation, multi-view perception, multi-expert distillation, capability balancing, sim-to-real transfer

TL;DR¶

MM-Nav combines the skills of depth-based RL experts for reaching, squeezing, and dynamic obstacle avoidance into a four-view RGB 7B vision-language-action (VLA) model through real-world VQA co-training and capability-balanced online distillation, achieving an average success rate of 88.1% on InternVLA-N1 System-1 and approximately 7 Hz continuous velocity prediction in server-assisted real-world deployment.

Background & Motivation¶

Visual navigation requires more than recognizing objects ahead: a robot must determine whether it fits between obstacles, whether its sides or rear are exposed to collisions, and when a moving obstacle will intersect its path, all without explicit depth. GNM, ViNT, and NoMaD demonstrate the potential of data-driven visual navigation, but a single front-facing camera cannot continuously observe all these hazards. VLA approaches such as NaVid and Uni-NaVid bring vision-language understanding to navigation, yet discrete forward and turning actions constrain immediate adjustments in narrow, dynamic environments.

Collecting suitable data introduces another constraint. Real demonstrations are expensive, and gathering large amounts of near-obstacle squeezing or reactive pedestrian avoidance experience is difficult to do safely. Simulation scales more easily but introduces visual domain differences, while rule-based planners struggle to generate these reactive behaviors. Training a single RL teacher in mixed environments does not automatically solve this problem: efficient reaching, decisive squeezing, and cautious avoidance impose different requirements, so a shared reward can compromise each skill.

The paper therefore trains three teachers in tailored environments before integrating their behavior into one large student model. The student learns both from successful teacher trajectories and from corrective labels at states it visits itself, while real-image question answering maintains cross-domain visual representations. Core Idea: use privileged depth-based RL specialists to acquire difficult navigation skills, use a shared RGB VLA to integrate those skills and generalize to real images, and allocate subsequent demonstrations according to the student's capability-specific deficits relative to its teachers.

Method¶

Overall Architecture¶

The task is point-goal navigation, not arbitrary natural-language instruction following. The robot receives a two-dimensional goal position relative to itself and RGB observation histories from front, right, back, and left cameras, then predicts longitudinal, lateral, and yaw velocities. Encoding the goal as text allows the language model to connect vision and action while supporting a VQA branch during training; this does not establish open-ended language navigation capability.

Training first constructs capability-specific experts and collects successful demonstrations to initialize the multi-view continuous-control student, with real VQA co-training. The student then runs closed-loop in simulation, where the corresponding teacher labels its visited states and capability-balanced online iterations address weak skills. Deployment retains only the student and requires neither RL teachers, depth input, nor VQA answers.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Capability-Specific<br/>Experts"] -->|Successful demonstrations| B["Multi-View<br/>Continuous Control"]
    B --> C["Real-World<br/>VQA Co-Training"]
    Q["Real-image QA"] -.->|Training supervision| C
    C --> D["Capability-Balanced<br/>Online Iteration"]
    A -.->|Labels at student states| D
    D -->|Update and collect again| D
    D -->|Training complete| E["Deployed student policy"]
    I["Four-view RGB history<br/>Relative point goal"] --> E
    E --> O["Continuous velocities<br/>Low-level controller"]

The first four design nodes describe training; the separate observation input to the deployed policy describes control-time data flow. The experts do not vote at inference time: the trained student produces one action output.

Key Designs¶

1. Capability-Specific Experts: obtain strong demonstrations for each difficult behavior

Three IsaacLab environments impose different training demands. Reaching randomizes the shapes, textures, and sizes of static obstacles, with goal distances up to 30 m. Squeezing uses dense pillars and walls with randomized narrow gaps, requiring side-view information to judge clearance. Avoiding introduces differently shaped obstacles moving at 0.5-1.5 m/s, with goal distances up to 10 m. Simulation abstracts the robot as a cuboid with a planar footprint of approximately 0.70 m by 0.35 m, making large-scale reactive navigation data collection more economical. These environments shape different control behaviors rather than merely partitioning an existing dataset.

Each teacher is trained separately with PPO and observes four depth images, the previous action, and the relative goal. ResNet-18 encodes the depth images; their features are concatenated with the other inputs and the previous hidden state before a three-layer MLP predicts velocity. Passing the hidden state to the next step provides limited memory. Rewards cover goal progress, time cost, behavioral regularization, and collisions, with different teacher-specific weights: Reaching uses stronger behavioral regularization, Avoiding omits it, and Squeezing receives a larger goal reward to discourage hesitation at narrow passages. The shared step and collision coefficients are -0.05 and -15; regularization coefficients are 0.05, 0.02, and 0, while goal coefficients are 1.2, 1.5, and 1.2, respectively. The cached reward and velocity-clipping equations are corrupted, so they do not support reconstructing the exact implementation.

2. Multi-View Continuous Control: compress surrounding geometry and motion history for responsive inference

The student uses a SigLIP visual encoder, a two-layer MLP cross-modal projector, and Qwen2-7B. Four cameras jointly cover 360 degrees, and each view retains a sliding observation history rather than contributing only its latest image. Current frames require finer spatial information for close-range obstacles and gap assessment; historical frames primarily provide motion cues and can be compressed more aggressively. The model therefore pools each current view to 16 tokens and each historical view at each timestep to 4 tokens, organizing these representations by time and view. This asymmetric encoding prioritizes the next control decision while preserving evidence about moving obstacles.

The relative goal is formatted as a prompt such as "Go to the position of " and supplied with the visual tokens to the language model. A two-layer MLP action head maps the resulting action token to three continuous velocity components instead of generating action words such as FORWARD or TURN LEFT. The robot can thus translate laterally while turning without waiting for a discrete action to finish. The paper states a window length of 8 and a total of 192 visual tokens; counting 7 historical timesteps and 1 current timestep under the described per-view allocation gives \(4\times7\times4+4\times16=176\). Both statements are preserved here rather than treating 192 as a verified reproducible sequence length.

3. Real-World VQA Co-Training: constrain simulated action learning with real visual understanding

Simulation experts provide action labels but cannot eliminate the difference between rendered images and real camera observations. MM-Nav therefore trains navigation regression and real-world visual question answering (VQA) on a shared vision-language backbone: navigation examples use the action head, while QA examples use the language head. VQA supplies no navigation velocities and is not an inference-time question-answer-then-plan routine. Instead, semantic and spatial supervision on real imagery keeps the hidden representations used by the action head exposed to real lighting, structures, and object appearances.

Initial training uses 500k navigation timesteps and 100k VQA examples, followed by 200k online navigation timesteps and 40k VQA examples per iteration. QA data come from a subset of the data used by LongVU, with video frames sampled at 1 FPS. Navigation ablations without VQA and UMAP visualizations of hidden states support its utility, but the visualization is representational evidence rather than proof of a causal transfer mechanism. In particular, "no real action demonstrations" must not be confused with "no real data": the reported zero-shot real-world transfer still uses real VQA training data.

4. Capability-Balanced Online Iteration: direct corrective supervision toward the student's weak skills

Imitating only successful teacher trajectories creates a state-distribution mismatch: small student errors lead to states rarely seen in demonstrations, after which errors can compound. The paper first initializes the student with successful demonstrations and then follows a DAgger-style procedure in which the student acts in the three environments and the corresponding teacher labels those visited states. The teacher supplies supervision, but the student determines the visited-state distribution. This differs from merely collecting more successful teacher trajectories and is not online reinforcement learning during real-world deployment.

Because the environments differ in difficulty, uniformly allocating data can undersupply a student that has not learned to squeeze. The authors compare student and teacher performance within each capability using weighted travel time (WTT), defined as mean successful-episode duration divided by success rate, with lower values being better. Larger gaps receive larger sampling shares, with a power exponent of 0.3 smoothing the distribution to prevent one skill from monopolizing data. A 0.1 floor term retains samples even for skills where the student exceeds its teacher. Symbols in cached Equations (5)-(6) are incomplete, so this note preserves the confirmed mechanism and parameters without guessing the precise normalization formula.

With success rate expressed as a fraction, WTT is:

\[ W=\frac{\overline{T}_{\mathrm{success}}}{\mathrm{SR}}. \]

For example, a mean successful-episode duration of 20 s gives WTT values of 40 s and 25 s at success rates of 0.5 and 0.8, respectively. This illustrates the metric rather than reporting an experiment. WTT penalizes both slow completion and failure but does not distinguish collisions from timeouts, so CR must still be reported separately.

A Worked Example¶

Suppose the goal is 10 m ahead, two pillars form a narrow passage, and a pedestrian approaches from the left. This is an illustrative scenario connecting the method's components, not an additional reported test episode.

The current front view shows the passage entrance, the side cameras provide clearance cues, and coarse representations of earlier frames indicate pedestrian motion. These visual representations and the goal prompt enter the same student; its action head predicts longitudinal, lateral, and yaw velocities. The robot adjusts its motion accordingly, and the low-level controller follows the command until the next response.

If the student stalls at the gap, training can collect teacher labels at student-visited states in the Squeezing environment. If the next capability evaluation shows a larger squeezing WTT deficit, that skill receives a larger training share rather than Reaching dominating merely because successful trajectories are easier to obtain.

Real VQA data constrain the shared representation during this process, but deployment neither calls the three teachers nor generates a textual answer such as "the pedestrian is on the left." The intended result is a policy that integrates skills within a single velocity decision, not a sequence of explicit skill switches.

Loss & Training¶

The navigation branch learns from teacher velocities using mean squared error, while the VQA branch retains cross-entropy. The paper specifies the joint objective as:

\[ \mathcal{L}=5\mathcal{L}_{\mathrm{action}}+\mathcal{L}_{\mathrm{QA}}, \qquad \mathcal{L}_{\mathrm{action}}=\operatorname{MSE}(\mathbf{v}_{\mathrm{Pred}},\mathbf{v}_{\mathrm{GT}}). \]

Each RL teacher uses 128 parallel environments and trains for approximately 8-12 hours on one RTX 4090. Initial demonstration collection runs 64 parallel robots per teacher and retains only trajectories that reach the goal. Camera height is randomized over 0.4-0.5 m and field of view over 100-140 degrees to reduce dependence on a fixed camera configuration.

Initial student fine-tuning takes approximately 5 hours on 8 H100 GPUs, totaling 40 GPU hours; each later iteration takes approximately 2 hours. The main text reports convergence after 4 iterations, whereas the headline totals are 1.5M expert demonstrations and 300k VQA examples. Adding the stated initial data and 4 incremental rounds yields 1.3M navigation timesteps and 260k VQA examples. The paper does not reconcile this difference, so demonstrations should not be silently reinterpreted as complete trajectories, nor should 5 iterations be assumed.

Real-world deployment uses a Unitree GO2 with four fisheye cameras whose images are undistorted before inference. The model runs on an RTX 5090 server, and the robot requests velocity commands at an average response rate of approximately 7 Hz. This is neither fully onboard 7B inference performance nor the frequency of low-level joint control.

Key Experimental Results¶

Main Results¶

Public evaluation follows the InternVLA-N1 System-1 point-goal protocol, covering 40 InternScenes scenes with the same 100 start-goal pairs as that benchmark and the omnidirectional Dingo platform. SR is success rate; SPL is success weighted by path length. The following representative results are from the paper's Table 1, reported on a percentage scale.

Method	Observation	Home SR	Home SPL	Commercial SR	Commercial SPL
InternVLA-N1(S1)	RGB-D	60.0	55.6	71.4	68.2
NavDP	RGB-D	60.3	54.7	74.1	70.5
SIDP	RGB-D	63.2	56.5	81.2	73.4
MM-Nav, without VQA	RGB	76.4	73.7	73.9	72.6
MM-Nav, single view	RGB	79.9	77.6	86.7	84.7
MM-Nav, multi-view	RGB	86.3	81.1	89.9	85.5

Average multi-view SR is 88.1%, versus SIDP's 72.2%, a difference of 15.9 percentage points; single-view MM-Nav averages 83.3%. Multi-view input adds an average of 4.8 SR points and approximately 2.2 SPL points over the single-view variant. External baseline comparisons also differ in camera coverage, making the single-view result important. The "without VQA" row does not explicitly identify its view configuration, so its difference from either variant should not be treated as a fully specified controlled comparison.

The custom IsaacLab evaluation runs 100 episodes per scene, with a 90 s limit for individual-capability scenes and 120 s for Mixed. Mixed combines static obstacles, moving obstacles, and narrow passages, uses training-unseen materials, and deliberately reduces the squeezing segment's difficulty. The following values are from the paper's Table 2; SR and CR are percentages, and WTT is in seconds.

Scene	NavDP SR / CR / WTT	MM-Nav SR / CR / WTT
Reaching	69 / 31 / 27.3	80 / 20 / 31.0
Squeezing	18 / 82 / 115.4	71 / 19 / 42.2
Avoiding	27 / 73 / 30.0	68 / 32 / 20.9
Mixed	23 / 77 / 178.6	47 / 26 / 127.5

Ablation Study¶

The paper's Table 3 compares specialist teachers, mixed-environment RL, and VLA models trained from different demonstration sources. Each scene uses 100 evaluation episodes, and the single-expert VLA variants use the same total data budget. All values below are SR (%). Mixed-VLA integrates demonstrations from all three specialists; it should not be interpreted as distilling only Mixed-RL.

Configuration	Reaching	Squeezing	Avoiding
Reaching-RL	65	2	33
Squeezing-RL	30	73	19
Avoiding-RL	59	0	58
Mixed-RL	45	58	42
Reaching-VLA	70	4	35
Squeezing-VLA	44	64	28
Avoiding-VLA	63	0	62
Mixed-VLA	80	71	68

Key Findings¶

The multi-expert student's advantage is capability coverage, not superiority to every best specialist: its Reaching SR of 80% exceeds the corresponding teacher's 65%, and Avoiding reaches 68% versus 58%, but Squeezing remains at 71% versus the specialist's 73%.
Higher success does not imply uniformly better efficiency: Reaching WTT is 31.0 s, worse than NavDP's 27.3 s. Mixed SR of 47% and CR of 26% also leave 27% timeouts, so reduced collisions do not imply successful task completion.
Figure 7 shows that capability balancing addresses squeezing and avoidance deficits faster, while unbalanced training performs better on Reaching. The strategy therefore favors multi-capability coverage rather than maximizing one skill.
The real-world VQA ablation uses 20 trials per model per scenario, covering thin wires, a cluttered corridor, pedestrians, and an outdoor alley at night. Interleaved plot labels in the cache do not support reliable transcription of individual bars. Figure 6 reports a representation gap-axis distance reduction from 1.86 to 0.38, which is auxiliary analytical evidence.

Highlights & Insights¶

Teachers need not each be generalists to train a student with broader coverage. Acquiring strong local skills under different environments and rewards, then integrating them in a shared visual representation, avoids making a single mixed RL teacher the initial multi-objective bottleneck.
Online sampling is controlled by student-teacher performance gaps. This reflects current skill deficits better than fixed proportions and suggests capability-progress-based annotation allocation for other multi-task distillation systems, although teacher limitations still influence the signal.
VQA contributes more than linguistic descriptions of actions. Real visual supervision without velocity labels helps shape action-relevant representations alongside synthetic demonstrations, offering a reusable approach to lower-cost sim-to-real transfer.

Limitations & Future Work¶

The authors identify cross-embodiment transfer as future work. The current task relies on relative point goals, four cameras, and omnidirectional velocity control, so the results do not directly extend to platforms with different kinematic constraints or long-horizon language tasks.
Mixed success remains at 47%, with nonzero collision risk. Safe deployment still needs dedicated evaluation of latency robustness, failure detection, and independent safety constraints; these are suggested extensions, not implemented paper features.
The reported 7 Hz depends on an RTX 5090 server and communication. The paper does not sufficiently quantify communication latency tails, disconnections, or onboard power requirements to establish unconditional standalone deployment.
Data totals, visual token counts, and some narrative claims have reporting inconsistencies. In particular, Table 1 does not directly support the statement that removing VQA causes drops exceeding 20% in both Home and Commercial; this note preserves the tabulated values instead of repeating that claim.

vs NavDP / LoGoPlanner: These methods use privileged-information planners for navigation supervision, whereas MM-Nav emphasizes reactive RL experts in tailored environments. Its focus on squeezing and dynamic avoidance comes with additional expert-training and online-collection costs.
vs NaVid / Uni-NaVid: MM-Nav builds on video VLA models and compressed history, extending them to four-view observations and continuous velocity regression. The distinction is surrounding perception and low-level responsiveness, not demonstrated broader language-instruction capability.
vs DAgger: Requesting teacher labels at student-visited states is established practice; the added mechanism is adjusting data proportions using capability-specific WTT gaps. A transferable research question is whether scheduling should incorporate uncertainty when teachers are weak or performance estimates fluctuate.

Rating¶

Novelty: 4/5. Specialist skill acquisition, RGB VLA integration, and capability-balanced iteration form a coherent navigation-specific solution, although many individual components have precedents.
Experimental Thoroughness: 4/5. Public benchmarks, difficult custom environments, teacher combinations, and real-world VQA ablations provide broad evidence, but larger-scale real-world safety statistics remain needed.
Writing Quality: 3/5. The main argument is clear, but token counts, data budgets, and discrepancies between narrative claims and tables complicate reproducibility; cached equations also have extraction damage.
Value: 4/5. The work offers a practical route from scalable simulated skill acquisition to real RGB control, with utility beyond simply increasing model size.