title: >- [Paper Note] MA-VLA: Multi-Arm Vision-Language-Action Model for Collaboration and Compositional Generalization description: >- [ECCV2026][Robotics & Embodied AI][Multi-Arm Collaboration] Introduces MA-VLA, a unified multi-arm VLA framework that leverages atomic action assignment and training-time Arm Shuffle to achieve zero-shot multi-arm compositional generalization under unseen collaboration patterns. tags: - ECCV2026 - Robotics & Embodied AI - Multi-Arm Collaboration - Compositional Generalization - Vision-Language-Action date: 2026-09-19 content_hash: af1297e71abb5964
MA-VLA: Multi-Arm Vision-Language-Action Model for Collaboration and Compositional Generalization¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/zhangzaibin/future-robots
Area: Robotics & Embodied AI
Keywords: multi-arm collaboration, compositional generalization, vision-language-action, atomic action assignment, Arm Shuffle
TL;DR¶
To overcome the severe arm-identity bias and static instruction limitations in multi-arm manipulation, MA-VLA decomposes complex instructions into per-arm atomic prompts and incorporates training-time Arm Shuffle, enabling robust zero-shot multi-arm compositional generalization across unseen collaboration topologies.
Background & Motivation¶
Embodied artificial intelligence is rapidly transitioning from unimanual manipulation to multi-arm and multi-robot cooperative systems. In applications ranging from industrial assembly lines to unstructured household chores, multi-arm setups utilize parallel execution and tight spatio-temporal coupling to achieve tasks that remain physically impossible for a single arm. While recent vision-language-action (VLA) models have demonstrated impressive end-to-end visuomotor mastery, existing extensions to multi-arm platforms predominantly inherit the single-arm paradigm: a monolithic, high-level instruction is provided to the network, expecting the latent dynamics to implicitly infer labor division, synchronization, and collision avoidance.
This implicit conditioning leads to profound brittleness. In imitation learning datasets, individual arms invariably overfit to narrow spatial roles and fixed hardware identities—for instance, memorizing that the left arm only picks containers while the right arm solely inserts objects. When test-time deployments demand previously unobserved collaboration patterns (such as inverted stacking sequences, cross-arm relay handovers, or swapped master-slave responsibilities), existing state-of-the-art VLAs suffer catastrophic failure despite each arm's required atomic physical behaviors remaining completely in-distribution.
Human teamwork demonstrates that scalable cooperation is intrinsically governed by the division of labor: complex macro-goals are factored into clear mid-level responsibilities and executed via modular atomic actions. Crucially, new cooperative workflows are formed by recomposing known atomic behaviors into novel task allocations. Motivated by this insight, the central challenge is whether a robotic system can generalize to unseen coordination structures without retraining—a capability defined as multi-arm compositional generalization. Core idea: decompose high-level instructions into temporally grounded atomic prompts allocated per arm, and introduce training-time Arm Shuffle (randomized permutation of arm inputs and prompts) within a unified flow-matching VLA to enforce role-agnostic coordination and zero-shot compositional generalization.
Method¶
Overall Architecture¶
MA-VLA adopts a hierarchical "high-level language planning + unified low-level execution" architecture. The framework comprises two core modules: (1) a pre-trained visual-language model (VLM) planner \(\mathcal{P}_\phi\) that translates macro-instructions into structured stage-wise atomic action prompts for each arm, and (2) a multi-arm VLA executor \(\pi_\theta\) that grounds these linguistic sub-goals into continuous, collision-free joint trajectories via flow matching.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["High-level instruction l + Scene images I"] --> B["Stage-wise Task Planner<br/>VLM task decomposition & per-arm atomic prompt assignment"]
B --> C["Sequential Atomic Prompts<br/>Arm0: prompt0, Arm1: prompt1, ..."]
C --> D["Stochastic Input Regularization<br/>Arm Shuffle & View Dropout during training"]
D --> E["Unified Multi-Arm VLA Executor<br/>Multi-view vision + proprioception + joint language context"]
E --> F["Flow-Matching Action Expert<br/>Multi-head projection outputting coordinated action trajectories"]
Key Designs¶
1. Stage-wise Task Planner: Atomic Action Decomposition and Explicit Assignment Directly delegating long-horizon symbolic planning and high-frequency motor control to a single model compromises policy capacity. MA-VLA introduces a frozen pre-trained VLM (such as GPT-4.1) as the planner \(\mathcal{P}_\phi\). The planner queries a standardized, task-agnostic library of atomic action predicates \(\mathcal{A}\) (e.g., grasp, lift, align, place). Conditioned on the global scene view \(I\) and high-level goal \(l\), the planner generates a complete schedule of \(T\) sequential stages in a single pass: \(\mathbf{l} = (\mathbf{p}_1, \dots, \mathbf{p}_T)\), where \(\mathbf{p}_t = (p_t^1, \dots, p_t^N)\) assigns an explicit prompt \(p_t^i \in \mathcal{A}\) to arm \(i\) at stage \(t\). Each stage executes until domain-specific termination criteria are met. This decouples temporal semantic reasoning from physical execution while keeping arm allocations fully transparent.
2. Unified Multi-Arm VLA Executor: Joint Representations with Multi-Head Projection Decentralized single-arm controllers suffer from inter-arm deadlock and linear computational scaling. MA-VLA utilizes a single, unified VLA policy \(\pi_\theta\). At control step \(t\), the per-arm atomic prompts are concatenated into an aggregated linguistic string: $\(u_t = \text{"Arm0: } p_t^0 \text{ Arm1: } p_t^1 \dots \text{ Arm}N: p_t^N\text{"}\)$ The executor consumes multi-view third-person and egocentric wrist observations \(I_t\), concatenated joint proprioceptions \(s_t = [s_t^1, \dots, s_t^N]\), and the prompt \(u_t\). Built on the \(\pi_0\) backbone, the multimodal Transformer models inter-arm spatial dependencies before splitting into arm-specific action heads via a multi-head projection layer. A flow-matching formulation denoises random latent action vectors into continuous multi-arm trajectories \([a_t^1, \dots, a_t^N]\).
3. Arm Shuffle: Breaking Identity Bias for Role-Agnostic Coordination Demonstration trajectories inevitably bind specific arm hardware slots to fixed subtasks (e.g., arm index 1 always grabbing the left cube). To prevent the policy from overfitting to positional identities, MA-VLA incorporates Arm Shuffle during training. With probability \(p_{\text{shuffle}}\), a random permutation \(\sigma \in \mathcal{S}_N\) re-indexes the per-arm proprioceptive states, wrist-camera visual streams, assigned atomic prompts, and ground-truth action tuples: $\((s_t^i, v_t^i, p_t^i, a_t^i) \xrightarrow{\text{shuffle}} (s_t^{\sigma(i)}, v_t^{\sigma(i)}, p_t^{\sigma(i)}, a_t^{\sigma(i)})\)$ This data-level permutation forces the policy to associate physical actions with explicit semantic prompt tokens rather than hardware index bindings, enabling zero-shot re-coordination when arm roles are reconfigured at inference.
4. View Dropout: Multi-View Spatial Redundancy and Occlusion Resilience In multi-arm manipulation, robotic links and grasped objects introduce frequent dynamic occlusions. To ensure visual robustness, View Dropout masks out a randomly chosen subset of camera views with probability \(p_{\text{drop}}\): $\(\tilde{I}_t = \text{Mask}(I_t, \mathcal{M})\)$ This forces the model to exploit spatial redundancy across third-person and wrist cameras, ensuring that losing line-of-sight on one camera does not derail cooperative execution.
Loss & Training¶
MA-VLA is optimized end-to-end using behavioral cloning with a flow-matching velocity field loss. With Arm Shuffle and View Dropout active during training, the loss computes the mean squared error across all \(N\) arms: $\(\mathcal{L}_{\text{BC}}(\theta) = \mathbb{E}_{(\tilde{I}_t, s_t, u_t, A_t, \tau)}\left[ \sum_{i=1}^N \left\| v_\theta^i(\tilde{I}_t, s_t^i, p_t^i, x_t^{\tau, i}, \tau) - u_t^{\text{target}, i} \right\|^2 \right]\)$ Models are trained on two NVIDIA A800 GPUs with batch size 32. In RoboFactory, 2-arm tasks are optimized for 10,000 steps and 3–4 arm tasks for 15,000 steps; RoboTwin 2.0 Hard tasks are trained for 30,000 steps; real-world SO101 policies are fine-tuned for 15,000 steps. Action chunking horizon is set to 50 across all environments.
Key Experimental Results¶
Main Results¶
Evaluations span two multi-arm simulation benchmarks (RoboFactory multi-arm suite and RoboTwin 2.0 bimanual benchmark) as well as a real-world dual-arm SO101 platform (12 DoF total, 6 per arm, equipped with global and dual wrist cameras).
Table 1: Performance on RoboFactory Simulation Benchmark (Success Rate %) Compares 2-arm cooperative tasks (Lift Barrier, Place Food, Stack2 Cube, Pass Shoe) and 3–4 arm long-horizon tasks (Stack3 Cube, Camera Alignment, Pipeline Delivery, Take Photo).
| Method | 2-arm Lift | 2-arm Place | 2-arm Stack2 | 2-arm Pass | 2-arm Avg | 3-arm Stack3 | 3-arm Align | 4-arm Deliver | 4-arm Photo | 3-4 arm Avg |
|---|---|---|---|---|---|---|---|---|---|---|
| DP | 58.0% | 20.0% | 20.0% | 12.0% | 27.5% | 22.0% | 19.0% | 0.0% | 20.0% | 15.3% |
| Pi0-FAST | 90.0% | 27.0% | 50.0% | 63.0% | 57.5% | 3.0% | 22.0% | 10.0% | 12.0% | 11.8% |
| Pi0 | 97.0% | 58.0% | 82.0% | 84.0% | 80.3% | 48.0% | 94.0% | 82.0% | 82.0% | 76.5% |
| MA-VLA (Ours) | 98.0% | 59.0% | 86.0% | 91.0% | 83.5% | 58.0% | 96.0% | 97.0% | 82.0% | 83.3% |
Table 2: Out-of-Domain Multi-Arm Compositional Generalization Evaluates unseen stacking orders of cube colors (Green-Blue-Red GBR, Red-Green-Blue RGB, Blue-Red-Green BRG) and novel collaboration topologies (Pass Two Shoes, Stack Two Bowls).
| Method | Stack GBR | Stack RGB | Stack BRG | Pass Two Shoes | Stack Two Bowls | Average |
|---|---|---|---|---|---|---|
| DP | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| Pi0-FAST | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| Pi0 | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| MA-VLA (Ours) | 28.0% | 9.0% | 9.0% | 9.0% | 10.0% | 13.0% |
Table 3: Real-World Dual-Arm SO101 Hardware Results (Success Count over 20 Trials) Evaluates in-domain (ID) and out-of-domain (OOD) tasks featuring inverted placement orders and swapped handovers.
| Method | Stack Bowls (ID) | Stack Bowls (OOD) | Place Cubes (ID) | Place Cubes (OOD) | Pass Toys (ID) | Pass Toys (OOD) | Stack Cubes (ID) | Stack Cubes (OOD) |
|---|---|---|---|---|---|---|---|---|
| Pi0 | 9/20 | 0/20 | 12/20 | 0/20 | 3/20 | 0/20 | 5/20 | 0/20 |
| MA-VLA (Ours) | 12/20 | 10/20 | 15/20 | 8/20 | 6/20 | 2/20 | 8/20 | 2/20 |
Ablation Study¶
Table 4: Component Ablation on Three Robots Stack Cube (Unseen Ordering)
| Atomic Prompts (Atom) | Arm Shuffle | View Dropout | In-Domain (ID) | Out-of-Domain (OOD) | Note |
|---|---|---|---|---|---|
| ✗ | ✗ | ✗ | 48.0% | 0.0% | Base Pi0 baseline |
| ✓ | ✗ | ✗ | 58.0% | 0.0% | Prompts boost ID (+10.0%), OOD remains 0.0% |
| ✓ | ✓ | ✗ | 52.0% | 7.3% | Arm Shuffle unlocks zero-shot OOD generalization |
| ✓ | ✓ | ✓ | 53.0% | 15.3% | View Dropout doubles OOD performance to 15.3% |
Table 5: Unified MA-VLA Model vs. Decentralized Independent Single-Arm Policies
| Configuration | Model Instances | In-Domain (ID) | Out-of-Domain (OOD) | Average |
|---|---|---|---|---|
| Pi0 (Global Instruction) | 1 | 48.0% | 0.0% | 24.0% |
| Separate Pi0 (1 policy per arm) | 3 | 61.0% | 0.0% | 30.5% |
| MA-VLA Unified (Ours) | 1 | 53.0% | 15.3% | 34.2% |
Key Findings¶
- Atomic prompts resolve coordination ambiguity: In 3–4 arm tasks where coordination complexity peaks, per-arm atomic prompts significantly clarify responsibilities, lifting 4-arm Delivery success from 82.0% to 97.0%.
- Arm Shuffle is essential for compositional transfer: Without Arm Shuffle, all baselines fail completely (0.0% success) when evaluated on unseen collaboration sequences. Arm Shuffle decouples motor control from hardware identities.
- Decentralized single-arm policies fail to coordinate: Training separate VLA models per arm improves in-domain accuracy (61.0%) due to unshared parameter capacity, but completely fails out-of-domain (0.0%), proving that cross-arm attention is necessary for emergent collaboration.
Highlights & Insights¶
- Permutation augmentation breaks combinatorial bottlenecks: Collecting demonstrations for every conceivable multi-arm collaboration pattern is intractable. Shuffling \((state, view, prompt, action)\) tuples at training time creates an effective inductive bias toward role-invariant execution.
- Hierarchical decoupling: Offloading temporal planning to a zero-shot frontier VLM and fine-grained motor trajectory execution to a flow-matching VLA prevents end-to-end compounding errors.
- Data-level regularizers offer zero inference overhead: Both Arm Shuffle and View Dropout function purely during training; at deployment time, the policy requires no special runtime wrappers.
Limitations & Future Work¶
- Open-loop planner limitations: The VLM planner schedules atomic prompts open-loop at task initialization; in the event of dropped objects or collisions, the system cannot trigger reactive closed-loop re-planning.
- Modest OOD absolute success: Out-of-domain success on complex multi-arm reordering tasks plateaus around 13%–15%, indicating that physical contact dynamics remain challenging under spatial perturbations.
- Future directions: Integrating closed-loop verification detectors for real-time recovery and incorporating multi-arm reinforcement learning for contact-rich interaction.
Related Work & Insights¶
- vs. \(\pi_0\) / OpenVLA: Standard VLAs process single global text tokens, which implicitly tie physical execution to training arm roles. MA-VLA shows that per-arm linguistic routing is essential for flexible multi-robot systems.
- vs. RoboBallet / RoboFactory: RoboBallet uses expensive multi-agent RL without language conditioning; RoboFactory relies on Diffusion Policy without semantic compositionality. MA-VLA bridges high-level semantic generalization and continuous multi-arm control.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates multi-arm compositional generalization and proposes an elegant, effective Arm Shuffle strategy.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive verification across 2–4 arm simulation suites and real-world dual-arm robots.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, thorough problem formulation, and crisp ablation analysis.
- Value: ⭐⭐⭐⭐⭐ Establishes a foundational paradigm for scaling vision-language-action policies to multi-robot collaboration.