MobileManiBench: Simplifying Model Verification for Mobile Manipulation

Conference: ECCV 2026
arXiv: 2602.05233
Paper: Project Website
Code: Project Website (Code, dataset, and models are all publicly released here)
Area: Robotics / Embodied AI
Keywords: Mobile Manipulation, Simulation Benchmark, Vision-Language-Action Models, Reinforcement Learning, Dexterous Manipulation

TL;DR

This paper proposes MobileManiBench, a large-scale mobile manipulation simulation benchmark based on Isaac Sim. By training reinforcement learning policies, it automatically generates 300K diverse manipulation trajectories (containing language instructions, multi-view RGB-D images, object/robot states, and actions) across 2 robot platforms, 630 objects, 5 skills, and 100 scenes. Multiple VLA models are systematically evaluated on this benchmark, providing a standardized platform for verifying VLA architectures in mobile manipulation tasks.

Background & Motivation

Background: Vision-Language-Action (VLA) models have made significant progress in robot manipulation, demonstrating strong generalization to novel objects and diverse visual environments. The training of current VLA models relies heavily on large-scale datasets, with Open X-Embodiment emerging as the de facto standard training resource. However, existing data is primarily collected via teleoperation in static tabletop scenarios, limited by third-person or head-mounted views, and features a constrained variety of objects.

Limitations of Prior Work: Building a mobile manipulation benchmark faces multiple technical barriers. First, the cost of teleoperation data collection is extremely high—once the hardware configuration changes (e.g., adding depth cameras, switching to dexterous hands, or mounting mobile bases), the data must be recollected from scratch. Teleoperation pipelines are particularly cumbersome for dexterous hands and mobile platforms. Second, existing simulation benchmarks (e.g., RLBench, ManiSkill2, LIBERO) mostly feature fixed bases and tabletop scenarios, lacking systematic support for mobile manipulation, dexterous hands, and multi-view sensors. Third, diverse manipulation skills (pull/push/open/close/pick) require unified modeling on the same platform, whereas the manipulation modes of different object structures (articulated vs. rigid objects) vary significantly.

Why this is possible now: The maturation of high-fidelity simulation engines (NVIDIA Isaac Sim) and reinforcement learning (PPO) makes the automated generation of diverse manipulation trajectories feasible. By training general RL policies rather than manually designing rules, multiple robot-object-skill combinations can be handled in a unified framework, automatically generating heavily annotated data.

Core Idea: Migrating VLA model verification from expensive, high-risk real-world teleoperation to simulation environments. First, RL is utilized to automatically train manipulation policies for each combination, which are then deployed in diverse scenarios to generate large-scale trajectories with multi-modal annotations. This allows evaluation of VLA architectural design choices under high-fidelity, low-cost, and reproducible conditions.

Method

Overall Architecture

The construction of MobileManiBench consists of three phases: MobileManiRL policy training, automatic generation of MobileManiDataset, and MobileManiVLA baseline model training. The workflow is illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Configuration<br/>2 Robots(G1/XHand)<br/>× 630 Objects × 5 Skills"] --> B["MobileManiRL Training<br/>Keypoint-based General RL Policy<br/>1182 combinations, avg 89.6%/92.9%"]
    B --> C["MobileManiDataset Generation<br/>Deploy policies to 100 scenes<br/>300K trajectories + Multi-modal annotations"]
    C --> D["MobileManiVLA Training<br/>Multi-view RGB-D fusion + DiT<br/>General VLA baseline model"]
    D --> E["Evaluate generalization on<br/>unseen objects/scenes"]

First, for each robot-object-skill triplet, a state-based reinforcement learning policy (MobileManiRL) is trained in simplified scenarios. The trained policies are then deployed in 100 diverse photorealistic scenes to automatically generate successful manipulation trajectories. Finally, all trajectories are aggregated to train a general VLA model, MobileManiVLA, which is evaluated for generalization capabilities on unseen objects and unseen scenes.

Key Designs

1. MobileManiRL: Keypoint-Based General Reinforcement Learning Policy

The key challenge this design addresses: manually designing policies for 1182 robot-object-skill combinations is impractical. The paper's solution is to abstract the manipulation problem as a planning problem of keypoint displacements.

Three categories of keypoints are defined: robot gripper/hand points, the object grasp point, and the target goal point. A unified reward function \(R = R_d + (1-f_g)R_a + f_g(R_g+R_m+R_s)\) drives policy learning: the global distance reward \(R_d\) penalizes discrepancies between the gripper and the grasp point, and between the grasp point and the goal point; when the gripper approaches the grasping point (distance below a threshold), the grasping flag \(f_g\) is set to 1, transitioning to the post-grasp phase, which sequentially triggers grasping reward \(R_g\), mobile movement reward \(R_m\), and success reward \(R_s\).

The policy input contains 5 categories of information: timestep encoding, object state (positions of the grasp point and goal point), robot proprioception (78-135 dimensions, including joint angles and poses), robot-object distance (22-31 dimensions, point-to-point distances from the gripper/fingers to the grasp point), and the previous action. The policy network is a 4-layer MLP (hidden layers 1024→1024→512→512), outputting \((6+D)\)-dimensional actions: the first 6 dimensions represent the wrist pose increment (analytically solved to obtain the target joint angles for the mobile base and the right arm), and the remaining \(D\) dimensions represent the target joint angles of the end-effector (\(D=1\) for the G1 robot, \(D=12\) for the XHand dexterous hand).

2. Automated Data Generation and Scene Diversification

This design addresses the bottlenecks of data diversity and annotation completeness. After being trained in simplified scenarios (ground and tabletop settings), MobileManiRL is deployed to 5 types of photorealistic scenes for trajectory rendering: space scenes (carts), wall-bound scenes (toilets, refrigerators, cabinets, etc.), door scenes (windows, lever doors, round-knob doors), outdoor scenes (cars), and tabletop scenes (boxes, laptops, dishwashers, etc.). For each scene category, 20 placement locations are manually annotated (totaling 100), of which 80 are used for VLA training and 20 are reserved for testing. The scenes cover daily environments such as kitchens, bedrooms, hospitals, offices, warehouses, and parking lots.

Each manipulation trajectory is recorded at 30 FPS, with an average length of 160 frames. Each trajectory \(\mathcal{T} = \{L, (I_1,S_1,A_1), \dots, (I_T,S_T,A_T)\}\) contains: a natural language instruction \(L\); synchronized \(520\times520\) RGB, depth, and segmentation images \(I_t\) from head and wrist views; object and robot states \(S_t\); and \((6+D)\)-dimensional actions \(A_t\). All states and actions are recorded in the global world coordinate system. The final dataset comprises a total of 15,232 training combinations and 920 test combinations. Generative collection yields 10 successful trajectories per training combination, making a total of 150K training trajectories.

3. MobileManiVLA: Multi-View Multi-Modal Diffusion Transformer Architecture

Addressing the problem that a single RGB view is insufficient for precise manipulation in mobile scenarios, this design constructs a VLA baseline model integrating 4 visual inputs and robot state information.

The vision and language modules are initialized from pre-trained PaliGemma-2 weights. The vision module processes 4 image streams from head and wrist views (two RGB + two depth, each resized to \(224\times224\times3\)), which are encoded into dense visual embeddings by SigLIP. Language instructions formatted as <skill> <object> (such as "open faucet") are tokenized and projected into language embeddings. After linear projection alignment, both are fed into the Gemma-2 language model along with a learnable cognitive token \(c\) to output features \(f_t^c\) that fuse perceptual and instruction context.

The action module adopts a Diffusion Transformer (DiT). During each inference step, DiT takes the noisy action sequence \((a_t^i, a_{t+1}^i, \dots, a_{t+N}^i)\) at the current denoising step \(i\), the cognitive features \(f_t^c\), and the 6-dimensional wrist pose state features \(f_t^s\) encoded by a lightweight MLP as conditional inputs. Through multi-step denoising, it progressively predicts clean actions \((\hat{a}_t, \hat{a}_{t+1}, \dots, \hat{a}_{t+N})\). The action chunk size is \(N=16\), and the adaptive ensembling strategy proposed in CogACT (with window size \(K=4\)) is used during inference to enhance trajectory smoothness. All states and actions are transformed into the robot's mobile base coordinate system to ensure cross-scene consistency.

Loss & Training

MobileManiVLA is trained end-to-end. The loss function is the standard MSE for diffusion models: \(\mathcal{L}_{\text{MSE}} = \mathbb{E}_{\epsilon \sim \mathcal{N}(0,1), i} \|\hat{\epsilon}^i - \epsilon\|_2\), where \(\hat{\epsilon}^i\) is the predicted noise at denoising step \(i\), and \(\epsilon\) is the corresponding ground-truth Gaussian noise. Training uses a batch size of 480, a learning rate of \(2\times10^{-5}\), and runs for 320K iterations, taking approximately 12 days per robot on 8 NVIDIA B200 GPUs.

Key Experimental Results

Main Results

The table below compares the success rates of MobileManiRL (state inputs, independent training and evaluation per combination) and MobileManiVLA (implicit sensory inputs, general model, evaluated on unseen objects and scenes) across 5 manipulation skills:

Skill	MobileManiRL G1 (%)	MobileManiRL XHand (%)	MobileManiVLA G1 (%)	MobileManiVLA XHand (%)
Open	86.6	91.9	42.9	34.5
Close	96.2	96.5	75.8	77.5
Pull	80.8	97.3	34.4	29.4
Push	93.1	97.2	85.1	90.0
Pick	66.4	72.6	28.8	40.2
Mean	89.6	92.9	56.7	57.3

Ablation Study

Input Modality Ablation (G1 robot, evaluated on three challenging skills: open, pull, and pick):

Visual Input	State Input	Success Rate (%)
Head RGB only	—	7.9
+ Head Depth	—	14.1
+ Wrist RGB	—	14.9
Head RGB-D + Wrist RGB-D	—	28.2
Head RGB-D + Wrist RGB-D	Wrist pose	32.3
Head RGB-D + Wrist RGB-D	Wrist pose + Grasp point + Goal point	36.6

Object/Scene Generalization Ablation (G1 robot):

Object	Scene	Success Rate (%)
Seen	Seen	59.6
Seen	Unseen	51.3
Unseen	Seen	39.2
Unseen	Unseen	28.2

VLA Model Comparison (G1 robot, default input configuration):

Model	Success Rate (%)
OpenVLA (7B)	4.5
CogACT (7B)	6.8
π₀ (3B)	11.2
π₀.₅ (3B)	18.8
MobileManiVLA (3B)	28.2

Key Findings

• Dexterous hands improve precision but increase collision risks: While the XHand dexterous hand overall outperforms the G1 gripper under MobileManiRL (92.9% vs. 89.6%), they perform comparably in MobileManiVLA (57.3% vs. 56.7%). On tasks like open/pull, the dexterous hand experiences degraded performance due to finger-object collisions, only excelling in the pick task due to its force-closure grasping advantage.
• Multi-view and multi-modality are crucial for mobile manipulation: Using only the head RGB yields a mere 7.9% success rate; incorporating synchronized head-and-wrist RGB-D boosts it to 28.2%, highlighting that self-occlusion of the robotic arm severely limits single-view setups. This is further validated by comparing with existing VLA models (OpenVLA/CogACT with only head RGB struggle at 4.5%/6.8%).
• Generalization to unseen objects is harder than to unseen scenes: The success rate for unseen objects in seen scenes is 39.2%, whereas seen objects in unseen scenes yield 51.3%. This gap of approximately 12% indicates that variations in object structure pose a greater challenge to manipulation policies than changes in scene layouts.
• The mobile base is vital: Under a fixed base, the success rate of MobileManiRL drops drastically from 82.8% to 25.4% (especially in tasks requiring mobility like open laptop/pull cart), confirming the core role of the mobile base in mobile manipulation scenarios.

Highlights & Insights

• Simulation-First Verification Paradigm: In the context of booming VLA models and skyrocketing data collection costs, shifting model verification from the real world to simulation is an elegant bridging strategy. It allows rapid iteration of architectures and sensor configurations in simulation prior to real-world deployment, drastically reducing verification overhead. This methodology is transferable to any robotic manipulation task beyond mobile manipulation.
• Keypoint-Based General Reward Function: Different object types (articulated, rigid) and manipulation skills (pull, push, open, close, pick) are modeled uniformly via the displacement of three keypoint groups (gripper points, grasp points, goal points). This unified description allows training RL policies for 1182 combinations with a single reward function, avoiding the necessity of manual reward engineering for each setting.
• Multi-View RGB-D Data Collection Paradigm: The paper clearly demonstrates the rigid necessity of four-channel visual inputs (head + wrist views, RGB + depth) for mobile manipulation (boosting success from 7.9% to 28.2%). This serves as a direct reminder to modern VLA models (which mostly employ single-view RGB) that the sensory challenges of mobile manipulation are far greater than those of tabletop setups.
• Limitations of Dexterous Hands in the VLA Paradigm: The paper honestly reports the sub-optimal performance of dexterous hands in the VLA model, where fingers easily collide when approaching object surfaces. This suggests that in VLA frameworks, the control of high-DOF dexterous fingers may require adding more constraints during training or designing better visual guidance strategies.
• Unified Multi-Dimensional Evaluation Platform: Concurrently supporting 2 end-effectors \(\times\) 630 objects \(\times\) 5 skills \(\times\) 100 scenes \(\times\) 2 views, this design space covers the primary variables of mobile manipulation, serving as a standardized platform for systematic studies on embodiment, sensory modalities, and policy architectures.

Limitations & Future Work

• Sim-to-Real Gap: While the paper demonstrates real-world robot deployment of MobileManiVLA (in the appendix), evaluations are heavily centered on simulation. Discrepancies in physical properties (friction, stiffness, contact dynamics) between simulation and the real world remain, and whether the conclusions drawn directly transfer to reality requires further verification.
• VLA Performance Lags Behind RL: MobileManiVLA (56.7%/57.3%) significantly underperforms MobileManiRL (89.6%/92.9%) by about 33%. This is because VLA models rely on implicit visual inputs rather than precise states, reflecting a fundamental limitation of current VLA architectures in fine-grained manipulation—where feature abstraction incurs far more information loss than explicit states.
• Trajectory Generation Upper-Bounded by RL: The dataset quality depends heavily on the success rate of the MobileManiRL policy. For challenging skills like pick (where RL only achieves 66.4%/72.6%), the ratio of successful trajectories in the subsequent training data is inherently constrained. Improving the RL policies themselves will directly enhance dataset quality and VLA training outcomes.
• Limited Skill Spectrum: Currently, only 5 manipulation skills are supported, lacking complex tasks (e.g., assembly, tool use, multi-object collaboration). Scenes are also restricted to static objects, lacking dynamic environments (e.g., human traffic, moving obstacles).
• Only Two Robot Platforms: Although dual-arm platforms with either parallel grippers or dexterous hands are included, other common end-effectors (e.g., suction cups, 3-finger claws) are absent, and both robots are dual-arm systems where only the right arm is activated.

Related Work & Insights

• vs. RLBench / ManiSkill2 / LIBERO: These are classic simulation benchmarks for tabletop manipulation, geared toward fixed-base platforms and tabletop objects without mobile bases or dexterous hands. MobileManiBench expands the manipulation space extensively in terms of mobility, end-effector diversity, and skill breadth.
• vs. SIMPLER: SIMPLER focuses on Sim-to-Real evaluation protocols for tabletop manipulation, but is limited to fixed bases and parallel grippers. MobileManiBench extends the evaluation to mobile manipulation and supports flexible sensor configurations.
• vs. DexArt: DexArt is a pioneer in studying dexterous articulated object manipulation in simulation, but supports only 2 skills (open/close), a single scene, and lacks a mobile base. MobileManiBench significantly surpasses it in scale and diversity.
• vs. Open X-Embodiment / DROID: These are real-world datasets that suffer from being limited mostly to fixed-base tabletop settings, and changing hardware configurations requires recollection. The simulation route of MobileManiBench bypasses this bottleneck.
• vs. π₀ / π₀.₅: These represent the latest VLA models, which are evaluated as baselines for comparative analysis. The results show that multi-view RGB-D fusion combined with wrist-pose state encoding yields substantial improvements over simple multi-view RGB concatenation (28.2% vs. 18.8%).

Rating

• Novelty: ⭐⭐⭐⭐ [Systematizes simulation-first validation for mobile manipulation, filling the gap of large-scale benchmarks; however, the overall concept of simulation data generation is not entirely original]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Spans 2 robots \(\times\) 630 objects \(\times\) 5 skills \(\times\) 100 scenes. Complete ablation designs cover input modalities, generalization dimensions, model comparisons, and mobile vs. fixed bases with multi-perspective analysis]
• Writing Quality: ⭐⭐⭐⭐ [Well-structured with concise takeaways and rich diagrams; however, the experiments in Section 5 could be more detailed, and some ablation results are only discussed in text without detailed tables]
• Value: ⭐⭐⭐⭐⭐ [Provides a rare, large-scale standardized evaluation platform for mobile manipulation VLA research, lowering validation cost and promoting reproducibility and architecture iteration; public release of code and datasets yields high practical value]

Related Papers

• [ICCV 2025] Learning Precise Affordances from Egocentric Videos for Robotic Manipulation
• [CVPR 2026] 3M-TI: High-Quality Mobile Thermal Imaging via Calibration-free Multi-Camera Cross-Modal Diffusion
• [CVPR 2026] RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulation
• [AAAI 2026] Segment and Matte Anything in a Unified Model (SAMA)
• [CVPR 2026] RobotSeg: A Model and Dataset for Segmenting Robots in Image and Video