Humanoid Whole-Body Manipulation via Active Spatial Brain and Generalizable Action Cerebellum

Conference: ECCV2026
Paper: https://eccv.ecva.net/virtual/2026/poster/3751
PDF: https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/2930.pdf
Project: Humanoid-WMAG
Area: Robotics & Embodied AI
Keywords: Whole-body manipulation, active perception, spatial reasoning, dexterous grasping, reachability optimization

TL;DR

An Active Spatial Brain decides where to observe next and what to execute, while a Generalizable Action Cerebellum translates those decisions into navigation, body repositioning, and dexterous manipulation without task-specific real-robot demonstrations, achieving 60.0% success on hard obstacle avoidance versus 30.0% for CaP.

Background & Motivation

Tabletop manipulation usually fixes the camera and robot base, with objects roughly inside the arm's workspace. A humanoid instead has to observe while moving, change body height when necessary, and coordinate its arms and dexterous hands. A target may be occluded, or visible but unreachable. Directly transferring a tabletop policy can confuse three distinct situations: needing more observations, needing to approach, and being ready to grasp.

Demonstration-based whole-body manipulation also faces a coverage problem: training heights, distances, and obstacle layouts cannot easily cover their combinations at test time. At the other extreme, a vision-language model (VLM) may understand object semantics and coarse spatial relationships without reliably producing collision-free paths or precise joint motions. The paper therefore does not assign all control to a large model. It separates active observation and task-stage assessment from geometric solving, allowing each execution outcome to change the next plan.

Core Idea: let large models change the observation conditions and select action intentions, let reachability optimization, motion planning, and grasp retargeting turn those intentions into executable actions, and connect the two through continuous feedback.

Method

Overall Architecture

The system takes a natural-language instruction, RGB-D observations, and the robot's proprioceptive state. Its outputs cover camera actions, dexterous hand poses, and upper- and lower-body joint actions, which PD controllers ultimately convert into joint torques. The high-level Active Spatial Brain maintains observation and execution histories and chooses between further observation and invoking the Generalizable Action Cerebellum. The latter provides Lower-body Locomotion and Reachability and Upper-body Dexterous Manipulation, returning execution feedback to the Brain.

The system does not generate a complete action sequence and then execute it open-loop. After each observation or subtask, it reassesses the target, current task stage, and next requirement. Lower- and upper-body actions can alternate across repeated invocations; not every iteration must involve walking followed by grasping.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Instruction, RGB-D<br/>and proprioception"] --> B["Active Spatial Brain"]
    B -->|Approach or reposition| C["Lower-body Locomotion and Reachability"]
    B -->|Grasp or manipulate| D["Upper-body Dexterous Manipulation"]
    C --> E["Joint actions and execution feedback"]
    D --> E
    E --> B
    B -->|Need more observations| A

Key Designs

1. Active Spatial Brain: determine which observations are missing before choosing an action

Failure to see a target from a fixed viewpoint does not establish that it is absent. The Active Spatial Brain is prompted with the camera neck's 2-DoF motion and a 4-DoF base action space covering body translation, height changes, and rotation about the vertical axis, together with their effects. The VLM uses the current image and instruction to select a new observation. Here, active perception means making viewpoint changes schedulable actions rather than merely collecting additional photographs: rotate the camera to search, move the body when a different vantage point is needed, and then assess whether there is enough evidence to manipulate.

Changing viewpoints also creates reference-frame ambiguity, so the memory bank does not accumulate images indefinitely. It stores only observations acquired since the last base movement, paired with their camera poses. After the base moves, images from the previous location no longer form the current local observation set. Meanwhile, a global textual log retains the Brain's commands to the Cerebellum and their feedback, preserving task progress when old visual records are discarded. The adaptive task planner first decomposes the instruction into subtasks, then repeatedly uses both memory types to assess execution progress. It adjusts the viewpoint when observations are insufficient, invokes an action agent when conditions are suitable, and schedules another attempt after a failed grasp. This preserves cross-stage execution history without treating pixel relationships from different body locations as interchangeable.

2. Lower-body Locomotion and Reachability: solve approaching the target separately from making it reachable

Locomotion combines a spatial navigation agent with an end-effector reachable space solver. The former handles coarse, long-range approach. It requests an RGB-D image containing both the target and the ground, back-projects depth into a point cloud, and projects that cloud onto the ground to form an occupancy grid. A cell is treated as occupied when it contains a point above a height threshold. The target's image coordinates undergo the same back-projection and ground projection to identify the goal cell. A* then searches for an obstacle-avoiding path, the Ramer–Douglas–Peucker (RDP) algorithm simplifies it into waypoints, and the robot tracks them using feedback about its position relative to the next waypoint. The VLM therefore need not infer an entire precise, collision-free trajectory from an image; it supplies geometrically meaningful inputs to a conventional planner.

Reaching the vicinity of an object does not ensure that the arm can grasp it. The reachable space solver jointly optimizes horizontal displacement and body height, penalizing excessive adjustment while bringing the object into an operable region near the shoulder. It also constrains shoulder-to-target distance using a safety margin on arm length. Crucially, changing a humanoid's body height changes the shoulder position relative to the base, so the shoulder cannot be treated as fixed. The authors use MuJoCo simulation to model how height affects the shoulder's horizontal and vertical offsets, then evaluate reachability in the adjusted base frame. The distance constraint alone can be summarized as:

\[ \left\|\mathbf{p}_{\mathrm{obj}}-\mathbf{p}_{\mathrm{shoulder}}\right\|_2 \leq 0.8L. \]

Both positions must be evaluated after body adjustment in the same base frame, and \(L\) denotes arm length. This is an equivalent summary of the reachability constraint based on the source's prose, not a verbatim recovery of its damaged equation layout. The full optimization equation is poorly extracted in the cache, so the exact weights and terms are not invented here. Navigation or fine-adjustment outputs are converted into forward velocity, turning velocity, and base height. A reinforcement learning locomotion policy following HOMIE then combines these commands with proprioception to generate lower-body joint actions. Target selection, geometric positioning, and stable low-level locomotion operate at different control scales.

3. Upper-body Dexterous Manipulation: connect human grasp priors and interpretable action primitives to robot joints

The upper-body module likewise avoids asking the VLM to predict every joint angle directly. A grasp pose generation agent uses a multimodal generative model to synthesize a human hand–object interaction image conditioned on the observation and instruction, drawing on large-scale pretraining for object-specific grasp priors. The image is not executable: human and robot hands differ, and an image does not supply joint states. The system therefore uses pretrained Hyper3D to reconstruct an object mesh from the observation, combines that mesh with the synthesized interaction image in EasyHOI to recover a human hand pose, and optimizes the robot fingers to align corresponding fingertips with the recovered human fingertips. The bridge is an image prior, a 3D hand–object relationship, fingertip correspondence, and finally a dexterous hand action—not the assumption that a generated image already satisfies every physical constraint.

After grasping, a trajectory generation agent predicts semantic 2D keypoints from the visual input and instruction and selects a parameterized primitive for pushing, pulling, placing, or rotating. Once depth lifts the keypoints into 3D, the primitive's kinematic rules convert them into a sequence of 6-DoF end-effector target poses. An initial alignment moves to a target keypoint or preparatory pose; pushing and pulling apply linear translation, placing preserves the original hand–object offset, and rotating applies a local rotation around a specified pivot. Inverse kinematics (IK) converts the end-effector poses into continuous joint commands. The generative model supplies grasp appearance priors that would be difficult to enumerate manually, while primitives and IK constrain subsequent motion explicitly. Generalization therefore remains bounded by the primitive set and reconstruction quality rather than covering arbitrary manipulation automatically.

A Worked Example

Consider the paper's occlusion-handling task: an instruction requires manipulating a target initially hidden by another object. The following illustrates the method's workflow; it is not presented as an exact execution trace recorded in the paper.

The Brain first uses active observation to identify the occlusion relationship and organizes the task as handling the occluder before manipulating the target. If the robot is too far away, it invokes the lower-body module to construct a grid and plan obstacle-avoiding waypoints. Near the object, it adjusts body height and position using the target and shoulder locations rather than maintaining a fixed standing distance.

The upper-body module generates a grasp for the object currently being handled and changes its position using an appropriate primitive. After feedback and new images return, the Brain checks whether the occlusion has been removed: a missed grasp triggers another attempt, whereas an exposed target triggers the target manipulation stage. If the base moved, visual memory uses images from the new location, while the textual log retains progress on handling the occluder. Already completed subtasks therefore need not be restarted.

Loss & Training

The paper does not introduce a new end-to-end loss for jointly training the entire whole-body policy. Execution primarily combines reachability optimization, fingertip alignment, kinematic solving, and a HOMIE-style reinforcement learning locomotion policy, alongside existing VLM and grasp-related foundation-model capabilities.

Requiring no task-specific real-robot data is not equivalent to having no training anywhere: foundation models rely on pretraining, and the locomotion policy also requires training. The comparison concerns avoiding additional real-robot demonstrations for learning these target tasks. By contrast, the data-driven baselines start from officially released post-pre-trained weights and receive at least 20 trajectories from each task's easy setting for post-training, before evaluation on both easy and hard settings.

The local main paper does not fully identify the VLM or generative model used by each deployed agent, nor provide all rewards, prompts, optimization weights, and control frequencies needed for reproduction. No unified training hyperparameters are invented here.

Key Experimental Results

Main Results

Experiments use a Unitree G1 with two 6-DoF Inspire FTP dexterous hands and a 2-DoF active RealSense D435i camera. The five task categories cover different heights, different positions, their combination, obstacle avoidance, and occlusion handling, each with easy and hard settings. Each setting receives 10–30 trials, with equal trial counts across methods within that setting. Success requires completing the manipulation without colliding with obstacles. The table reports success rates in %, higher is better, selecting three compound spatial tasks from the paper's Table 1.

Setting	Task	TrajBooster	Ψ0	CaP	Ours
Easy	Combined height and position changes	55.0	70.0	70.0	80.0
Easy	Obstacle avoidance	60.0	70.0	80.0	80.0
Easy	Occlusion handling	40.0	60.0	65.0	70.0
Hard	Combined height and position changes	10.0	25.0	35.0	55.0
Hard	Obstacle avoidance	0	0	30.0	60.0
Hard	Occlusion handling	20.0	30.0	35.0	60.0

Hard obstacle avoidance introduces multiple obstacles rather than the single obstacle in the easy setting; hard occlusion handling increases occluder diversity. These comparisons therefore test generalization across task layouts, not universal superiority on in-distribution tests. In particular, both the proposed method and CaP achieve 80.0% on easy obstacle avoidance, so the paper's broad claim of outperforming baselines should not be read as a strict lead in every column.

Ablation Study

AP denotes active perception; removing it fixes the robot's viewpoint. EES denotes the end-effector reachable space solver; removing it substitutes a predefined distance and relative height to the target. Results below come from the paper's Table 3 and again report success rates in %, higher is better. The paper does not specify the easy/hard mixture or per-entry trial denominators for these ablations, so they cannot be equated directly with individual difficulty levels in the preceding table.

AP	EES	Combined height and position changes	Obstacle avoidance
Off	Off	18.8	0
On	Off	55.6	16.7
Off	On	24.9	33.4
On	On	67.5	70.0

Key Findings

• Removing only AP lowers success on combined height and position changes from 67.5% to 24.9%, a drop of 42.6 percentage points. Obstacle avoidance falls from 70.0% to 33.4%, or 36.6 percentage points. Dynamic observation is particularly relevant to maintaining valid visual context after long-distance movement.
• Removing only EES reduces the two task results by 11.9 and 53.3 percentage points, respectively. Fine repositioning matters especially after obstacle avoidance, consistent with the authors' explanation about perception errors accumulated during nonlinear motion. The table does not directly measure those errors, however, so this remains a mechanistic interpretation rather than a separate causal measurement.
• The independent spatial benchmark asks VLMs to reconstruct shuffled 3×3 image grids, actively locate targets, and propose obstacle-avoiding waypoints. gemini-3.1-pro reaches 98.84% correctness and 94.11% success on the first two tasks, but 21.05% of its waypoint predictions are still classified as collision failures. These are component benchmarks, not end-to-end robot success rates, and do not establish that every deployed module uses that model.
• Object generalization is uneven: on tools unseen during TrajBooster fine-tuning, the proposed method achieves 52.0% versus 50.0% for TrajBooster; on unseen everyday items, the rates are 69.6% and 50.0%. Robustness to spatial changes is therefore better supported than a claim of substantial gains on every new object.

Highlights & Insights

• The memory bank is organized around body movement rather than simply increasing context length. Updating local visual records after base movement while retaining global execution text targets errors caused by changing spatial references.
• Reachability is formulated as an optimization problem involving the body, not a fixed check before grasping. Explicitly modeling shoulder displacement with height explains why navigation to an object's vicinity is insufficient.
• The boundary between large models and geometric algorithms is relatively clear. Semantic reasoning selects targets and operation types, while grid planning, kinematic models, and action primitives handle precise execution; the component spatial benchmark also supports this division.

Limitations & Future Work

• The paper does not develop a separate limitations section. From the method's scope, grasping depends on synthesized images, object reconstruction, and hand retargeting, while subsequent manipulation is restricted to a defined primitive family. The main paper does not establish coverage of arbitrary contact-rich operations requiring continuous force control.
• Evidence comes from one robot platform and limited trials, without confidence intervals in the result tables. Systematic measurements of end-to-end latency, model-call cost, and failure types are also absent, leaving long-term autonomous reliability unresolved.
• Table 3 does not separately ablate the memory strategy, adaptive replanning, or the upper-body grasping chain, and does not explain difficulty aggregation. Those measurements and implementation details are needed for stronger performance attribution and reproducibility assessment.

Related Work & Insights

• vs TrajBooster / Ψ0: The baselines adapt policies using real-robot task data, whereas this work mainly combines existing model capabilities with explicit planning in a task loop. Hard-setting results support advantages beyond the given training coverage, not superiority over learned policies at arbitrary data scales.
• vs CaP: CaP uses language models to generate executable control programs. This work adds mechanisms specifically addressing humanoid viewpoint changes and whole-body reachability. Its clear advantage on hard obstacle avoidance but tie on the easy setting shows that benefits depend on task conditions.
• Relationship to HOMIE, Hyper3D, and EasyHOI: These support lower-body locomotion, object reconstruction, and human hand pose recovery, respectively; they are not competing full systems replaced by this work. The main contribution lies in connecting these capabilities with active perception, task planning, and action execution in a closed loop rather than reinventing every foundation module.

Rating

• Novelty: 4/5. The combination of movement-conditioned memory, active spatial decisions, and whole-body execution is targeted, although many foundation modules come from prior work.
• Experimental Thoroughness: 3/5. Real-robot comparisons, difficulty shifts, and key ablations are included, but sample sizes, statistical uncertainty, and some protocol details remain limited.
• Writing Quality: 3/5. Module responsibilities are relatively clear; deployed models, optimization details, and ablation aggregation need fuller specification.
• Value: 4/5. A concrete, analyzable system route reduces task-specific real-robot demonstrations, with broader platform and long-term validation still needed.