Skip to content

Learning Physics-Based Full-Body Human Reaching and Grasping from Brief Walking References

Conference: CVPR 2025
arXiv: 2503.07481
Code: https://liyitang22.github.io/phys-reach-grasp/
Area: Robotics
Keywords: Physical Simulation, Full-Body Grasping, Walking Transfer, Active Data Augmentation, Shallow Feature Alignment

TL;DR

By utilizing only approximately 30 seconds of walking MoCap data and combining transferable movement patterns from walking (shallow network feature alignment) with kinematic-method-generated grasping poses (active data augmentation strategy), this work achieves physically feasible and natural full-body human reach-and-grasp motion generation, achieving a 99.8% grasp success rate in simple scenarios.

Background & Motivation

  1. Background: Physically simulated human-object interaction motion generation typically relies on massive MoCap data. Existing works like ASE and AMP replicate reference motions through adversarial learning or motion tracking, but the generated motions are limited by the coverage of the reference data.
  2. Limitations of Prior Work:
    • The collection of MoCap data for grasping motions is highly costly and has limited coverage—objects have diverse shapes and scenarios vary significantly, making it difficult to enumerate.
    • Methods relying on large-scale MoCap datasets (such as AMASS) perform well but suffer from high barriers to data acquisition.
    • Grasping motions generated by kinematics are flexible but lack physical plausibility and natural movement patterns.
  3. Key Challenge: How to generate diverse, physically plausible, and natural full-body interaction motions with extremely limited real motion data?
  4. Goal: Can reference walking data, which is easy to acquire, be used to drive the learning of full-body reaching and grasping motions?
  5. Key Insight: A key observation is that walking motions contain rich local movement patterns and balancing capabilities (e.g., reaching with the right hand while extending the left foot), which are transferable across tasks. A pilot study reveals that in a critic network trained on walking data, shallow features can capture common patterns of real motions, unaffected by semantic differences.
  6. Core Idea: Transferring local walking "movement patterns" to grasping—using shallow feature alignment to maintain naturalness while employing active data generation to address task coverage.

Method

Overall Architecture

A multi-iteration training pipeline: Each iteration consists of low-level policy training (establishing a latent motion space) and high-level policy training (selecting actions to complete downstream tasks). The first iteration uses only walking data to build the motion space. Subsequently, task performance is evaluated, and an active policy is used to generate interpolated motion data for difficult scenarios. After expanding the dataset, the low-level policy is fine-tuned (with shallow feature alignment regularization), followed by training the next round of the high-level policy, iterating until convergence.

Key Designs

  1. Shallow Transferable Patterns Discovered in the Pilot Study:

    • Function: Verifies the transferable motion features inherent in walking movements, guiding the subsequent design of feature alignment.
    • Mechanism: A critic network is trained on walking MoCap data to extract features from different layers for real walking, real grasping (CIRCLE dataset), and interpolated grasping, calculating the FID. Results show that in the shallow layer (sub-features of \(f_0\)), the FID between real walking and real grasping is very small (2.35), while the FID to generated grasping is large (2.47). Conversely, this gap diminishes in deeper features, as deeper layers focus more on "walking-like" semantics. t-SNE visualizations also validate the clustering phenomenon of real motions in the shallow layers.
    • Design Motivation: This discovery serves as the theoretical foundation of the paper—shallow features represent low-level motion patterns (balance, coordination) while deep features represent high-level semantics (walking/grasping). Thus, aligning only the shallow features is sufficient to make grasping motions appear realistic.

  2. Active Data Generation Strategy:

    • Function: Intellectually generates training data targeting difficult scenarios, maximizing data utilization efficiency.
    • Mechanism: Tasks are discretized based on key parameters (such as table height), and the success rate \(sr\) and discriminator score \(\bar{p}\) of each task category are evaluated. A comprehensive weighted score is formulated as \(W_j = s_0 + w_{succ} \frac{\max_i sr_i - sr_j}{\max_i sr_i - \min_i sr_i} + w_{disc} \frac{\max_i \bar{p_i} - \bar{p_j}}{\max_i \bar{p_i} - \min_i \bar{p_i}}\). Tasks with higher scores (worse performance) receive more generated data. The data is generated via FLEX using SLERP interpolation from standing to grasping.
    • Design Motivation: Randomized data addition is inefficient—some scenarios are already sufficiently covered by walking data, while others (e.g., extremely high or low tabletops) urgently require new motions. The active strategy precisely allocates data to where it is needed.

  3. Local Feature Alignment Mechanism:

    • Function: Constrains generated motions using the feature distribution of walking when fine-tuning the low-level policy on the augmented dataset, maintaining naturalness.
    • Mechanism: The feature distribution \((\mu_i, \sigma_i)\) of walking data in the shallow layers of the critic is pre-computed. During training, for each state, the Mahalanobis distance of its shallow features \(f_i(s,z)\) to the walking distribution is calculated: \(d^{ma}_{f_i} = \sqrt{(f_i - \mu_i)(\sigma_i + \epsilon I)^{-1}(f_i - \mu_i)}\). When the distance exceeds a threshold, a penalty reward is applied: \(r^{feats} = -\sum_{f_i} w_{f_i} d^{ma}_{f_i} \mathbb{1}(d^{ma}_{f_i} > \text{thres}_{f_i})\). The threshold prevents over-constraining motion diversity.
    • Design Motivation: Although the generated interpolated motions provide correct task guidance, they lack the natural patterns of human movement. Shallow alignment enables new motions to "inherit" the local coordination and balance of walking, avoiding robot-like appearance.

Loss & Training

  • Low-level Policy: Adversarial imitation reward (discriminator \(D\)) + skill discovery reward (encoder \(q\)) + feature alignment reward (\(r^{feats}\)): \(r_t = -\log(1-D(s_t, s_{t+1})) + \beta \log q(z|s_t, s_{t+1}) + r^{feats}\).
  • High-level Policy: Task reward \(r_G\) (four-phase: directional walking, pre-grasping, grasping, post-grasping) + motion prior reward \(r_{p1}\) (preventing frequent skill switching) + walking guidance prior \(r_{p2}\) (guiding sampling in the first phase).
  • Trained using the PPO algorithm.

Key Experimental Results

Main Results

Overall Task Performance:

Method Simple Scenario SR(Grasp) Simple Scenario SR(Goal) Complex Scenario SR(Grasp) Complex Scenario SR(Goal)
ASE 55.7% 13.4% 40.2% 10.5%
AMP 85.3% 58.1% 65.5% 38.0%
AMP* (With Data) 85.9% 72.3% 66.7% 55.3%
Ours 99.8% 88.8% 69.7% 55.8%
Oracle Grasp Policy 100.0% 95.8% 75.8% 72.1%
Oracle Policy (Real Data) 97.4% 59.0% 69.7% 53.4%

Comparison with SOTA Methods:

Method SR(Grasp) SR(Goal) GPT-4o/Kimi Score User Score
WANDR 32%(reach) - 8.03/7.75 8.33
Braun et al. 59.6% 22.2% 6.00/5.00 5.83
Omnigrasp 54.4% 52.6% 6.50/6.13 5.67
Ours 69.7% 55.8% 7.38/7.25 7.55

Ablation Study

Comparison of Data Augmentation Strategies (Simple Scenario, Different Data Ratios):

Strategy 5% SR(Grasp/Goal) 10% SR(Grasp/Goal) 20% SR(Grasp/Goal)
Random 55.6% / 15.3% 81.2% / 20.7% 92.1% / 64.1%
Active-S (Success Rate Only) 70.1% / 30.1% 92.8% / 36.6% 95.2% / 64.2%
Active-Both Best Best Best

Ablation of Feature Alignment:

Configuration SR(Grasp) SR(Goal) User(G)
No Alignment Lower Lower Highly Unnatural
\(f_0\) Alignment Improved Improved Natural
\(f_0 + f_1\) Alignment Best Best Most Natural

Key Findings

  • Walking Reference Outperforms Real Grasping Reference (in some scenarios): In simple scenarios, the proposed method (using only walking data) achieves an SR(Goal) of 88.8%, surpassing the Oracle Policy (trained on real grasping data) which achieves 59.0%. This suggests that walking data offers better balancing capabilities.
  • Feature Alignment Boosts Naturalness and Success Rate: The balance skills transferred from walking via the alignment mechanism assist in completing challenging grasping tasks.
  • Active Strategy is Most Effective with Scarce Data: At a 5% data ratio, Active-S doubles the SR(Goal) compared to Random (30.1% vs. 15.3%).
  • Implications of AMP* Failure: Even when AMP is augmented with generated data, because the generated data contains artifacts that AMP's discriminator fails to distinguish, the resulting motions remain unnatural. This paper avoids this issue through decoupling (low-level policy fine-tuning + shallow alignment regularization).

Highlights & Insights

  • The Bold Vision of "Leveraging Limited Walking Data for Full-Body Grasping": This challenges the conventional wisdom that task-specific MoCap data is required, demonstrating the cross-task transferability of local motion patterns. This holds significant value for humanoid robotics, where collecting only basic walking data can enable transfer to complex interactions.
  • Elegant Experimental Design of the Pilot Study: Utilizing a hierarchical critic network to extract features from shallow to deep layers, the study quantitatively demonstrates that shallow layers capture cross-task motion patterns while deep layers capture semantics, providing solid experimental evidence for the methodology.
  • Practicality of the Active Strategy: Feeding diagnostic details of "which tasks are difficult" from RL training back into the data generation loop creates a closed-loop system. This paradigm is highly transferable to other data-efficient RL scenarios.

Limitations & Future Work

  • Limited Success Rate in Complex Scenarios: A 69.7% grasp rate compared to the Oracle's 75.8% indicates room for improvement in extreme scenarios (such as very low or high tabletops).
  • Quality of SLERP Interpolation: Generated data relies purely on linear SLERP interpolation from standing to grasping, which lacks authentic intermediate dynamics. Employing advanced kinematic generation methods or diffusion model generation could provide more substantial improvements.
  • Coverage of Walking References: The data only covers straight walking and turning, lacking side stepping or backward walking patterns. Incorporating more diverse walking patterns could further improve generalization.
  • Naturalness Gap Compared to WANDR in the User Study: Since WANDR is a kinematics-based approach and utilizes a larger motion dataset, it rates higher in naturalness (8.33 vs. 7.55), but it does not guarantee physical feasibility.
  • vs. ASE [Peng et al.]: ASE also constructs a motion space using adversarial learning and skill encoding but relies entirely on the coverage of the reference data. The proposed work adds data augmentation and feature alignment to the ASE framework, expanding its applicability from "data-heavy tasks" to "zero-data tasks."
  • vs. Omnigrasp [Zhang et al.]: Omnigrasp depends on the massive AMASS dataset and fails to generalize to unseen scenes. This work achieves comparable success rates with less than 30 seconds of walking data, signifying several orders of magnitude higher data efficiency.
  • vs. AMP [Peng et al.]: AMP's discriminator is trained alongside the task, making it vulnerable to artifacts in generated data. This work mitigates artifact propagation by decoupling low-level/high-level training and employing shallow feature alignment.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ "Walking to grasping" cross-task transfer framework is entirely novel, and the discovery of shallow feature transferability is highly inspiring
  • Experimental Thoroughness: ⭐⭐⭐⭐ Broad baselines, comprehensive ablations, and user study, though testing across more diverse object classes and environments is lacking
  • Writing Quality: ⭐⭐⭐⭐ Clear presentation of the pilot study, with intuitive system architecture schematics
  • Value: ⭐⭐⭐⭐⭐ Significant reference value for data-efficient physically simulated motion generation, especially within the humanoid robotics domain