WristMimic: Full-Body Humanoid Control with Wrist-Guided Manipulation¶

Conference: ECCV 2026
Paper: ECCV paper
Area: Video Understanding
Keywords: Whole-body control / Human-object interaction / Physics-based retargeting / Wrist guidance / Reinforcement learning

TL;DR¶

WristMimic separates human–object interaction retargeting into two supervision regimes — kinematic pose guidance for contact-free body parts and wrists, and fingers shaped only by object trajectory and contact outcomes — and secures wrist placement through reward modulation and phase-specific reset thresholds within a contact window; across 20 sequences each from OMOMO and ParaHome it attains a 91.1% average success rate, versus 43.3% for InterMimic despite the latter's full finger pose supervision.

Background & Motivation¶

Retargeting human demonstrations into physics-based simulation is a long-standing problem in robotics and vision: successful manipulation requires reproducing not only body poses but also the object motion and contact dynamics that make manipulation actually succeed. The common approach of tracking kinematic trajectories frame by frame works reasonably for parts that do not directly touch the object — torso, reaching arms, wrists — but manipulation happens at the hands, and kinematic trajectories carry no contact forces: the same finger positions can correspond to a light pinch or a firm grip, so accurate pose tracking does not guarantee that the object follows the reference. The authors therefore ask a contrarian question: rather than pursuing ever more precise finger tracking, can hand behavior be guided by object and contact dynamics?

Translating this into whole-body control yields asymmetric supervision divided by contact regime: contact-free body parts are well described in position space and suit standard kinematic imitation, whereas contact-rich finger motion is dominated by contact forces and must be learned indirectly through object-pose outcomes and contact alignment. The difficulty is that fingers exploring from arbitrary configurations rarely reach the right contact regions. The wrist is the natural bridge between the two regimes — it is largely contact-free and thus amenable to kinematic guidance, yet it determines the global finger configuration and the reachable grasp affordance (Fig. 1 contrasts grasp success and failure under wrist poses inside and outside a feasible bound).

Core Idea: provide kinematic pose targets only for the body and wrists, exclude all 30 finger joints from pose supervision entirely, shape finger behavior through object-pose tracking and contact alignment, and use reward modulation plus phase-specific reset thresholds inside a contact window to keep the wrist as the bridge connecting contact-free body control and contact-rich hand manipulation — enabling finger-agnostic retargeting across hand embodiments.

Method¶

Overall Architecture¶

WristMimic trains a policy in a physics-based simulator to reproduce a reference human–object interaction sequence: the policy outputs actions for 51 actuated SMPL-X joints, converted to torques by PD controllers; the reward multiplicatively combines joint position/rotation tracking, object position/rotation tracking, and contact alignment. Supervision is decoupled by contact regime — 19 body joints and 2 wrists form the contact-free set that receives kinematic pose targets, while the 30 finger joints receive no kinematic reference and learn only through object motion and contact outcomes. Around the first contact frame, a contact window applies reward weight modulation (relaxing the upper arm, keeping the wrist, reducing other body joints) and phase-specific wrist reset thresholds (7 cm / 0.2 rad during the grasping phase). Training uses PPO; at inference the policy maps proprioceptive and goal states directly to all joint actions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Reference body, object, contacts"] --> B["Decoupled supervision regimes"]
    B -->|Goal state| P["Single control policy"]
    S["Simulation state"] -->|Proprioception| P
    P -->|All joint actions| D["PD control and physics"]
    D --> S
    B -.->|Training: contact window| C["Prioritized wrist alignment"]
    C -.->|Modulate pose reward weights| U["PPO update"]
    C -.->|Same contact window| E["Phase-specific reset thresholds"]
    S -.->|Training: wrist deviation| E
    E -.->|Terminate and resample episode| U
    S -.->|Training: object and contact rewards| U
    U -.->|Update parameters| P

Solid arrows show the execution-time state–action loop; dashed arrows denote training rewards, episode resets, and parameter updates. Execution still requires a goal state constructed from the reference sequence: this is not a proprioception-only free-motion generator. Reward modulation and reset thresholds are not sequential inference layers. Resets determine whether sampling continues rather than acting as a multiplicative reward component.

Key Designs¶

1. Decoupled supervision regimes: assign supervision sources by contact within one policy

The SMPL-X humanoid has 52 rigid bodies and 51 actuated joints with 3 DoFs each, partitioned by contact: the contact-free set contains 19 body joints plus 2 wrist joints, and the contact-rich set contains 30 finger joints. The goal state provides kinematic pose targets (rotation and position differences relative to the reference, expanded over multiple time horizons) only for contact-free joints; finger joints are entirely absent from the goal. Finger behavior is optimized indirectly through object-pose targets and contact alignment. The action space covers all 51 joints with exponential-map target rotations converted to torques via PD controllers, so fingers remain fully actuated — they simply receive no pose reference.

The contact target has 23 elements: explicit contact labels for 21 non-finger rigid bodies plus one binary hand-level indicator per hand (set to 1 if any rigid body of that hand touches the object). Finger supervision therefore does not require per-finger contact labels. The reward multiplies five components, \(r_t = r^{p}_t \cdot r^{\theta}_t \cdot r^{p,\mathrm{obj}}_t \cdot r^{\theta,\mathrm{obj}}_t \cdot r^{c}_t\), each of the form \(r=\exp(-\lambda E)\); joint tracking errors are computed only over contact-free joints, with all 30 finger joints excluded from pose tracking. This design is a supervision split, not a claim of lightweight or efficient training — the exploration cost of emergent finger behavior does not disappear.

2. Prioritized wrist alignment: relax the proximal arm inside the contact window

Motion capture is inherently imperfect when executed in physics-based simulation — floating root frames can violate physical constraints — so aligning every joint exactly at once is unrealistic. Within the contact window \([t_c-t_b,\, t_c+t_a]\) centered on the first contact frame \(t_c\), the framework deliberately relaxes proximal supervision: upper-arm (shoulder, elbow) weights drop to zero so the policy can freely adjust upper-arm configurations to place the wrist optimally; wrist weights stay at 1; the remaining contact-free body joints are down-weighted to \(w_{red}<1\), letting the policy concentrate on hand–object contact and object dynamics inside the window. Outside the window, all weights return to normal.

This "selective relaxation" differs from globally amplifying the wrist reward: it frees degrees of freedom by subtraction rather than pressing other joints with larger weights. The paper motivates it by the need to let some joints deviate when mocap is imperfect, in exchange for accurate wrist alignment. Specific weight values are deferred to supplementary material, which is not in the cache.

3. Phase-specific reset thresholds: tighten the wrist feasibility region during the grasping phase

On top of base early termination (severe body-pose deviation, root height below threshold indicating falling, object pose drifting too far, contact error persisting over frames), the grasping hand's wrist receives an additional phase-specific reset scheme inside the contact window. The window is divided into three phases around \(t_c\): approach \([t_c-t_b,\, t_c+\tau_1]\), grasping \([t_c+\tau_1,\, t_c+\tau_2]\), and stabilization \([t_c+\tau_2,\, t_c+t_a]\), with \(\tau_1=-2\) and \(\tau_2=12\) frames. The grasping phase uses tight thresholds of 7 cm / 0.2 rad, terminating the episode when the wrist deviates beyond these bounds from the reference; the approach and stabilization phases use looser thresholds of 15 cm / 0.5 rad, preserving exploration room.

The design intent is "reduced reward weights + strict reset boundaries" rather than "high reward weights": exploration is confined near feasible wrist configurations so the policy can discover optimal wrist poses within a feasible search space, while absorbing mocap inaccuracies and maintaining natural grasp configurations. This shapes the exploration space explicitly; it imposes no direct constraint on the fingers.

One source inconsistency must be retained: Section 3.3 requires \(0\leq\tau_1<\tau_2\leq t_a\), whereas the implementation in Section 4.1 sets \(\tau_1=-2\). The values above follow the implementation paragraph rather than silently making the offset nonnegative; this places the start of grasping before first contact. Several state and reward equations are also damaged in the extracted text, so this note uses only structures supported by the prose and algorithm.

Loss & Training¶

Multiplying the five rewards requires pose, object trajectory, and contact objectives to succeed together. Reward modulation changes relative joint contributions to pose error, whereas early termination determines which trajectories continue to be sampled. The ablation tests reward modulation together with wrist resets, not whether multiplicative aggregation outperforms alternative reward combinations.

The training objective is the multiplicative reward above, optimized with PPO. Training runs in Isaac Gym with 2,048 parallel environments at 30 Hz control frequency on a single NVIDIA RTX 3090; the policy consists of separate actor and critic MLPs with SiLU activations. Contact-window parameters are \(t_b=10\) and \(t_a=15\) frames. Each sequence trains a scene-specific policy — an explicit authorial setting, not a cross-scene general policy. Complete hyperparameters are in supplementary material, absent from the cache.

Key Experimental Results¶

Main Results¶

Evaluation covers ParaHome and OMOMO, which provide object and subject geometries, human motion trajectories in SMPL-X (or bone vectors), and object trajectories; episodes span roughly 100–250 frames. All methods train scene-specific policies and are evaluated over 10,000 rollout trials per scene on three metrics: success rate (sequence completes without early termination, average object position error below 10 cm, and contact maintained for at least 80% of the reference contact duration), object position error (cm), and object rotation error (degrees). SkillMimicV2 is evaluated only on ParaHome because its bone-vector representation is available only there. The table reproduces the paper's Table 1 over 40 sequences (20 each from OMOMO and ParaHome), distinct from the later 8-sequence ablation. Entries are success rate (%), object position error (cm), and object rotation error (degrees), in that order.

Method	OMOMO	ParaHome	Average
InterMimic	86.6 / 14.2 / 22.2	0.1 / 83.5 / 73.3	43.3 / 48.8 / 47.7
SkillMimicV2	Not applicable	1.3 / 72.1 / 80.3	1.3 / 72.1 / 80.3
WristMimic	98.9 / 7.3 / 12.2	83.3 / 15.3 / 33.9	91.1 / 11.3 / 23.1

WristMimic leads on every metric, nearly halving object position and rotation errors on OMOMO; on ParaHome it retains an 83.3% success rate while InterMimic achieves 0.1%. The authors attribute the ParaHome gap to that dataset's more dexterous scenarios — grasping small handles or objects with narrow contact regions — where wrist constraints provide structural guidance. All methods use scene-specific policies, and cross-dataset comparisons are affected by task difficulty and dataset differences, so the gap between the two columns cannot be attributed entirely to the method itself. Success rate is a composite criterion (completion, error, and contact retention all required) and should not be read as a pure grasp-difficulty metric.

Ablation Study¶

The paper's Table 2 evaluates wrist constraints on 8 representative ParaHome sequences; format as above, units % / cm / degrees.

Config	ParaHome
Decoupled, no wrist constraints	0.0 / 56.4 / 49.7
+ Reward weight modulation only	0.0 / 50.3 / 43.6
+ Phase-specific reset only	67.5 / 28.8 / 56.9
WristMimic (full)	86.5 / 9.9 / 36.0

The third table reports the paper's Table 5 hand-morphology experiment on 10 ParaHome sequences, with separately trained policies for scene-specific SMPL-X, InterMimic, and OmniGrasp hand configurations. Its caption says “the same policy,” but Section 4.4 explicitly describes retraining and independently trained policies. This note follows the prose: the same method adapts to different hands, rather than demonstrating zero-shot transfer of one set of policy weights. Entries are success rate (%), object position error (cm), and rotation error (degrees), in that order.

Hand morphology	ParaHome
InterMimic hand	95.2 / 10.8 / 28.1
OmniGrasp hand	76.4 / 16.9 / 27.1
Scene-specific (SMPL-X)	75.8 / 20.1 / 39.6

Key Findings¶

Decoupling alone is insufficient: removing wrist constraints drops success to 0.0%, showing that excluding fingers from pose supervision alone cannot produce reliable manipulation. Phase-specific resets provide the main gain (0.0 → 67.5%), reward modulation alone gives only marginal improvement, and the combination is best.
The grasping-phase threshold involves a trade-off: 3.5 cm / 0.1 rad is too tight (0.0% success) and 15 cm / 0.5 rad too loose (17.5%); the default 7 cm / 0.2 rad achieves the best balance of 96.3% / 9.5 cm / 18.1° over 4 representative ParaHome scenes (paper's Table 4).
Hand-agnostic behavior holds but is asymmetric: the InterMimic hand performs best among the three, which the authors attribute to its constrained joint ranges (others use \([-\pi,\pi]\)) excluding physically implausible poses; wrist guidance's effectiveness is still bounded by hand-model kinematics.
Non-hand contact scenarios (pushing a table with the foot, sitting on a chair; paper's Table 3) need no wrist-like design and reach 97.2–99.9% success, showing the framework extends naturally to coarse body-support interactions.

Highlights & Insights¶

Treating "what to supervise" as a structural question rather than a data question. Instead of denser finger motion capture, identify the structurally decisive joint subset — here the wrist — and concentrate supervision there. The perspective generalizes to other embodied control tasks: first analyze which joint subset determines task success, then decide supervision density.
Constraints via reset boundaries rather than reward weights. Phase-specific resets bound the exploration space, while reward modulation only biases the search. When a degree of freedom "must land somewhere but the exact pose is not unique," this hard-boundary + soft-bias combination is steadier than pure reweighting.
Finger-agnostic retargeting without finger data. Finger configurations emerge from object–contact dynamics, and switching hand morphologies only requires retraining the same finger-supervision-free policy — a practical direction for reducing capture cost, especially finger-level capture.

Limitations & Future Work¶

Limitations admitted by the authors. The method targets affordance-aware grasping and object manipulation, not fine-grained in-hand manipulation requiring precise finger control (finger repositioning, in-hand object reorientation); training is currently scene-specific — one policy per sequence — with no cross-scene generalization.
Implicit dependence on mocap quality. Phase thresholds and reward modulation are designed around mocap error, so reference-trajectory quality directly shapes the feasibility bounds' appropriateness. Behavior under extreme noise is not verified in the cache's evidence and remains untested here.
Small evaluation scale. The main experiment covers 40 sequences (20 per dataset) with 10,000 rollouts per scene; hand-morphology generalization uses only 10 ParaHome sequences. Extrapolating conclusions to broader object/scene distributions needs more systematic evaluation — this is this note's judgment based on the paper's stated scope, not an authorial claim.
Possible improvements. Combine wrist constraints with scene-level generalization, e.g., training cross-scene policies with a shared wrist-structure prior; or make phase thresholds adaptive to mocap uncertainty rather than fixed frame counts and thresholds.

vs InterMimic: InterMimic achieves universal whole-body HOI control via teacher–student distillation with contact-guided rewards, relying on relatively complete kinematic supervision. WristMimic uses a single policy, decoupled supervision, and wrist-specific design to overtake it on ParaHome's fine-grained grasping (83.3% vs 0.1%), at the cost of scene-specific training.
vs SkillMimicV2: SkillMimicV2 handles noisy demonstrations with skill-level bone-vector representations yet reaches only 1.3% success on ParaHome. The qualitative comparison (Fig. 5) shows plausible finger closure defeated by incorrect wrist placement, supporting this paper's "wrist first" judgment.
vs OmniGrasp: OmniGrasp grasps 1,200+ objects via a universal dexterous motion representation without paired full-body MoCap. WristMimic performs wrist guidance within whole-body HOI retargeting; the hand-morphology ablation borrows its hand model, but the positioning differs — this work does not pursue cross-scene generality.
vs GraspXL / Chen et al.: The two study large-scale grasp motion generation and wrist-stratified object-centric dexterous manipulation respectively, both at the hand level. WristMimic embeds wrist guidance into a whole-body control policy while keeping manipulation coupled with body motion.

Rating¶

Novelty: 4/5 — Combining supervision decoupling with the wrist-bridge idea forms an actionable framework, and the wrist as the bridge between contact regimes is insightful; individual components (reward modulation, phase resets) are specific combinations of known techniques.
Experimental Thoroughness: 4/5 — Main results, wrist-constraint ablation, threshold ablation, hand-morphology generalization, and non-hand contacts are all covered; sequence counts are small (40 main sequences) and training is entirely scene-specific.
Writing Quality: 4/5 — Motivation, design, and experiments map clearly, and damaged formulas do not impede the main understanding; some details (weight values, full hyperparameters) depend on supplementary material.
Value: 4/5 — Offers a practical route to reducing finger-level mocap dependence, useful for follow-up work on hand-morphology generalization and structured joint selection.