World Models for Learning Dexterous Hand-Object Interactions from Human Videos

Conference: ECCV 2026
Paper: ECCV Paper
Area: Robotics & Embodied AI
Keywords: dexterous hand-object interaction, world models, keypoint action representation, hand consistency, model predictive control

TL;DR

DexWM conditions a latent world model on hand-keypoint differences and camera motion, using hand-consistency supervision to preserve local detail; after pretraining on human and parallel-jaw robot videos and fine-tuning on about 4 hours of target-robot simulation exploration, model-based planning achieves 10/12 successful real-robot grasps, where zero-shot means unseen test skills on the target embodiment and no real-robot fine-tuning—not the absence of all target-embodiment data.

Background & Motivation

Dexterous manipulation requires more than knowing where the wrist should move: the model must predict how opening, bending, and contacting fingers changes an object. Text, navigation, and whole-body pose conditions describe coarse motion but can omit the finger movements that determine grasp success. Policies that directly predict actions from images also depend on the behavior distribution in their training trajectories. If the data consists mainly of exploration rather than successful task demonstrations, a policy may not know how to reach a specified goal. World models offer another route: learn action-induced state changes first, then search for actions at test time.

Training such a model requires fine-grained action-vision correspondences, yet dexterous robot data is scarce. Egocentric human videos contain abundant hand-object interactions, and hand keypoints offer a geometric interface across embodiments. The difficulty is that ordinary visual-feature prediction may pay little attention to the small hand region. Even when the overall image semantics match, incorrect fingertip locations can derail planning, so both the action representation and training supervision must become more precise.

Core idea: condition the world model on finger-level geometric changes, use keypoint-heatmap supervision to make predicted states retain hand locations, and reuse the same state and keypoint representations for goal-conditioned planning rather than directly imitating training actions.

Method

Overall Architecture

Inputs are historical egocentric RGB observations and action sequences; outputs are future DINOv2 patch features. A keypoint-difference action interface maps different hand types into a shared space. A deterministic latent predictor directly predicts the next state from state history and actions, and a hand-consistency readout recovers fingertip and wrist locations from predicted features. Predicted states can be rolled out autoregressively without first generating pixels and encoding them back into features.

For robot use, a goal image supplies target latent features and keypoints. Goal-conditioned CEM planning searches joint-angle sequences, converts them to model actions through forward kinematics, evaluates terminal costs inside the predictor, and passes waypoints to low-level controllers. Losses in the diagram provide training supervision, whereas CEM performs test-time search. The keypoint head supports both training and planning-cost computation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Human annotations or robot joint angles"] --> K["Keypoint-difference action interface"]
    I["Historical RGB images"] --> E["Frozen DINOv2 encoder"]
    K --> P["Deterministic latent predictor"]
    E --> P
    P --> H["Hand-consistency readout"]
    P -->|Predicted states| C["Goal-conditioned CEM planning"]
    H -->|Predicted keypoints| C
    G["Goal-image features and keypoints"] --> C
    C -->|Candidate joint sequences: forward kinematics| K
    C -->|Selected waypoints: low-level control| O["Robot execution"]
    L["Ground-truth states and heatmaps"] -.->|Training supervision| P
    L -.->|Training supervision| H
    P -->|Autoregressive state history| P

Key Designs

1. Keypoint-difference action interface: model finger changes separately from camera motion

Each hand uses 21 MANO-style 3D keypoints, for 42 across both hands. Between consecutive states, the future keypoints are first expressed in the current camera frame using a known rigid transformation, then differenced, preventing camera motion from being mistaken for hand motion. Camera translation and Euler-angle changes are appended. The two hands contribute 126 coordinate-difference dimensions, and camera changes add 6, giving 132 dimensions in total. The paper's 44×3 notation groups the total dimensions; it does not indicate 44 hand keypoints.

This interface combines different data sources. EgoDex provides human hand and pose annotations. DROID's parallel-jaw grippers are approximated by dummy keypoints on concentric circles at the end effector, with radii changing with gripper opening. Simulation and real-robot keypoints are obtained from joint angles through forward kinematics. Allegro has only four fingers, so the authors reuse its last finger's keypoints to represent the human pinky. The main text describes the performance cost as small, but the supporting evidence is in the unavailable supplementary material, so the effect cannot be independently quantified here.

This is not a unified representation of contact forces, joint torques, or physical parameters. It is a geometric action interface available across several datasets. Keypoint trajectories describe how the hand moves but do not guarantee identical contact mechanics across embodiments; subsequent target-embodiment simulation fine-tuning remains necessary.

2. Deterministic latent predictor: predict visual features directly instead of iteratively generating video

A frozen DINOv2 encoder maps observations to patch-level latent states, preserving spatial information without modeling every pixel. The predictor adopts a Conditional Diffusion Transformer (CDiT) architecture, injecting the flattened action into each Transformer block through AdaLN. However, it performs no diffusion denoising: it directly regresses the next state. Future-state prediction tokens are initialized from the current state, and the model can also read multiple historical states.

Training randomly skips states to vary the temporal intervals between inputs, which the authors report improves generalization. Multistep prediction feeds predicted states and subsequent actions back into the predictor. Compared with iterative denoising, direct regression better suits planning that repeatedly evaluates candidate actions. The trade-off is a deterministic transition assumption, which does not naturally represent multiple possible contact outcomes for the same observation and action. Qualitative examples show action following, not a proof of general physical accuracy.

3. Hand-consistency readout: keep small keypoint regions from being overwhelmed by global feature losses

The base objective is mean squared error between predicted features and DINOv2 features of the true future image. Since hands occupy little image area, this objective may tolerate incorrect fingers while preserving the overall scene. A Transformer readout therefore recovers fingertip and wrist locations from predicted states as 12 keypoint heatmaps, supervised by mean squared error against ground-truth heatmaps. These are image-space localization heatmaps, not an additional complete 42-point 3D hand prediction.

The cache contains corrupted equations for the individual losses, so their exact normalization is not reconstructed here. The main text explicitly gives the combination:

\[ \mathcal{L}=\mathcal{L}_{\mathrm{state}}+\lambda\mathcal{L}_{\mathrm{HC}},\qquad \lambda=100. \]

The encoder remains frozen while the predictor and readout components receive supervision. This head is not a training-only auxiliary module: evaluation uses it to measure keypoint localization, and planning uses it to compare hand locations at the predicted endpoint and the goal. The local information training requires the model to retain thus serves a direct decision-making purpose rather than merely improving an auxiliary metric.

4. Goal-conditioned CEM planning: search for actions that reach the goal rather than reproduce exploration behavior

Given start and goal images, the Cross-Entropy Method (CEM) searches robot joint-angle sequences. Each candidate produces corresponding keypoint actions through forward kinematics and is rolled out inside DexWM. Scoring combines the L2 distance between terminal and goal latent states with the Euclidean distance between keypoint pixels read from their heatmaps. The latter has coefficient μ=0.001. Grasping additionally uses an end-effector orientation cost to favor neutral poses. The corrupted full optimization equation is not reconstructed here.

This process is organized within a Model Predictive Control (MPC) framework, planning waypoint trajectories for low-level controllers rather than treating world-model outputs as direct joint commands. It can learn action consequences from exploration transitions that do not complete tasks, then constrain search with a goal image. Its benefits still depend on the predictor covering candidate actions and on the cost reflecting task success. Model-based search can also exploit prediction errors; it does not automatically avoid all out-of-distribution problems. CEM sample counts, horizon, iteration count, and computational cost are described only in the supplementary material.

Loss & Training

Pretraining uses EgoDex and DROID. EgoDex contains 829 hours of 1080p human video, and the abstract describes the combined data as exceeding 900 hours. The default predictor is the 450M-parameter DexWM-XL; scaling experiments span 30M to 450M. Open-loop baseline comparisons train on EgoDex for 40 epochs in a shared DINOv2 space. This configuration should not be treated as the full schedule for every fine-tuning experiment.

Robot transfer additionally uses about 4 hours of RoboCasa exploration. Default collection starts from Lift trajectories, adds noise, and randomizes scenes and objects. The authors state that this prevents successful grasps while broadening state coverage. An alternative fully programmatic procedure samples random 3D targets and controls the robot toward them without Lift initialization. Their average simulation success rates are 53% and 49%, respectively; the main table uses the former. Although the exploration is not a set of successful skill demonstrations, it still supplies target-embodiment experience, and the default procedure retains Lift initialization as a data source.

The real-robot stage uses no real-robot fine-tuning data. The paper's zero-shot boundary is unseen test skills on the same embodiment. It does not mean going directly from human videos to a robot without adaptation, nor that pretraining never contains grasping interactions. Encoder ablations also compare alternative visual backbones using task success rather than raw L2 across different feature spaces, an important choice for avoiding incompatible feature scales.

Key Experimental Results

Main Results

Open-loop evaluation supplies an initial state and action sequence and predicts 4 seconds into the future: 20 frames at 5 Hz. Embedding L2 measures prediction error in the shared DINOv2 space. PCK@20 is the fraction of predicted keypoints within a 20-pixel radius of ground truth, reported as percentages. NWM uses only camera motion, while PEVA uses upper-body poses without finger articulation; both are variants modified in this paper.

Model	Embedding L2 @4s ↓	Mean embedding L2 ↓	PCK@20 @4s ↑	Mean PCK@20 ↑
NWM*	0.74	0.57	34	48
PEVA*	0.62	0.49	56	63
DexWM	0.67	0.51	60	68

Data are from Table 4. PEVA* has lower embedding error, but DexWM's mean PCK is 5 percentage points higher, showing that overall perceptual similarity and local hand accuracy are different objectives. The prose says “over 5 points,” whereas the tabulated average difference is 5; this note follows the table.

Model	Simulation Reach	Simulation Place	Simulation Grasp	Real-robot Grasp
Diffusion Policy	16	8	0	0
DexWM, no pretraining	18	8	14	0
DexWM	72	28	58	83

Data are from Table 5, all in success-rate %. Simulation uses 50 trials per task, assessing reach/place by positional error and additionally checking object-robot contact for grasping. The real-robot evaluation comprises 12 grasping trials, with manual judgment of whether the object is in the hand. DexWM succeeds in 10, approximately 83%. All three skills were not evaluated on the real robot.

Ablation Study

The following combines mean open-loop metrics from Tables 2 and 3. Each ablation group is compared separately rather than treated as a single configuration sweep. Full-model values in Table 3 differ slightly from Table 4; values are preserved as reported in each table rather than forced to agree.

Ablation group and configuration	Mean embedding L2 ↓	Mean PCK@20 ↑
Table 2: hand-only actions	0.60	56
Table 2: camera pose in state	0.57	59
Table 2: camera pose in action	0.50	69
Table 3: no hand-consistency loss	0.61	52
Table 3: with hand-consistency loss	0.50	69

In Table 3, endpoint PCK at 4 seconds rises from 26 to 60: 34 percentage points, not a relative increase of 34%. The prose's “34% increase” should be interpreted against the table. Table 1 additionally shows that adding EgoDex to DROID reduces RoboCasa's 4-second embedding error from 1.3 to 0.79 and raises PCK from 2 to 7. This supports the contribution of human data to cross-embodiment prediction, but the open-loop keypoint score remains low and should not be equated with robot task success.

Key Findings

• Pretraining and planning work well together. Table 5 gives a mean simulation success rate of 52.7% for DexWM, 13.3% without pretraining, and 8% for Diffusion Policy. The gain over DP is therefore approximately 44.7 percentage points. The abstract's “over 50%” should not be rewritten as an average success-rate increase exceeding 50 percentage points.
• Identical pretraining does not guarantee policy success. The authors additionally pretrain DP on the same human data and fine-tune it, reporting only 4% average simulation success versus roughly 53% for DexWM. This supports dynamics modeling plus planning under the tested exploration-data regime, not a universal claim that behavior cloning also fails with successful demonstrations.
• Action following has qualitative rather than exhaustive physical evidence. Atomic-action examples move the hand 1 cm per frame and show a cup moving forward after contact. They support local controllability, not coverage of stochastic contact and external disturbances by a deterministic model.

Highlights & Insights

• A geometric action interface connects large-scale human data with different robot morphologies without requiring all data to come from the same dexterous hand. Dummy keypoints are nevertheless a lossy approximation, and embodiment adaptation remains necessary.
• The hand readout connects local training supervision with the planning objective. Rather than merely reducing global feature error, it specifies which information downstream control must be able to recover.
• Exploration data can teach state transitions even without successful task completions. Goal-conditioned planning can then use those transitions to seek new skills, a practical distinction between world models and direct action fitting.

Limitations & Future Work

• About 4 hours of target-robot simulation exploration is required, and the default procedure uses Lift initialization. Zero-shot claims must retain this adaptation stage.
• The authors assume static scenes without external agents, while still modeling the moving hand and manipulated objects. “Static” does not mean objects never move. External dynamic interactions may require stochastic prediction through latent variables or diffusion.
• There are only 12 real-robot grasping trials, and simulation place success is only 28%, insufficient for broad claims about general dexterity or complex in-hand manipulation.
• Image goals, hand heatmaps, and visual states do not replace touch or force feedback. In contact-rich tasks, looking close to the goal need not indicate a stable grasp.
• The supplied cache lacks supplementary material, so CEM costs, operating frequency, four-finger mapping ablations, and failure-case details cannot be checked. These are important gaps for deployment and reproduction.

Related Work & Insights

• vs NWM / PEVA: shares their world-modeling perspective but conditions on finger-level actions and substitutes deterministic latent regression for iterative video denoising. The starred baselines in this paper must be distinguished from the original models.
• vs Cosmos-Predict2: text can describe a goal but does not precisely specify geometric changes in every finger. The qualitative comparison emphasizes action control rather than a comprehensive ranking of video-generation capabilities.
• vs Diffusion Policy: DP learns a goal-conditioned action distribution directly; DexWM learns transitions and then searches. The tested exploration data better supports learning “what will happen” than “what successful behavior looks like.”
• vs HOP / MAPLE: also leverages dexterous priors from human video, but DexWM primarily learns state transitions and produces robot behavior through planning rather than action regression alone.
• Resources: The project page with code and data is listed in the abstract; online availability has not been independently verified.

Rating

• Novelty: 4/5. Combines a finger-action interface, local consistency supervision, and latent planning in a targeted approach to dexterous interaction.
• Experimental Thoroughness: 4/5. Includes open-loop evaluation, data and supervision ablations, and robot transfer, while real-robot scale and computational reporting remain limited.
• Writing Quality: 4/5. The core method and zero-shot definition are clear; some percentage wording and dependence on supplementary material require care.
• Value: 4/5. Demonstrates the potential of human-video-pretrained world models for dexterous manipulation, but not a general robot controller requiring no adaptation.