Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation

Conference: ECCV 2026
arXiv: 2606.30598
Code: https://sid2697.github.io/epic-contact
Area: 3D Vision
Keywords: Hand-Object Interaction, Egocentric, 3D Pose Estimation, Contact Annotation, Cross-Attention

TL;DR

This work constructs the first in-the-wild egocentric hand-object contact dataset, EPIC-Contact, using a low-cost annotation pipeline of "hand contact area \(\to\) bijective transfer to object \(\to\) optimization fitting." It also proposes HOPformer, an end-to-end network that injects pre-trained hand priors into object features via cross-attention to jointly regress the 3D poses of both hands and the object in a single forward pass. This improves the success rate from 76.2% to 82.4% on ARCTIC, while nearly doubling the success rate and reducing the contact deviation by 75% on in-the-wild data.

Background & Motivation

Simultaneously estimating the 3D poses of both hands and manipulated objects from a single egocentric RGB image is a fundamental capability for human-centric applications such as AR/VR, robotics, and assistive technologies. However, in real-world scenarios such as kitchens and desktops, objects are highly diverse, hand gestures vary wildly, and severe hand-object mutual occlusions and ambiguous contact areas exist, causing learning-based methods to fail when leaving laboratory settings. Currently, most mainstream joint hand-object estimation methods, such as ArcticNet-SF and JointTransformer, are trained and evaluated in controlled, in-lab environments (e.g., ARCTIC, HOI4D, DexYCB). A common shortcoming of these methods is their lack of modeling the interaction between the hand and object poses—the hand is capable of providing strong constraints for the object pose, yet this is largely underutilized.

The underlying bottleneck is the scarcity of supervision signals. To train or evaluate on in-the-wild interactions, paired 3D ground truth of the hands and objects is essential. Until now, such ground truths could almost exclusively be obtained using expensive motion capture (MoCap) systems, which require meticulous calibration and dedicated equipment. As a result, the backgrounds are often clean and monotonous, failing to reflect the clutter of real egocentric scenes. The only in-the-wild dataset, MOW, only roughly validates object poses and completely ignores fine-grained hand-object contact. Consequently, "wanting to perform in-the-wild estimation but lacking in-the-wild ground truth" becomes a deadlock.

On one hand, single-hand reconstruction methods (e.g., HaMeR, WiLoR) can robustly estimate hand poses in various complex environments due to large-scale data and foundation models. On the other hand, joint hand-object estimation has stagnated. The two starting points of this paper address these two gaps: since MoCap ground truth is unobtainable, a highly efficient manual annotation pipeline for "bijective contact" is designed to obtain ground truth meshes through optimization at a low cost; since hand priors are already strong, they can be utilized as conditions to guide object pose estimation. Core Idea: A pipeline of "painting contact regions on hand meshes, mapping them bijectively to the object, and then optimized-fitting the pose meshes" is used to convert in-the-wild egocentric interactions into trainable 3D supervision (EPIC-Contact). Concurrently, an end-to-end transformer (HOPformer) is proposed to inject pre-trained hand priors into object features via cross-attention, enabling the joint regression of both hands and object poses in a single forward pass.

Method

Overall Architecture

There are two mutually cooperative products in this work. The first is the EPIC-Contact dataset: constructed by selecting 2.3K "stable grasp" egocentric videos (covering 9 object categories, totaling 62.3K frames) from EPIC-Kitchens/EPIC-Grasps. The pipeline consists of three steps: annotators "paint" contact vertices on the subdivided MANO hand mesh, bijectively transfer the contact region to the object surface, and then use the EC-fit optimization pipeline to fit paired hand-object pose meshes. This yields in-the-wild data with bijective contact correspondences and 3D mesh ground truths. Since each video contains a stable grasp, only the center frame needs annotation before being propagated across the clip. The second is the HOPformer network: taking a single RGB image as input, it uses DINOv2 (ViT-G) to extract object tokens and pre-trained WiLoR to extract hand pose tokens. It feeds object features as queries and hand features as memory into a 12-layer transformer decoder for iterative refinement. The aggregated features are then fed into multiple MLP heads to regress the left/right hand MANO parameters, the object 6D rotation + translation + joint angles, and the object category (which is used to retrieve the corresponding mesh from a model library) in a single pass. Both components can be used independently: the pose meshes generated by EPIC-Contact can train either HOPformer or other methods.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Egocentric Grasp Videos<br/>(EPIC-Grasps, 9 Categories)"] --> B
    subgraph DS["EPIC-Contact Data Construction"]
    direction TB
    B["Bijective Contact Annotation<br/>MANO contact painting + transfer via axis"] --> C["EC-fit Optimization Fitting<br/>Contact alignment + occlusion-aware mask"]
    end
    C --> D["Pose Hand-Object Mesh GT<br/>(62.3K Frames)"]
    D --> E["Hand Prior Conditioning<br/>WiLoR hand tokens injected into object features via cross-attn"]
    E --> F["Object Mesh Retrieval<br/>Predict category -> fetch CAD from library"]
    F --> G["Output: Two-hand MANO + Object 6D Pose"]

Key Designs

1. Bijective Contact Annotation: Reducing "Paired 3D Ground Truth" Acquisition from MoCap to a Few Clicks

In-the-wild interactions lack MoCap ground truth, which blocks the entire pipeline. Instead of trying to approximate the object pose directly, this paper shifts the focus to annotating the bijective contact correspondence between the hand and the object, because contact points are the only strictly necessary constraints during hand-object pose optimization. The first step only annotates the hand: the standard 778-vertex MANO mesh is uniformly subdivided to \(N_V=3106\) vertices, and annotators "paint" the vertices that contact the object while watching the video (the video offers richer multi-view context than a single image, letting annotators infer contact even in occluded regions based on grasping motion). Since the hand mesh has a fixed topology and serves as a canonical representation, this is much easier than annotating on objects of diverse shapes. The second step transfers the contact region to the object: following ContactEdit, each contact region is parameterized with a 2-DoF "contact axis," enabling the annotator to transfer the region and its correspondence to the object surface with just two clicks (origin + direction). To balance speed and accuracy, the hand is divided into three parts: thumb, fingers, and palm; at most three axes (six clicks) are needed per video to obtain the complete bijective correspondence. The authors validate consistency using Fleiss' Kappa, obtaining \(\kappa_h=0.61\) on the hand and \(\kappa_o=0.62\) on the object. This is comparable to DECO's 0.65, demonstrating that this manual pipeline is highly reliable while being extremely low-cost (averaging 3–4 minutes per video).

2. EC-fit Optimization Fitting: Solving Paired Pose Meshes from Contact Points

With the bijective contact correspondence, it must be converted into physical pose meshes to serve as training ground truths. EC-fit is a joint multi-loss optimization pipeline. For pose initialization, the hand uses WiLoR's prediction on the center frame as the starting point \(\Theta_0\); for the object, instead of a single initialization, multiple initializations combining random poses and category-aware priors are used to avoid local optima. The core contact alignment minimizes the distance between the bijective vertex pairs \(\mathbb{C}=\{(h_i,o_i)\}\):

\[\mathcal{L}_{con}=\frac{1}{|\mathbb{C}|}\sum_{i=1}^{|\mathbb{C}|}\left\|h_i-o_i\right\|_2\]

Then, refinement is performed using the center frame image—the key here is the occlusion-aware mask loss: in real-world scenarios, the object is often occluded by hands or other objects (e.g., a plate covered by food). Aligning rendered masks directly with image masks can be misled by partial occlusions. Thus, an occluded region \(M_{occ}\) is first identified, and the IoU is only calculated on unoccluded areas, i.e., \(\mathcal{L}_m^o=1-IoU([\hat{M}_o\setminus M_{occ}],[M_o\setminus M_{occ}])\). In addition, a penetration loss is applied to prevent hand-object interpenetration (using Signed Distance Fields of the hand mesh to penalize object points inside the hand), along with a regularization term to prevent the hand pose from deviating too far from the initial WiLoR prediction. The authors validate EC-fit on ARCTIC using MoCap ground truth: the fitted meshes yield a pose L2 error of only 1.9 mm, an MRRPE of 8.0 mm, and a penetration depth of 0.79 cm (comparable to the MoCap data's 0.80 cm), indicating that the quality of these optimization results is sufficient to serve as ground truth.

3. Hand-Prior-Conditioned Cross-Attention Decoding: Letting the Hand "Guide" Object Pose

Previous methods simply concatenated hand and object features (Concat+MLP), completely failing to model their interaction and collapsing under occlusion. HOPformer progressively modulates rather than simply merges the object features with the hand pose. It projects the object tokens extracted by DINOv2 into query sequences \(X_o\) and the pose-specific hand tokens extracted by WiLoR into memory sequences \(X_h\). These are then fed into a full 12-layer decoder (instead of a single cross-attention layer) for iterative refinement. Within each layer, object representations first interact via self-attention, then attend to the hand context via cross-attention, and finally pass through a feed-forward network:

\[\text{CA}(X_o^{(\ell-1)},X_h)=\delta\!\left(\frac{(X_o^{(\ell-1)}W_Q^\ell)(X_h W_K^\ell)^\top}{\sqrt{d_k}}\right)(X_h W_V^\ell)\]

The output after 12 layers is combined via a residual connection, and passed through a learnable aggregation module \(\mathcal{A}\) that compresses the 256 tokens into 39 tokens with layer normalization to produce the interactive features \(z_i\). The reason for using "pose-specific" WiLoR features instead of general DINOv2 features as hand priors is that ablation studies show using DINOv2 as hand features degrades hand MPJPE from 16.1 to 20.6 mm, which consequently drops the object success rate from 82.4% to 70.0%—the quality of the hand prior directly limits the object estimation performance. This design is the primary source of performance gains across the paper: replacing cross-attention with naive concatenation collapses the success rate from 82.4% directly to 29.4%, proving that the gains stem from "conditioning" rather than the backbone itself.

4. Decoupled Multi-Head Output and Object Mesh Retrieval: Separating Pose Regression and Object Recognition

The 39 aggregated tokens are routed to a set of independent small MLP heads, following the idea of DETR where different tokens handle different duties: the first 16 tokens regress the right hand \(\theta_r\) (including global orientation), the 17th token outputs the root, and the 18th token outputs the shape \(\beta_r\). The next 18 tokens are responsible for the left hand, and the final 3 tokens regress the articulated object pose \(\omega\). The pose heads only regress rotation + translation without hardcoding specific object geometries. When rendering meshes, a classification head predicts the category \(c\) using the rich semantic object features \(z_o\) to retrieve the corresponding CAD mesh from a predefined model library \(\mathcal{M}\), on which the regressed pose is applied. This decoupled design of "regressing pose / recognizing category" reuses the same features while allowing the inference to run fully automatically (achieving 52.9% classification accuracy on EPIC-Contact). The authors argue in the appendix that current CAD-free methods (e.g., SAM 3D) still misclassify bottles as cans or glasses as bowls under heavy in-the-wild occlusion or with transparent objects. Thus, "retrieving from a fixed CAD library" remains a more practical choice for egocentric pose estimation at this stage, which can be seamlessly replaced with predicted meshes once CAD-free reconstruction becomes reliable.

Loss & Training

Training utilizes a set of frame-level losses. The hand loss \(\mathcal{L}_h=\lambda^h_{2D}\mathcal{L}^h_{2D}+\lambda^h_{3D}\mathcal{L}^h_{3D}+\lambda^h_\theta\mathcal{L}^h_\theta+\lambda^h_\beta\mathcal{L}^h_\beta+\lambda^h_T\mathcal{L}^h_T\) covers 3D joints, 2D projection, MANO pose and shape, and weak perspective camera parameters. The object loss \(\mathcal{L}_o\) is similar, with an additional classification loss \(\mathcal{L}^o_c\) and object pose loss \(\mathcal{L}^o_\omega\) (geodesic loss for rotation, MSE for the rest). An interaction loss \(\mathcal{L}_{int}\) based on CDev is added for frames where the hand is in contact with the object. The total loss is \(\mathcal{L}=\mathcal{L}_r+\mathcal{L}_l+\mathcal{L}_o+\mathcal{L}_{int}\). AdamW is used for optimization. Following ARCTIC, a two-phase training strategy is adopted: pre-training on 8 egocentric-outer camera views for 25 epochs (linear warmup + cosine decay), followed by fine-tuning on egocentric views for 30 epochs; and training for 125 epochs on EPIC-Contact. The backbone consists of DINOv2 ViT-G (object) + WiLoR (hand), with a decoder depth of 12 and a batch size of 256/128 distributed across 4 GH200 GPUs.

Key Experimental Results

Main Results

ARCTIC is an in-lab dataset with MoCap ground truths (used for ablation), while EPIC-Contact is the proposed in-the-wild benchmark. The baselines are ArcticNet-SF and the current SOTA, JointTransformer (the latter uses the same DINOv2 backbone, providing the fairest direct comparison).

Dataset	Metric	HOPformer	JointTransformer (SOTA)	Gain
ARCTIC (Ego)	[email protected] ↑	82.4	76.2	+6.2 pts
ARCTIC (Ego)	CDev (mm) ↓	31.9	35.0	-3.1
ARCTIC (Ego)	MDev (mm) ↓	7.3	10.4	-3.1
ARCTIC (Ego)	MPJPE (mm) ↓	16.1	20.0	-3.9
EPIC-Contact	[email protected] ↑	29.8	17.6	+12.2 pts (≈2x)
EPIC-Contact	CDev (mm) ↓	20.7	30.1	-31% (~78% vs ArcticNet's 94.2)
EPIC-Contact	MDev (mm) ↓	11.4	20.0	-43%

The in-the-wild benchmark is much harder than the in-lab one: JointTransformer's [email protected] drops from 76.2% on ARCTIC to 17.6% on EPIC-Contact, because EPIC-Contact features unseen instances of known categories, which are often transparent, heavily occluded, and cluttered. HOPformer nearly doubles the success rate on in-the-wild data, and reduces CDev to 20.7 mm compared to 94.2 mm of the weakest baseline ArcticNet-SF (an ~78% reduction).

Ablation Study

Ablation studies are performed on the ARCTIC egocentric split (with 82.4 as the full-reference [email protected]).

Configuration	[email protected] ↓	CDev ↑	Description
Full HOPformer	82.4	31.9	Full model
Concat + MLP (replacing cross-attn)	29.4	74.4	Collapses, proving gains stem from conditioning, not the backbone
DINOv2 as hand features (replacing WiLoR)	70.0	44.2	Hand prior quality determines the upper bound for the object
No self-attention	55.8	61.3	Token communication is critical for object estimation
No aggregation module \(\mathcal{A}\)	74.0	40.4	Learnable pooling is effective
Decoder depth L=1	21.9	101.6	Iterative refinement is a necessity
w/o Object Losses	66.8	33.8	Object supervision contributes the most
w/o Interaction Loss	79.2	40.7	Mainly degrades physical consistency (CDev)

Key Findings

• Cross-attention conditioning is critical: replacing it with naive concatenation drops the SR from 82.4 to 29.4 and increases CDev from 31.9 to 74.4 using the same strong backbone, demonstrating that performance stems from the "hand guiding object" mechanism itself.
• Hand priors must use pose-specific features: using general DINOv2 tokens as hand priors deteriorates MPJPE from 16.1 to 20.6 mm, which negatively affects the object (SR 82.4 \(\to\) 70.0), confirming that the "prior must be task-specific."
• Performance improves with more object categories: training on 3/6/9/11 categories steadily improves metrics like MDev, demonstrating strong generalization rather than being bogged down by increased difficulty.
• ARCTIC pre-training is crucial: comparing training from scratch on EPIC-Contact versus fine-tuning from ARCTIC initialization, the latter improves [email protected] from 24.2 to 69.7 and reduces MRRPE from 99.5 to 65.8 mm.
• EC-fit ground truth is highly effective: training HOPformer using EC-fit fitted pose meshes versus MoCap ground truths yields almost identical validation metrics (SR 71.9 vs 71.8), proving that this low-cost annotation is sufficient to replace MoCap for training.

Highlights & Insights

• Replacing "hard-to-acquire pose ground truth" with "easy-to-annotate contact correspondence": The core insight is that hand-object pose optimization only requires contact point constraints. Thus, bypassing MoCap and adopting the "paint points + two clicks for axis transfer" bijective annotation followed by optimization fitting compresses the acquisition cost of in-the-wild 3D supervision to 3-4 minutes per video. This "change of annotation target" can be transferred to any interaction task where paired pose ground truths are hard to obtain.
• Contact axis parameterization saves clicks: Representing the contact region with a 2-DoF axis compresses the tedious "vertex-by-vertex mapping" operation into just two clicks per region, and at most six clicks for the whole hand, which is the key trick for annotation efficiency.
• Occlusion-aware mask loss: In-the-wild objects are frequently occluded. Aligning only the unoccluded regions (excluding \(M_{occ}\)) rather than strictly aligning the fragmented masks is a minor adjustment that is critical to the fitting quality in cluttered scenes.
• Non-isotropic VLM scale estimation: Prompts customized by category are sent to Gemini 2.5 to output multiple "scale dimensions" (e.g., asking for the pan's diameter and handle length separately). Performing non-uniform scaling to fit template meshes to instances yields an MAE of only 0.94 cm, which is a clever way to adapt fixed templates to highly variable real-world instances.

Limitations & Future Work

• The authors acknowledge that although the annotation pipeline is robust, it is still time-consuming (3–4 minutes per center frame). They envision using trained models to provide a better initialization for contact, enabling annotators to start from an optimal estimate.
• Broadcasting single-frame ground truth to the entire video clip may introduce noise (e.g., WiLoR hand misidentifications or camera jumps), but this does not affect the results since the metrics are relative/root-aligned. The authors release frame-by-frame confidence scores to allow users to filter the data.
• HOPformer currently only covers a few categories (9 in EPIC-Contact). Extending to more categories and handling articulated objects in the wild are left as future work.
• Relying on a fixed CAD model library for retrieval is a pragmatic compromise at present—it fails when encountering out-of-library novel objects. The authors argue that this can be seamlessly replaced with predicted meshes once CAD-free reconstruction matures, though they note that this still requires substantial effort in diverse real-world scenes.

Related Work & Insights

• vs JointTransformer: Both use a DINOv2 backbone as an encoder-decoder to regress hand-object poses. The difference is that JointTransformer does not explicitly model the hand's guidance on the object, whereas HOPformer injects hand priors into object features via cross-attention. Our advantages include nearly doubling the in-the-wild success rate and significantly reducing CDev; it also remains the former SOTA that is surpassed on the ARCTIC leaderboard.
• vs ArcticNet-SF: ArcticNet-SF utilizes a ResNet-50 backbone, whereas HOPformer replaces it with the stronger DINOv2+WiLoR combination and adds conditioning, leading to comprehensive improvements across all metrics.
• vs MOW: MOW is the only existing in-the-wild paired hand-object mesh dataset, but its object poses are only roughly validated and it ignores fine-grained contact. EPIC-Contact provides bijective contact correspondences and quality-controlled pose meshes, making it more suitable for training and evaluation.
• vs DECO / PICO: DECO uses "vertex painting" to annotate contact on the human body, while PICO transfers contact regions to the object. This work is inspired by them but specializes in hand-object interactions, creates bijective correspondences (which allow back-solving poses), and adapts to egocentric occluded scenes.
• vs CAD-free Methods (HOLD-Net / G-HOP / SAM 3D): These methods do not estimate object poses relative to a canonical CAD, and the shapes they generate under heavy in-the-wild occlusions or on transparent objects often deviate from reality. HOPformer instead retrieves from a known CAD database to provide accurate poses, which represents a more practical route at this stage.

Rating

• Novelty: ⭐⭐⭐⭐⭐ The two points of "changing the annotation target (contact correspondence instead of pose ground truth)" and "hand-prior-conditioned object features" both address the core pain points of in-the-wild hand-object estimation, and they mesh together seamlessly.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across dual in-lab + in-the-wild benchmarks, 7 sets of architecture/loss ablations, category scaling, cross-dataset transfer, and validations of EC-fit ground truth quality and robustness.
• Writing Quality: ⭐⭐⭐⭐ Motivation and methodology are clearly explained, and the main lines of the annotation pipeline and network are distinct. There are many symbols, and the appendix contains a large amount of information.
• Value: ⭐⭐⭐⭐⭐ The first in-the-wild egocentric hand-object contact dataset + open-source code/weights, providing a solid foundation for training and evaluation in this direction.

Related Papers

• [CVPR 2025] EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Vision
• [ECCV 2026] Audio-Visual Camera Pose Estimation with Passive Scene Sounds and In-the-Wild Video
• [CVPR 2026] Egocentric Visibility-Aware Human Pose Estimation
• [CVPR 2025] HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos
• [CVPR 2026] WildPose: A Unified Framework for Robust Pose Estimation in the Wild