GenHOI: Generalized Hand-Object Pose Estimation with Occlusion Awareness¶
Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: Hand-Object Pose Estimation, Unseen Object Generalization, Hierarchical Semantic Prompts, Multi-Modal Masked Modeling, Hand Kinematic Prior
TL;DR¶
To overcome severe occlusion and shape variation in monocular 3D hand-object pose estimation, GenHOI integrates InstructBLIP-derived hierarchical textual semantic prompts, multi-modal masked reconstruction across RGB-point cloud-text, and hand structural/SDF priors to achieve 89.34% ADD-0.5D accuracy on unseen objects in the DexYCB S3 benchmark.
Background & Motivation¶
Monocular 3D hand-object pose recovery from a single RGB image is a fundamental perception capability for augmented reality interaction, collaborative robotics, and dexterous embodied manipulation. However, real-world hand-object interactions present drastic variations in object geometry, diverse manipulation patterns, and severe mutual occlusions between the hand and the object. These frequent occlusions corrupt crucial visual cues and cause conventional models to suffer catastrophic performance degradation when confronted with unseen object categories or unfamiliar grasp configurations.
Current mainstream approaches primarily fall into direct appearance regression or explicit keypoint-based correspondence pipelines. Direct regression models rely purely on shallow visual textures to map image features to geometric parameters, ignoring intrinsic object semantics and underlying interaction intent, which severely impairs cross-category generalization. Keypoint correspondence methods depend on the accurate 2D/3D detection of sparse landmarks, making them extremely fragile when critical keypoints are occluded or entirely out of view. Crucially, neither paradigm leverages structured semantic knowledge or rigid physical geometric anchors to compensate for missing visual evidence. While manipulated objects exhibit arbitrary topologies and large intra-class variations, the human hand adheres to a constrained kinematic skeleton whose contact and articulation naturally impose physical constraints on the 6D pose of the held object.
This paper addresses the bottleneck through a dual perspective: since raw visual cues degrade under occlusion, a vision-language foundation model can extract high-level semantic descriptions spanning object state, hand configuration, and contact intent to provide generalization priors; meanwhile, multi-modal masked modeling compels the network to infer missing cues, and the stable hand skeleton and implicit signed distance fields serve as rigid spatial reference anchors for object pose estimation. Core idea: introduce hierarchical textual semantic prompts to compensate for ambiguous visual evidence, apply multi-modal masked modeling across RGB, point clouds, and text to cultivate occlusion awareness, and leverage hand kinematic and implicit SDF priors as spatial reference anchors to guide generalized object pose reasoning.
Method¶
Overall Architecture¶
The GenHOI framework comprises three coordinated stages: hierarchical textual semantic embedding, multi-modal masked modeling, and hand prior guided pose estimation. Given an input RGB image, a vision-language model (InstructBLIP) is guided by a structured three-tier template to produce detailed textual descriptions capturing object attributes, hand articulation, and interaction patterns. During training, the visual image, dynamically sampled point cloud, and generated textual description undergo modality-specific masking and are passed through cross-modal attention transformers to reconstruct the original signals. Finally, the learned representations first regress the hand's kinematic parameters and continuous surface signed distance field (SDF), and the resulting hand joint positions and rotational descriptors are fed directly as spatial reference anchors into the object pose decoder to output 3D rotation and translation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Single RGB Image"] --> B["Hierarchical Textual Semantics<br/>Object-hand-interaction prompt via VLM"]
A --> C["Multi-Modal Masked Modeling<br/>Modality-specific masking & cross reconstruction"]
B --> C
C --> D["Hand Prior Guided Estimation<br/>Hand SDF & MANO kinematic decoding"]
D --> E["Object Pose Decoder<br/>Hand joint & rotation anchored R & T"]
Key Designs¶
1. Hierarchical Textual Semantics: Multi-Level Context for Disambiguation
Relying solely on visual cues or generic single-word category tags fails to guide fine-grained interaction reasoning under partial visibility. GenHOI introduces a structured three-tier prompt template: object-level semantics \(s_o\) describe category, geometry, and orientation (the object is a {category} with {appearance, shape} and {orientation}); hand-level semantics \(s_h\) articulate finger flexion and rotation (the hand {fingers_detail} and {rotation_description}); and interaction-level semantics \(s_i\) specify contact topology and verbs (the hand is {interaction_verb} it, {contact_detail}). Feeding this template along with the RGB image into InstructBLIP generates a unified description \(T = \mathcal{G}(I, \mathcal{T})\). This semantic embedding acts as an invariant conceptual anchor across unseen objects and provides high-level geometric hints when visual features are occluded. Random token perturbations are injected during training to prevent over-reliance on noisy descriptions.
2. Multi-Modal Masked Modeling: Cross-Modal Completion and Occlusion-Aware Feature Learning
To teach the model to infer missing visual cues during heavy hand-object occlusion, GenHOI constructs a multi-modal masked autoencoding pipeline over images, point clouds, and text. For image \(I\), a target-centric masking strategy selectively drops patches from the hand-object interaction zone (ratio \(\alpha\)) and background to form \(\tilde{I}\); for point cloud \(P\) (sampled from ground-truth meshes in training, and adaptively sampled from a visual voxel grid during inference), local patches are randomly dropped by ratio \(\beta\) to yield \(\tilde{P}\); for text \(T\), words are randomly replaced by [MASK] with ratio \(\gamma\) to produce \(\tilde{T}\). Unmasked representations \(F_I, F_P, F_T\) interact via bidirectional cross-attention to produce text-aligned visual features \(F_{I\leftarrow T}\) and geometric features \(F^h_{P\leftarrow I\leftarrow T}\). Decoders \(\psi_I, \psi_T, \psi_P\) supervise the reconstruction of the masked inputs. This forces each modality to draw on complementary context from the others, cultivating strong occlusion awareness and cross-modal consistency.
3. Hand Prior Guided Pose Estimation: Kinematic Anchoring for Generalization Unlike arbitrary object geometries, the human hand exhibits stable structural topology and limited shape variation. GenHOI leverages this physical asymmetry by using the hand as an anchor for inferring object pose. The network first computes Fourier positional encodings \(\gamma(\cdot)\) over sampled spatial points \(P_h\) and fuses them with enhanced visual features \(F_{I\leftarrow T}\) to predict an implicit hand signed distance field: $\(SDF_h = \text{MLP}_{\text{SDF}}\left(\gamma(P_h) \oplus F_{I\leftarrow T} \oplus P_h\right)\)$ Using a Sigmoid activation and learnable temperature \(\beta_h\), surface-weighted features \(\frac{1}{\beta_h \cdot \sigma_h(SDF_h / \beta_h)}\) aggregate information to regress MANO hand parameters (16 joint rotations \(\theta\) and 10 shape coefficients \(\alpha\)) and 21 3D joint locations \(J\). Rather than decoding object pose in isolation, the object rotation and translation decoders explicitly concatenate pooled hand rotation descriptors and 3D joint coordinates with object features \(F^o_{\text{agg}}\): $\(R_{\text{obj}} = \text{MLP}_R\left(F^o_{\text{agg}} \oplus \text{Pool}_{\text{rot}}(\theta)\right), \quad T_{\text{obj}} = \text{MLP}_T\left(F^o_{\text{agg}} \oplus \text{Pool}_{\text{pos}}(J)\right)\)$ This geometric coupling restricts the feasible 6D spatial configuration of the object according to the physical grasp boundary of the hand.
Loss & Training¶
The entire network is trained end-to-end using a multi-task objective: $\(L_{\text{total}} = \lambda_1 L_{\text{rec}} + \lambda_2 L_{\text{text}} + \lambda_3 L_{\text{pc}} + \lambda_4 L_{\text{mano}} + \lambda_5 L_{\text{obj}} + \lambda_6 L_{\text{sdf}} + \lambda_7 L_{\text{others}}\)$ Image pixel reconstruction \(L_{\text{rec}}\), point cloud coordinate reconstruction \(L_{\text{pc}}\), implicit field error \(L_{\text{sdf}}\), hand parameters \(L_{\text{mano}}\), and object pose error \(L_{\text{obj}}\) are optimized with \(L_1\) loss functions; text reconstruction \(L_{\text{text}}\) uses token cross-entropy loss; auxiliary contact and non-penetration terms \(L_{\text{others}}\) follow HOISDF. Training is executed on a single NVIDIA RTX 4090 GPU for 60 epochs using the Adam optimizer with an initial learning rate of \(1\times 10^{-4}\) decayed by a factor of 0.7 every 5 epochs and a batch size of 24.
Key Experimental Results¶
Main Results¶
On the challenging DexYCB S3 split targeting unseen objects and novel interactions, GenHOI is evaluated against leading Hand Pose Estimation (HPE) and Hand-Object Pose Estimation (HOPE) baselines:
| Method | Type | Object: gelatin_box (AUC↑) | Object: bleach_cleanser (AUC↑) | Object: wood_block (AUC↑) | Object Average (avg↑) | Hand MJE (mm↓) | Hand PA-MJE (mm↓) |
|---|---|---|---|---|---|---|---|
| HandOccNet | HPE | – | – | – | – | 14.58 | 6.73 |
| MobRecon | HPE | – | – | – | – | 15.40 | 7.25 |
| Liu et al. | HOPE | 26.31 | 25.07 | 68.56 | 38.89 | 15.43 | 6.61 |
| Keypoint Trans. | HOPE | 0.00 | 1.31 | 32.61 | 10.47 | 18.79 | 7.77 |
| HFL-Net | HOPE | 25.88 | 32.08 | 70.16 | 41.59 | 14.77 | 6.64 |
| UniHOPE | HOPE | 26.23 | 32.32 | 74.29 | 43.06 | 13.31 | 6.08 |
| GenHOI (Ours) | HOPE | 86.70 | 89.27 | 92.06 | 89.34 | 13.67 | 6.39 |
In terms of absolute Euclidean distance error ADD-S (mm) on DexYCB S3, GenHOI lowers the average error across unseen objects from 84.04 mm (HOISDF) and 110.79 mm (UniHOPE) to 21.76 mm (13.14 mm for gelatin_box, 26.91 mm for bleach_cleanser, and 25.23 mm for wood_block). Furthermore, on the HO3Dv2 unseen object evaluation ("019 pitcher base"), GenHOI obtains 92.7% ADD-0.5D, substantially outperforming HFL-Net (24.1%) and HOISDF (88.4%).
Ablation Study¶
Ablations on the full DexYCB S3 test split quantify the contribution of each architectural component:
| Configuration | Hand MJE (mm↓) | Hand PA-MJE (mm↓) | Mesh Error V-PE (mm↓) | PA-V-PE (mm↓) | Object ADD-0.5D (%↑) | Note |
|---|---|---|---|---|---|---|
| GenHOI (Full) | 13.42 | 5.81 | 12.96 | 5.63 | 90.44 | Full proposed architecture |
| w/o Text Generation Template | 16.78 | 6.73 | 15.12 | 6.23 | 86.51 | Removing structured prompt template |
| w/o Multi-level Semantic Knowledge | 17.04 | 6.98 | 15.75 | 6.41 | 86.32 | Removing tiered textual guidance |
| w/o Image Masking | 15.33 | 6.52 | 14.78 | 6.03 | 87.65 | Disabling target-centric image masking |
| w/o Text Masking | 14.98 | 6.21 | 14.50 | 5.88 | 88.12 | Disabling token mask reconstruction |
| w/o Point Cloud Masking | 15.21 | 6.47 | 14.91 | 6.07 | 87.90 | Disabling geometric patch masking |
| w/o Hand Rotation Prior | 14.65 | 6.12 | 14.32 | 5.78 | 85.45 | Removing hand joint rotation input to object decoder |
| w/o Hand Joint Prior | 14.88 | 6.29 | 14.55 | 5.91 | 84.01 | Removing 3D joint position input to object decoder |
Key Findings¶
- Hand spatial priors anchor unseen object pose: Removing hand joint location cues degrades object ADD-0.5D by 6.43% (from 90.44% to 84.01%), while hand error changes only slightly. This confirms that hand articulation provides an indispensable physical support envelope for estimating unmodeled object geometries.
- Image masking drives occlusion robustness: Among the three masked modalities, removing image masking causes the steepest degradation in both hand error (MJE worsens by 1.91 mm) and object pose (ADD-0.5D drops by 2.79%), proving that forced image completion trains the encoder to handle missing visual regions.
- Resilience under severe occlusion: In the extreme 75%-100% hand occlusion interval, GenHOI maintains an error of 26.28 mm J-PE and 7.55 mm PA-J-PE, matching or slightly exceeding UniHOPE (26.42 mm / 7.64 mm) despite not requiring dedicated occlusion-segmentation modules.
Highlights & Insights¶
- Bridging High-Level Foundation Models and Low-Level 3D Geometry: Rather than using VLMs for coarse scene tagging, GenHOI factors interaction into a structured triple ("object geometry + hand posture + contact verb"), creating an explicit bridge between natural language prompts and SE(3) pose parameters.
- Exploiting Asymmetric Kinematic Stability: By leveraging the topological invariance of the human hand to constrain the unbounded shape variations of unseen objects, the method effectively narrows the search space for object rotation and translation.
- Transferable Multimodal Masking Paradigm: The joint image-point cloud-text masking and cross-reconstruction strategy offers an effective recipe for other heavily occluded 3D perception problems, such as bimanual robot teleoperation and in-hand tool manipulation.
Limitations & Future Work¶
- Static Exocentric View Bias: Current evaluations are restricted to fixed-view exocentric benchmarks (DexYCB and HO3Dv2). Generalization under rapid egocentric head-mounted camera movements remains unverified.
- Two-Stage Latent Latency: Calling InstructBLIP alongside voxel-guided point cloud sampling introduces notable computational latency compared to monolithic single-stage direct regression networks, challenging real-time high-frequency (>30 FPS) robotic control.
- Future Directions: Exploring student-teacher distillation from the VLM to a unified visual encoder, or compiling hierarchical prompt tokens into compact discrete codebooks to unlock real-time streaming 6D tracking.
Related Work & Insights¶
- vs UniHOPE (CVPR 2025): UniHOPE focuses on unifying isolated hand and hand-object pipelines using an explicit hand occlusion-removal module. In contrast, GenHOI uses multi-modal completion and hand spatial anchoring, delivering an 89.34% vs 43.06% ADD-0.5D advantage on DexYCB S3.
- vs HOISDF (CVPR 2024): HOISDF models continuous hand-object interfaces via signed distance fields without high-level semantic generalization priors; GenHOI combines SDF surface modeling with VLM prompts and masked autoencoding, cutting DexYCB S3 ADD-S error from 84.04 mm to 21.76 mm.
- vs HFL-Net (CVPR 2023): HFL-Net applies dual-stream disentanglement but collapses on novel objects under occlusion (41.59% ADD-0.5D); GenHOI demonstrates that explicit hand joint position and rotation injection provides the necessary geometric backbone for cross-category generalization.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ [Cohesive design connecting hierarchical VLM prompts, tri-modal masked pretraining, and kinematic spatial anchoring]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Exhaustive evaluation across DexYCB S3/S0, HO3Dv2, occlusion brackets, and component ablations]
- Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous problem motivation, clean formulation, and well-structured comparative analysis]
- Value: ⭐⭐⭐⭐☆ [Sets a strong benchmark for generalized 6D pose estimation in complex hand-object manipulation scenarios]