Skip to content

title: >- [Paper Note] TaxoGrasp: Taxonomy-Guided Human Grasp Synthesis with Sparse Contact Constraint description: >- [ECCV2026][Robotics & Embodied AI][Human Grasp Synthesis] TaxoGrasp introduces the Feix grasp taxonomy as a discrete intent bottleneck across cognition, instantiation, and grounding stages to achieve intent-consistent and physically executable grasp synthesis under sparse thumb-contact cues. tags: - ECCV2026 - Robotics & Embodied AI - Human Grasp Synthesis - Hand-Object Interaction - Grasp Taxonomy date: 2026-09-19 content_hash: 5289e4e9be2a23e1

TaxoGrasp: Taxonomy-Guided Human Grasp Synthesis with Sparse Contact Constraint

Conference: ECCV 2026
Paper: ECCV Official Page
Area: Robotics & Embodied AI
Keywords: Human Grasp Synthesis, Hand-Object Interaction, Human Grasp Taxonomy, Sparse Contact Constraint, Diffusion Models

TL;DR

Addressing the severe intent ambiguity and contact-semantics decoupling under sparse thumb-contact constraints, TaxoGrasp establishes the Feix grasp taxonomy as an explicit semantic bottleneck across a three-stage "Cognition-Instantiation-Grounding" framework to synthesize intent-consistent, physically plausible, and geometrically executable human grasps.

Background & Motivation

Controllable human grasp synthesis is a cornerstone technology for dexterous robotic manipulation, AR/VR immersive interaction, and 3D digital content generation. In practical interactive scenarios, specifying intentions via natural language often fails to provide deterministic and precise geometric boundary constraints, while densely defined contact maps are prohibitively tedious and impractical for end users to provide. Consequently, sparse yet informative geometric cues—such as designating a contact patch for the thumb on an object's surface—have emerged as an appealing control interface. However, such extreme sparsity inevitably introduces severe ambiguity in intent specification, as localized surface contacts exhibit an underdetermined many-to-many correspondence with full-hand anatomical configurations.

Recent advances have largely embraced a contact-centric control paradigm, predicting or prescribing contact maps first and subsequently deriving hand poses conditioned on those geometric cues. While this contact-first, pose-second formulation enhances local geometric touching, it becomes remarkably fragile under sparse control inputs. Without explicit intent modeling, generators struggle to infer global functional hand organizations purely from a localized thumb patch. Furthermore, during test-time refinement, unconstrained contact attraction objectives frequently disrupt the global hand pose, causing unnatural anatomical distortions, severe penetrations, or semantic mode collapse.

Resolving these challenges requires overcoming three fundamental barriers: the ill-posed many-to-many mapping between sparse contacts and hand postures, the lack of quantifiable semantic supervision beyond raw distance metrics, and the optimization conflict where contact-driven refinement sacrifices semantic structure for geometric proximity. The central angle of attack is to treat sparse contact not merely as a hard geometric target, but as a semantic index for inferring functional grasp intention. Core idea: introduce the classic Feix grasp taxonomy as an explicit discrete semantic bottleneck across Cognition (supervised template reasoning), Instantiation (template-conditioned diffusion synthesis), and Grounding (anchor-level contact alignment), bridging discrete biomechanical semantics with continuous physical execution.

Method

Overall Architecture

TaxoGrasp takes a 3D object mesh \(\mathcal{O}\) and a user-specified thumb-contact region \(C_{\mathrm{th}}\) (derived from a surface Capacity-Constrained Centroidal Voronoi Tessellation) as inputs, synthesizing a physically plausible, intent-consistent continuous MANO hand pose \(\mathcal{H}^*\). The pipeline unfolds in three cohesive stages: 1. Cognition Stage: A surface graph Transformer reasons over object geometry and the thumb patch to infer a discrete Feix grasp taxonomy template \(\hat{y}\) and extract a unified condition embedding \(\mathbf{c}\); 2. Instantiation Stage: Conditioned on \((\mathbf{c}, \hat{y})\), a Transformer-based conditional diffusion model samples an initial continuous MANO pose \(\mathcal{H}_0 \sim p(\mathcal{H} \mid \mathbf{c}, \hat{y})\) on the feasible grasp manifold; 3. Grounding Stage: Leveraging the inferred taxonomy label \(\hat{y}\), hand surface anchors are partitioned into taxonomy-guided contact and non-contact sets, followed by differentiable Signed Distance Function (SDF) optimization to produce the executable pose \(\mathcal{H}^*\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input: Object Mesh O + Thumb Contact Region Cth"] --> Cog["Grasp Template Reasoning<br/>Patch Transformer + Surface Graph Attention predicts Feix template"]
    Cog --> Ins["Template-Conditioned Diffusion Sampling<br/>Denoise initial MANO pose conditioned on template feature and geometry"]
    Ins --> Gro["Taxonomy-Guided Anchor Optimization<br/>Differentiable SDF alignment over contact/non-contact anchor sets"]
    Gro --> Out["Output: Executable and Intent-Consistent Grasp H*"]

Key Designs

1. Grasp Template Reasoning: Resolving Sparse Contact-to-Intent Ambiguity

Inferring full-hand grasping posture from a sparse thumb contact patch is fundamentally ill-posed. To establish a structured semantic anchor, the Cognition stage partitions the object surface into \(N\) roughly uniform patches via Capacity-Constrained Centroidal Voronoi Tessellation (CCVT), constructing an adjacency graph \(\mathcal{G}=(\mathcal{V}, \mathcal{E})\). A shared Transformer encodes point clouds within each patch into patch descriptors \(\mathbf{z}_i\), which are combined with geometric attributes and processed through \(L\) layers of edge-conditioned graph Transformer updates. By injecting a one-hot thumb indicator \(\tau_i\), the network aggregates thumb-focused and global attention-pooled features into a unified condition embedding \(\mathbf{c}\): $\(\mathbf{c} = \left[ \sum_{i=1}^N \tau_i \mathbf{h}_i^{(L)};\; \mathrm{AttPool}\big(\{\mathbf{h}_i^{(L)}\}\big) \right]\)$ A lightweight MLP projects \(\mathbf{c}\) into classification logits over the Feix taxonomy, supervised via standard cross-entropy loss. This transforms ambiguous continuous search into structured classification across 28 functional grasp categories.

2. Template-Conditioned Diffusion Sampling: Balancing Archetypal Structure with Continuous Variation

Directly using a static canonical posture for each taxonomy class leads to severe stiffness and optimization traps on diverse object shapes. The Instantiation stage formulates grasp synthesis as a conditional diffusion process in MANO parameter space, learning \(p(\mathcal{H}_0 \mid \mathbf{c}, \hat{y})\). To provide continuous structural guidance, each Feix category is represented by a precomputed canonical pose prototype (medoid) extracted from the training set, encoded into feature token \(\mathbf{z}_{\mathrm{feix}}(\hat{y})\). During diffusion denoising, a Transformer architecture fuses noisy hand parameters \(\mathbf{x}_t\), condition token \(\mathbf{c}\), and the taxonomy token through cross-attention at every layer. This mechanism preserves the characteristic finger coordination of the intended grasp family while enabling continuous physical deformation to accommodate diverse object geometries.

3. Taxonomy-Guided Anchor Optimization: Preserving Functional Semantics during Physical Grounding

Initial diffusion samples \(\mathcal{H}_0\) reside near the valid manifold but inevitably exhibit subtle interpenetrations or floating gaps. Conventional post-hoc optimizations apply undifferentiated surface attraction, which frequently collapses the intended grasp configuration. The Grounding stage maps the inferred class \(\hat{y}\) to a predefined template over hand surface anchors \(\{\mathbf{a}_k(\mathcal{H})\}_{k=1}^K\), partitioning them into functional contact groups \(\{\mathcal{G}^{(m)}_{\hat{y}}\}\) and a non-contact set \(\mathcal{N}_{\hat{y}}\). For contact groups, a smooth soft-min aggregation pulls anchors toward the object surface (or strictly to the thumb patch center \(\mathbf{c}_{\mathrm{th}}\) for the thumb group): $\(\mathcal{L}_{\mathrm{cg}}(\mathcal{H}, \hat{y}) = \frac{1}{M_{\hat{y}}} \sum_{m=1}^{M_{\hat{y}}} \operatorname{softmin}_{k \in \mathcal{G}^{(m)}_{\hat{y}}} \big( d_k^{(m)}(\mathcal{H}) \big)\)$ where \(\operatorname{softmin}(\{u_k\}) = -\frac{1}{\alpha} \log \sum_k \exp(-\alpha u_k)\) with \(\alpha=20\). Crucially, non-contact anchors are enforced with a clearance margin of \(m_{\mathrm{nc}}=2\,\mathrm{mm}\) via repulsive loss: $\(\mathcal{L}_{\mathrm{nc}}(\mathcal{H}, \hat{y}) = \frac{1}{|\mathcal{N}_{\hat{y}}|} \sum_{k \in \mathcal{N}_{\hat{y}}} \Big(\max\big(0, m_{\mathrm{nc}} - s_{\mathcal{O}}(\mathbf{a}_k(\mathcal{H}))\big)\Big)\)$ Combined with vertex penetration penalties \(\mathcal{L}_{\mathrm{vertex}}\), anatomical swing-twist joint angle limits \(\mathcal{L}_{\mathrm{joint}}\), and deviation regularization \(\mathcal{L}_{\mathrm{reg}}\), this optimization refines \(\mathcal{H}_0\) into an executable posture \(\mathcal{H}^*\) within 100 gradient steps without corrupting the intended grasp semantics.

Loss & Training

The Cognition and Instantiation models are trained independently in stages. Cognition utilizes cross-entropy loss on Feix taxonomy classification (learning rate 0.001, batch size 64, 150 epochs). The diffusion generator is trained on MANO parameters using standard mean squared error (MSE) noise prediction loss: $\(\mathcal{L}_{\mathrm{diff}} = \mathbb{E}_{\mathbf{x}_0, t, \boldsymbol{\epsilon}} \big[ \| \boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, \mathbf{c}, \mathbf{z}_{\mathrm{feix}}(y), t) \|^2 \big]\)$ Inference applies classifier-free guidance with scale 1.5. Test-time grounding optimizes pose parameters over precomputed SDF grids for 100 iterations (learning rate 0.01) with loss weights \(\lambda_{\mathrm{vertex}}=1\), \(\lambda_{\mathrm{joint}}=0.5\), \(\lambda_{\mathrm{cg}}=0.5\), \(\lambda_{\mathrm{nc}}=0.1\), and \(\lambda_{\mathrm{reg}}=2\). All stages operate efficiently on a single consumer NVIDIA RTX 4060 Ti GPU.

Key Experimental Results

Main Results

Evaluation is performed on the thumb-patch augmented HOGraspNet benchmark across interaction physical feasibility (Contact Area CA, Maximum/Mean Penetration Depth \(\text{PD}_{\max}\)/\(\text{PD}_{\text{mean}}\), Penetration Volume PV, PyBullet Simulation Displacement SIM), generation fidelity (classifier-based Fréchet distance cFID, Taxonomy Distribution Divergence TDD), and condition hit rate.

Method CA↑ (cm²) \(\text{PD}_{\max}\)↓ (cm) \(\text{PD}_{\text{mean}}\)↓ (cm) PV↓ (cm³) SIM↓ (mm) cFID↓ TDD↓
GraspTTA (Jiang et al., 2021) 113.4 0.561 0.236 8.085 12.89 11.75 0.953
ContactGen (Liu et al., 2023) 106.2 0.494 0.218 5.437 12.59 6.754 0.502
ClickDiff (Li et al., 2024) 102.5 0.392 0.188 3.420 11.13 9.149 0.655
RegionGrasp (Wang et al., 2024) 110.8 0.486 0.212 4.309 11.11 2.968 0.435
TaxoGrasp (Ours) 120.1 0.340 0.131 1.265 10.72 1.714 0.223

Ablation Study

Ablation experiments evaluate the contribution of each stage: Cognition (Cog.), Instantiation (Ins.), and Grounding (Gro.).

Configuration Cog. Ins. Gro. CA↑ \(\text{PD}_{\max}\)↓ \(\text{PD}_{\text{mean}}\)↓ PV↓ SIM↓ cFID↓ TDD↓
Random template conditioning \(\times\) \(\checkmark\) \(\checkmark\) 116.6 0.364 0.146 1.938 11.34 8.590 1.009
Fixed canonical pose template \(\checkmark\) \(\times\) \(\checkmark\) 113.2 0.238 0.095 1.461 11.44 3.135 0.444
Without Grounding stage \(\checkmark\) \(\checkmark\) \(\times\) 107.3 0.455 0.197 3.588 11.78 2.625 0.409
Full Model \(\checkmark\) \(\checkmark\) \(\checkmark\) 120.1 0.340 0.131 1.265 10.72 1.714 0.223

Key Findings

  • Simultaneous optimization of contact and collision: TaxoGrasp achieves the highest effective contact area (120.1 cm²) while cutting penetration volume to 1.265 cm³ (a 70.6% reduction over RegionGrasp), proving that establishing firm contact does not require excessive surface interpenetration.
  • Cognition anchors prevent mode drift: Providing random taxonomy labels causes cFID to degrade from 1.714 to 8.590 and TDD to quadruple to 1.009, verifying that downstream optimization cannot compensate for an ungrounded or absent semantic template.
  • Diffusion sampling outclasses static template fitting: Replacing the diffusion generator with fixed category medoid poses reduces contact area to 113.2 cm² and nearly doubles cFID (3.135), as the optimization degrades into a posture-shrinking shortcut to resolve collision.
  • Robustness to Top-1 classification ambiguity: The Cognition classifier achieves 72.8% Top-1 accuracy and 99.8% Top-3 accuracy. The majority of Top-1 discrepancies reflect valid multi-modal grasp affordances (e.g., Tripod vs. Quadpod) on symmetric objects, which still serve as physically viable semantic priors.

Highlights & Insights

  • Discrete biomechanical taxonomy as an intent bottleneck: Reformulating underdetermined sparse-contact synthesis as supervised taxonomy inference elegantly resolves the contact-semantics decoupling without requiring dense user guidance.
  • Dual-force anchor guidance: Constraining contact anchors to object surfaces while enforcing a 2 mm clearance margin on non-contact anchors guarantees both geometric adhesion and functional posture integrity.
  • Accessible computation footprint: Full training and inference run seamlessly on a single mid-range GPU (RTX 4060 Ti), presenting a practical recipe for embodied AI manipulation and interactive graphics systems.

Limitations & Future Work

  • Single-label supervision vs. multi-modal affordance: Training relies on deterministic Top-1 labels from recorded demonstrations, failing to represent the true multi-modal distribution of compatible grasp types for a given contact region.
  • Post-hoc thumb patch annotations: The evaluation benchmark derives thumb contact patches from projected recording data rather than prospective, user-driven interactive trials.
  • Future directions: Developing probabilistic mixture models or diffusion priors over the taxonomy space, alongside collecting human-in-the-loop datasets with diverse grasp alternatives.
  • vs RegionGrasp (Wang et al., 2024): RegionGrasp conditions directly on thumb regions via continuous generation without intermediate semantic abstractions, often producing mismatched hand configurations (e.g., failing to distinguish holding from clinking on a wine glass). TaxoGrasp eliminates this ambiguity with explicit taxonomy guidance.
  • vs ContactGen / ClickDiff (Liu et al., 2023; Li et al., 2024): Dense contact predictors struggle when conditioned on a single sparse click or patch, leading to collapsed contact distributions. TaxoGrasp leverages structural hand anchors and taxonomy routing to maintain global posture consistency.
  • vs Dexonomy / OmniDexVLG (Chen et al., 2025; Zhang et al., 2025): While prior works utilize taxonomy conditioning for robotic hands under explicit language prompts, TaxoGrasp demonstrates how taxonomy can serve as an inferred latent bridge under sparse geometric constraints.

Rating

  • Novelty: ⭐⭐⭐⭐☆ (Insightful integration of Feix taxonomy as an intent bottleneck under sparse interaction)
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Comprehensive validation spanning physical metrics, physics simulation, generative distribution fidelity, and ablations)
  • Writing Quality: ⭐⭐⭐⭐⭐ (Clear three-stage architectural narrative with precise geometric formulations)
  • Value: ⭐⭐⭐⭐☆ (Highly applicable to AR/VR interaction, embodied manipulation, and controllable avatar generation)