CoToGrasp: Contact-Topology-Conditioned Dexterous Grasp Synthesis via Canonical Workspace Learning¶
Conference: ECCV2026
Paper: Official page · PDF
Project: https://cea-list.github.io/cotograspweb/
Area: Robotics / Dexterous Grasping
Keywords: Contact topology, canonical workspace, object-agnostic training, conditional variational autoencoder, force closure
TL;DR¶
CoToGrasp learns requested contact topologies using robot-hand geometry alone, transfers them to unseen objects through a canonical workspace, and validates and optimizes the resulting grasps, improving topology-averaged success from 21.13% to 26.72% and topology compliance from 14.28% to 17.18% on DexGraspNet without universally outperforming unconditioned planners in physical success.
Background & Motivation¶
The value of a multifingered hand extends beyond holding an object securely: selecting which fingers, phalanges, and palm surfaces make contact can leave room for subsequent manipulation. Optimizing stability or force closure alone tends to produce enveloping power grasps. For precision pinches or constrained tool-use contacts, a large collection of stable candidates may still contain nothing appropriate. Large variation in joint configurations does not necessarily imply functional contact diversity.
Human grasp taxonomies offer a direct way to control generation, but coupling each type to object meshes and fixed joint templates requires extensive object-interaction data with type annotations. When an unfamiliar object's local geometry cannot accommodate a template, optimization may abandon the requested contacts and fall back to a more stable enveloping grasp. CoToGrasp builds on object-agnostic training by learning over the structurally fixed, kinematically bounded hand surface instead of the effectively unbounded variety of object surfaces.
The semantics here are explicit combinations of contact regions, not arbitrary language instructions. Core idea: learn which hand regions should contact where in a hand-centered canonical workspace, project object geometry into that same space, and let topology specify the target, geometric validation determine feasibility, and joint optimization realize the grasp.
Method¶
Overall Architecture¶
At inference, the inputs are an object point cloud, a requested contact topology, and a candidate global hand pose; the output is a joint configuration realizing the requested contacts at that pose. Training uses only hand point clouds under different joint configurations and topology labels to predict contact regions in a canonical workspace. At inference, the object is transformed into the hand frame and its normals are reversed before it enters the same geometric encoding and generation network.
This is not an end-to-end predictor of the complete grasp pose. Global rotation and translation come from an external planner, an operator, or the paper's topology-conditioned sampling heuristic. The network generates local contacts, while the backend rejects unsuitable candidates and solves for joint angles.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
Input["Training: hand cloud<br/>Inference: object and pose"] --> Topology["Hand-Centered<br/>Contact Topology"]
Topology --> Workspace["Canonical<br/>Workspace Transfer"]
Workspace --> Generation["Topology-Conditioned<br/>Contact Generation"]
Generation --> Validation["Cascaded Validation<br/>and Joint Optimization"]
Validation --> Output["Topology-compliant joints<br/>or candidate rejection"]
Validation -->|"Force closure fails: resample latent"| Generation
Key Designs¶
1. Hand-Centered Contact Topology: define types by contact regions, not fixed poses
The method adopts 21 contact topologies from the Gonzalez taxonomy and maps the hand onto 22 anatomical contact zones. The topologies belong to precision, power, and object-specific functional groups. The object-specific group denotes tightly constrained contacts associated with particular tool uses; it does not mean that training requires meshes of those tools. Each topology specifies active regions, assigning zero to inactive points and the corresponding zone ID to active points.
The hand representation is not an indiscriminate sampling of the full mesh. It discretizes the active grasping surfaces into a handprint. Point indices and zone identities remain fixed, while coordinates and normals change with joint angles through forward kinematics. A single contact template can therefore label many hand shapes: it preserves which surfaces must touch while allowing their articulation to adapt to object geometry. The taxonomy acts as a labeling interface rather than an immutable joint template.
2. Canonical Workspace Transfer: put hand and object geometry at shared spatial locations
Training on hands and testing on objects introduces a substantial geometric domain gap. CoToGrasp fixes the palm at the origin and defines a canonical workspace using a fixed set of three-dimensional basis points in that frame. Input points contain positions and surface normals. A modified DGCNN extracts local pointwise features, which are projected onto the fixed basis through k-nearest-neighbor aggregation gated by an aligned distance. The model consequently reads the contact geometry near shared locations rather than a sequence indexed by a particular object's surface.
At inference, the candidate hand pose is inverted to transform the object into the hand frame, and object normals are reversed. Reversal turns the opposing object surface at a contact into a negative mold of the expected hand surface, making local orientations comparable between training and inference. The aligned distance considers proximity and orientation, preventing nearby but incorrectly oriented geometry from being treated as valid contact. Training labels are projected into the same workspace: a basis point receives the label of its nearest kinematically aligned hand point, or zero outside a threshold. The main text specifies a threshold of 0.8, which should not be interpreted directly as a contact distance in meters.
This is more than normalizing two point clouds to a common size. Fixed basis points provide shared indexing, local features describe geometry, and orientation gating constrains contact feasibility. Together, they allow structure learned on the hand's bounded kinematic manifold to transfer to unseen objects. The cached extraction corrupts the aligned-distance and projection equations, so this note describes the verifiable mechanism without inventing their exact weighting functions.
3. Topology-Conditioned Contact Generation: constrain the broad type before sampling within it
Each projected workspace point becomes a token. A learnable topology embedding is concatenated to every token, sinusoidal positional encoding is added, and a Transformer processes the resulting set. Repeating the condition at every location protects the requested type from dilution through deep attention layers. Self-attention lets spatially separated fingertips, phalanges, and palm regions jointly determine whether a contact pattern is structurally plausible instead of classifying points independently.
During training, an MLP embeds the ground-truth contact template. These embeddings are concatenated with Transformer features and compressed by the Set Transformer's Pooling by Multihead Attention and Set Attention Block. A CVAE encoder then produces a latent posterior. The decoder uses AdaLN to receive the geometric condition and predicts contact-zone probabilities at workspace points. Inference requires neither ground-truth contact templates nor the posterior encoder: it samples the latent from a standard Gaussian prior. The division of labor is important: topology conditioning handles broad grasp categories, while latent sampling models contact-patch variation within a category. The model does not rely on one unconditioned Gaussian to discover every grasp type.
4. Cascaded Validation and Joint Optimization: enforce both intended and forbidden contacts
A network prediction does not guarantee that the object supports the requested type at the current pose. First, label consistency compares the required and predicted sets of contact zones, allowing at most one missing or hallucinated zone. This is a region-combination check, not a demand for identical pointwise masks. For surviving candidates, the method extracts nonzero workspace regions and uses their barycenters to approximate the grasp wrench space and test force closure. A failed force-closure check triggers latent resampling, capped at 20 iterations to avoid indefinite search at unsuitable poses.
Only candidates passing both checks enter joint optimization. Its objective combines contact alignment, hand-object penetration, self-collision, joint limits, and a repulsive term for unused fingers and palm regions. The first four terms make the intended contacts physically realizable; repulsion enforces a 5 mm safety margin around surfaces that should remain unused. This discourages the optimizer from quietly adding palm or extra-finger contacts to gain stability. Where the hand must not touch thus becomes part of contact semantics rather than merely a post-hoc classification criterion.
A Worked Example¶
Consider an unseen object and a precision-grasp request that excludes palm contact. An external candidate pose first determines the approach. After transforming the object into the fixed palm frame, the model projects surface features into the workspace and generates contacts labeled by hand region. If too many required regions are missing, a prediction resembling a stable enveloping grasp is still rejected by the label check.
Once the region combination is acceptable, the system checks whether the contact distribution has force-closure potential. Failure leads to another latent sample instead of immediately optimizing all joints. If validation succeeds, optimization brings the requested finger surfaces toward their predicted contact regions while repelling the unused palm. This example illustrates the control flow, not an additional successful trial reported by the authors. Geometric incompatibility can ultimately result in rejection rather than forced generation of the requested type.
Loss & Training¶
Training begins with 10,000 uniformly sampled, kinematically valid joint configurations and their forward-kinematic handprints. Pairing each configuration with all 21 templates yields 210,000 training examples. This construction requires neither object meshes nor physical simulation, but it still needs known hand geometry, kinematics, and a contact-zone mapping. Object-agnostic does not mean free of hand-specific priors.
The network is trained end to end with multiclass cross-entropy reconstruction over workspace points and a beta-weighted KL regularizer toward the standard Gaussian prior. Detailed losses, architectural hyperparameters, projection parameters, and backend energy weights are delegated to the supplement. The available cache contains the main paper and references only, so missing values are not inferred here.
Key Experimental Results¶
Main Results¶
Simulation uses the Shadow Hand. The taxonomy-conditioned comparison uses the DexGraspNet test set after mapping Dexonomy's Feix types to the paper's contact taxonomy. SR measures physical grasp success under the cited evaluation protocols. TC computes the fraction of attempts matching the requested semantic template for each topology, then averages across topologies; it is not interchangeable with SR. Automatic classification uses Tversky similarity with a stronger penalty for extra contacts than for missing ones. A grasp scoring below 0.5 against every template is labeled unknown. The exact asymmetric weights are not expanded in the available main text.
The following values come from the paper's Table 2. Topology-averaged and object-averaged SR use different averaging units and should not be substituted for one another.
| Method | Power SR (%) | Precision SR (%) | Object-Specific SR (%) | Topology-Averaged SR (%) | Object-Averaged SR (%) | TC (%) | Semantic Entropy |
|---|---|---|---|---|---|---|---|
| Dexonomy | 27.16 | 12.36 | 19.62 | 21.13 | 23.80 | 14.28 | 0.77 |
| CoToGrasp | 29.75 | 22.71 | 25.50 | 26.72 | 27.56 | 17.18 | 0.84 |
Semantic entropy is the Shannon entropy of the normalized per-topology TC values, divided by the natural logarithm of the number of topologies. Stability entropy uses per-topology SR instead. A value near one indicates a more balanced distribution, not necessarily a high absolute success rate. Stability entropy is 0.96 for CoToGrasp and 0.91 for Dexonomy.
Ablation Study¶
These results also come from Table 2. They ablate inference-time validation, not the Transformer or canonical workspace itself.
| Configuration | Precision SR (%) | Topology-Averaged SR (%) | Object-Averaged SR (%) | TC (%) | Semantic Entropy |
|---|---|---|---|---|---|
| Full CoToGrasp | 22.71 | 26.72 | 27.56 | 17.18 | 0.84 |
| Without label consistency | 14.77 | 21.14 | 22.97 | 14.45 | 0.81 |
| Without force closure | 14.87 | 22.08 | 23.65 | 16.26 | 0.81 |
| Neither check | 14.73 | 21.06 | 23.00 | 14.72 | 0.81 |
Key Findings¶
- Against Dexonomy, the full model gains 10.35 percentage points in precision SR, 5.59 in topology-averaged SR, and 2.90 in TC. The clearest category-level improvement is precision grasping rather than already easier-to-stabilize power grasping.
- Removing label consistency reduces TC by 2.73 points; removing force closure reduces it by 0.92 points. Both substantially reduce precision SR. Removing both does not produce a TC strictly below every single-removal configuration, so component contributions cannot simply be added.
- In a separate comparison with taxonomy-unaware planners on a MultiDex subset, CoToGrasp reports 36.94% SR, 0.83 semantic entropy, and 0.11 seconds per grasp; GOAG reports 77.90%, 0.6527, and 0.20 seconds. These results support greater semantic diversity and faster reported generation, not universally better physical success. The planners also operate under different topology constraints.
- The real platform is a UR10 with a four-finger Allegro Right Hand. Demonstrations on YCB objects show forming and statically holding multiple grasp types; M6 is disabled because it requires five fingers. The main text does not report real-world success rates, repetition counts, or downstream dynamic-task scores, so example images do not establish deployment-level reliability.
Highlights & Insights¶
- Contact duality changes the learning domain from arbitrarily varied object surfaces to a bounded hand-capability space. The canonical workspace also addresses mismatched training and inference point clouds, rather than merely reducing object annotation requirements.
- Semantic control has positive and negative constraints. Required regions determine where to touch, while repulsion on inactive regions determines where not to touch; this is useful for functional grasps that must preserve manipulation space.
- Distributional balance and individual grasp stability are evaluated separately. Joint diversity or aggregate SR can hide type collapse, whereas TC and normalized entropy test whether the generator actually honors requests.
Limitations & Future Work¶
- Absolute TC is only 17.18%, well short of reliably realizing arbitrary requested topologies. The authors attribute part of this to stringent penalties on small simulation-induced contact changes, but that does not remove the remaining semantic-control gap.
- Global 6D pose comes from an external prior or heuristic, not joint learning with contact generation. More local resampling cannot repair every unsuitable approach direction; pose feasibility and contact synthesis should be integrated more closely.
- The main-paper ablations establish the importance of validation, but do not isolate the canonical workspace, attention architecture, AdaLN, or repulsion term. The entire improvement cannot be credited to any one network component.
- Shadow Hand simulation and Allegro hardware demonstrations do not establish adaptation-free transfer of a single model between hands. Contact mappings depend on morphology, and disabling M6 shows that a human taxonomy cannot always be realized in full by a robot hand.
- Hardware evidence concerns static formation and holding, not verified tool use, finger gaiting, or continuous contact switching. Follow-up evaluation should include repeated trials per topology, dynamic-task success, and end-to-end latency.
Related Work & Insights¶
- vs GOAG: CoToGrasp inherits object-agnostic training and approximate force-closure validation, adding explicit topology conditioning, canonical workspace features, and semantic checks. Training without objects alone should not be presented as an entirely new contribution of this paper.
- vs Dexonomy: The comparison contrasts type generation closely tied to joints and object geometry with generation based on active contact regions. The latter is more flexible under articulation, but still needs validation and optimization and does not eliminate geometric feasibility constraints.
- vs GenDexGrasp / RobotFingerPrint: These approaches share an interest in contact representations and geometric alignment. CoToGrasp combines explicit topology requests with a fixed workspace and emphasizes whether the generated contact pattern was actually requested, not just whether the grasp holds.
- Research direction: A high-level task planner could provide topology and approach pose, with a local model checking contact feasibility. This is a direction enabled by the modular interface, not an already demonstrated language-to-action system.
Rating¶
- Novelty: 4/5. A clear combination of contact topology, canonical workspace, and object-agnostic learning, while building on prior hand-centered generation.
- Experimental Thoroughness: 3/5. Includes a conditioned baseline, unconditioned analysis, and validation ablations, but hardware evidence is primarily qualitative and static, with limited architectural ablation.
- Writing Quality: 4/5. The training-to-inference geometric transfer and semantic motivation are clear; important parameters and evaluation details depend on supplementary material.
- Value: 4/5. A useful representation and evaluation perspective for functional dexterous grasping, with a substantial gap remaining to reliable dynamic manipulation.