AdaDexGrasp: Adaptive Dexterous Grasping via 3D Visuo-Tactile Representation Fusion¶
Conference: ECCV2026
Paper: Official paper page / PDF
Area: Robotics / Dexterous Grasping
Keywords: visuo-tactile fusion, contact semantics, 3D point clouds, closed-loop adaptation, dexterous hands
TL;DR¶
AdaDexGrasp predicts which hand part should contact each object region to generate an initial grasp, then maps measured touch back into 3D geometry to assess and correct unstable configurations, raising simulated success on unseen categories from 59% without adaptation to 83%.
Background & Motivation¶
A dexterous grasp involves more than placing the hand at a geometrically plausible location. Vision reveals an object's shape, but does not directly establish whether the fingers have actually pressed against its surface, whether their contacts provide adequate support, or whether the object will slip during lifting. Generative approaches such as DexGraspAnything and DexDiffuser can propose useful configurations from geometry, yet a static proposal cannot explain a failed execution caused by local misalignment. Tactile-only exploration has the opposite weakness: it can disregard useful geometry already available from vision and require additional attempts.
Existing visuo-tactile systems often encode the modalities separately and concatenate their global features. Having touch somewhere in the input is not enough: the model must still infer which finger produced a signal and where that interaction occurred on the object. This correspondence matters for a multi-finger hand because an equal number of contacts can provide very different support depending on their locations and hand-part assignments. AdaDexGrasp therefore performs fusion at the geometric representation level and distinguishes planning before contact from feedback after contact.
Core idea: use finger and palm identities as shared semantics between visual geometry and touch, predicting these correspondences before execution to organize a grasp and measuring them after contact to assess stability and guide closed-loop correction.
Method¶
Overall Architecture¶
The input is an object point cloud obtained from RGB-D observations; the output specifies palm position, orientation, and finger joint configuration. In simulation, the hand state contains a three-dimensional position, a four-dimensional orientation representation, and 22 joint angles. The four orientation coordinates do not imply four rotational degrees of freedom: including root translation and rotation, the system has 28 DoF. Touch is obtained from six sensing locations: five fingertips and the palm.
The pipeline combines semantic contact-guided generation, tactile mapping and stability prediction, and paired correction and closed-loop updates. Initial planning has no measured touch, so the first module predicts hand-part assignments from object geometry. Only after executing a contact action does the system construct an observation containing measured tactile information. If the classifier predicts failure, the correction model updates the configuration and the system observes contact again. Refinement stops when predicted success exceeds a threshold or the iteration budget is exhausted, followed by the final lift. Exhausting the budget is a stopping condition, not a guarantee of physical success.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["RGB-D object point cloud"] --> B["Semantic contact-guided generation<br/>Contact assignments and initial pose"]
B --> C["Execute contact<br/>Read fingertip and palm touch"]
C --> D["Tactile mapping and stability prediction<br/>Hand geometry and measured contact"]
D -->|Failure predicted and budget remains| E["Paired correction and closed-loop updates<br/>Generate a pose correction"]
E --> C
D -->|Success predicted or budget exhausted| F["Final lift and success evaluation"]
Key Designs¶
1. Semantic contact-guided generation: assign hand parts to object regions before decoding the pose
A conventional contact map primarily identifies object regions suitable for contact, without necessarily specifying which part of the hand should touch them. AdaDexGrasp's Cmap model processes the object point cloud with PointNet++ and predicts per-point contact categories carrying finger or palm identities. The five fingers and palm provide six hand-part identities. This intermediate representation turns a generic graspable region into a proposed division of contact responsibilities across the hand. It is a planning prior, not a sensor reading obtained before contact exists, and it does not equate proximity with measured contact.
The Gpose model then conditions on both the object geometry and the predicted contact map to decode the initial palm pose and joint state. The readable conditional relationship in the paper's Equation (3) is:
Here, \(\mathcal{M}\) is the predicted contact map and \(s^{(0)}\) is the initial grasp state. This design does not require an exploratory touch before planning. Instead, contact observations collected during training teach the model which hand-object assignments are associated with feasible grasps. Compared with directly regressing an entire hand configuration from shape, the intermediate map makes those assignments explicit. Its predictions can still be wrong, however, which is why initial generation does not end the pipeline.
2. Tactile mapping and stability prediction: attach actual contact to 3D locations instead of only concatenating features
After execution of the initial configuration, contact forces determine which sensing locations are active, and nearby object points receive the corresponding hand-part labels. The activation threshold used during data collection is 0.01 N. Each tactile-mapped point carries its 3D coordinates, RGB values, a contact-state identifier, and a six-dimensional tactile intensity vector normalized to \([0,1]\). Identity indicates which hand part is involved, intensity indicates contact strength, and position identifies where the interaction occurs. The readable text does not fully specify the numeric label convention or neighborhood radius, so neither is invented here.
This observation differs fundamentally from the predicted contact map: the latter is a pre-execution categorical prior inferred from shape, whereas the tactile-mapped point cloud contains post-contact measurements. In Figure 3, the classifier uses two PointNet++ encoders for the hand point cloud and the visuo-tactile point cloud, followed by multi-head attention and an MLP for success prediction. Including hand geometry allows the model to interpret contact in relation to the actual finger arrangement. Object-side tactile labels alone cannot fully express how the current hand configuration supports the object.
The classifier predicts whether the grasp will remain stable after lifting, rather than merely detecting contact. It is trained on binary success labels and supplies the decision signal for adaptation. The paper also invokes its differentiability when describing stability guidance in the correction objective. This should not be confused with differentiable physics, nor should a predicted success probability be treated as a proof of mechanical stability.
3. Paired correction and closed-loop updates: learn a local recovery target from the failed interaction
Resampling after failure may produce another proposal with the same underlying defect. AdaDexGrasp instead makes the failure observation part of the correction input. For each object, successful and failed states are pooled from PPO trajectories immediately before lifting, transformed into an object-centric coordinate frame, and paired by assigning each failed state its nearest successful counterpart. Aligning the interaction stage limits phase mismatch, while the object-centric frame removes dependence on absolute placement. The distance considers palm translation, orientation geodesic distance, and joint configuration; Equation (8) is corrupted in the cached extraction, so its exact weights and combination are not reproduced.
The pairing aims to provide a nearby successful target rather than asking the model to jump from an arbitrary failure to an unrelated grasp. The correction model conditions on the failed state's tactile-mapped point cloud and hand configuration to predict changes to palm pose and joints. Figure 3 depicts a conditional diffusion denoiser using PointNet++ features, while the prose describes its output as a pose adjustment. Together they support the interpretation of observation-conditioned corrective generation. The cached text does not provide a reproducible noise schedule, denoising step count, or full diffusion-training specification; a standard diffusion loss must not be inserted as though it were the paper's own equation.
The important feature of the loop is renewed execution and sensing after each adjustment. The classifier evaluates updated contact observations, not a succession of imagined poses with unchanged touch. The textual training objective combines approaching the paired successful pose with increasing predicted grasp stability. Equation (11) and the loop update equations are also corrupted, so their signs, weights, and rotation-update rules are not reconstructed here. The readable text names a success threshold \(\tau_{succ}\) and a maximum iteration count \(K\), but does not give their numerical settings.
A Worked Example¶
Consider an unseen object with a loose initial grasp. This is an illustrative walkthrough of the reported pipeline, not an additional measured trial. From the object's shape, the model first predicts contact regions assigned to different fingers and the palm, then generates a hand configuration. Once contact is attempted, the six tactile channels may reveal that only some fingers actually press against the surface, producing a sparse measured contact pattern on the object point cloud.
The classifier evaluates that pattern together with hand geometry. If it predicts failure, the correction model proposes a new configuration and the system measures contact again, rather than simply closing every finger because contact was missing. A success prediction above threshold leads to the final lift; persistent uncertainty or failure ends refinement when the budget is exhausted. Figure 5 qualitatively illustrates a transition from sparse, loose contact to denser support, but does not report a per-iteration probability or force trace for this example.
Loss & Training¶
Training observations come from PPO rollouts following the first-stage UniDexGrasp setup. Both successful and failed attempts are retained, rather than collecting successful demonstrations alone. Samples include hand state, the binary outcome, and the tactile-mapped point cloud; nearest-success pairing supplies additional supervision for correction. Successful examples teach feasible proposals, while failed observations help the classifier and correction model respond to contact deficiencies.
The readable prose specifies cross-entropy supervision for contact maps, mean-squared error for initial grasp poses, binary cross-entropy for success classification, and a correction objective that pursues both the successful target and predicted stability. Equations (2), (4) through (11), and (14) have missing or reordered characters in the extraction. This note does not reconstruct them from familiar textbook forms. In particular, a damaged objective does not establish whether the classifier is frozen, how gradients pass through contact updates, or whether all modules are jointly trained end to end.
Key Experimental Results¶
Main Results¶
Simulation uses IsaacGym and ShadowHand, with reaching, grasping, and lifting stages. The text reports 50 objects from 6 categories, with 20 training objects and 30 unseen-category test objects. It also reports a separate same-category unseen-object column without clearly accounting for that subset within these counts. The table below preserves the paper's three splits without inventing an allocation.
A simulated grasp succeeds if the object remains stable without slipping for at least 1 second after lift-off. All entries below are grasp success rates from Table 1; the selection emphasizes generation, resampling, feature concatenation, and corrective feedback.
| Method | Seen objects | Unseen objects | Unseen categories |
|---|---|---|---|
| DexGraspAnything | 77% | 72% | 67% |
| DexDiffuser | 71% | 69% | 55% |
| ContactDexNet | 72% | 68% | 63% |
| UniDexGrasp-CL | 59% | 52% | 46% |
| DexDiffuser-VT | 75% | 71% | 57% |
| AdaDexGrasp | 91% | 82% | 83% |
Against DexGraspAnything, the strongest baseline in all three columns of the original table, the gains are 14, 10, and 16 percentage points. UniDexGrasp-CL adds failure detection and resampling, while DexDiffuser-VT adds tactile conditioning, but neither reaches the proposed method's performance. These comparisons support the importance of relating contact to local geometry, although differences between complete systems cannot be attributed exclusively to one component. The text separately reports 87% on the DexGraspAnything dataset with 15k+ objects, but does not provide settings comparable in detail to the main table; this is not another fully specified controlled benchmark.
Ablation Study¶
The following results come from Table 2. The final column is calculated by subtracting each unseen-category result from the full model's result. These variants do not represent independent, additive contributions.
| Configuration | Seen objects | Unseen objects | Unseen categories | Unseen-category drop |
|---|---|---|---|---|
| Full model | 91% | 82% | 83% | 0 pp |
| Shared contact ID during adaptation | 84% | 72% | 73% | 10 pp |
| Shared contact ID during generation | 79% | 75% | 69% | 14 pp |
| No pose adaptation | 81% | 78% | 59% | 24 pp |
| Object-only point cloud with tactile mapping | 72% | 67% | 61% | 22 pp |
| Full hand-object point cloud without touch | 78% | 74% | 67% | 16 pp |
| Tactile-labeled images instead of point clouds | 74% | 71% | 64% | 19 pp |
The shared-ID adaptation variant changes the inputs to both the classifier and the adaptation model, not only the latter. Removing adaptation produces the largest unseen-category drop, revealing a substantial limitation of initial planning for novel categories. On same-category unseen objects, the same ablation changes success only from 82% to 78%, so the benefit depends on the type of distribution shift. Removing hand geometry costs 22 percentage points on unseen categories, supporting the need to interpret touch jointly with hand-object relationships.
Real-World Validation and Representation Analysis¶
Physical experiments use a Psibot SynHand-6 equipped with high-resolution tactile sensors. The setup figure identifies an 11-DoF hand and a 7-DoF arm, with more than 20 objects of different shapes and materials. Success requires holding the lifted object stably for at least 3 seconds. This differs from the 1-second simulation criterion, so the two tables cannot be directly interpreted as a same-protocol sim-to-real performance gap.
| Method | Seen objects | Unseen objects | Unseen categories |
|---|---|---|---|
| DexGraspAnything | 81% | 71% | 73% |
| DexDiffuser | 77% | 65% | 52% |
| DexGraspVLA | 64% | 57% | 55% |
| AdaDexGrasp | 90% | 87% | 81% |
These values are from Table 3. Gains over DexGraspAnything are 9, 16, and 8 percentage points. The main text does not give trial counts, confidence intervals, or a complete account of training across the two hand embodiments.
The representation analysis compares the current compact tactile encoding with denser modalities that include force directions and shear forces. Reported accuracy is 88% versus 81% in a small-data test and 85% versus 72% in a real-world analysis. The precise task and aggregation behind this accuracy are not sufficiently defined, so these values should not be inserted into the main tables as grasp success rates. For Figure 7, the cache supports only qualitative trends: successful grasps tend to involve more contacting fingers and higher Avg. TIF, while very small or very large objects are harder to grasp. Missing metric definitions and curve values are not reconstructed.
Highlights & Insights¶
- Correspondence is central to fusion. Attaching contact identity to a 3D location exposes which finger interacts where, instead of requiring the model to rediscover that relation from two global embeddings.
- Predicted and measured contact have different jobs. Prediction organizes the proposal; measurement validates and corrects execution. The pattern may be useful for insertion or tool grasping, but those tasks require their own contact semantics and outcome criteria.
- Failures can supervise actions, not just classification. Pairing a failure with a nearby success supplies a recovery target. The transferable idea is the data construction, not an assumption that every failed configuration is locally recoverable.
Limitations & Future Work¶
- Representation trade-off discussed by the authors: force directions and shear forces increase data requirements, hardware demands, and sim-to-real alignment difficulty. The compact representation works well in these tests, but this does not establish that richer touch is generally unhelpful; slipping and deformable objects may still require it.
- Incomplete reproducibility details: the readable text lacks contact-neighborhood selection, diffusion configuration, stopping thresholds, iteration budgets, and latency statistics. Corrupted cached equations further restrict what can be verified about objectives and state updates. Claims of real-time use do not establish a specific control frequency.
- Protocol gaps: object counts do not clearly reconcile all three evaluation splits, and simulation and physical hands have different DoF. The adaptation or retraining procedure is insufficiently detailed to claim zero-shot cross-hand transfer. Without repeated trials and uncertainty estimates, the stability of small performance differences is unknown.
- Reliance on classification and reachable corrections: false success predictions can terminate the loop early, while a nearby successful pose may not be reachable along a feasible trajectory. Useful follow-up tests would compare calibration, collision constraints, and recovery rates under a fixed contact-attempt budget, including dynamic disturbances rather than only short post-lift holds.
Related Work & Insights¶
- Compared with ContactDexNet: both use contact semantics, but AdaDexGrasp additionally uses measured touch after execution to guide correction. The distinction is between predicting how contact should occur and checking whether the current contact is actually stable.
- Compared with DexDiffuser and DexDiffuser-VT: the tactile variant evaluated in this paper concatenates touch with visual conditioning. AdaDexGrasp preserves local 3D contact identities and conditions updates on the failed state. This comparison concerns the paper's particular baseline implementations, not a comprehensive judgment of the original methods.
- Compared with UniDexGrasp and its closed-loop variant: AdaDexGrasp reuses the PPO stage for data collection, but deployment is not simply execution of that PPO policy. Resampling does not exploit a failed contact pattern as a corrective target, whereas paired adaptation explicitly conditions on the current failure.
Rating¶
- Novelty: 4/5. Combining hand-part identity, local geometry, and measured touch directly addresses multi-finger coordination; the contribution lies mainly in representation and closed-loop organization.
- Experimental Thoroughness: 3/5. Baseline comparisons, component ablations, and physical tests are present, but split definitions, budgets, repeatability, and hardware adaptation need more detail.
- Writing Quality: 3/5. The distinction between predicted and measured contact is clear, while diffusion implementation and analysis metrics need clarification. Extraction damage is not evidence of faulty typesetting in the original PDF.
- Value: 4/5. The 24-percentage-point unseen-category gain from adaptation is informative; safe deployment still requires validation of constraints and real-time behavior.