RAGrasp: A Retrieval-Augmented Framework with Diversity-Aware Modeling for Dexterous Grasp Generation¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://yz-alpha.github.io/ragrasp/
Area: Robotics / Embodied AI
Keywords: Dexterous Grasp Generation, Retrieval-Augmented Generation, Normalizing Flow, Diversity Modeling, Embodied AI
TL;DR¶
Addressing the twin challenges of grasp success rates being bottlenecked by random initial latent states and grasp diversity collapsing under greedy single-label classification, RAGrasp introduces a distribution-aligned geometry-aware retrieval initialization strategy alongside an exploration-plus-fallback dual-head grasp-type selector, achieving substantially improved grasp success rates while enabling flexible diversity tuning.
Background & Motivation¶
Dexterous multi-finger grasping is a fundamental capability enabling embodied robots to perform intricate physical manipulation tasks. However, generating stable and diverse grasps from single-view partial point cloud observations remains an open challenge. Conventional analytical approaches rely on complete 3D object meshes to solve mathematical force-closure optimizations, leaving them brittle under occlusions and real-world sensing noise. Learning-based pipelines have largely superseded them, with recent state-of-the-art methods decoupling global hand pose estimation from high-DoF joint configuration regression using hybrid architectures—such as mobius normalizing flows over the rotation manifold—to prevent the severe gradient conflicts typical of joint end-to-end optimization.
Nevertheless, current generative and hybrid frameworks uniformly overlook the decisive impact of the initial latent state on the resulting hand pose. Standard models sample initial noise from uninformative isotropic Gaussian priors, ignoring the fact that aligning the initial latent state with the target object's geometric features and grasp taxonomy drastically increases the likelihood of converging to physically plausible contact configurations. In parallel, existing taxonomy-guided frameworks employ single-label classifiers to infer the most probable grasp type from 3D shapes. This winner-take-all formulation artificially collapses multimodal grasp affordances—such as a mug admitting both pinch grips and palm wraps—into a single dominant mode, crippling grasp diversity.
A naive integration of retrieval into grasp generation introduces a catastrophic domain shift: retrieving ground-truth grasps computed by analytical solvers injects poses that fall outside the empirical support of the neural generative model. Core idea: build a distribution-aligned retrieval database generated by the model itself and validated through physics simulation, invert retrieved geometry-aligned poses through an invertible normalizing flow to initialize the base latent distribution, and deploy a dual-head "multi-label exploration + single-label fallback" classifier to balance success rate and grasp diversity.
Method¶
Overall Architecture¶
The RAGrasp framework is built upon two core pillars: the Geometry-Aware hand pose Initialization strategy (GAI) and the Diversity-Aware grasp-type Selection network (DAS). Given a single-view partial point cloud, the pipeline first passes it through a 3D point cloud completion network to reconstruct a view-invariant complete shape. Next, DAS fuses the object's geometric embedding with empirical grasp success statistics of geometrically similar objects to select the target grasp type. Finally, GAI retrieves successful grasp candidates of similar shapes from a precomputed model-generated database, inverts them into the normalizing flow's latent base distribution via the inverse flow mapping to serve as informative initialization priors, and rolls the forward flow model out to produce high-fidelity rotation and translation parameters.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Partial Point Cloud<br/>O_partial"] --> P["Point Cloud Completion Network<br/>View-Invariant Shape P_pred"]
P --> F["Sparse 3D Conv Feature Extraction<br/>Global Shape Feature f_test"]
F --> D1["Distribution-Aligned Retrieval DB<br/>Model-Generated Valid Grasps D_succ"]
D1 --> D2["Grasp-Type Selector DAS<br/>Multi-Label Head + Fallback Head"]
D1 --> G1["Geometric Retrieval & Flow Inversion<br/>Inverse Mapping to Prior (z_rot, z_trans)"]
D2 -->|Selected Type E_i| G2["Mobius Normalizing Flow<br/>Global Pose (R_pred, t_pred)"]
G1 --> G2
G2 --> Reg["Lightweight MLP Regressors<br/>Joint Angles, Pre-Grasp & Squeeze Poses"]
Key Designs¶
1. Distribution-Aligned Model-Generated Retrieval Database: Eliminating the Training-Retrieval Domain Shift Retrieving ground-truth poses computed via analytical solvers introduces an acute distribution mismatch: the continuous manifold learned by a parameterized neural generative model does not perfectly overlap with the discrete analytical solution space. Inverting an analytical grasp into the learned flow often maps into low-likelihood latent regions, causing generation quality to severely degrade. To resolve this, GAI executes an offline synthesis-and-filter stage. The trained mobius normalizing flow generates candidate grasps \(\mathcal{H}^\text{pred}_i\) for all training objects, which are then evaluated in a rigid-body physics simulator (MuJoCo) against multi-directional disturbance forces and penetration checks. Only verified stable grasps are retained to form the success set \(\mathcal{H}^\text{succ}_i\): $\(\mathcal{H}^\text{succ}_i = \left\{ H^\text{pred}_j \in \mathcal{H}^\text{pred}_i \mid \text{Valid}(H^\text{pred}_j) = 1 \right\}\)$ The resulting database \(\mathcal{D}^\text{succ} = \{(\mathcal{P}_i^\text{pred}, \mathcal{H}^\text{succ}_i)\}_{i=1}^{N_D}\) is completely generated by the model itself, ensuring exact distribution consistency between the retrieval source and the inference generator.
2. Geometric Similarity Retrieval and Flow Inversion: Seeding Latent Space with Geometry-Aligned Priors During test-time inference on an unseen object, its completed point cloud is encoded into a global descriptor \(f_i^\text{test}\), and pairwise squared Euclidean distances to all training descriptors \(f_j^\text{train}\) are computed via \(M_{i,j} = \|f_i^\text{test} - f_j^\text{train}\|^2\). Training objects are ranked in ascending distance order, and successful grasps \(H_j^\text{succ}\) from the top-ranked neighbors are retrieved. Taking advantage of the exact invertibility of mobius normalizing flows on the \(\text{SO}(3)\) rotation manifold and Euclidean translation space, the retrieved rotation \(R_s^\text{succ}\) and translation \(t_s^\text{succ}\) are passed through the backward flow: $\((\mathbf{z}_s^\text{rot}, \mathbf{z}_s^\text{trans}) = (f_K^{-1} \circ \cdots \circ f_1^{-1})\left((R_s^\text{succ}, t_s^\text{succ}); \mathbf{E}_j; \mathcal{P}_j^\text{pred}\right)\)$ The resulting latent variables \((\mathbf{z}_s^\text{rot}, \mathbf{z}_s^\text{trans})\) replace uninformative Gaussian noise, acting as structured, geometry-aware seeds that accelerate sampling convergence and bias generation toward physically stable grasp configurations.
3. Dual-Head Grasp-Type Selector (DAS): Balancing Multimodal Exploration with Deterministic Fallback To resolve grasp-type collapse caused by greedy single-label classification, DAS employs a retrieval-augmented dual-head architecture. The input representation fuses the query object feature \(f_i^\text{test}\) with the historical grasp-type success rates \(s_j \in \mathbb{R}^T\) of the retrieved neighbor. The multi-label head, optimized with binary cross-entropy loss, predicts whether the success probability of each taxonomy category exceeds a configurable threshold \(\tau\). When one or more categories surpass \(\tau\), a grasp type is sampled uniformly at random from this candidate set, directly fostering grasp diversity. If no category exceeds \(\tau\) (e.g., for geometrically difficult objects), the single-label classification head, trained with standard cross-entropy loss, acts as a deterministic fallback by selecting the highest-confidence mode. This dual design automatically promotes diversity on versatile objects while guaranteeing robustness on challenging geometries.
Loss & Training¶
The core mobius normalizing flow is trained on the \(\text{SO}(3)\) manifold via negative log-likelihood (NLL) optimization, combining mobius transformations and quaternion affine maps to preserve topological fidelity. Joint angle predictors and auxiliary pose heads are trained using Huber regression loss coupled with contact distance consistency objectives. For DAS, the multi-label branch is supervised using BCEWithLogitsLoss over independent category probabilities, while the fallback branch is trained via standard cross-entropy loss.
Key Experimental Results¶
Main Results¶
RAGrasp is comprehensively evaluated on two large benchmarks: BODex (1.4M fingertip grasps across 2.4k objects with a single grasp type) and Dexonomy (9.5M multi-type grasps across 10.7k objects covering 31 grasp taxonomy types). Metrics include Grasp Success Rate (GSR, resistance to 6 external wrench perturbations without severe penetration), Object Success Rate (OSR), Contact Distance Consistency (CDC), Hand-Object Penetration Depth (PD), Self-Penetration Depth (SPD), and Grasp Diversity (D, percentage of variance explained by the first PCA component; lower indicates greater diversity).
Quantitative results are summarized below:
| Dataset | Method | GSR (%)↑ | OSR (%)↑ | PD (mm)↓ | SPD (mm)↓ | CDC (mm)↓ | D (%)↓ |
|---|---|---|---|---|---|---|---|
| BODex | GTTA | 14.8 | - | 27.3 | 0.15 | 29.6 | 39.2 |
| BODex | ISAG | 20.7 | - | 17.3 | 0.08 | 22.4 | 38.7 |
| BODex | DP3 | 62.4 | - | 8.91 | 0.10 | 13.9 | 32.6 |
| BODex | UDG | 80.1 | - | 3.47 | 0.04 | 8.85 | 33.4 |
| BODex | RAGrasp (Ours) | 86.7 | - | 3.38 | 0.04 | 9.55 | 37.7 |
| Dexonomy (Conditioned) | nflow (Baseline) | 63.9 | 91.3 | 8.6 | - | 13.9 | 25.7 |
| Dexonomy (Conditioned) | Ours (Diversity-favored) | 65.6 | 91.3 | 8.5 | - | 14.2 | 25.5 |
| Dexonomy (Conditioned) | Ours (Success-favored) | 71.5 | 91.4 | 8.3 | - | 13.4 | 26.2 |
Under the unconditioned grasp-type setting on Dexonomy, RAGrasp achieves an average grasp success rate of 39.1% across all 31 taxonomy types, outperforming the official normalizing flow baseline (34.6%) by 13.0% relative (4.5% absolute) and substantially outperforming diffusion-based baselines (24.6%). Notable gains are recorded on both contact-sensitive precision grasps (Precision Sphere: 56.8% vs. 46.9%; Writing Tripod: 46.1% vs. 40.2%) and large power grips (Prismatic 4 Finger: 57.7% vs. 48.8%).
Ablation Study¶
The ablation study on Dexonomy isolates the contribution of each component within GAI across all 31 grasp types. Base denotes the baseline normalizing flow, Comp denotes point cloud completion, Ret denotes geometric similarity retrieval, and Data denotes the self-generated physics-validated database (Base+Comp+Ret evaluates retrieval using dataset ground-truth poses).
| Config | Avg. GSR (%)↑ | Note |
|---|---|---|
| Base | 34.2 | Baseline normalizing flow without GAI |
| Base + Comp | 36.0 | Adding point cloud completion for shape invariance |
| Base + Comp + Ret (Ground Truth) | 30.4 | Retrieving ground-truth poses causes severe distribution shift (-5.6%) |
| Base + Comp + Data + Ret (Full GAI) | 39.1 | Full GAI with self-generated database, yielding a +4.9% gain over Base |
Varying the classification threshold \(\tau\) in DAS reveals clear controllability: when \(\tau=1.1\), the multi-label head is deactivated and the model reverts to greedy single-label prediction with minimal diversity. As \(\tau\) decreases, additional feasible modes are unlocked. Setting \(\tau=1.0\) achieves an optimal balance, yielding 71.5% GSR while maintaining strong diversity (D=25.5).
Key Findings¶
- Distribution alignment is mandatory for retrieval-augmented generation: The most revealing ablation outcome is that retrieving analytically derived ground-truth grasps (Base+Comp+Ret) degrades performance from 36.0% down to 30.4%. Because analytical optimizers follow fundamentally different manifold geometries than trained neural networks, inverting ground-truth poses pushes latent variables into low-probability boundary regions. Populating the database via model self-generation with physical filtering (\(\mathcal{D}^\text{succ}\)) perfectly preserves distribution alignment, recovering an 8.7% net improvement.
- Structured latent initialization accelerates training convergence: Convergence tracking across 10 representative grasp types reveals that GAI provides high-quality latent priors that allow RAGrasp to surpass the final performance of the 50k-iteration baseline within only 30k training steps.
- Physical real-world transferability: In hardware trials using a Franka Research 3 arm equipped with an anthropomorphic LEAP hand and an Orbbec Femto Bolt depth camera, RAGrasp achieved a 70% physical grasp success rate across 10 novel everyday objects in 30 grasp trials under collision-aware motion planning.
Highlights & Insights¶
- Self-referential retrieval database construction: Instead of treating external data as an immutable retrieval source, RAGrasp has the generative model generate its own retrieval database, using a physics engine as the oracle discriminator. This eliminates representation discrepancy and offers a general template for flow- and diffusion-based robotic manipulation.
- Exploiting bidirectional flow invertibility for latent seeding: By leveraging the exact invertibility of mobius normalizing flows on \(\text{SO}(3)\), the method projects successful 6-DoF poses directly into the latent base distribution, avoiding heuristic coordinate-space perturbations.
- Adaptive diversity control via dual-head classification: The collaborative single-label/multi-label formulation prevents mode collapse while avoiding execution failures on hard objects, offering an elegant mechanism to tune the exploration-exploitation trade-off via a single scalar threshold \(\tau\).
Limitations & Future Work¶
- Computational cost of offline database generation: Populating \(\mathcal{D}^\text{succ}\) requires sampling candidate poses and executing rigid-body force-closure simulations across all training assets, which becomes computationally demanding when scaling to millions of 3D objects.
- Floating-gripper assumption vs. environmental collision: The model generates grasps under floating object assumptions, meaning certain high-scoring poses may be rejected by downstream motion planners due to arm or tabletop collisions. Future work should integrate collision-aware constraints directly into the latent generative process.
- Dependence on point cloud completion fidelity: Under extreme camera viewpoints with severe self-occlusion, inaccuracies in the reconstructed point cloud can misguide nearest-neighbor geometric retrieval toward topologically mismatched objects.
Related Work & Insights¶
- vs. Dexonomy: While Dexonomy pioneered multi-type grasp generation via normalizing flows, its greedy single-label classifier collapses grasp diversity, and its flow relies on random noise. RAGrasp resolves mode collapse via DAS and incorporates GAI priors to lift average success from 34.6% to 39.1%.
- vs. Diffusion-Based Grasping (DP3 / UDG): Diffusion models support multimodal distributions but suffer from slow iterative denoising and unconditioned noise initialization. RAGrasp demonstrates that pairing invertible normalizing flows with distribution-aligned retrieval initialization outperforms diffusion baselines on BODex (86.7% vs. 62.4% / 80.1%) while offering substantially faster inference.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ [Pioneers distribution-aligned retrieval-augmented latent initialization for continuous rotation flows in dexterous grasping]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations on BODex and Dexonomy across 31 grasp types, deep ablations on domain shift, and real Franka+LEAP robot trials]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary clarity in identifying distribution shift pitfalls and motivating the dual-head taxonomy selection scheme]
- Value: ⭐⭐⭐⭐⭐ [Delivers an effective paradigm for balancing success and diversity in high-DoF embodied manipulation]