content_hash: b7b1b18b777d06d0
Fixed Reality, Diffused Possibility: Disentangling Stochastic and Deterministic Latent for Cluttered Grasping¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Robotics
Keywords: Cluttered Grasping, Hierarchical Diffusion, Latent Disentanglement, Harmonic Grasp Field, Dexterous Grasping
TL;DR¶
Addressing the entanglement of deterministic geometric structure and stochastic strategic uncertainty in heavily cluttered scenes, this paper proposes a split-latent hierarchical diffusion framework (FRDP) that decomposes scene representations into a fixed geometric latent and a diffused contextual latent, regularized by a differential-geometric Harmonic Grasp Field to produce smooth, obstacle-aware, and physically consistent 6-DoF grasps.
Background & Motivation¶
Dexterous robotic manipulation in unstructured and densely cluttered 3D environments represents a cornerstone challenge for embodied intelligence. When interacting with stacked or jumbled objects in tabletop or bin environments, robots face severe visual occlusions, complex physical interlocking, and ambiguous multi-object contact manifolds. In such environments, a successful grasp cannot be achieved purely through localized pick-and-place heuristics; it demands a fine-grained understanding of contact surface geometry coupled with high-level tactical reasoning regarding object removal order and collision avoidance. However, existing grasp synthesis paradigms struggle under these demands: conventional deterministic regression models collapse in the presence of multimodal grasp distributions—often predicting physically meaningless mean poses—while single-latent diffusion models focus predominantly on local geometric cues and fail to maintain global scene-level structural coherence.
The fundamental tension underlying these failures is that current frameworks cannot disentangle structural certainty from contextual uncertainty. Within any partially observed cluttered scene, certain properties are geometrically deterministic (such as surface normals, sharp edges, and the rigid geometry of visible handles), whereas other factors are intrinsically stochastic and dependent on manipulation strategy (such as whether to singulate an occluding block or grasp the target directly, and from which approach angle). Existing neural encoders either collapse both dimensions into a single entangled latent space—sacrificing robustness against spatial perturbations—or treat the entire latent space as stochastic Gaussian noise during diffusion, destroying critical geometric invariants.
This work addresses this dilemma by establishing a principled divide-and-conquer hierarchical generation architecture. Core idea: Disentangle deterministic geometric priors (Fixed Latent) from stochastic contextual uncertainty (Diffused Latent) via a split-latent hierarchical diffusion model, and regularize the local grasp manifold with a Harmonic Grasp Field to bridge deterministic physical structure with probabilistic strategic reasoning.
Method¶
Overall Architecture¶
The proposed system takes as input a partial 3D point cloud \(X \in \mathbb{R}^{N \times 3}\) and paired RGB scene context, outputting dense 6-DoF grasp poses \([R, T]\) or grasp affordance fields. The overall pipeline integrates four key stages: multi-view masked encoder pretraining, split-latent hierarchical diffusion (Global Conditional Diffusion GCD and Local Conditional Diffusion LCD), Harmonic Grasp Field (HGF) regularization, and variational grasp pose decoding. First, a Point Transformer encoder is pretrained via 3D masked autoencoding to extract a rigid, viewpoint-invariant geometric latent \(h_{\text{fixed}}\). Next, GCD synthesizes a strategic latent \(h_{\text{diffused}}\) conditioned on \(h_{\text{fixed}}\) and RGB features, forming the composite global latent \(h_G\). LCD then generates dense point-level affordance latents \(h_L\) conditioned on \(h_G\) via cross-attention. This local manifold is further smoothed and constrained using harmonic potential gradients before being decoded into collision-free 6-DoF grasp configurations by a variational grasp decoder.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
InPoint["Point Cloud & RGB Observations"] --> Enc["Multi-View Pretrained Encoder<br/>Extract deterministic geometric prior"]
Enc --> GCD["Global Conditional Diffusion GCD<br/>Sample stochastic strategic latent"]
GCD --> LCD["Local Conditional Diffusion LCD<br/>Point-level cross-attention denoising"]
LCD --> HGF["Harmonic Grasp Field HGF<br/>Laplacian potential & collision-aware smoothing"]
HGF --> Dec["Variational Grasp Decoder<br/>Generate 6-DoF collision-free grasps"]
Key Designs¶
1. Multi-View Masked Pretraining: Constructing Invariant Geometric Priors Raw point cloud observations in cluttered bins are heavily fragmented due to severe occlusion. If local geometric representations are extracted directly from noisy partial scans, they suffer from acute viewpoint drift. To endow the global encoder with robust, view-consistent 3D geometric priors, this paper introduces a pretraining scheme based on 3D masked autoencoding (3D-MAE) paired with cross-view contrastive alignment. Partial point cloud patches sampled across virtual camera viewpoints are randomly masked by 50%–70%, and a lightweight point decoder is trained to reconstruct the unmasked surface geometry using Chamfer Distance (CD). Simultaneously, a multi-view contrastive alignment objective maximizes the latent representation consistency across distinct viewpoints: $$ \mathcal{L}{\text{align}} = -\frac{1}{K}\sum $$ Following pretraining, the four-layer Point Transformer backbone } \log \frac{\exp(\text{sim}(z_i, z_j)/\tau)}{\sum_l \exp(\text{sim}(z_i, z_l)/\tau)\(\mathcal{E}_\phi\) is frozen, acting as a dedicated deterministic feature extractor that produces the rigid geometric latent \(h_{\text{fixed}}\).
2. Global & Local Conditional Diffusion: Decoupling Strategic Variance from Local Grasp Fields To resolve the entanglement of invariant geometry and multimodal decision-making, the framework decomposes diffusion into a two-level hierarchical architecture. At the macro level, the Global Conditional Diffusion (GCD) uses a 12-layer residual MLP to model scene-level strategic variability. It is conditioned jointly on \(h_{\text{fixed}}\) and image features \(c = f_I(\text{Img})\), optimizing reverse denoising over an initial distribution \(q(h_{\text{diffused}}^0 \mid h_{\text{fixed}}, c) = \mathcal{N}(\mu, \sigma^2)\): $$ \mathcal{L}{\text{GCD}} = \mathbb{E}, c, t)|_2^2 \right] $$ The resulting denoised feature is concatenated to produce the combined global prior }}^0, \epsilon^t} \left[ |\epsilon^t - \epsilon_\theta(h^t, h_{\text{fixed}\(h_G = [h_{\text{fixed}}; h_{\text{diffused}}^0]\). At the micro level, the Local Conditional Diffusion (LCD) utilizes a 4-scale point-voxel convolution (PVConv) U-Net. Conditioned on \(h_G\) via cross-attention, LCD denoises point-wise local representations to output a fine-grained 32-dimensional local grasp latent field \(h_L \in \mathbb{R}^{N \times 32}\), grounding global strategic decisions firmly into localized 3D surface coordinates.
3. Harmonic Grasp Field Regularization: Continuous Collision-Aware Manifold Constraints Even with conditional local diffusion, raw grasp candidates in dense clutter can exhibit spatial discontinuities and surface boundary artifacts. To enforce physical smoothness, the method introduces the Harmonic Grasp Field (HGF), grounded in differential geometry. Within the object and manipulation volume \(\Omega\), an electric-like potential field \(\Phi(\mathbf{x})\) is defined to satisfy Laplace's equation: $$ \nabla^2 \Phi(\mathbf{x}) = 0, \quad \mathbf{x} \in \Omega $$ subject to Dirichlet boundary conditions: \(\Phi(\mathbf{x}) = 0\) on obstacle/background boundaries \(\partial\Omega_o\), and \(\Phi(\mathbf{x}) = 1\) on valid contact/grasp boundaries \(\partial\Omega_g\). Because harmonic functions obey the maximum principle and exhibit zero interior local extrema, their gradient field \(\nabla \Phi(\mathbf{x})\) provides a smooth, collision-free vector flow directing the robot hand toward optimal grasp regions. The local diffusion latents are projected onto this harmonic gradient flow: $$ \mathbf{h}'{L,i} = \mathbf{h}_i) $$ coupled with a harmonic regularization loss } + \lambda_h \nabla \Phi(\mathbf{x\(\mathcal{L}_{\text{harm}} = \mathbb{E} [ \|\nabla^2 \Phi(\mathbf{x})\|_2^2 + \gamma \|\mathbf{h}'_L - \mathbf{h}_L\|_2^2 ]\), penalizing non-harmonic curvatures and locking latent representations to smooth geometric manifolds.
4. Variational Grasp Decoding: ELBO-Aligned 6-DoF Pose Synthesis To map the continuous, harmonically regularized local latent field into executable SE(3) end-effector transformations, the pipeline incorporates a variational grasp autoencoder (\(\mathcal{E}_{\text{gr}}, \mathcal{D}_{\text{gr}}\)). During training, ground-truth grasp poses \(y = [R, T]\) are encoded into grasp distribution \(q(h_{\text{gr}} \mid X, y)\), supervised under an Evidence Lower Bound (ELBO): $$ \mathcal{L}{\text{ELBO}} \approx \mathbb{E}_q [ |y - \hat{y}|_2^2 ] + D(\mu_L, \sigma_L^2)\big) $$ where }}\big(\mathcal{N}(\mu_{\text{gr}}, \sigma_{\text{gr}}^2) \parallel \mathcal{N\((\mu_L, \sigma_L)\) represent the parameterized distributions predicted from local latent \(h_L\). During deployment, the harmonically regularized features \(h_L'\) and global context \(h_G\) are passed directly into decoder \(\mathcal{D}_{\text{gr}}\) to predict 6-DoF grasp poses \(\hat{y}_i = [\hat{R}_i, \hat{T}_i]\) across all viable surface points.
Loss & Training¶
The framework is trained using an explicit two-stage optimization strategy to prevent feature collapse: 1. Stage 1 (Encoder Pretraining): Point Transformer \(\mathcal{E}_\phi\) is pretrained for 8,000 epochs on synthetic and real 3D point scans using masked reconstruction \(\mathcal{L}_{\text{MAE}}\) and contrastive cross-view alignment \(\mathcal{L}_{\text{align}}\), after which \(\mathcal{E}_\phi\) is frozen. 2. Stage 2 (Hierarchical Diffusion & Grasp Learning): Keeping \(\mathcal{E}_\phi\) frozen, GCD, LCD, HGF regularization, and grasp VAE modules are jointly trained for 24,000 epochs under the combined loss: $$ \mathcal{L}{\text{total}} = \mathcal{L}}} + \lambda_1 \mathcal{L{\text{LCD}} + \lambda_2 \mathcal{L} $$ Optimization uses Adam with an initial learning rate of }} + \lambda_3 \mathcal{L}_{\text{ELBO}\(1 \times 10^{-4}\) and cosine annealing schedule. Loss weights are set to \(\lambda_h = 0.5, \gamma = 0.05, \lambda_1 = 0.8, \lambda_2 = 1.0, \lambda_3 = 0.5\). Experiments were conducted on 8 NVIDIA V100 GPUs (32GB VRAM each).
Key Experimental Results¶
Main Results¶
The framework was evaluated across dexterous multi-finger grasping (DexGraspNet-2.0), dense parallel-jaw grasping (GraspClutter6D), un-cluttered real-world grasping (GraspNet-1Billion), and closed-loop policy rollouts (\(A^2\) Simulation Dataset).
Table 1: Dexterous Grasp Success Rate (%) on DexGraspNet-2.0 (Dense: 8–11 objects, Random: 1–10 objects, Loose: 1–2 objects)
| Method | GraspNet-1B Dense | GraspNet-1B Random | GraspNet-1B Loose | ShapeNet Dense | ShapeNet Random | ShapeNet Loose |
|---|---|---|---|---|---|---|
| HGC-Net (ICRA 2022) | 46.0 | 37.8 | 26.7 | 46.4 | 44.8 | 30.4 |
| DDGC (RA-L 2021) | 47.9 | 38.2 | 30.4 | 50.9 | 48.7 | 46.4 |
| DexGraspNet-2 (CoRL 2024) | 90.6 | 83.7 | 73.2 | 81.0 | 85.4 | 74.2 |
| DexMGNet (RA-L 2025) | 89.5 | 84.5 | 75.2 | 81.4 | 86.7 | 73.9 |
| ClutterDexGrasp (arXiv 2025) | 91.6 | 85.1 | 74.9 | 80.7 | 86.9 | 73.5 |
| DexGraspVLA (arXiv 2025) | 91.7 | 85.0 | 75.5 | 81.8 | 85.4 | 72.8 |
| Ours | 93.1 | 85.9 | 76.1 | 83.2 | 87.8 | 74.9 |
Table 2: Parallel-Jaw Grasp Performance on GraspClutter6D (GSR: Grasp Success Rate %, DR: Declutter Rate %)
| Method | Packed 5 GSR | Packed 5 DR | Pile 5 GSR | Pile 5 DR | Pile 10 GSR | Pile 10 DR | Pile 15 GSR | Pile 15 DR |
|---|---|---|---|---|---|---|---|---|
| Contact-GraspNet (ICRA 2021) | 84.9 | 86.1 | 77.6 | 75.4 | 77.2 | 64.7 | 77.3 | 54.0 |
| MVGrasp (RAS 2023) | 83.7 | 84.4 | 75.8 | 76.1 | 76.2 | 63.9 | 77.5 | 54.2 |
| BiGraspFormer (arXiv 2025) | 82.9 | 84.4 | 77.2 | 75.4 | 76.9 | 64.2 | 77.1 | 54.7 |
| ThinkGrasp (arXiv 2024) | 85.1 | 86.7 | 77.6 | 77.1 | 76.4 | 65.2 | 77.3 | 55.9 |
| AffordGrasp (IROS 2025) | 84.9 | 86.8 | 78.0 | 76.2 | 76.8 | 65.1 | 77.9 | 56.2 |
| Ours | 86.2 | 87.3 | 78.6 | 78.6 | 77.6 | 66.1 | 79.0 | 56.8 |
Ablation Study¶
Table 3: Ablation Analysis of Key Framework Components (GSR reported on GraspClutter6D; AP reported under RealSense rendered depth on GraspNet-1Billion)
| Configuration | DexGraspNet Dense | DexGraspNet Loose | Clutter6D Packed (5) | Clutter6D Pile (15) | GraspNet AP Seen | GraspNet AP Novel | Note |
|---|---|---|---|---|---|---|---|
| (1) No Split (Single Global Latent) | 87.0 | 69.1 | 84.5 | 75.0 | 75.4 | 34.3 | Conflates geometry with variance, limiting expressive capacity |
| (2) Fixed Latent Only | 86.5 | 68.2 | 84.0 | 76.4 | 74.2 | 33.5 | Purely deterministic; fails to model multimodal valid grasp hypotheses |
| (3) Fixed + Diffused (GCD) | 88.5 | 69.8 | 85.0 | 77.1 | 76.5 | 35.8 | Introduces stochastic global context to capture macro uncertainty |
| (4) + Local Conditional Diffusion (LCD) | 89.3 | 73.6 | 85.8 | 78.2 | 77.9 | 37.4 | Aligns global strategic intent with fine-grained local surface latents |
| (5) Full Model (+ HGF Regularization) | 93.1 | 76.1 | 86.2 | 79.0 | 79.8 | 40.9 | Enforces obstacle-aware harmonic potential, achieving peak performance |
Key Findings¶
- Harmonic Potential Regularization Yields Major Improvements: Incorporating HGF produces the single largest margin of improvement, boosting Dense grasp success on DexGraspNet-2.0 from 89.3% to 93.1% (+3.8 pp), and raising Novel object AP on GraspNet-1Billion from 37.4 to 40.9 (+3.5 pp). This confirms that smooth Laplacian potentials eliminate the local discontinuities and inter-object collisions typical of unregularized diffusion.
- Split Latents Counteract Heavy Occlusion: In the challenging Pile-15 setting of GraspClutter6D, the proposed model achieves a GSR of 79.0% and DR of 56.8%, significantly outperforming strong baselines like ThinkGrasp (77.3% GSR, 55.9% DR).
- Substantial Gains in Downstream Policy Execution: Integrating the generated grasps into the \(A^2\) simulation policy framework reduces planning steps by 16%–19% in Pick tasks while elevating Place success rates by up to +14.7%, proving that high-fidelity grasp pose distributions translate directly into efficient long-horizon manipulation.
Highlights & Insights¶
- Explicit Disentanglement of Structural Invariance and Strategic Diversity: Instead of assuming that deep networks can automatically decouple geometry from decision stochasticity, the paper explicitly enforces \(h_{\text{fixed}}\) for geometric rigidity and \(h_{\text{diffused}}\) for strategic variation. This split prevents diffusion noise from corrupting physical contact surfaces.
- Harmonic Field Differential Regularization: By defining Dirichlet boundary conditions on obstacle and grasp surfaces, Laplace's equation naturally constructs a smooth, monotonic potential field. The resulting gradients serve as physics-grounded approach vectors that bypass heuristic collision checkers.
Limitations & Future Work¶
- Lack of Microscopic Contact Mechanics: The formulation treats grasp generation as kinematic pose matching over potential fields without explicitly modeling dynamic friction cones, surface deformation, or tactile shear forces.
- Computational Overhead of Potential Field Solving: Solving 3D Dirichlet Laplace boundary problems over high-density point clouds introduces numerical overhead, which could constrain closed-loop high-frequency visual servoing.
- Future Directions: Extending the framework to incorporate real-time tactile sensor feedback into the reverse diffusion sampling loop, and combining harmonic grasp fields with compliant reinforcement learning policies.
Related Work & Insights¶
- vs DexGraspNet-2.0 / ClutterDexGrasp: While prior dexterous grasp systems rely on direct regression or single-step sampling that struggle in dense clutter, the proposed hierarchical split-latent diffusion captures multimodal strategies and achieves significant success rate gains in Dense clutter splits.
- vs Contact-GraspNet / ThinkGrasp: Contact-GraspNet uses local geometric heuristics without global clutter reasoning, whereas ThinkGrasp utilizes VLMs that lack continuous differential geometric smoothness. The proposed HGF formulation unifies global semantic conditioning with local potential flow constraints.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Proposes a novel split-latent hierarchical diffusion framework coupled with differential harmonic grasp fields.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensively validated across 4 major benchmarks spanning multi-finger dexterous grasping, parallel-jaw grasping, and downstream manipulation policies.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulation with clear conceptual intuitions and clean empirical explanations.
- Value: ⭐⭐⭐⭐⭐ Provides a foundational blueprint for integrating differential geometry and diffusion models in cluttered robot manipulation.