Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch

Conference: ECCV 2026
arXiv: 2604.09100
Code: https://github.com/hsp-iit/physical-generative-reconstruction
Area: 3D Vision
Keywords: hand-occlusion reconstruction, tactile sensing, flow matching diffusion, physical constraints, proprioception

TL;DR

This work integrates monocular RGB vision, proprioception (hand pose), and multi-contact touch within a flow matching diffusion framework to generate the metric-scale SDF of an object under severe hand occlusion. It guides shape reconstruction using two physical losses—non-penetration and contact consistency—during both training and inference sampling stages, yielding physically feasible outputs and implicitly estimating the object pose.

Background & Motivation

In robotic grasping and manipulation, object geometry is crucial for planning and control—it dictates the selection of stable contact points, free space for collision avoidance, and the precise orchestration of subsequent motions. However, precisely when geometric information is most critical, visual signals are least reliable: hand occlusion of the object during grasping and in-hand manipulation makes monocular RGB reconstruction an inherently ill-posed problem. Although recent 3D generative models based on diffusion and flow matching have made significant strides in single-view 3D inference by learning strong shape priors, purely vision-driven methods under heavy occlusion generate outputs that violate physical feasibility—objects penetrate the hand, real contact regions are missing, and scale drifts occur, rendering such reconstructions useless for robot control. Although approaches like Amodal3R and Sam3D model partial occlusion using foreground masks, they essentially perform "visual completion" without incorporating physical evidence to resolve ambiguities.

A natural yet underutilized idea is that the manipulation process itself continuously generates physical evidence. Proprioception precisely provides the spatial pose and geometry of the hand via joint angles, tactile sensors indicate which positions the surface must pass through, and the hand geometry itself constrains where the object cannot exist. These interaction signals provide the most direct geometric constraints precisely where vision is most ambiguous. While tactile sensing is widely used in robotics for object property estimation, pose estimation, and grasping robustness, current 3D generative methods rarely integrate these cues as explicit geometric constraints to drive dense 3D reconstruction. They typically target fully visible or slightly occluded object scenes.

The core insight of this work is that 3D reconstruction under hand occlusion is inherently a physically constrained generative inference problem, rather than simple visual completion. Core Idea: The object geometry is represented as a Signed Distance Field (SDF) sharing the same camera-aligned 3D grid with the hand. A conditional flow matching diffusion model is trained on the latent space of a Structure-VAE to generate the object SDF conditioned on visual evidence from RGB, the hand SDF latent variable, and multi-point tactile encoding. Two physical losses—non-penetration and contact consistency—are used to guide reconstruction via gradients during both training and inference sampling stages, thereby yielding metric-scale, physically viable 3D shapes and poses.

Method

Overall Architecture

This work decouples in-hand object reconstruction into two stages. Stage A is the core contribution: given a monocular RGB image, hand joint angles, and fingertip tactile readings, it outputs a camera-aligned coarse object SDF (which implicitly contains the metric scale and pose). Internally, Stage A consists of a Structure-VAE (compressing the SDF grid into a compact latent space) and a conditional flow matching diffusion model (generating on the latent space). The diffusion model is first pre-trained on unoccluded, object-only rendered images to learn a shape prior, and then fine-tuned on simulated grasping scenes—additionally conditioned on the visible region image, occlusion masks, hand SDF latent variable, and tactile encoding. Two physical losses are applied during both fine-tuning and inference. Stage B is an optional SLat-style refinement module that further refines the coarse SDF from Stage A into a textured triangular mesh.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Input: RGB + Joint Angles + Touch"] --> B["Visible Region Extraction<br/>I ⊙ Mo + Mask"]
    A --> C["Forward Kinematics→Hand Mesh<br/>→ SDF Grid Sh"]
    A --> D["Tactile Readings→Dual-Channel Field<br/>Binary Occupancy C + Distance Field D"]

    C --> E["Structure-VAE<br/>Encode to Hand Latent"]
    D --> F["Lightweight 3D CNN Encoder"]

    E --> G["1×1×1 Fusion Projection"]
    F --> G

    B --> H["Flow Transformer<br/>(Multi-stream Cross-attention)"]
    G --> H

    H --> I["Predict Object Latent"]
    I --> J["VAE Decoder→Object SDF"]

    subgraph K["Physical Constraints (Training + Inference)"]
        direction TB
        L1["Non-penetration Loss Lni<br/>Penetration of Hand→Penalize"]
        L2["Contact Consistency Loss Lc<br/>SDF at contact points→0"]
    end

    J --> L1
    J --> L2
    L1 --> M["Gradient-guided Sampling Trajectory"]
    L2 --> M

    M --> N["Stage A Output<br/>Coarse SDF + Pose"]
    N --> O["Stage B: SLat<br/>→ Textured Mesh"]

Key Designs

1. Camera-Aligned SDF Representation: A Shared Space Unifying Vision, Proprioception, and Touch

To align the RGB image, hand geometry, and tactile contact points into the same 3D physical space, this work defines a Cartesian grid aligned with the camera view: the grid's \(+z\) axis aligns with the line of sight of the rendering camera, and the object SDF \(S_o\) and hand SDF \(S_h\) are both computed on this grid. This natively aligns 2D evidence with the 3D representation—projecting the SDF along the \(z\)-axis corresponds directly to the conditional images, avoiding coordinate transformation ambiguities. The grid has a resolution of \(64^3\), and its coverage is normalized to \([-1,1]^3\). During data generation, random similarity transformations (scale 0.5–1.0 + in-plane translation) are applied to the object SDF to align its position and scale with the grasping scene (where the object position is determined by its physical scale and the relative hand-object pose, rather than being centered), enhancing the model's adaptability to different relative object-grid placements.

2. Structure-VAE: Differentiable, Geometry-Aware SDF Latent Space

Generative modeling directly on \(64^3\) grids is computationally inefficient and lacks smoothness. This work uses a 3D convolutional VAE to encode the SDF into a compact latent variable \(\mathbf{z}\). In addition to the L1 reconstruction loss and KL divergence, two geometric regularizations are introduced: Eikonal Loss \(\mathbb{E}(||\nabla\hat{S}||_2 - 1)^2\) to force the gradient norm to be 1 everywhere (an inherent property of SDFs), and a normal consistency loss to align predicted surface normals with the ground truth within a narrow band near the surface. These two terms guarantee correct geometric properties of the decoded SDF, laying a differentiable foundation for subsequent gradient-based physical guidance (which requires computing \(\nabla_{\mathbf{x}}E\)).

3. Multi-Modal Condition Injection: Hand Geometry for Global Constraints, Tactile Sensing for Local Refinement

During the fine-tuning stage, the diffusion model receives three classes of conditions. Visual conditions: visible region image \(I_o = I \odot M_o\) and occlusion masks \((M_o, M_h)\), following Amodal3R. Proprioceptional conditions: hand SDF \(S_h\) is encoded by the shared Structure-VAE encoder to obtain the hand latent variable \(\mathbf{x}_{0,h}\), which encodes the complete geometry of the hand in the shared grid. Tactile conditions: dual-channel 3D tensor \(T=[C,D]\), where \(C\) is a binary contact occupancy grid and \(D\) stores the distance from each voxel to the nearest contact point. The design of \(D\) is clever—it diffuses sparse, discrete binary contact signals into a continuous smooth field, allowing gradients to propagate to non-contact voxels, significantly easing the learning process. \(T\) is encoded by a lightweight 3D CNN and fused with \(\mathbf{x}_{0,h}\) via a \(1\times1\times1\) convolution, then tokenized and sent to the denoiser via an additional cross-attention stream.

4. Physically-Guided Training and Inference: Constraining Generation with Non-Penetration and Mandatory Contact

This is the core technical contribution, featuring a dual-level design spanning both training and inference.

During training, in addition to the standard flow matching regression loss \(\mathcal{L}_{FM}\), two physical losses computed on the decoded SDF are added. Non-penetration loss \(\mathcal{L}_{NI}\): applies a saturation function \(\psi_\tau(s)=\tau\tanh(\text{ReLU}(s)/\tau)\) for smoothing (preventing deep penetration from producing unstable gradients) to the object's SDF values inside the hand volume, and takes the average. Contact consistency loss \(\mathcal{L}_{C}\): directly penalizes \(|\hat{S}_o(\mathbf{x})|\) at contact voxels, forcing the reconstructed surface to pass through the contact points. Both losses are down-weighted by \((1-t)^2\) over the diffusion timestep \(t\)—since early predictions are highly noisy, and physical constraints are not yet applicable.

During inference, the exact same physical losses are injected into the sampling process via decoder gradient guidance. For the current latent variable \(\mathbf{x}_k\), we decode \(\hat{S}_o\), compute the physical energy \(E = \lambda_{NI}\mathcal{L}_{NI} + \lambda_C\mathcal{L}_C\) and its gradient with respect to \(\mathbf{x}_k\): \(\mathbf{g}_k = \nabla_{\mathbf{x}_k} E\). We update the control term using an exponential moving average: \(\theta_{k+1} = \beta\theta_k + \eta\mathbf{g}_k\) and add it to the velocity field: \(\mathbf{x}_{k+1} = \mathbf{x}_k - \Delta t_k (\mathbf{v}_\vartheta + \theta_k)\). Gradient normalization and trust-region constraints ensure that the guidance does not deviate too far from the flow matching prior.

Loss & Training

VAE training objective: $\(\mathcal{L}_{VAE} = \lambda_{L1}\mathcal{L}_{L1} + \lambda_{eik}\mathcal{L}_{eik} + \lambda_n\mathcal{L}_n + \lambda_{KL}\mathcal{L}_{KL}\)$

Total loss for flow matching fine-tuning (including physical terms): $\(\mathcal{L} = \mathcal{L}_{FM} + \lambda_{NI}\mathcal{L}_{NI} + \lambda_C\mathcal{L}_C\)$

where \(\lambda_{NI}\) is gradually warmed up from zero, and \(\lambda_C\) is fixed. Inference guidance uses the same coefficients.

Key Experimental Results

Main Results

Tested on simulated grasping scenes of 36 objects from the YCB dataset, split into 5 bins by occlusion severity (B1 being the lightest, B5 being the heaviest), compared against Amodal3R (a vision-only baseline with the same architecture) and Sam3D (a stronger single-image 3D baseline).

Metric	Method	B1 (low)	B3 (mid)	B5 (high)	All
CD ↓	Amodal3R	0.126	0.212	0.439	0.188
	Sam3D	0.020	0.034	0.153	0.039
	Ours	0.021	0.027	0.109	0.033
NC ↑	Amodal3R	0.704	0.644	0.602	0.669
	Sam3D	0.837	0.081	0.732	0.803
	Ours	0.868	0.834	0.803	0.844
Voxel IoU ↑	Amodal3R	0.386	0.302	0.262	0.339
	Sam3D	0.558	0.490	0.337	0.504
	Ours	0.616	0.576	0.532	0.586

Regarding pose estimation, Ours comprehensively outperforms Sam3D in 3D IoU (0.53 vs. 0.406) and [email protected] (0.73 vs. 0.658). In real-world robot transfer experiments, Ours achieves CD 0.035, NC 0.903, and Voxel IoU 0.648, also outperforming Sam3D's 0.037, 0.888, and 0.510 respectively.

Ablation Study

Configuration	CD ↓	NC ↑	Voxel IoU ↑	Description
Vision-Only	0.142	0.696	0.357	Vision only, equal to the information of Amodal3R
No-Touch (w/ Proprioception)	0.085	0.760	0.430	Drastic improvements after adding hand geometry
W/o Physical Loss in Training	0.052	0.774	0.487	Physical loss is a key component
W/o Guidance in Inference	0.036	0.811	0.530	Guidance contributes minimally but positively
Binary Contact Only	0.046	0.798	0.505	Distance field is a crucial supplement
Distance Contact Only	0.037	0.816	0.532
Ours (Full)	0.033	0.844	0.586

Key Findings

• Proprioception (hand geometry) brings a far greater improvement than tactile sensing: No-Touch decreases CD by 40% compared to Vision-Only (0.142 \(\rightarrow\) 0.085), while the full model further reduces it to 0.033. Hand geometry provides large-scale shape constraints, while tactile sensing provides fine-grained local contact refinements.
• The physical loss during the training phase is a decisive component: Removing it degrades CD from 0.033 to 0.052 (a 57% increase), with all other metrics deteriorating comprehensively. Inference-time guidance contributes positively but to a much lesser extent, indicating that the model has already internalized physical priors during training.
• The model is sensitive to kinematic noise but robust to tactile noise: A 5mm kinematic error elevates CD to 0.042, whereas a 5mm tactile noise only increases it to 0.036. This is consistent with the voxel size of the \(64^3\) grid (approx. 3cm), allowing tactile noise to be partially absorbed by quantization effects.
• Real-world performance is slightly lower than simulation (CD 0.035 vs. 0.033), with the main bottleneck stemming from hand-eye calibration and forward kinematics errors. The sensitivity to kinematic noise shown in the ablation study explains this gap.

Highlights & Insights

• The dual-level design of physical constraints (training loss + inference guidance) is the most notable highlight: Training determines the upper bound of the model's capabilities, while inference fine-tunes on top of the fixed model. The two are complementary, and the ablation study justifies this design (where the training physical loss has a much larger impact than inference guidance).
• The dual-channel touch encoding (binary occupancy + distance field) is highly ingenious: \(C\) provides sparse but precise surface constraints, while \(D\) diffuses them into a continuous, smooth signal that allows gradients to propagate globally. Ablation studies show that combining both outperforms using either alone, with \(D\) contributing more.
• Sharing a camera-aligned 3D grid with the hand enables implicit pose estimation without requiring an extra pose regression head. This cleverly exploits the readily available hand-eye calibration in robotic manipulation scenarios, serving as an excellent example of task-driven representation design.
• Although Stage B is a straightforward adaptation of SLat rather than a core contribution of this work, the overall pipeline design clearly embodies a decoupled philosophy of "physical reasoning \(\rightarrow\) visual refinement", facilitating easy replacement of the downstream refinement module.

Limitations & Future Work

• Sensitivity to hand-eye calibration accuracy: A 5mm kinematic error leads to perceptible performance degradation, which may become more severe on low-cost or uncalibrated robotic platforms. The authors acknowledge this as the primary source of the Sim2Real gap.
• The \(64^3\) fixed grid limits thin-walled structures and concave details, and the effective resolution is even lower when the object and hand share the grid. Adaptive sparse volumes or octree representations could be considered.
• Multi-stream cross-attention is relatively inefficient: Stage A inference takes approximately 11s on an A6000 (about 5s on an H200), which is roughly twice as slow as Sam3D. A further trade-off between inference speed and performance is required.
• The dataset is entirely synthetic, with real-world scenarios only verified on 5 YCB objects. The authors note that richer and more diverse object-interaction datasets are a future direction, potentially extending to tactile reasoning of physical properties such as friction and compliance.

Related Work & Insights

• vs Amodal3R: Both share a latent-space diffusion model of the TRELLIS architecture. By fully configuring proprioception, touch, and physical losses, Ours reduces the CD under heavy occlusion from 0.188 to 0.033 (an 82% reduction). The comparison clearly demonstrates that pure vision hits a performance ceiling under severe hand occlusion, making physical interaction signals a critical increment.
• vs Sam3D: Sam3D is a stronger single-image 3D baseline (larger model, more data, and depth maps). Ours comprehensively outperforms it on the YCB dataset even without depth inputs, raising the 3D IoU of pose estimation from 0.406 to 0.53. In addition, the output of Stage A in Ours can replace Sam3D's geometric module, showing complementarity between the two approaches.
• vs TouchSDF / NeuralFeels: Purely tactile approaches require active multi-step exploration to construct the shape point-by-point. Ours integrates tactile sensing into a generative framework, completing reconstruction in a single frame—tactile cues perform local refinements given the global coarse structure from shape priors, proving much more efficient and data-realistic.
• vs FollowMyHold / EasyHOI: These methods designed for human-hand manipulation require minutes for inference (>10 mins/sample), whereas Ours only takes dozens of seconds. On DexYCB, Ours achieves a reconstruction rate of 99% (vs. 58% and 30%), without needing specific adaptations to robotic hands, showing strong cross-embodiment generalization.

Rating

• Novelty: ⭐⭐⭐⭐⭐ A 3D generative method that injects physical constraints into both training and inference guidance is proposed for the first time in robotic manipulation scenarios. The distance-field tactile encoding is also highly ingenious.
• Experimental Thoroughness: ⭐⭐⭐⭐ The ablation study is comprehensive (sensory modalities, noise robustness, independent ablations of training/inference components), but the real-world validation is limited to only 5 objects, and the simulation data is entirely synthetic.
• Writing Quality: ⭐⭐⭐⭐⭐ Precise methodological descriptions, standardized equation numbering, a complete logical chain of ablations, and a candid and clear Limitations section.
• Value: ⭐⭐⭐⭐⭐ Opens a new direction of "physically-guided generation" for 3D reconstruction under occlusion, providing significant insights to both robotic manipulation and tactile sensing domains.

Related Papers

• [CVPR 2026] PhyGaP: Physically-Grounded Gaussians with Polarization Cues
• [CVPR 2026] Artiverse: A Diverse and Physically Grounded Dataset for Articulated Objects
• [CVPR 2025] Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation
• [CVPR 2025] Touch2Shape: Touch-Conditioned 3D Diffusion for Shape Exploration and Reconstruction
• [ECCV 2026] Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation