Affordance-Guided Diffusion Prior for 3D Hand Reconstruction

Conference: ECCV2026
Paper: Official page · PDF
Area: 3D Vision / Hand Reconstruction
Keywords: Affordance descriptions, grasp classification, conditional diffusion, occlusion awareness, pose refinement

TL;DR

AffHandGen conditions a hand-pose diffusion prior on detailed descriptions of how an object is grasped and used, together with image features, refining occluded joints while retaining visible 2D evidence and reducing WiLoR's high-occlusion PA-MPJPE on HOGraspNet from 12.41 to 9.37 mm.

Background & Motivation

Reconstructing a hand from a single RGB image requires more than recognizing finger outlines. Large pretrained models such as HaMeR and WiLoR often align projected meshes with visible image regions, but when an object or another finger hides a joint, the same projection can support multiple 3D configurations. A mesh that looks like a hand does not necessarily grasp the object plausibly. Hidden ring and little fingers are particularly vulnerable to errors in depth and articulation.

Objects offer another source of evidence: holding an elongated tool, supporting a large bowl, and pinching a thin sheet impose different requirements on thumb placement and finger opening. Yet a generic caption about holding an object is too coarse, and an isolated action or grasp label is incomplete. The authors treat affordances as relationships between object properties and possible interactions, seeking to encode both action intent and grasp geometry. The difficulty is that general-purpose vision-language models can describe a scene without reliably identifying its precise grasp taxonomy.

The method therefore neither asks a language model to regress 3D joints directly nor replaces all visual estimates with generated poses. It first organizes affordance information into a more reliable textual condition, learns a corresponding hand-pose distribution, and uses that distribution to repair uncertain parts of an initial reconstruction. Core idea: narrow the plausible configurations of occluded joints using grasp-classification-enhanced affordance text, while constraining visible joint projections so that the semantic prior cannot freely overwrite existing visual evidence.

Method

Overall Architecture

The input is an RGB image of a hand-object interaction; description generation also uses hand and object bounding boxes. WiLoR or Hamba supplies the initial pose to refine. The pipeline generates an affordance description, learns a MANO pose distribution conditioned on that description and the image through AffHandGen, and starts inference from an intermediate noisy version of the initial pose to produce a refined hand mesh.

The generative target consists of MANO's internal joint pose parameters, while the shape estimate remains fixed. Text and images condition the prior, whereas joint visibility determines which 2D errors are trustworthy during refinement. Visibility is neither another textual attribute nor an explicit object-collision model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["RGB image and boxes"] --> DESC["Stepwise affordance<br/>description"]
    INPUT --> BASE["Initial pose and<br/>image features"]
    DESC --> PRIOR["Multimodal token<br/>diffusion prior"]
    BASE --> PRIOR
    BASE --> REFINE["Occlusion-aware<br/>pose refinement"]
    PRIOR --> REFINE
    REFINE --> OUTPUT["Refined hand mesh"]

Key Designs

1. Stepwise affordance description: obtain general scene understanding and specialized grasp knowledge separately

Qwen2.5-VL first reads the image and hand-object bounding boxes to extract object category, shape, relative size, interaction action, and intention. The purpose is not simply to lengthen the caption: a sentence about holding a bottle can omit object scale or grasp configuration, both of which affect finger geometry. The supplementary prompt additionally contains a short hand-action caption and permits unknown or undetermined values for uncertain fields. These descriptions are model inferences, not physical ground truth.

A separate supervised classifier supplies the grasp type. It reuses HaMeR's ViT image features and MANO pose predictions, feeds both into self-attention and an MLP, and learns from HOGraspNet labels. It uses 28 of the original 33 grasp types and achieves 87.6% evaluation accuracy. Pose parameters supply finger configuration while visual features add interaction context, avoiding reliance on an unspecialized VLM to guess fine-grained taxonomy labels. Finally, Mistral-7B integrates the fields and predicted grasp type into a detailed sentence for the text encoder. Classification, field extraction, and sentence integration belong to one description-generation stage, not three independent pose optimizers.

2. Multimodal token diffusion prior: let pose features query specific words and image regions

AffHandGen trains its denoiser in MANO's \(15\times3\) pose-parameter space and directly predicts a clean pose rather than generating RGB images. Linear layers encode the pose and timestep, whose representations are concatenated and processed by Transformer decoder layers before a linear pose decoder produces the output. The clearly recoverable conditional prediction relation is:

\[ \hat{x}_0=f_\phi(x_t,t,c),\qquad c=\{c_D,c_I\}. \]

Here, \(x_t\) is the noisy pose, and \(c_D\) and \(c_I\) are the text and image conditions. DistilBERT provides 64 text tokens, while HaMeR's ViT provides 192 image tokens. Decoder cross-attention accesses these tokens, and self-attention models relationships within the pose representation. The important distinction is not collapsing each modality into a global vector: even for the same object category, different grasp words and local finger regions can remain distinct attention targets. This enables fine-grained conditioning but does not imply supervised one-to-one correspondences between individual joints and words.

This choice also explains why adding text is not automatically effective. The adapted InterHandGen baseline can receive images and descriptions too, but represents them as global condition vectors rather than providing the same localized conditional interactions. During training, the full model randomly masks text and image conditions with probability 0.1 to accommodate missing conditions and reduce dependence on a single modality. That strategy is not evidence of robustness to arbitrary incorrect descriptions.

3. Occlusion-aware pose refinement: constrain visible projections while freeing uncertain joints

Inference starts from a WiLoR or Hamba estimate, not by generating a new hand from pure noise. The initial pose is diffused to an intermediate timestep and iteratively denoised under text and image conditioning. After each clean-pose prediction, a gradient update uses the 2D projection error of visible joints. Its reference comes from projecting the initial 3D prediction itself, rather than from an independent detector or manually annotated keypoints. A final 2D fitting stage follows diffusion. Preserving visible evidence is consequently a soft constraint, not an exact freeze of every visible joint's 3D coordinates.

Two criteria determine visibility. For self-occlusion, a ray is cast from each joint of the initial MANO mesh toward the camera; at least 2 surface intersections indicate occlusion by another part of the hand. For object occlusion, projected initial joints are compared with the SAM2 hand mask, and joints outside the mask are considered object-occluded. Hidden joints are thus no longer forced to match potentially wrong initial 2D positions, leaving the prior room to correct articulation and depth. Errors in the initial mesh or segmentation can nevertheless corrupt this decision.

The starting noise timestep is adjusted linearly with the number of occluded joints. More occlusion permits stronger correction from a noisier state, while less occlusion requires fewer changes and less computation. Reverse timesteps are also sampled sparsely. This allocates refinement freedom according to uncertainty instead of executing a full diffusion chain for every sample.

A Worked Example

In the controlled generation example of Figure 8, the description keeps a precision disk grasp fixed and changes only the mug size from small to large. The generated hand changes from a tighter grasp to a wider configuration. This is qualitative evidence of textual control, not a physical grasping experiment with a measured success rate.

For reconstruction, the same mechanism operates after an initial mesh estimate: image evidence retains the visible thumb position, object size and grasp type constrain plausible hidden-finger geometry, and the visibility mask prevents denoising from being forced to reproduce unreliable projections. Both uses share the pose prior, but the main reconstruction results come from refinement with an initial estimate and 2D guidance.

Loss & Training

Training uses an L2 loss between predicted and ground-truth clean poses, plus L2 losses on mesh vertices and 3D joints after the MANO layer. The pose, vertex, and joint loss weights are 1, 10000, and 10000, respectively. The cached forward-diffusion and total-loss equations contain corrupted characters, so this note reports the unambiguous prose rather than reconstructing missing symbols, square roots, or coefficient structure.

The latent dimension is 512 with 4 attention heads. Training uses Adam at a learning rate of \(10^{-3}\), batch size 64, and 80 epochs. The maximum diffusion length is 1000 steps with a cosine forward schedule. HOGraspNet uses a 2D guidance weight of 0.005, starting steps in [100, 1000], and reverse sampling interval 20. Because HO3D initial predictions are more reliable, it uses weight 0.5, starting steps in [10, 20], and interval 3. Cross-dataset results therefore involve different refinement settings, not an entirely tuning-free unified configuration.

Key Experimental Results

Main Results

The authors sample every 10 frames to reduce near-duplicate descriptions from adjacent images. HOGraspNet uses 119K training and 3K evaluation images, with unseen subjects in its S1 split. HO3D uses 65K/1K images and evaluates unseen views, subjects, and objects. HO3D and HInt lack grasp labels, so the classifier trained on HOGraspNet supplies pseudo-labels.

The following values are selected from Table 1. PA-MPJPE and PA-MPVPE are Procrustes-aligned mean joint and vertex errors in mm; lower is better. High occlusion means 11–21 occluded joints, while low and medium mean 0–5 and 6–10. Aligned errors should not be interpreted as absolute camera-coordinate reconstruction errors.

Dataset	Method	Average PA-MPJPE	High-Occlusion PA-MPJPE	PA-MPVPE
HOGraspNet	WiLoR	8.22	12.41	7.57
HOGraspNet	WiLoR + InterHandGen	8.99	14.61	7.91
HOGraspNet	WiLoR + global text/image condition baseline	8.06	11.12	7.29
HOGraspNet	WiLoR + AffHandGen	7.53	9.37	6.90
HOGraspNet	Hamba	8.46	12.72	7.57
HOGraspNet	Hamba + AffHandGen	7.85	9.46	7.11
HO3D	WiLoR	7.43	7.87	7.63
HO3D	WiLoR + AffHandGen	7.37	7.80	7.46
HO3D	Hamba	7.42	8.71	7.66
HO3D	Hamba + AffHandGen	7.29	8.39	7.37

WiLoR's average HOGraspNet error drops by 0.69 mm, and its high-occlusion error drops by 3.04 mm; the average HO3D improvement is only 0.06 mm. Benefits clearly depend on the initial estimate and occlusion level. Hamba's medium-occlusion HOGraspNet error even increases from 9.00 to 10.21 mm, so overall improvement does not imply improvement on every subset.

Ablation Study

The following table selects clearly readable average HOGraspNet PA-MPJPE values from Table 2. Rows represent separate experimental configurations, not cumulative removal of multiple components.

Configuration	PA-MPJPE (mm)	Interpretation
Image conditioning only	7.90	No text
Simple VLM caption + image	7.76	Coarse textual condition
Affordance description only	8.43	No image condition
Full affordance description + image	7.53	Standard configuration
Full configuration + ground-truth grasp labels	7.51	Replaces predicted grasp types
Keywords without Mistral sentence integration	7.59	Alternative description generation
Without object-size field	7.59	Field ablation
Without grasp-taxonomy field	7.71	Field ablation
CLIP condition encoder	11.36	Encoder replacement
Without segmentation mask	7.55	Visibility-mask ablation

Full descriptions improve over simple captions by 0.23 mm and over image-only conditioning by 0.37 mm. Ground-truth grasp labels add only 0.02 mm of improvement. This supports the usefulness of the current classifier's conditions, not a claim that classification errors are always harmless. Removing the segmentation mask costs only 0.02 mm, which also cannot establish a large contribution from the entire occlusion-aware strategy: that row does not jointly remove ray casting, adaptive timesteps, and 2D guidance. CLIP replacement causes the largest degradation in this table, but changes both condition representations and encoders, preventing attribution of the full difference to a single attention operation.

Key Findings

Supplementary Table 4 evaluates cross-domain performance on HInt. The following New Days results retain a counterexample. [email protected]/0.1/0.15 follows the HInt threshold protocol, with higher being better; it should not be mixed with the pixel-threshold PCK@5/10/15 in the main paper's Table 1.

New Days Joint Subset	Method	[email protected]	[email protected]	[email protected]
All	WiLoR	24.9	54.9	71.6
All	WiLoR + AffHandGen	23.6	54.5	72.8
Occluded	WiLoR	15.4	42.1	61.2
Occluded	WiLoR + AffHandGen	15.1	42.1	63.0

Improvement at a looser threshold does not guarantee improvement at a stricter one. The supplementary hyperparameter curve reports 7.51 mm at weight 0.005 under a setting with 100 post-processing steps, whereas the main table reports 7.53 mm. These results retain their respective experimental contexts; the curve's optimum does not replace the main result.

Computational cost must also be separated by component. Table 3 reports 138.06 GFLOPs and 21.0 FPS for WiLoR, versus 7.94 GFLOPs and 48.6 FPS for AffHandGen at approximately 8 diffusion steps per sample. However, Table 2 reports only 0.31 FPS for Qwen+Mistral description generation and 8.1 FPS for SAM2. A lightweight diffusion module does not make the entire pipeline, including initial reconstruction, descriptions, and segmentation, faster. The lightweight affordance-classifier alternative reports 7.57 mm and 96 FPS, again as a component-level replacement experiment.

Highlights & Insights

• Useful linguistic conditioning comes from information that affects geometric decisions, not caption length. Expressing grasp taxonomy together with object scale can exclude implausible configurations more effectively than a generic holding caption.
• The division of labor between a specialized classifier and a general VLM is reusable. The classifier handles supervised, well-defined taxonomy details, while the VLM handles open-ended properties and intentions, reducing dependence on general models for specialized knowledge.
• A prior is useful for repairing uncertain regions rather than overwriting every observation. Separating reliable projections from occluded joints could transfer to other articulated reconstruction tasks, provided visibility estimates are sufficiently reliable.

Limitations & Future Work

• The authors explicitly limit the method to static single-hand reconstruction without explicit 3D object geometry. Text proxies object information, so functional plausibility is not proof of collision-free geometry, correct contact, or stable grasping.
• Supplementary failures involve unusual grasps and out-of-distribution object affordances, where less reliable descriptions weaken refinement. Broader vocabulary and data coverage, together with explicit 3D object constraints, are proposed directions rather than established capabilities.
• Reviewer observation: the initial estimate determines both the 2D reference and part of the visibility decision, potentially preserving errors as trusted supervision. Fixed shape also limits the error types that refinement can correct.
• Reviewer observation: strict-threshold HInt metrics and some visible-joint results degrade, while refinement hyperparameters differ across datasets. A stronger robustness evaluation would include incorrect descriptions, visibility errors, and fixed-configuration cross-domain tests.
• Reviewer observation: real-time claims require end-to-end latency measurements across all components, not diffusion-module FPS. Future work could test combined lightweight description and segmentation replacements rather than assuming individual speed results transfer directly to their combination.

Related Work & Insights

• Relation to HaMeR, WiLoR, and Hamba: these models provide initial visual reconstruction; the paper adds a semantic conditional prior and inference-time refinement. It augments a base estimator rather than eliminating the need for one.
• Difference from InterHandGen: the original method targets two-hand interaction generation. Here it is adapted to a single hand, with a separate global text/image-conditioned variant. AffHandGen instead uses fine-grained modality tokens and cross-attention; comparisons must acknowledge the task adaptation.
• Relation to SCGen: the method builds on intermediate-noise diffusion refinement with 2D keypoint guidance, adding affordance conditioning and hand-specific occlusion handling. The 2D reference comes from the initial hand estimate rather than directly reusing a body keypoint detector.
• Difference from G-HOP: G-HOP uses object geometry to build a hand-object interaction prior, whereas this paper models hand poses conditioned on text and images. Whether object geometry enters the model explicitly is a substantive distinction.

Rating

• Novelty: 4/5. Affordance descriptions, specialized grasp classification, and occlusion-aware refinement form a clear combination, while diffusion refinement itself builds on prior work.
• Experimental Thoroughness: 4/5. Multiple estimators, main datasets, cross-domain results, and detailed ablations are covered, but physical correctness and end-to-end efficiency need stronger evidence.
• Writing Quality: 4/5. Modules and supplementary details are well covered, but generalization and computational-cost claims require careful reading of the tables; cache equation corruption is not counted against the paper.
• Value: 4/5. The method offers actionable gains for heavily occluded hand reconstruction, with practical value depending on condition reliability and full-pipeline cost.