AnchorHOI: Zero-shot Generation of 4D Human-Object Interaction via Anchor-based Prior Distillation

Conference: AAAI 2026
arXiv: 2512.14095v1
Code: None
Area: 3D Vision
Keywords: 4D HOI generation, Zero-shot, Anchor prior distillation, NeRF, Video diffusion models

TL;DR

This paper proposes AnchorHOI, which distills interaction and motion priors from image/video diffusion models via two intermediate bridges—anchor NeRF and anchor keypoints. This achieves zero-shot text-driven 4D human-object interaction (HOI) generation, outperforming existing methods in both static 3D and dynamic 4D HOI generation.

Background & Motivation

Text-driven 4D human-object interaction (HOI) generation has wide applications in AR/VR, gaming, and robotics. Existing supervised methods rely on scarce and expensive motion capture (mocap) paired data, which severely limits scalability. Recent zero-shot methods like AvatarGO attempt to replace mocap data using pre-trained image diffusion models, but suffer from two critical flaws: (1) human poses are fixed to canonical poses during interaction composition, lacking adaptability; (2) motions originate from object-agnostic text-to-human motion models, neglecting interaction-aware motion synthesis. Therefore, richer sources of priors and more effective prior distillation techniques are urgently needed.

Core Problem

How to effectively distill interaction priors from pre-trained image and video diffusion models to generate 4D HOI sequences with realistic poses and interaction-aware motions without relying on paired mocap data? The key challenges lie in: (1) optimizing high-degree-of-freedom SMPL-X human joint poses guided by image diffusion models is extremely difficult; (2) severe inter-subject occlusions exist in synthetic videos generated by video diffusion models, making it difficult to directly extract reliable interaction motion information.

Method

Overall Architecture

AnchorHOI adopts a two-stage pipeline: first generating static 3D HOI instances (interaction composition), and then extending them to dynamic 4D HOI sequences (motion synthesis). The input is a natural language description (including descriptions of human, action, and object), and the output is a multi-frame 3D human-object interaction sequence. The core innovation is the introduction of "anchors" as intermediate bridges, decomposing the otherwise difficult direct optimization problem into two manageable steps.

Key Designs

1. Anchor NeRF for Interaction Composition: Direct optimization of poses in the SMPL-X parameter space using SDS is almost infeasible due to high dimensionality and non-linearity. Therefore, SDS is first used to generate a coarse, entangled human-object NeRF from an image diffusion model. The human part is then extracted as the anchor NeRF via multi-view feature alignment. Afterwards, 2D skeletal keypoints are detected from the anchor NeRF rendered images using OpenPose. The SMPL-X pose parameters are optimized by minimizing the gap between the projected 3D joints and the detected 2D keypoints, achieving pose-adaptive interaction composition. The object is initialized from the object part of the anchor NeRF and further refined via SDS.

2. Anchor Keypoints for Motion Synthesis: In HOI videos generated by video diffusion models, severe occlusions exist in the contact regions between human and object, making pure pixel-level cues insufficient to capture interaction motion. Therefore, two types of anchor keypoints are defined: (a) Body keypoints—18 2D human keypoints detected from each video frame using OpenPose, which provide robust pose cues even under occlusion; (b) Contact keypoints—based on 3D geometric proxies (sampled points on the object mesh surface and SMPL-X mesh candidate contact vertices), valid contact point pairs are identified using normal-vector alignment and geometric proximity constraints to capture interaction information in occluded areas.

3. Motion Optimization: Using the anchor keypoints as tracking cues, the human and object motion parameters for each frame are optimized through a joint loss. The losses include: joint projection alignment loss (SMPL-X joint re-projection to 2D aligned with detected keypoints), contact constraint loss (minimizing the distance between human-object contact pairs), penetration penalties, and regularization terms (rendering consistency, self-penetration penalty, and temporal smoothness).

Loss & Training

• Pose Alignment Loss \(\mathcal{L}_{\text{align}}\): Geman-McClure robust distance between SMPL-X 3D joint projections and OpenPose-detected 2D keypoints under multi-view settings.
• Total Motion Loss \(\mathcal{L}_{\text{total}} = \lambda_J \mathcal{L}_J + \lambda_C \mathcal{L}_C + \lambda_{\text{pen}} \mathcal{L}_{\text{pen}} + \lambda_{\text{reg}} \mathcal{L}_{\text{reg}}\)
  • \(\mathcal{L}_J\): Joint re-projection alignment (with confidence-weighted Geman-McClure).
  • \(\mathcal{L}_C\): Euclidean distance between contact keypoint pairs.
  • \(\mathcal{L}_{\text{pen}}\): Human-object penetration penalty.
  • \(\mathcal{L}_{\text{reg}}\): Rendering MSE + self-penetration penalty + temporal smoothness.
• 3000 iterations for the interaction composition stage, 1000 iterations for the motion synthesis stage, using Adam optimizer, lr=0.01, and an A6000 GPU.

Key Experimental Results

Method	CLIP Score ↑	GPT-4V Overall ↑	User Semantic ↑	User Contact ↑	User Motion ↑	User Overall ↑
DreamGaussian4D	0.2833	25.00%	2.33	2.38	2.63	3.33
TC4D	0.3017	20.83%	3.11	2.32	2.39	3.66
AnchorHOI	0.3149	54.17%	4.79	4.75	4.87	4.83

3D static HOI comparison (vs MVDream/InterFusion/AvatarGO): CLIP Score is 0.3173 (highest), and GPT-4V Overall selection rate is 52.63% (vs InterFusion 26.32%).

Ablation Study

• Without Anchor NeRF: GPT-4V selection rate drops to only 5.88% (vs 94.12% for the full model), and the human pose fails to converge to an appropriate interaction pose.
• Without Body Keypoints: 5.89%, leading to implausible motion poses.
• Without Contact Keypoints: 17.65%, where the actions are visually reasonable but lack physical contact.
• Full Model: 76.47%, optimizing both plausible poses and interaction-aware contact.

Highlights & Insights

• The design philosophy of the anchor strategy is highly ingenious: Instead of directly performing difficult optimization in high-dimensional parameter spaces, it introduces intermediate representations (NeRF and keypoints) as bridges to decompose the problem into a two-step controllable process. This "build a bridge first, then cross the river" idea is highly generalizable.
• Hybrid prior utilization: This is the first work to combine image diffusion models (static interaction priors) and video diffusion models (dynamic motion priors) for zero-shot 4D HOI generation.
• Definition of contact keypoints: It identifies valid contact by combining normal alignment and geometric proximity constraints, featuring clear physical mechanisms (opposite normal vectors and close distance at contact regions).

Limitations & Future Work

• It assumes the human and the object maintain continuous contact throughout, failing to handle dynamic interactions where contact is broken and then restored (e.g., throwing and catching an object).
• It only supports rigid objects and cannot handle articulated or deformable objects (e.g., opening a door, folding clothes).
• Generation speed is limited by the iterative optimization of SDS (3000 + 1000 iterations), resulting in poor real-time performance.
• It relies heavily on the quality of OpenPose 2D keypoint detection, which can be unstable under severe occlusion.

Related Work & Insights

• vs AvatarGO: AvatarGO is the most direct prior work, but during the interaction composition stage, its human pose is fixed to a canonical standing pose, failing to adapt to interaction-specific poses such as sitting or squatting. Furthermore, its 4D generation part is not publicly released. AnchorHOI achieves pose-adaptive interaction composition via the anchor NeRF.
• vs DreamGaussian4D: While using the same VDM video guidance, DG4D only relies on RGB and mask pixel cues to drive the HOI animation, failing to capture interaction motions in occluded areas. AnchorHOI provides more robust interaction motion cues through anchor keypoints.
• vs InterFusion (3D): InterFusion retrieves fixed poses from a pre-built pose library, lacking adaptability to specific interaction scenarios. AnchorHOI achieves pose optimization aligned with interaction semantics via the anchor NeRF.

Inspirations & Connections

• The concept of the anchor strategy (generating intermediate representations first, then distilling them into target representations) can be extended to other diffusion model-guided 3D/4D generation tasks, especially when target representations (such as parametric models) struggle to directly receive gradients from diffusion models.
• The definition of contact keypoints (normal alignment + geometric proximity) can be transferred to robotic grasp planning and contact modeling in physical simulations.

Rating

• Novelty: ⭐⭐⭐⭐ The anchor strategy transforms infeasible direct optimization into a two-step controllable process, which is clear and effective.
• Experimental Thoroughness: ⭐⭐⭐⭐ Quantitative (CLIP/GPT-4V/user studies) + qualitative + ablation studies provide comprehensive coverage, though comparison with more 4D methods is missing.
• Writing Quality: ⭐⭐⭐⭐ Problem motivation and methodology are clearly described, with intuitive anchor illustrations.
• Value: ⭐⭐⭐⭐ Makes significant progress in the new direction of zero-shot 4D HOI, though limited by rigid objects and continuous contact assumptions.

Related Papers

• [ICML 2026] HOI-PAGE: Zero-Shot Human-Object Interaction Generation with Part Affordance Guidance
• [CVPR 2026] CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction
• [CVPR 2026] ReGenHOI: Unifying Reconstruction and Generation for 3D Human-Object Interaction Understanding
• [AAAI 2026] EPSegFZ: Efficient Point Cloud Semantic Segmentation for Few- and Zero-Shot Scenarios
• [CVPR 2026] ConceptPose: Training-Free Zero-Shot Object Pose Estimation using Concept Vectors