PartCHOI: Part-Aware Guidance for Clothed Human-Object Interaction Generation

Conference: ECCV 2026
Paper: ECCV 2026 Official
Area: Human Understanding
Keywords: Clothed Human Generation, Human-Object Interaction (HOI), 3D Scene Generation, Part-Aware Guidance, Surface Deformation

TL;DR

By reformulating single-view clothed human-object interaction (CHOI) as a human-centered scene where interacting objects are treated as body parts, PartCHOI introduces training-free H-GPE for closed-form pose estimation and O-GRM for Jacobian-based contact deformation, delivering physically plausible and penetration-free 3D interactive scenes.

Background & Motivation

Generating 3D digital humans and modeling their interactions with objects from a single image is vital for virtual reality, gaming, animation, and embodied intelligence. However, real-world images rarely depict isolated humans in neutral poses; instead, they are saturated with complex human-object interactions (HOI)—such as people carrying backpacks, holding handheld props, wearing hats, or sitting on chairs. Synthesizing single-view clothed human-object interactions (CHOI) requires simultaneously recovering high-fidelity clothed human geometries, diverse object meshes, and accurate spatial arrangements with physically plausible contacts.

Existing methodologies tackling this problem fall predominantly into two paradigms, both of which suffer from severe bottlenecks. Monocular HOI reconstruction methods infer interaction parameters from 2D images, yet they heavily depend on predefined CAD templates and simplified parametric human bodies (e.g., SMPL-X), making them incapable of synthesizing unconstrained object geometries, realistic fabric wrinkles, or high-fidelity textures. On the other hand, compositional 3D scene generation methods generate individual entities separately and assemble them via layout optimization. These approaches rely on coarse spatial bounding constraints that easily get trapped in local minima, require excessively slow optimization routines (often tens of minutes), and frequently result in unrealistic floating or severe interpenetration between the human body and the object.

This paper tackles the challenge from a novel conceptual perspective: rather than treating human and objects as disjoint, weakly coupled entities, it reformulates CHOI as generating a coherent human-centered scene where interacting objects are viewed as functional parts of the human body. Core idea: reformulate single-view CHOI into a part-aware mutual guidance paradigm that extracts object-part voxels using the human body surface to solve poses in closed form via SVD, and drives a 3D Jacobian deformation field using object parts as anchors to eliminate surface interpenetration without task-specific retraining.

Method

Overall Architecture

Given a single image \(I\) (or a text prompt converted into an image via a 2D generator), PartCHOI aims to generate an interactive 3D scene \(S = \{M_h, \{O_i\}_{i=1}^N\}\) comprising the clothed human mesh \(M_h\) and separate object meshes \(\{O_i\}\). The pipeline first deploys a Separation Module (using a VLM to infer semantics and occlusion topology, Grounded-SAM to segment instances, and diffusion-based inpainting to complete occluded regions) to obtain isolated images \(I_h\) and \(\{I_{o_i}\}\). The pipeline then executes three collaborative branches: an object branch generating individual objects via TRELLIS; a Pose-Aligned Reconstruction (PAR) branch optimizing human pose parameters \(\Theta'_H\) to align with the scene structure; a Human-Guided Pose Estimator (H-GPE) calculating object poses \(\Theta_O\) via closed-form alignment; and an Object-Guided Refinement Module (O-GRM) executing localized mesh deformation to establish natural physical contact.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Single Input Image I<br/>(Decomposed into human & object masks and images)"] --> B["Pose-Aligned Reconstruction (PAR)<br/>Progressively aligns human pose with scene voxels"]
    A --> C["Human-Guided Pose Estimator (H-GPE)<br/>Voxel ray-casting & closed-form Kabsch alignment"]
    B --> C
    C --> D["Initial Scene Composition<br/>Multi-entity spatial layout alignment"]
    D --> E["Object-Guided Refinement Module (O-GRM)<br/>Poisson Jacobian deformation removes penetration"]
    E --> F["Output High-Fidelity 3D CHOI Scene S<br/>Disentangled, editable clothed human & objects"]

Key Designs

1. Pose-Aligned Reconstruction (PAR): progressively reconciling multi-view texture cues with generative voxel priors

Because the clothed human appearance is synthesized via multi-view human diffusion models (e.g., PSHuman), directly forcing the raw SMPL-X parameters \(\Theta_H = \{\theta, \beta, \psi\}\) to align with the coarse voxel grid from TRELLIS would induce severe texture-structure misalignments. PAR resolves this through a progressive dual-guidance optimization scheme. During early iterations, camera-rendered normal maps, silhouette masks, and 2D keypoints derived from the coarse TRELLIS scene mesh supervise the SMPL-X parameters, anchoring the human posture firmly to the 3D interaction space. In later iterations, a weighting factor \(\mu \in [0, 1]\) linearly scales up to transfer primary supervision to multi-view renderings generated by PSHuman. This progressive shift simultaneously corrects initial pose initialization errors and ensures geometric consistency between the human body and fine surface textures.

2. Human-Guided Pose Estimator (H-GPE): reformulating layout optimization as voxel-level part localization

Conventional layout optimization relies on score distillation sampling (SDS) or noisy monocular depth alignment, which converges sluggishly and frequently yields distorted poses. H-GPE exploits the implicit spatial relationships embedded within the scene's native 3D voxel representation \(S_v\), utilizing the PAR-aligned SMPL-X surface mesh as a geometric boundary. Camera rays are projected from the 2D foreground mask of each object \(O_i\) through the voxel grid, categorizing active voxels into five occlusion topological cases (e.g., voxels behind the human surface when human occludes object, or voxels bounded by inner-facing limb surfaces for grasped items). Once the candidate voxel subset \(\mathcal{V}_i\) is isolated, it is aligned with the canonical object voxel set \(\mathcal{V}'_i\) via a closed-form Kabsch algorithm. Normalizing both sets by their centroids \(\bar{v}\) and \(\bar{v}'\), the spatial covariance matrix \(H = (\mathcal{V}'_i - \bar{v}')^\top (\mathcal{V}_i - \bar{v})\) is decomposed via a single Singular Value Decomposition (SVD): $\(R = \mathbf{V} \mathbf{U}^\top, \quad T = \bar{v} - S \cdot R \bar{v}'\)$ This closed-form formulation completely circumvents iterative gradient descent, estimating accurate 6-DoF object poses in approximately 7 seconds with remarkable geometric robustness.

3. Object-Guided Refinement Module (O-GRM): resolving interpenetration and modeling pressure via Jacobian deformation

Even with accurate 6-DoF poses, subtle geometric discretization artifacts between the clothed human mesh \(M_h\) and the object surface inevitably produce non-physical interpenetration. Instead of perturbing global poses or introducing destabilizing rigid collision penalties, O-GRM constructs a per-face 3D Jacobian deformation field \(J_t \in \mathbb{R}^{3\times 3}\) across the human mesh, solving a Poisson equation to update vertices while strictly preserving high-frequency garment folds across non-contact areas. Identifying the interaction faces \(\{t_r\}\) via object projection, the module minimizes a bidirectional signed distance field (SDF) penetration penalty: $\(\mathcal{L}_{\text{int}} = \sum_{t \in \{t_r\}} \left( \max(0, -s_o(c_t)) + \max(0, -s_h(c_t)) \right)\)$ where \(s_o(c_t)\) and \(s_h(c_t)\) denote the signed distances from face centroid \(c_t\) to the object surface and the underlying SMPL-X body surface, respectively. Combined with an \(L_1\) Jacobian regularization term \(\|J\|_1\) and a Laplacian smoothness term, this loss forces garments to conform naturally into the gap between the skin and object surfaces, faithfully simulating the physical flattening of fabric under external contact pressure.

Loss & Training

PartCHOI requires zero task-specific pre-training and operates entirely at test-time inference. In the PAR stage, the Adam optimizer runs for 700 steps to minimize \(\mathcal{L}_{\text{align}} = \mu \mathcal{L}_{\text{align-H}} + (1 - \mu) \mathcal{L}_{\text{align-G}}\), balancing normal map L1 distance \(\mathcal{L}_n\), silhouette loss \(\mathcal{L}_s\), and 2D keypoint loss \(\mathcal{L}_j\). In the O-GRM stage, optimization runs for 100 iterations with fixed object placements, optimizing the composite objective \(\mathcal{L}_{\text{geo}} = \lambda_1 \mathcal{L}_{\text{int}} + \lambda_2 \|J\|_1 + \lambda_3 \|L V'\|_2^2\).

Key Experimental Results

Main Results

Quantitative evaluations are conducted across the DNA-Rendering, THuman-Sit, and in-the-wild HOI benchmarks. Across multi-view 2D rendering metrics (PSNR, SSIM, LPIPS, CLIP) and perceptual user study indicators—appearance fidelity (A), layout consistency (L), and interaction realism (I)—as well as holistic GPT-5 evaluation (G-S), PartCHOI consistently outperforms all baseline approaches.

Method	Type	User Study A(%)↑	User Study L(%)↑	User Study I(%)↑	GPT-5 Selection G-S(%)↑	PSNR↑	SSIM↑	LPIPS↓
Wonder3D	General 3D	0.000	4.625	2.250	0.00	15.52	0.6833	0.1956
Unique3D	General 3D	0.000	5.250	1.875	0.00	15.93	0.7015	0.1716
Ouroboros3D	General 3D	0.000	1.500	0.750	0.00	16.35	0.6933	0.1699
TRELLIS	General 3D	0.000	33.250	19.375	2.50	17.28	0.7188	0.1513
SiTH	Human-focused	1.250	2.250	1.875	0.00	18.09	0.8219	0.1231
IDOL	Human-focused	0.250	0.000	0.250	0.00	17.73	0.7968	0.1584
MultiGO	Human-focused	6.375	2.250	2.125	1.25	18.38	0.8006	0.1330
PSHuman	Human-focused	23.125	4.625	9.625	3.75	19.47	0.8420	0.1104
PartCHOI (Ours)	Interaction-aware	69.000	46.250	61.875	92.50	20.04	0.8532	0.0967

Ablation Study

The ablation study systematically quantifies the contribution of each module toward rendering fidelity and penetration reduction. The Penetration Ratio (PR) is defined as the fraction of sampled points inside the object volume that exhibit negative signed distance (SDF < 0) relative to the human mesh.

Config	PSNR↑	SSIM↑	LPIPS↓	CLIP↑	PR(%)↓	Note
Baseline	17.25	0.7375	0.1438	0.2601	61.5	Both human and object placed naively at origin
+ H-GPE	19.55	0.8429	0.1001	0.2927	10.4	Human-guided closed-form pose alignment
w/ PAR w/o O-GRM	20.10	0.8528	0.0956	0.2955	9.1	Accurate human pose without contact refinement
w/o PAR w/ O-GRM	19.49	0.8520	0.0977	0.2932	6.5	Surface deformation without pose refinement
Full model	20.04	0.8532	0.0967	0.2962	4.9	Complete framework achieving lowest penetration

In terms of execution time, while optimization-based compositional baselines like LayoutDreamer (65 min) and Flash Sculptor (6 min) are extremely slow, PartCHOI requires only 7 seconds for H-GPE, 25 seconds for O-GRM, and 1.5 minutes for PAR, bringing the total scene generation runtime to ~3 minutes (160–180 seconds).

Key Findings

• H-GPE is foundational to realistic spatial layouts: Naively placing entities at the origin produces a disastrous 61.5% penetration ratio, which drops sharply to 10.4% upon introducing H-GPE, proving the efficacy of SVD-based voxel alignment in single-shot spatial layout recovery.
• O-GRM substantially improves physical contact plausibility: Incorporating O-GRM cuts the penetration ratio from 9.1% down to 4.9% (a >46% relative reduction). Although subtle local deformations slightly lower reprojection metrics (PSNR shifts from 20.10 to 20.04), it completely eliminates glaring geometric intersections.
• PAR resolves pose drift and geometric inconsistencies: Disabling PAR causes PSNR to drop from 20.04 to 19.49, demonstrating that grounding SMPL-X parameters against native 3D generative voxels effectively corrects 2D monocular pose estimation errors.

Highlights & Insights

• Conceptual Shift to Part-Aware Scene Generation: By treating interacting objects as parts of the human body, the method adapts part-level 3D generative concepts to complex multi-entity compositional generation, neatly sidestepping brittle global layout optimization.
• Closed-Form Pose Inference via Voxel Geometry: Harnessing 3D generative foundation models (TRELLIS) for coarse spatial occupancy and coupling it with SMPL-X ray-casting turns an intractable layout search problem into a single SVD operation taking only 7 seconds.
• Jacobian Deformation as a Physical Proxy: Leveraging per-face Poisson surface deformations guided by SDF constraints successfully mimics cloth compression under contact, producing physically convincing human-object interaction interfaces without running slow cloth physics simulators.

Limitations & Future Work

• Voxel Ambiguity under Heavy Occlusion: Ray-casting and voxel filtering rely heavily on 2D VLM occlusion reasoning. Complex, multi-limb intertwined occlusions can introduce boundary noise during voxel assignment.
• Absence of Material-Aware Physical Simulation: Contact deformation is purely driven by geometric signed distance relaxation, without explicitly modeling cloth material properties such as Young's modulus, Poisson's ratio, or gravity-induced draping.
• Future Directions: Extending the part-aware framework to dynamic 4D video generation with two-person or multi-agent physical interactions, and coupling geometric deformation fields with lightweight neural physics engines.

Related Work & Insights

• vs PSHuman [17]: PSHuman excels at isolated clothed human reconstruction but collapses in interactive scenes, misinterpreting held objects as severe body deformities or missing limbs. PartCHOI preserves PSHuman's photorealistic human synthesis while enabling interaction modeling via part-aware decoupling.
• vs TRELLIS [39]: While TRELLIS reconstructs unified 3D scenes with interaction context, its geometry is coarse, lacks fine details, and fuses human and objects into an un-editable mesh. PartCHOI uses TRELLIS strictly as a coarse structural and pose prior.
• vs ComboVerse [4] / Flash Sculptor [13]: These compositional scene generation methods rely on iterative score distillation or bounding box layout optimization, taking 6–65 minutes and regularly suffering from floating or penetration. PartCHOI achieves cleaner physical contact in under 3 minutes via closed-form H-GPE and O-GRM.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant human-centered part-aware formulation that bypasses iterative layout optimization with closed-form SVD.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive multi-view rendering metrics, penetration ratio analysis, human user studies, and automated GPT-5 evaluations.
• Writing Quality: ⭐⭐⭐⭐⭐ Structured problem exposition, clear mathematical explanations, and well-designed comparative diagrams.
• Value: ⭐⭐⭐⭐⭐ Establishes an efficient, training-free, and editable pipeline for single-view 3D clothed human-object interaction generation.