Reconstructing Humans and Objects in Interaction using Large Reconstruction Models¶

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://ac5113.github.io/MILO
Area: 3D Vision
Keywords: 3D human-object interaction / Large Reconstruction Models / SMPL-H fitting / geometric scaffold / semantic correspondence

TL;DR¶

MILO reframes monocular 3D human-object interaction reconstruction as interpreting joint meshes from Large Reconstruction Models (LRMs) as non-parametric geometric scaffolds, decomposing them via multi-view segmentation, fitting a parametric SMPL-H body, and optionally aligning CAD templates to establish a new state of the art across benchmarks without contact supervision.

Background & Motivation¶

Daily human activities inherently involve complex physical interactions with diverse everyday objects, ranging from handheld utensils and smartphones to massive furniture and tools. Accurately recovering 3D human-object interactions (3D HOI) from a single RGB image is a cornerstone for immersive AR/VR experiences, robot manipulation learning, and embodied AI. However, monocular 3D HOI estimation remains an ill-posed and challenging inverse problem: single-view projection suffers from severe depth-scale ambiguities, generic objects exhibit extreme geometric diversity without universal parametric representations (unlike human bodies with SMPL), and severe mutual occlusions between people and objects frequently derail independent estimation pipelines.

Existing mainstream methods generally adhere to two paradigms: retrieving CAD models from pre-existing repositories and fitting them under 2D reprojection constraints, or relying on explicit contact maps to prevent interpenetration and unrealistic floating. Both strategies face severe scalability bottlenecks. Retrieval-based frameworks are bounded by the limited coverage of controlled lab datasets (such as BEHAVE and InterCap) and collapse when encountering unseen object geometries in the wild. Meanwhile, contact-regularized optimization depends heavily on privileged ground-truth contact labels or fragile contact predictors; in out-of-domain scenarios, predicted contacts become unreliable, causing joint optimization to diverge or collapse into physically implausible configurations.

This paper tackles the challenge from a fresh angle by investigating the emergent spatial capabilities of Large Reconstruction Models (LRMs). While prior work has confined LRMs to isolated object reconstruction, the authors observe that feeding a joint person-object image into an LRM produces a holistic non-parametric mesh that, despite surface noise and lack of kinematic rigging, faithfully preserves relative human-object arrangements and proximity cues. Core idea: reframe monocular 3D HOI reconstruction as explaining the joint interaction mesh from an LRM, treating it as a non-parametric geometric scaffold to fit a parametric SMPL-H body and segment/align object geometries without relying on contact annotations.

Method¶

Overall Architecture¶

Given a single RGB image and an instance segmentation mask (RGBA), MILO reconstructs a kinematically rigged SMPL-H human mesh and an aligned 3D object geometry (either as a segmented point cloud or a fitted CAD mesh). The pipeline operates in four major phases: first, an off-the-shelf LRM (Hunyuan3D-2.0) generates a combined human-object 3D mesh scaffold from the RGBA input; second, the mesh is rendered from 60 virtual viewpoints, where 2D keypoint detectors and robust multi-view consensus triangulation lift body and hand landmarks into reliable 3D keypoints; third, these 3D keypoints and visible-surface Chamfer constraints guide a two-stage SMPL-H optimization (root coarse alignment followed by anchored pose fitting); finally, multi-view semantic segmentation separates the object point cloud, which can optionally be aligned with a CAD template via semantic correspondences and iterative Sim(3) registration.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Single Input Image (RGBA)"] --> B["Combined Mesh Generation<br/>Hunyuan3D-2.0 produces joint geometric scaffold"]
    B --> C["Multi-view Rendering & 3D Keypoint Triangulation<br/>60 virtual views + ViTPose/HaMeR + consensus filtering"]
    C --> D["Two-stage Parametric Human Optimization<br/>Root coarse alignment + anchored pose fitting"]
    B --> E["Multi-view Point Cloud Segmentation<br/>Viewpoint quality & boundary-weighted voting"]
    D --> F["Template-free Mode: Composed Human Mesh & Object Point Cloud"]
    E --> F
    E --> G["Bidirectional Semantic Correspondence & Alignment<br/>Forward matching + cycle consistency + iterative Sim(3)"]
    G --> H["Template-based Mode: Composed Human Mesh & CAD Template"]

Key Designs¶

1. Combined Mesh Generation & Geometric Scaffold: Bypassing Monocular Depth Ambiguities

Prior methods typically reconstruct humans and objects independently and attempt to compose them in 3D camera space, often succumbing to relative scale and depth ambiguities. MILO fundamentally circumvents this by feeding the combined human-object RGBA image directly into Hunyuan3D-2.0 to generate a holistic interaction mesh. The resulting mesh is not treated as the final artifact but as a non-parametric geometric scaffold. Because the pre-trained LRM inherently infers occlusion boundaries, support surfaces, and spatial proximities from large-scale 3D training data, it provides a stable 3D reference frame that eliminates the need for heuristic placement heuristics.

2. Multi-view Robust 3D Keypoint Triangulation & Two-stage Human Optimization: Articulated Decoupling with Occlusion Anchoring

To transform the unrigged LRM mesh into an articulated, kinematically valid SMPL-H human model, MILO renders the mesh across 60 virtual camera viewpoints on a sphere. In each view, ViTPose extracts COCO-25 body keypoints while HaMeR estimates 2D hand keypoints. Across all view pairs, multi-view ray triangulation identifies the hypothesis with the largest consensus inlier set under a reprojection threshold, followed by non-linear bundle adjustment to yield sparse 3D keypoints $X \in \mathbb{R}^{67 \times 3}$ with aggregated confidence $\bar{c}$.

Human parameter optimization then proceeds in two stages, initialized from monocular predictions by HMR2.0 and HaMeR: The first stage is Root Fitting (30 iterations), which fixes pose and shape and optimizes only global orientation $\Phi$, root translation $\Gamma$, and a global scale factor against triangulated keypoints using a robust Geman-McClure penalty: $$\mathcal{L}_{rf} = \sum_{i} \rho(X_i^S - X_i)$$ The second stage is Pose Fitting (60 iterations), optimizing body pose $\Theta$, hand pose $\Theta_h$, and shape $\beta$. To prevent the human mesh from distorting when the LRM hallucinates occluded limbs, an anchoring loss penalizes large deviations from the initial HMR2.0 VPoser latent embedding $\zeta_{bp}^{\mathrm{hmr}}$ beyond a tolerance threshold $\tau=2.5$: $$\mathcal{L}_{ha} = \sum_{j} \max\left(0, |\zeta_{bp, j} - \zeta_{bp, j}^{\mathrm{hmr}}| - \tau\right)$$ Simultaneously, a one-way robust Chamfer loss aligns observed LRM surface points $P^{\mathrm{obs}}$ only to visible predicted SMPL-H vertices $P_{\mathcal{V}}^{\mathrm{pred}}$, preventing unobserved backside vertices from warping into ungrounded geometry.

3. Multi-view Point Cloud Segmentation & Boundary-Weighted Filtering: Clean Object Extraction

With the human body accurately fitted, the object geometry must be isolated from the shared scaffold. Because the human and object meshes are often topologically fused at contact interfaces, naive spatial distance cuts leave detached body fragments on the object surface. MILO renders the multi-view frames and prompts an open-vocabulary segmentation model (Grounding DINO + SAM) with person.<obj_name>. Vertices are labeled by projecting them into each view and aggregating visibility quality scores with a boundary-closeness penalty that downweights ambiguous contact boundaries. Connected-component filtering subsequently strips isolated floating artifacts, producing a clean object point cloud $P^{\mathrm{lrm}}$.

4. Semantic Correspondence & Iterative Sim(3) Alignment: Dual-Mode Versatility

When a target object CAD template is provided, MILO can register it to the segmented LRM point cloud. To bridge the domain gap between untextured CAD renders and textured LRM geometry, geometric-aware semantic features are extracted to compute mutual nearest-neighbor matches across top-ranked view pairs. Two-dimensional matches passing local cosine similarity and pixel distance criteria are lifted to 3D vertex pairs, refined through neighborhood-based reverse cycle consistency to yield sparse correspondence set $\mathcal{C} = \{(u_{i_k}, v_{j_k}, w_k)\}$. An initial similarity transform $T(u) = s\mathbf{R}u + \mathbf{t}$ is solved in closed form via weighted Kabsch, followed by iterative refinement combining semantic anchor constraints with nearest-neighbor surface points under a progressively tightening distance inlier threshold $\tau_t$.

Loss & Training¶

The entire pipeline runs at test-time optimization without retraining network backbones. The total loss for the pose fitting stage is: $$\mathcal{L}_{pf} = \lambda_{rf}\mathcal{L}_{rf} + \lambda_{bp}\mathcal{L}_{bp} + \lambda_{hp}\mathcal{L}_{hp} + \lambda_{\beta}\mathcal{L}_{\beta} + \lambda_{ha}\mathcal{L}_{ha} + \lambda_{3D}\mathcal{L}_{3D}$$ with hyperparameters set to $\lambda_{rf} = 10$, $\lambda_{bp} = \lambda_{hp} = 0.04$ (VPoser and MANO latent priors), $\lambda_{\beta} = 0.05$ ($L_2$ shape prior), $\lambda_{ha} = 10$ (human anchor loss), and $\lambda_{3D} = 50$ (visible-surface Chamfer distance).

Key Experimental Results¶

Main Results¶

MILO was comprehensively evaluated on three challenging 3D HOI benchmarks: InterCap, HODome, and IMHD. Evaluation follows the standard out-of-domain protocol where baseline models are trained on BEHAVE. Performance is measured using Procrustes-Aligned Chamfer Distance (PA-CD in cm) for human ($h$), object ($o$), and joint ($h+o$) meshes.

The quantitative comparison on InterCap is shown below (Table 1):

Methods	Type	Needs Template?	Needs Contact?	PA-CDh (cm) ↓	PA-CDo (cm) ↓	PA-CDh+o (cm) ↓
CONTHO [CVPR 2024]	Reg.	✓	✓	8.36	24.30	13.14
HOI-TG [CVPR 2024]	Reg.	✓	✗	8.22	25.05	14.63
PHOSA [ECCV 2020]	Opt.	✓	✓	10.07	23.36	13.38
Open3DHOI [ECCV 2024]	Opt.	✗	✗	6.88	31.18	10.17
PICO [CVPR 2025]	Opt.	✓	✓	7.43	21.85	10.33
Ours (w/ template)	Opt.	✓	✗	6.96	18.97	7.45
Ours (w/o template)	Opt.	✗	✗	6.85	20.74	9.36

On HODome (Table 2) and IMHD (Table 3), MILO similarly outperforms PICO by a wide margin: - HODome: Ours (w/ template) achieves PA-CDh+o of 6.68 cm and Ours (w/o template) achieves 6.38 cm, compared to PICO's 10.07 cm (a >33% error reduction). - IMHD: Ours (w/o template) achieves PA-CDh+o of 6.98 cm and Ours (w/ template) achieves 10.10 cm, substantially lower than PICO's 13.24 cm.

In downstream contact evaluation on InterCap (Table 4), contact inferred purely from MILO's 3D geometry achieves an F1 score of 30.00% and a geometric error of 39.57 cm, drastically outperforming the image-regressed DECO baseline (F1 of 9.37%, geometric error of 126.70 cm).

Ablation Study¶

To verify the core mechanisms, the authors conducted thorough ablation experiments on InterCap.

1. Joint Scaffold vs. Independent Reconstruction (Table 6)

Baseline	Contact Loss	PA-CDh (cm) ↓	PA-CDo (cm) ↓	PA-CDh+o (cm) ↓
EasyHOI* (full-body, LRM object + reprojection)	✗	41.57	77.06	37.21
EasyHOI* (full-body, LRM object + contact loss)	✓	9.48	27.93	12.08
SAM3D (independent, separate + MoGe alignment)	✗	18.18	31.07	15.89
Ours (Joint LRM Scaffold)	✗	6.85	20.74	9.36

2. Effect of Human Optimization Stages (Table 7)

Human Fitting Stage	Uses Template	PA-CDh (cm) ↓	PA-CDo (cm) ↓	PA-CDh+o (cm) ↓
No fitting (raw LRM human point cloud)	✗	30.58	23.53	14.10
Root Fitting	✗	7.98	22.53	10.15
Pose Fitting (Full model)	✗	6.85	20.74	9.36
No fitting (raw LRM human point cloud)	✓	14.97	37.02	14.57
Root Fitting	✓	7.79	18.63	7.99
Pose Fitting (Full model)	✓	6.96	18.97	7.45

Key Findings¶

The joint reconstruction scaffold is the primary performance driver: As demonstrated in Table 6, reconstructing human and object separately and aligning them via depth pointmaps yields a joint error of 15.89 cm, whereas using an object-only LRM without contact blows up to 37.21 cm. The joint LRM scaffold immediately drops the joint error to 9.36 cm without contact guidance.
Parametric SMPL-H fitting resolves non-parametric noise: As shown in Table 7, raw LRM human geometry suffers from mesh artifacts and missing anatomy, exhibiting a human error of 30.58 cm. Two-stage optimization brings this down to 6.85 cm while simultaneously reducing joint error from 14.10 cm to 9.36 cm.
Agnostic to specific LRM backbones: Table 5 demonstrates that swapping Hunyuan3D-2.0 (9.36 cm) for SAM3D joint (9.80 cm) achieves comparable accuracy, indicating that MILO is a general-purpose formulation ready to scale with future LRM advances.

Highlights & Insights¶

Paradigm shift in HOI estimation: Rather than viewing LRM 3D generation as an end product or restricting it to isolated object CAD synthesis, MILO recognizes its unique strength as a monocular geometric scaffold that encodes spatial layout, elegantly pairing it with classical kinematic optimization.
Liberation from privileged contact supervision: Prior state-of-the-art frameworks required ground-truth or pre-estimated dense vertex contact maps to constrain scale and penetration. MILO discards contact losses during optimization while outperforming all contact-dependent baselines and inferring superior contact retrospectively.
Seamless template-aware and template-free duality: MILO supports both open-world free-form object point cloud reconstruction and high-fidelity CAD alignment via cycle-consistent semantic matching, providing maximum flexibility across varied deployment settings.

Limitations & Future Work¶

Bottlenecked by upstream LRM geometry: In cases involving tiny handheld objects (e.g., pens, small keys) or severe physical occlusion, LRMs may struggle to generate distinct geometry, creating topological artifacts that downstream point cloud filtering cannot fully repair.
Point cloud segmentation relies on 2D vision models: Multi-view aggregation using Grounding DINO and SAM can occasionally produce jagged or ambiguous boundaries near fine-grained contact interfaces like finger grasps.
Static single-image scope: The current framework is designed for static single images without modeling temporal dynamics, mass, or friction, and is currently focused on single-human single-object scenarios.

vs EasyHOI [CVPR 2025]: EasyHOI uses LRMs strictly to generate the object mesh before optimizing hand-object spatial relationships via 2D reprojection and contact heuristics. MILO generates a holistic human-object scaffold, eliminating relative drift and scale collapse.
vs PICO [CVPR 2025]: PICO requires CAD template retrieval and relies heavily on contact maps to guide optimization. MILO operates without contact constraints and supports template-free reconstruction for arbitrary in-the-wild objects.
vs CONTHO [CVPR 2024]: CONTHO trains a neural contact regressor to refine joint meshes, suffering significant performance drops when evaluated out-of-domain. MILO leverages robust zero-shot foundation models (ViTPose, HaMeR, Hunyuan3D) with test-time optimization, achieving superior cross-dataset generalization.

Rating¶

Novelty: ⭐⭐⭐⭐☆ [Pioneering use of LRMs as joint geometric scaffolds for monocular HOI reconstruction]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive multi-benchmark evaluation, rigorous scaffold isolation ablations, and contact benchmarking]
Writing Quality: ⭐⭐⭐⭐⭐ [Clean, highly structured exposition with clear mathematical formulations and intuitive figures]
Value: ⭐⭐⭐⭐⭐ [Provides a practical and powerful blueprint for integrating foundation 3D models with embodied human understanding]