HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human–Scene Interactions

Conference: ECCV2026
Paper: ECCV Paper
Project: HSImul3R
Area: 3D Vision
Keywords: human-scene interaction, physical simulation, 3D reconstruction, motion tracking, generative geometry prior

TL;DR

HSImul3R turns the simulator into a reconstruction supervisor, using scene-targeted reinforcement learning to refine human motion and direct simulation reward optimization to refine object geometry, raising easy-case HSIBench interaction stability from HSfM's 10.52% to 53.68%, although hard-case stability remains only 13.92%.

Background & Motivation

Visual reconstruction accuracy does not guarantee executable physical interaction: an image of a person sitting on a chair may reproject accurately while the chair lacks legs or the body penetrates its seat. Once this reconstruction enters a physics engine, collision responses may push the chair away. The person and chair can eventually stand independently, but the original sitting interaction is lost. Checking only whether the final state is stationary would incorrectly count this failure as success.

DUSt3R recovers environmental geometry, methods such as 4DHumans estimate people, and HSfM further aligns both in image space. However, under occlusion, 2D constraints cannot fully identify true contact and support relationships. Even replacing geometry with watertight image-to-3D outputs leaves bumps or missing slender supports that undermine load bearing. Optimizing human motion alone cannot repair those object defects.

Core Idea: use simulation feedback to supervise both human control and object generation, learning contact-preserving motion in the forward direction and interaction-supporting geometry in the reverse direction, rather than treating simulation as a visualization step after reconstruction.

Method

Overall Architecture

The standard input consists of 4 uncalibrated sparse views, and the output is human motion and scene geometry validated in simulation. Reconstruction and alignment with an explicit 3D structural prior come first. Optimization then proceeds in two directions: refine motion under fixed geometry, and use simulation outcomes to fine-tune the object generator.

“Bidirectional” does not mean end-to-end back-propagation through the entire simulator: the forward direction uses reinforcement learning, while the reverse direction trains a generative model with discrete stability labels. The paper describes both directions but does not provide a scheduling specification for reproducing an arbitrary number of alternating optimization rounds.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["4-view images"] --> Align["Structural Prior and<br/>3D Alignment"]
    Align --> Motion["Scene-Targeted<br/>Motion Optimization"]
    Motion --> Geometry["Simulation-Reward<br/>Geometry Optimization"]
    Geometry -->|Update object generator| Align
    Motion --> Output["Simulation-validated interactions<br/>and downstream robot deployment"]

Key Designs

1. Structural Prior and 3D Alignment: place humans and objects in a shared, contact-compatible space

DUSt3R estimates scene point maps and cameras, SAM2 associates people and identities across frames, 4DHumans recovers SMPL motion, and ViTPose extracts 2D keypoints. Human-centric bundle adjustment and global human-scene alignment minimize keypoint reprojection error to reconcile independently reconstructed coordinate systems. These initialization steps build on HSfM rather than training a new unified visual backbone.

Point maps can nevertheless contain open surfaces or disconnected supports, while image-space alignment cannot fully constrain occluded contact. For each object, the method selects the clearest input view, segments it with SAM, and passes it to pretrained MIDI to generate structurally more complete explicit geometry. The generative prior therefore supplies geometric completion, not merely better texture or rendering.

Human and object positions are subsequently optimized together. Before contact, distances between the nearest human body part and object vertices encourage proximity; after contact, negative signed-distance values penalize human vertices inside the object. The cached layout of Eq. (3) has misaligned indices, so its exact summation is not reconstructed here. Eq. (4) clearly specifies a nonnegative penetration-depth penalty. These 3D constraints prevent small visual overlaps from becoming strong simulated collisions.

2. Scene-Targeted Motion Optimization: track contact relationships as well as the person

After initialization enters the simulator, conventional PHC-style motion tracking may reproduce an approximate action while kicking away the relevant object. The proposed method supplements motion-fidelity supervision with a scene-distance constraint that keeps contact keypoints near the corresponding local object surface. The objective is not only to match the demonstrated joints, but to perform the action at the correct support surface.

Eq. (5) averages squared Euclidean distances between contact keypoints and locally sampled surface points:

\[ \ell_{\mathrm{scene}}= \frac{1}{N_{\mathrm{contact}}N_{\mathrm{surf}}} \sum_{j=1}^{N_{\mathrm{contact}}}\sum_{i=1}^{N_{\mathrm{surf}}} \|\mu_i^o-k_j^h\|_2^2. \]

Here, \(k_j^h\) is a human contact keypoint and \(\mu_i^o\) is an object surface sample in the corresponding local contact region. The two counts denote contacts and local surface samples. Although the prose calls this Euclidean distance, the displayed equation explicitly uses a squared norm; this note follows that equation. Since an object center is not the contact location on a seat or tabletop, the authors also compare against a CLoSD-style center-distance alternative.

This stage holds scene geometry fixed while adapting motion control to contact and collision. It can correct movement, but cannot restore the support structure of a chair with missing legs, motivating reverse geometry optimization.

3. Simulation-Reward Geometry Optimization: fine-tune object generation using interaction outcomes

Direct Simulation Reward Optimization, or DSRO, follows the idea of DSO and generates binary training labels from simulation. A positive example must satisfy all three conditions: the object is stable under gravity alone, the human-scene system reaches a stable final state, and actual human-object contact remains. Failing any condition makes it negative. Figure 3 distinguishes gravity instability, interaction instability, stability without meaningful interaction, and stable interaction; only the last category receives a positive label.

This labeling specifically rejects the shortcut in which the person and object stand independently on the ground. The generator must consequently learn more than an upright object: it must preserve local geometry suitable for leaning, sitting, or support. Missing slender legs and occlusion-induced surface bumps are explicit failure modes targeted by the method.

Eq. (6) inserts the binary label into a signed denoising objective:

\[ \ell_{\mathrm{DSRO}}=-T\,\mathbb{E}_{I\sim\mathcal{I},\,x_0\sim\mathcal{X}_I,\,t\sim\mathcal{U}(0,T),\,x_t\sim q(x_t\mid x_0)} \left[w(t)(1-2l(x_0))\|\epsilon-\epsilon_\theta(x_t,t)\|_2^2\right]. \]

Here, \(x_0\) is an image-conditioned generated 3D sample, \(x_t\) its noised version, \(w(t)\) a time weight, and \(l(x_0)\) equals 1 for stable samples and 0 otherwise. For positives, the outer minus sign cancels the label factor, so minimization reduces denoising error. For negatives, the direction reverses to suppress those generated outcomes. The generator thus uses nondifferentiable simulation feedback without differentiating contact dynamics.

A Worked Example

Consider sitting on a chair as an explanatory walkthrough, not an additional experiment. Initialization from 4 views provides body pose, chair placement, and cameras. After MIDI generates the chair mesh, 3D alignment reduces penetration into the seat, and PHC-style tracking uses the local surface-distance term to preserve sitting contact.

If defective legs cause the chair to fall under gravity or human loading, the sample becomes a DSRO negative. If the chair is pushed away while the person stands alone, it remains negative. Only a stable outcome that preserves sitting contact is positive and feeds back to the generator, rather than simply retaining whichever candidate best reprojects into the images.

Loss & Training

DSRO starts from pretrained MIDI and uses AdamW with LoRA rank 64, batch size 1, a learning rate of \(10^{-5}\), and 3000 training steps on 4 NVIDIA A100 GPUs. These settings do not specify step time or the complete reinforcement-learning reward weights, which should not be inferred.

HSIBench contains 300 interaction cases captured from 16 views each. For each case and view, the authors run reconstruction and simulation with 15 different random seeds to collect supervision. Multi-view 2DGS and SMPL estimation provide pseudo ground truth for geometry and motion, respectively, rather than precision scans paired with motion-capture ground truth.

The monocular-video extension uses MegaSAM for scenes and TRAM for humans, with SAM2 bounding boxes assisting interaction identification and dynamic alignment. It assumes a static scene in which only the human moves; this is not evidence of support for arbitrary dynamic-object scenes.

Key Experimental Results

Main Results

HSIBench was captured with 3 volunteers. The introduction reports 19 objects and more than 50 motion sequences; the dataset section specifically lists 8 chairs, 3 tables, and 3 sofas without individually explaining the remaining categories. Cases are divided into Easy, Medium, and Hard by interaction complexity, but the available main text does not specify clear thresholds or training/test split details.

Stability-HSI is the fraction passing gravity stability, final scene stability, and meaningful-contact checks. SP-3D measures reconstruction penetration; W-MPJPE measures world-coordinate joint error, while PA-MPJPE measures aligned pose error. Table 1 does not explicitly state units for these two motion errors, so neither centimeters nor millimeters are assumed here.

Method (paper Table 1)	Easy stability ↑	Medium stability ↑	Hard stability ↑	SP-3D ↓	W-MPJPE ↓	PA-MPJPE ↓
HSfM	10.52%	4.50%	2.66%	69.51%	5.02	2.79
V1: HSfM with MIDI watertight geometry	13.96%	8.81%	4.17%	77.12%	6.18	3.20
HSImul3R	53.68%	30.56%	13.92%	22.90%	4.09	2.17

Against HSfM, easy-case stability increases by 43.16 percentage points and penetration decreases by 46.61 percentage points. V1 instead increases penetration and motion errors, showing that watertight meshes do not automatically produce correct spatial alignment or physical contact.

Ablation Study

Config (paper Table 1)	Easy stability ↑	Medium stability ↑	Hard stability ↑	W-MPJPE ↓	PA-MPJPE ↓
V2: without bidirectional optimization	39.56%	22.71%	7.05%	4.91	2.71
V3: object-center distance replacing Eq. (5)	42.57%	23.84%	10.18%	4.60	2.42
V4: without reverse-pass DSRO	29.56%	16.62%	5.17%	4.57	2.39
Full model	53.68%	30.56%	13.92%	4.09	2.17

The full model improves easy-case stability by 24.12 percentage points over V4 and 11.11 percentage points over V3. However, V2 is more stable than V4, so these results do not establish monotonically improving performance as each component is added. The text does not adequately explain this nonmonotonicity or provide error bars.

Generator (paper Table 2)	Gravity stability ↑	Hard interaction stability ↑	Chamfer Distance ↓	F-Score ↑
MIDI	79.19%	5.17%	0.198	81.95
DSO*: also fine-tuned from MIDI	87.23%	7.88%	0.191	86.26
DSRO / Ours	91.50%	13.92%	0.173	88.25

DSO* uses MIDI instead of its original TRELLIS backbone to control initialization differences. DSRO improves gravity stability, interaction stability, and geometric metrics; nevertheless, 91.50% gravity stability is not a substitute for the 13.92% hard-interaction success rate.

Key Findings

• In Table 3, 4-view and 16-view easy-case stability are 53.68% and 55.16%, respectively, while hard-case stability is 13.92% and 13.59%. More views do not consistently improve interaction success.
• In the same comparison, PA-MPJPE decreases from 2.17 to 1.99 and SP-3D from 22.90% to 21.81%. Table 3's caption claims increased penetration with more views, which conflicts with these entries; the numerical values are retained here.
• Figure 7 demonstrates GMR retargeting to Unitree G1, whole-body policy training in IsaacGym, and deployment through the Unitree SDK. This is a qualitative hardware demonstration, not a robotics evaluation table reporting success rates and repeated trials.

Highlights & Insights

• Including meaningful contact in the success definition blocks the shortcut of separating the person and object. This evaluation is better aligned with interaction reconstruction than merely reducing collisions or checking whether an object stands upright.
• Physical feedback changes both movement and geometry, recognizing that not every failure originates in the controller. Motion correction cannot replace restoring a missing support structure.
• Local surface distance better represents a physical support location than object-center distance. The transferable principle is to constrain actions around functional interaction regions rather than substituting whole-object position for contact points.

Limitations & Future Work

• The authors acknowledge low success for complex interactions, especially with more than 3 objects. Hard-case stability is only 13.92%, so “simulation-ready” describes improved executable reconstruction capability rather than reliable handling of every input.
• A common failure still leaves the human and object standing independently without meaningful interaction. Finer contact states, support relationships, or sustained-interaction constraints are potential directions.
• Biases from MIDI's training distribution and HSIBench's 3 volunteers may limit generalization across object categories and human populations.
• The main text does not clearly report training/test splits, stability thresholds, motion-error units, full reinforcement-learning hyperparameters, or statistical variation. These gaps affect reproducibility and conclusion strength, while pseudo ground truth can also bias evaluation.
• The monocular extension assumes a static scene, and hardware results are primarily demonstrations. Dynamic movable objects and systematic repeated robot trials require separate validation.

Related Work & Insights

• vs HSfM: HSfM jointly recovers humans and scenes mainly through image-space consistency. This work adds structural priors, explicit 3D contact constraints, and simulation supervision to target executable interaction.
• vs PHC / CLoSD: PHC provides the motion-tracking foundation, while CLoSD motivates the center-distance comparison. The proposed local surface-distance term aims to prevent pose similarity from masking failed interaction.
• vs DSO / MIDI: MIDI supplies generative geometry priors and DSO motivates simulation-feedback optimization. DSRO additionally requires positive samples to preserve actual human-scene interaction, not merely gravity stability.
• Research implication: evaluate reconstruction error, contact relationships, and dynamic executability separately, then check whether they improve together. Better rendering or more input views do not directly establish more reliable physical interaction.

Rating

• Novelty: 4/5. Combines contact-aware control with interaction-label-driven geometry fine-tuning in a concrete bidirectional physics-supervision framework.
• Experimental Thoroughness: 3/5. Includes main comparisons, component ablations, generation quality, and view-count analysis, but split and statistical details are insufficient.
• Writing Quality: 3/5. The overall workflow is clear, with ambiguities in equation layout, captions, and ablation explanations.
• Value: 4/5. Offers a concrete route from visual reconstruction to embodied simulation, although difficult-interaction reliability remains a major obstacle.