ComplexMimic: Human–Scene Interaction Imitation in Complex 3D Environments

Conference: ECCV2026
Paper: Official page · PDF
Code: https://github.com/LuPan23/ComplexMimic
Area: Robotics / Human-Scene Interaction / Physics-Based Character Control
Keywords: whole-body control, dual experts, difficulty-aware distillation, learnability filtering, motion capture

TL;DR

ComplexMimic trains a strict motion-tracking expert and a collision-adaptive interaction expert, then routes supervision by motion regime and prioritizes difficult trajectories that still improve, distilling a single controller with 90.6% success on TRUMANS while retaining cross-domain pose-error trade-offs.

Background & Motivation

Motion capture describes how human joints move, but does not guarantee that a simulated body with mass and contact constraints can execute the recorded trajectory. Dense furniture, narrow passages, and irregular meshes can turn small penetrations or alignment errors in the reference into consequential collisions. Strong tracking in scene-free imitation or single-object interaction therefore need not transfer to cluttered indoor environments. The task here is not text-to-motion generation: it is closed-loop control under a given reference trajectory, preserving its intent while respecting scene constraints.

The paper connects this difficulty to the early-termination threshold used during training. Ending an episode as soon as tracking drifts encourages precision, but can terminate exploration before the policy learns to accommodate an obstacle. Relaxing the threshold allows more feasible interactions, yet also permits the motion to diverge from the reference. A single threshold cannot supply both forms of supervision, and averaging two teachers' actions does not guarantee a valid compromise between collision avoidance and fidelity.

The proposed response is to first establish complementary expertise, then decide which teacher each motion should follow. Meanwhile, repeated failure alone does not make a trajectory worth training on: a reference that conflicts severely with the scene may remain infeasible. Core idea: separate motion fidelity and scene feasibility into two experts, then combine fixed motion-level teacher routing with progress-dependent hard-motion sampling to learn a unified controller through online physical interaction.

Method

Overall Architecture

Inputs are the MoCap reference motion, current simulated humanoid state, and 3D scene. Outputs are control actions executed by the physics simulator, producing a dynamically constrained human-scene interaction sequence. Training has a teacher stage and a student stage; inference uses one student rather than requiring both experts to make online decisions.

Dual-flow expert training supplies two capabilities: the imitation expert tracks strictly without scene meshes, while the interaction expert adapts to full-scene geometry using a local height map. Before distillation, the imitation expert is evaluated on each training motion in the scene-aware simulator to establish teacher routing. During student training, failure history and return improvement update sampling priorities, and teacher-distribution supervision is combined with the student's own PPO updates.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Reference motion and scene"] --> B["Dual-flow expert training<br/>Imitation / interaction experts"]
    B --> C["Motion-level teacher routing<br/>Evaluate imitation expert in scene"]
    C --> D["Difficulty and learnability sampling<br/>Inter-group / intra-group selection"]
    D --> E["Distillation and online RL<br/>Frozen teachers / student updates"]
    E -->|Failure and return feedback| D
    E --> F["Single student controller<br/>Physical human-scene interaction"]

Key Designs

1. Dual-flow expert training: give precise tracking and collision adaptation their own learnable conditions

The imitation expert uses a physics simulator without scene meshes and an early-termination threshold of \(\tau=0.25\). The interaction expert uses full scene meshes and \(\tau=0.5\). Both are trained with imitation rewards and PPO; they differ in scene constraints, observations, and tolerated tracking deviation. Scene-free training does not remove physics. It removes interference from complex scene meshes so the teacher can first learn faithful control, while the scene-aware teacher can deviate moderately to discover executable motions.

The interaction expert additionally observes a character-centered, heading-aligned \(20\times20\) height grid with 8 cm between neighboring samples. This local representation supports geometric awareness, but the full-scene simulator still handles physical collisions: the height map is not the collision model. Both experts are MLPs with hidden sizes 2048, 1024, 1024, and 512. Their specialization therefore primarily comes from training conditions, not from choosing two different network architectures.

2. Motion-level teacher routing: request supervision only from the expert suited to the trajectory

Before distillation, the imitation expert is evaluated on all training motions in the scene-aware simulator. Successfully tracked motions enter the tracking-friendly group; failed motions enter the interaction-critical group. The first group follows the imitation expert and the second follows the interaction expert. This assignment depends on execution outcomes rather than textual action labels, and it is not a fresh teacher choice at every timestep. The paper does not describe rebuilding the groups during distillation.

The student still observes its own state and the height map, but its target is the selected teacher's action distribution evaluated at the student's current state. The imitation teacher does not receive the height map, whereas the interaction teacher does. Hard routing avoids repeatedly giving one motion inconsistent teacher targets, and avoids assuming that the average of two control actions is a feasible intermediate solution. Routing determines whom to learn from; sampling determines how often to learn from each motion.

3. Difficulty and learnability sampling: prioritize hard motions without letting stagnant ones monopolize training

Every \(K\) iterations, the student is evaluated on the motions. A motion fails if its average joint distance from the reference exceeds 0.5 m at any timestep. An exponential moving average tracks difficulty, preventing one isolated failure from immediately dominating the sampling distribution:

\[ D_k(m)=(1-\beta)D_{k-1}(m)+\beta f_k(m),\qquad f_k(m)\in\{0,1\}. \]

Here \(m\) is a motion clip, \(k\) is an evaluation checkpoint, and \(f_k\) is the failure indicator. Mean difficulty within each teacher group first determines a softmax distribution over groups. A second softmax then chooses a motion inside the selected group. This handles both a generally harder interaction regime and a small set of particularly difficult motions within a regime, instead of spending most updates repeating solved trajectories.

Difficulty alone would favor infeasible or heavily corrupted references. The method therefore also tracks \(I_k(m)\), an EMA of the return increment \(R_k(m)-R_{k-1}(m)\), and downweights motions below a progress threshold. The following sampling relations are reconstructed from the surrounding explanation; some symbols in the cached equations are damaged, so these are not presented as verbatim, fully verified transcriptions:

\[ \begin{aligned} s_k(m)&=D_k(m)-\lambda_u\max(0,\epsilon_u-I_k(m)),\\ p_k(m\mid c)&\propto\exp\!\left(\tau_{\mathrm{intra}}s_k(m)\right),\\ p_k(m)&=(1-\epsilon)p_k(c(m))p_k(m\mid c(m))+\frac{\epsilon}{|\mathcal M|}. \end{aligned} \]

Here \(c(m)\) is the fixed teacher group and \(\mathcal M\) is the training-motion set. The first line expresses the downweighting mechanism described as \(s_k=D_k+\log u_k\). The final uniform mixture preserves opportunities to revisit difficult or temporarily stagnant motions. Thus, learnability filtering is soft downweighting rather than permanent removal; it does not equate low return with proof that a motion can never be learned.

4. Distillation and online RL: the student must refine behavior on the states it actually creates

The student is a four-layer Transformer encoder with 4 attention heads, hidden size 512, and feed-forward size 2048. Both teachers are frozen. The student executes its own actions in the scene-aware simulator, receiving teacher supervision and PPO updates from environmental returns. When it drifts into states different from those visited by the teachers, actual contact and failure signals can still refine its behavior instead of restricting learning to a static teacher dataset.

Policies are Gaussian action distributions, and distillation uses KL divergence from the student to the selected teacher. The expression below combines the two routing branches of the paper's Equation (3) into equivalent notation; it is not an additional loss:

\[ \mathcal L_{\mathrm{KD}}=\mathbb E_t\!\left[D_{\mathrm{KL}}\!\left(\pi_S(\cdot\mid s_t,hm_t)\,\|\,\pi_{T(m)}(\cdot\mid o_t^{T(m)})\right)\right]. \]

\(T(m)\) denotes the motion's assigned teacher, whose observation \(o_t^{T(m)}\) includes or excludes the height map according to expert type. The key is the coexistence of distribution supervision and online student interaction, not replacing the entire control objective with KL. Extraction of the PPO equation and signs in the joint objective is incomplete, so an unverified total-loss expression is not reproduced. What is supported is that PPO improves environmental return while KL constrains the student to retain teacher expertise.

A Worked Example

Consider an illustrative reference that passes beside furniture, not an additional measured experiment. The scene-free imitation expert first learns the original trajectory. If its average joint deviation crosses 0.5 m when evaluated with full meshes, that motion is assigned to the interaction expert. The student then executes under scene constraints and learns from the height-map-aware teacher rather than mechanically following a reference that may intersect furniture.

Repeated student failures increase the difficulty EMA and training priority. If returns continue improving, the policy may be learning obstacle accommodation or balance recovery, and the motion remains a useful hard example. If the reference conflicts severely with the scene and returns stop improving, the learnability term lowers its weight. The teacher group does not switch with this failure history; the student's sampling probability changes, while the 0.05 uniform mixture retains a chance to retry.

Loss & Training

All policies train only on TRUMANS. Motions are segmented into 121-frame clips, with 10% randomly selected for testing. LINGO evaluates out-of-domain, unseen-scene conditions; GIMO evaluates real scanned scenes with reconstruction noise. The latter still uses simulated execution, not physical robot experiments.

Implementation uses one NVIDIA 4090 and Isaac Gym, with control at 30 Hz and simulation at 60 Hz, and the mean SMPL body shape. The student's termination threshold stays at 0.5. The critic is an MLP with hidden sizes 1024 and 512. Both inter-group and intra-group softmax scale parameters are 1.0; the uniform mixture is 0.05, progress threshold \(\epsilon_u=0.1\), and penalty strength \(\lambda_u=1.0\).

The text reports EMA settings of 0.95 and 0.9 for failure and return statistics, without clearly mapping that convention to the new-sample weight in the recurrence. This note therefore does not assert \(\beta=0.95\). The cache contains the main paper and references, not the supplement. Reward components, evaluation interval \(K\), full training budget, and joint-loss weights cannot be verified here and are not invented.

Key Experimental Results

Main Results

Succ is the fraction of successfully tracked sequences; a sequence fails if average joint distance exceeds 0.5 m at any timestep. \(E_{g\text{-mpjpe}}\) measures global joint-position error and \(E_{\mathrm{mpjpe}}\) measures root-relative error. Acceleration and velocity differences use mm/frame² and mm/frame, respectively. Main-paper table headers do not explicitly state units for the two position errors, so their original values are retained without supplying an unverified unit. All baselines receive height maps at the same scale as the proposed method.

The table selects PHC, MaskedMimic, and the full model from original Tables 1–3, covering all three evaluation distributions. It does not combine different baselines' best metrics into a fictitious comparator.

Dataset	Method	Succ ↑	Global MPJPE ↓	Root-relative MPJPE ↓	Acceleration diff. ↓	Velocity diff. ↓
TRUMANS	PHC	0.853	85.793	65.365	13.559	12.055
TRUMANS	MaskedMimic	0.872	120.260	105.099	26.913	24.717
TRUMANS	ComplexMimic	0.906	77.840	64.128	12.931	10.721
LINGO	PHC	0.615	120.472	84.782	23.648	20.383
LINGO	MaskedMimic	0.719	163.525	132.315	38.387	36.499
LINGO	ComplexMimic	0.746	124.665	90.575	20.641	17.303
GIMO	PHC	0.443	173.580	109.696	47.151	38.556
GIMO	MaskedMimic	0.405	286.629	191.210	69.821	68.460
GIMO	ComplexMimic	0.579	161.969	104.590	36.845	29.749

On TRUMANS, success increases by 3.4 percentage points over MaskedMimic, approximately 3.90% in relative terms. Global error decreases by approximately 9.27% relative to PHC, calculated from the table. The introduction's 9.36% does not match 85.793 → 77.840; this note uses the reproducible table calculation and explicitly distinguishes the baselines for these two gains.

Ablation Study

The following TRUMANS ablations come from original Table 4. Root-relative error and acceleration difference are retained to avoid turning a strong overall result into a claim of superiority on every metric.

Configuration	Succ ↑	Global MPJPE ↓	Root-relative MPJPE ↓	Acceleration diff. ↓	Velocity diff. ↓
Without imitation expert	0.876	86.059	64.487	13.170	11.477
Without interaction expert	0.847	82.489	62.464	12.565	11.416
Without inter-group sampling	0.899	82.045	62.816	12.779	10.918
Without intra-group prioritization	0.898	80.735	61.016	13.349	11.248
Without learnability filtering	0.877	88.497	67.465	14.231	11.876
Without PPO	0.830	110.880	79.460	18.788	16.475
Random teacher routing	0.874	83.016	63.715	12.625	10.945
Teacher action averaging	0.873	83.698	64.075	12.712	11.030
Full model	0.906	77.840	64.128	12.931	10.721

Key Findings

• Online PPO has the largest success-rate effect among these ablations: removing it loses 7.6 percentage points and increases global error from 77.840 to 110.880. Teacher distillation alone does not adequately address the student's own state distribution.
• Removing the interaction expert loses 5.9 percentage points of success; removing the imitation expert increases global error to 86.059. The supervision is complementary, but removing interaction supervision improves some local-error metrics, so the trade-off is not eliminated.
• Removing learnability filtering loses 2.9 percentage points and raises global error to 88.497. Hard-example mining benefits from tracking continued progress rather than failure rate alone.
• On LINGO, the full model outperforms PHC in success but has higher errors under both position metrics. GIMO's strongest external success baseline is actually AMP at 0.446; the proposed 0.579 is 13.3 percentage points higher. This does not establish superiority over every configuration on every metric.
• The original no-DAS variant has a better TRUMANS acceleration difference, 12.759 versus 12.931, and a better GIMO root-relative error, 102.009 versus 104.590. The defensible claim is a stronger overall compromise.

Highlights & Insights

• Turning termination-induced behavioral preferences into teacher specialization is more interpretable than searching for one compromise threshold. Scene constraints change not just the input features, but which behaviors can be explored during training.
• Teacher routing and curriculum sampling address separate problems: conflicting supervision and misallocated training effort. This separation is useful for multi-teacher control tasks with different feasibility regimes.
• Failure EMA and return improvement jointly define hard-but-learnable motions without manual difficulty labels. Uniform mixing prevents early difficulty judgments from permanently excluding some motions.

Limitations & Future Work

• The authors do not provide a dedicated limitations section; the following points mostly analyze the experimental scope. Scanned scenes are still evaluated in simulation, with no physical robot deployment and only the mean SMPL shape. Cross-body and hardware robustness are not established.
• TRUMANS uses a random 10% clip holdout. The main text does not establish strict separation of source sequences or scenes. In-domain performance should not be treated as unseen-scene generalization; the main cross-domain evidence comes from LINGO and GIMO.
• Success checks a mean joint-distance threshold, not an independent collision-free rate, contact-correctness score, or semantic task-completion rate. The main text also does not specify whether errors are computed only over successful clips. Contact forces, penetration depth, and scene-specific failure categories would help.
• Stagnant return can indicate infeasibility or simply a need for longer exploration. Soft downweighting and uniform mixing only mitigate this ambiguity. Longer windows, uncertainty estimates, or explicit reference repair are possible extensions, not validated results.
• The cache lacks the supplement, repeated-run variance, and complete training-time information, limiting assessment of sampling's cost-benefit ratio. The height map's ability to represent complicated 3D occlusion structures also warrants direct evaluation.

Related Work & Insights

• Versus DeepMimic and AMP: the former learns physical skills with reference rewards, while the latter introduces adversarial motion priors. ComplexMimic focuses on conflicts between reference fidelity and complex scene constraints, not on introducing a new general motion prior.
• Versus PHC and MaskedMimic: these supply strong tracking or flexible whole-body control. ComplexMimic creates complementary teachers through different training conditions and distills them into one student. Since every baseline receives a height map, the result cannot simply be attributed to only this method observing the scene.
• Versus motion generation in LINGO and TRUMANS: this work uses recorded motions as references and studies closed-loop physical execution rather than replacing a text-to-motion generator. Generation and execution could be connected as upstream and downstream stages, but that combination is not quantified here.
• Versus UNIHSI and TokenHSI: these emphasize contact-chain or task-representation-driven interaction control, whereas this paper emphasizes faithful reproduction of a supplied trajectory. The transferable idea is to use feasibility differences in supervision assignment instead of introducing a generic action averager.

Rating

• Novelty: 4/5. Dual-flow training, motion-level routing, and learnability-aware sampling form a targeted combination, although PPO, KL distillation, and hard-example mining are established tools.
• Experimental Thoroughness: 4/5. Three datasets, four external baselines, and component ablations support the central claim; variance, training cost, and real deployment remain unverified.
• Writing Quality: 3/5. Motivation and module relationships are clear, but blanket best-on-all-metrics statements and some improvement figures require correction against the tables.
• Value: 4/5. The approach offers a transferable training strategy for physical imitation under noisy references and complex scenes, within the demonstrated simulation scope.