Training-free Controllable Motion Generation under Heterogeneous Constraints

Conference: ECCV 2026
Paper: ECCV Official
Area: Human Understanding
Keywords: motion generation, controllable diffusion model, training-free control, stochastic optimal control, heterogeneous constraint coordination

TL;DR

This paper introduces Motion-Inference-as-Control (MIC), a training-free controllable motion generation framework that casts the diffusion sampling process as a stochastic optimal control problem, unifying non-differentiable criterion-based rules and differentiable objective-based constraints within a shared control interface while resolving multi-constraint conflicts via bilevel coordination.

Background & Motivation

Synthesizing text-driven 3D human motion is essential for virtual reality, video game character animation, and whole-body control of humanoid robots. In practical scenarios, synthesized motions are routinely required to satisfy diverse real-world constraints, ranging from environmental boundaries and physical feasibility to target keyposes. Traditional controllable generation pipelines predominantly rely on constraint-specific training or task-tailored fine-tuning, which severely limits scalability whenever deployment requirements or environmental layouts shift. Consequently, training-free controllable motion generation—leveraging pre-trained motion diffusion models at inference time—has emerged as a highly flexible and practical paradigm.

However, real-world motion constraints are intrinsically heterogeneous across mathematical formulations and spatial-temporal scopes. Some constraints are continuous and objective-based, naturally parameterized by smooth, differentiable loss functions (e.g., reaching a target Cartesian waypoint or maintaining a designated velocity). In contrast, a vast array of practical requirements are criterion-based, providing feedback only through discontinuous events, discrete thresholds, or external black-box evaluators (e.g., safety boundary crossings, contact logic, or physical simulation validity in MuJoCo). Prior training-free methods fundamentally depend on back-propagating analytic gradients, forcing them either to abandon criterion-based rules or to rely on fragile, hand-crafted continuous proxies. Furthermore, when multiple heterogeneous constraints are applied concurrently to a single sequence, disparate signal magnitudes and mismatched spatial-temporal scopes cause dominant constraints to crush weaker ones and local joint adjustments to distort global posture coherence.

Addressing both the non-differentiability of criterion-based feedback and the interference among concurrent constraints requires shifting beyond heuristic test-time gradient patching toward a principled dynamical systems formulation. Core idea: cast inference-time diffusion motion generation as a stochastic optimal control problem, where Hamilton-Jacobi-Bellman theory and Stein's lemma yield a gradient-free expectation control law that handles criterion-based rules using only forward evaluations and naturally subsumes differentiable objectives, augmented by a bilevel coordination mechanism featuring integral feedback regulation and spatial-temporal control allocation.

Method

Overall Architecture

MIC (Motion-Inference-as-Control) reformulates the reverse diffusion stochastic differential equation (SDE) as a controlled dynamical system governed by step-wise external control inputs. At each denoising step, MIC seeks an optimal control law that minimizes terminal constraint violation penalties while penalizing excessive deviations from the pre-trained natural motion prior. Control signals derived from heterogeneous constraints are subsequently reconciled via a bilevel coordination module—comprising a closed-loop feedback regulator and a spatial-temporal control allocator—before being injected into the diffusion drift to steer the sampling trajectory.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Text Prompt + Heterogeneous Constraints<br/>Criterion-based rules/simulations & differentiable objectives"] --> B["Unified Stochastic Optimal Control Law<br/>Path-integral expectation & analytic gradient formulation"]
    B --> C["Step-wise Control Instantiation<br/>Cross-entropy importance sampling & Tweedie clean motion estimation"]
    C --> D["Bilevel Constraint Coordination<br/>Integral feedback regulation & spatial-temporal weighted least-squares allocation"]
    D --> E["Controlled Diffusion Sampling Step<br/>Constraint fulfillment while preserving natural motion prior"]

Key Designs

1. Unified Stochastic Optimal Control Law: Harmonizing forward evaluations and analytic gradients To bridge the mathematical divide between non-differentiable criteria and differentiable targets, MIC defines the reverse diffusion dynamics under forward time as: $$ \mathrm{d}\mathbf{z}t = \mathbf{b}(\mathbf{z}_t, t)\mathrm{d}t + \sigma(t)\big(\mathbf{u}_t(\mathbf{z}_t, t)\mathrm{d}t + \mathrm{d}\mathbf{\varepsilon}_t\big) $$ where \(\mathbf{z}_t = \mathbf{x}_{T-t}\) is the evolving state and \(\mathbf{u}_t\) denotes the external control guidance. Optimal control aims to minimize a cumulative cost combining terminal constraint violations \(\mathcal{E}(\mathbf{z}_T)\) and running control effort. Applying the Hamilton-Jacobi-Bellman (HJB) equation and the Feynman-Kac formula characterizes the optimal control law via the logarithmic gradient of the desirability function. By exploiting the Gaussian structure of diffusion noise increments through Stein's lemma and the tower property, MIC rewrites this gradient for non-differentiable criterion-based constraints as a conditional expectation weighted by the terminal penalty: $$ \mathbf{u}^*_t(\mathbf{z}_t, t)\mathrm{d}t = \frac{\mathbb{E}}^0}\big[\exp(-\mathcal{E}(\mathbf{zT))\,\mathrm{d}\mathbf{\varepsilon}_t \mid \mathbf{z}_t\big]}{\mathbb{E} $$ This path-integral formulation requires only forward black-box evaluations of final candidate poses, bypassing gradient computation entirely. Concurrently, for differentiable objective constraints, the control law naturally simplifies to standard score-based gradients }^0}\big[\exp(-\mathcal{E}(\mathbf{z}_T)) \mid \mathbf{z}_t\big]\(\sigma(t)\nabla_{\mathbf{z}_t}\log p_t(\mathbf{d}\mid\mathbf{z}_t)\), thereby unifying heterogeneous constraints within a single control interface.

2. Step-wise Control Instantiation: Cross-entropy importance sampling with Tweedie estimation Because evaluating expectations under uncontrolled diffusion trajectories \(\mathcal{P}^0\) is analytically intractable, MIC implements a tractable step-wise Monte Carlo approximation. At each denoising step \(t\), Tweedie's formula projects the noisy state \(\mathbf{z}_t\) onto an estimated clean motion sequence \(\hat{\mathbf{z}}_T\), enabling immediate evaluation of the terminal evaluator \(v_d(\cdot)\). To approximate the expectation over noise increments \(\mathrm{d}\mathbf{\varepsilon}_t\), MIC draws \(M=16\) candidate samples from a Gaussian proposal distribution \(q = \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})\) and weights them according to their terminal exponentiated costs. To counteract variance explosion in high-dimensional motion spaces, MIC integrates the Cross-Entropy Method (CEM): after scoring candidates, it selects the top 20% elite samples to adaptively update the proposal's mean and covariance. This iterative refinement focuses sampling probability on low-cost trajectories, yielding accurate, low-variance guidance estimates at minimal computational cost.

3. Bilevel Constraint Coordination: Integral feedback regulation and spatial-temporal control allocation When an animation must respect multiple constraints simultaneously (e.g., reaching a spatial handle while respecting floor-boundary safety rules and avoiding foot skating), simple summation of control vectors causes signal domination and mutual cancellation. MIC introduces a two-tiered control-theoretic mechanism. First, the feedback regulator dynamically adapts individual constraint weights \(W_{k,t}\) via an integral control law driven by instantaneous normalized violations \(c_{k,t}\) computed on Tweedie estimates: $$ W_{k,t+1} = \Pi_{[0, W_{\max}]}(W_{k,t} + \gamma\,c_{k,t}) $$ Unresolved constraints accumulate weight to exert growing corrective pressure, while satisfied constraints (\(c_{k,t}=0\)) cease weight growth to prevent budget hogging. Second, the control allocator treats individual joint-frame segments as multi-channel control budgets. Using diagonal spatial-temporal projection masks \(\mathbf{M}_k\), the final applied control consensus \(\mathbf{u}_t\) is computed via a regularized weighted least-squares quadratic program admitting a closed-form solution: $$ \mathbf{u}t = \left(\sum}^K W_{k,t}^2 \mathbf{Mk^\top \mathbf{M}_k + \lambda \mathbf{I}\right)^{-1} \sum}^K W_{k,t}^2 \mathbf{Mk^\top \mathbf{M}_k \mathbf{u} $$ This closed-form allocation restricts local corrections strictly to relevant joints and frames, preventing localized adjustments from corrupting global trajectory smoothness.

Key Experimental Results

Main Results

MIC is evaluated on the comprehensive benchmark established by ProgMoGen using the pre-trained Motion Diffusion Model (MDM) with DDIM sampling. Tasks span Human-Scene Interaction (HSI-1, HSI-2, HSI-3), Geometric constraints (GEO-1), and Human-Object Interaction (HOI-1). Evaluation metrics include foot skating ratio (Skating), maximum joint acceleration (Max Acc.), constraint error (C.Err.), unsuccess rate (Unsucc. Rate), and physical simulation pass rate in MuJoCo (Pass).

Table 1: Quantitative comparisons on open-set constraint specifications (reproduced from Table 1 of the paper, MDM backbone)

Task	Method	Skating↓	Max Acc.↓	C.Err.↓	Unsucc. Rate↓	Pass↑
HSI-2 (Barrier Avoidance)	MDM (Unconstrained)	0.096	0.126	0.454	1.000	0.000
	DNO (CVPR 2024)	0.196	0.162	0.051	0.375	0.531
	ProgMoGen (CVPR 2024)	0.180	0.150	0.097	0.219	0.563
	ReAlign (AAAI 2026)	0.245	0.155	0.067	0.250	0.469
	MIC (Ours)	0.172	0.140	0.009	0.094	0.875
HSI-3 (Limited-area Walk)	ProgMoGen (CVPR 2024)	0.125	0.093	0.012	0.344	0.594
	MIC (Ours)	0.112	0.089	0.004	0.250	0.719
GEO-1 (Wall Touching)	ProgMoGen (CVPR 2024)	0.110	0.104	0.023	0.531	0.406
	MIC (Ours)	0.102	0.088	0.008	0.094	0.781
HOI-1 (Object Moving)	ProgMoGen (CVPR 2024)	0.109	0.067	0.028	0.188	0.750
	MIC (Ours)	0.036	0.062	0.004	0.031	0.938

Table 2: Performance on known HumanML3D constraint specifications (HSI-1 Head Height, reproduced from Table 2)

Method	Skating↓	Max Acc.↓	C.Err.↓	Unsucc. Rate↓	Pass↑	FID↓	Diversity→	R-Precision (Top3)↑
MDM (Unconstrained)	0.086	0.097	0.118	0.718	0.193	0.545	9.656	0.610
ProgMoGen	0.075	0.094	0.012	0.088	0.776	0.556	9.611	0.597
ReAlign	0.096	0.104	0.024	0.189	0.757	0.619	9.433	0.635
MIC (Ours)	0.074	0.093	0.009	0.068	0.857	0.494	9.656	0.635

Ablation Study

Table 3: Ablation of constraint formulation and coordination mechanisms on HSI-2 (reproduced from Tables 3 & 5)

Variant	Configuration Description	Skating↓	Max Acc.↓	C.Err.↓	Unsucc. Rate↓	Pass↑
Baseline A	Differentiable surrogate loss + gradient guidance	0.180	0.152	0.097	0.219	0.594
Baseline B	Zeroth-order gradient estimation	0.242	0.267	0.205	0.281	0.594
Baseline C	Reinforcement learning policy gradient approximation	0.274	0.243	0.231	0.250	0.625
Baseline D	Evolutionary strategies black-box search	0.208	0.238	0.147	0.219	0.625
w/o regulation	Only control allocation, fixed constraint weights	0.208	0.143	0.014	0.125	0.813
w/o allocation	Only feedback regulation, uniform spatial aggregation	0.189	0.142	0.048	0.125	0.750
w/o coordination	Direct averaging of control vectors	0.220	0.146	0.054	0.156	0.688
MIC (Full Model)	Path-integral control + integral regulation + spatial allocation	0.172	0.140	0.009	0.094	0.875

Key Findings

• Superiority of direct criterion-based control: On HSI-2 barrier avoidance and HOI-1 object moving, MIC reduces the unsuccess rate to 0.094 and 0.031 respectively, outperforming ProgMoGen (0.219 and 0.188) by over 50%. This verifies that replacing discrete safety rules with artificial continuous proxies introduces detrimental target drift, whereas expectation-based control adheres precisely to true criteria.
• Physical validity with uncompromised realism: MIC achieves an 87.5% pass rate in MuJoCo physical simulation on HSI-2 (vs. 56.3% for ProgMoGen) and 93.8% on HOI-1 (vs. 75.0%), while improving FID on HSI-1 from 0.556 to 0.494. The quadratic control penalty \(\lambda \|\mathbf{u}\|^2\) in optimal control effectively prevents steering commands from breaking the pre-trained motion manifold.
• Indispensability of bilevel coordination: Ablation studies show that removing integral feedback regulation increases unsuccess rate from 0.094 to 0.125 due to constraint starvation; omitting spatial control allocation inflates constraint error from 0.009 to 0.048 due to cross-joint bleed; and abandoning both drops physical simulation pass rate from 87.5% to 68.8%.

Highlights & Insights

• Reinterpreting diffusion sampling through stochastic control: Rather than relying on heuristic guidance hacks, MIC establishes a rigorous optimal control foundation. Utilizing HJB theory and Stein's lemma converts non-differentiable boundary criteria into tractable path-integral expectations, unlocking gradient-free test-time control with theoretical grounding.
• Cross-entropy importance sampling for efficient Monte Carlo estimation: By pairing Tweedie clean pose estimation with CEM adaptive elite sampling, MIC achieves low-variance guidance estimates with only 16 candidate rollouts per diffusion step, rendering gradient-free guidance computationally viable.
• Plug-and-play modular coordination: The closed-loop integral feedback regulator combined with the closed-form weighted least-squares allocator forms a general control pipeline that can readily integrate any black-box simulator, rule evaluator, or vision-language judge into generative diffusion architectures.

Limitations & Future Work

• Sampling overhead from Monte Carlo rollouts: Evaluating \(M=16\) forward candidates per denoising step incurs non-trivial computational latency compared to standard unconditional sampling. Future research could explore temporal caching of proposal distributions or amortized surrogate value estimators.
• Feasibility collapse under extreme constraint conflict: In scenarios where mutually exclusive constraints are imposed (e.g., an extremely low ceiling obstacle combined with a high jump prompt), the controlled dynamics can fall into local minima. Incorporating a high-level kinematic feasibility planner could help prune infeasible trajectories early.

Related Work & Insights

• vs ProgMoGen (CVPR 2024): ProgMoGen relies strictly on differentiable objectives and requires smooth surrogate approximations for discrete rules. MIC establishes a unified control framework that natively evaluates non-differentiable, black-box criteria without surrogate drift.
• vs DNO (CVPR 2024) / ReAlign (AAAI 2026): DNO optimizes initial latent noise vectors with high compute demands, while ReAlign applies step-aware reward alignment still tied to explicit gradients. MIC injects controlled drift perturbations into the SDE and resolves spatial joint contention via closed-form allocation.
• vs PhysDiff (ICCV 2023) / CLoSD (ICLR 2025): Physics-guided diffusion approaches commonly require specialized training or explicit physical projections. MIC remains strictly training-free, treating physics simulators as black-box forward evaluators within a terminal cost structure.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneers the formulation of diffusion motion synthesis as stochastic optimal control, unifying gradient-free criteria and differentiable targets under a shared control interface]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across diverse open-set HSI, HOI, and geometric tasks, backed by rigid-body MuJoCo physics simulation validation and user studies]
• Writing Quality: ⭐⭐⭐⭐⭐ [Mathematically rigorous, clearly structured, and logically lucid in articulating optimal control concepts and algorithmic design choices]
• Value: ⭐⭐⭐⭐⭐ [Removes the differentiability bottleneck for test-time diffusion guidance, offering a powerful, plug-and-play tool for physically plausible character animation and robotics]