RL from Physical Feedback: Aligning Large Motion Models with Humanoid Control¶

Conference: ECCV2026
Paper: ECCV Official Page
Area: Robotics & Embodied AI; Human Understanding
Keywords: text-to-motion, RL from physical feedback, humanoid robots, semantic alignment, hierarchical co-training

TL;DR¶

RLPF fine-tunes a large motion model with both simulated humanoid tracking success and text-motion semantic correspondence, while Hierarchical RLPF alternately improves the generator and controller to make motions executable without collapsing into instruction-independent standing.

Background & Motivation¶

Text-to-motion models can turn natural language into coherent human joint trajectories, but a motion that resembles a human jump is not necessarily executable by a robot. Motion capture datasets such as AMASS provide motion priors, while video-derived datasets such as MotionX expand coverage but introduce floating feet, ground penetration, abrupt transitions, and reconstruction noise. Retargeting these trajectories to a robot addresses body proportions and joint correspondence; it does not guarantee sufficient balance, torque, or tracking capability. A generator can therefore perform well on text retrieval and visual quality metrics yet fail frequently once connected to a controller.

Existing physical constraints often rely on kinematic measures such as skating and penetration, or project human motion into a physically feasible space. They do not directly answer whether the target robot and its current tracking policy can execute the motion. This paper instead asks a low-level controller to attempt generated motions in simulation and feeds execution outcomes back to the high-level generator. However, this creates a reward-hacking opportunity: standing is easy to track but does not satisfy instructions to jump or turn. A further difficulty is that tracking failure may reflect a controller that has not learned a skill, rather than an inherently infeasible reference motion. A frozen controller can consequently impose its own capacity limit on the generator.

The generator must therefore preserve instruction meaning, while the controller needs exposure to the new motions the generator continually produces. Core Idea: reward both executability and correspondence to the requested motion, then reuse the generator's on-policy samples to train the controller, creating an alternating improvement loop across text, motion, and control.

Method¶

Overall Architecture¶

The input is a text instruction; the outputs are a robot-trackable reference motion and the resulting control behavior. The base generator uses a VQ-VAE to compress human motion into discrete motion tokens, extends the vocabulary of LLaMA2-7B, and predicts motion tokens autoregressively from text. A decoder reconstructs human motion from those tokens. This pretraining stage follows established discrete motion modeling; the main contribution is subsequent alignment through physical feedback, rather than a new motion representation.

During training, multiple candidate motions are sampled for the same instruction. Each receives Execution-Based Physical Feedback and Semantic Alignment Verification before Dual-Reward GRPO updates the generator. Plain RLPF freezes the tracking policy; Hierarchical Alternating Training additionally updates the tracker using trajectories produced by the current generator. Deployment requires only motion generation, decoding, retargeting, and tracking, without the reward scoring or GRPO updates used for post-training.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Text instruction"] --> Generator["Motion generator<br/>Decode tokens into trajectories"]
    Generator --> Physical["Execution-Based Physical Feedback"]
    Generator --> Alignment["Semantic Alignment Verification"]
    Input --> Alignment
    Physical --> Update["Dual-Reward GRPO"]
    Alignment --> Update
    Update -.->|Training updates| Generator
    Update --> Hierarchy["Hierarchical Alternating Training"]
    Generator -.->|On-policy motions| Hierarchy
    Hierarchy -.->|Update tracker| Physical
    Generator -->|Deployment| Execution["Retargeting and tracking<br/>Robot actions"]

Key Designs¶

1. Execution-Based Physical Feedback: ask the target controller to attempt the motion

Body mismatch must be resolved before feedback is collected. The authors place the SMPL body and robot in a canonical T-pose and optimize SMPL shape parameters so that proportions such as limb lengths match the robot. With shape fixed, they optimize the robot's root pose and joint angles frame by frame to align the 3D positions of 14 semantically corresponding joint pairs. This produces a robot reference trajectory, but does not establish its dynamic feasibility. A control policy and physical simulation are therefore still needed; retargeting error alone is not treated as the reward.

The tracker uses two-stage teacher-to-student training. The teacher learns with PPO in IsaacGym, receives privileged information available only in simulation, and outputs 23 target joint positions executed by PD controllers. The student receives only deployment-observable information and tracking targets. It first rolls out in simulation, queries the teacher for actions at those visited states, and learns through DAgger-style data aggregation and action MSE. This deployable student then evaluates candidate motions: exceeding a positional deviation threshold between the reference and execution, or losing balance, marks a rollout as a failure; otherwise, it receives a success reward. The feedback evaluates a combination of motion, robot morphology, and controller capability, not an absolute physical truth independent of the controller.

2. Semantic Alignment Verification: prevent high success through trivial standing

Contrastively pretrained text and motion encoders map both modalities into a shared embedding space. Matching text-motion pairs should lie closer together, and the verification reward is the negative squared Euclidean distance between their embeddings. A motion that moves farther from the instruction's meaning consequently receives a lower reward. The verifier compares the decoded human motion with the original instruction; the simulator need not understand language, and no human annotation is required for each rollout.

This component does not merely improve visual quality: it constrains physical success by task semantics. An instruction to jump forward cannot be satisfied by standing, even if standing almost never fails under the current tracker. The verifier is nevertheless a learned semantic proxy, not a guarantee that every instruction detail is respected. The experiments therefore also report text retrieval, distributional quality, and diversity, rather than treating tracking success alone as complete task success.

3. Dual-Reward GRPO: compare executability among candidates for the same instruction

Group Relative Policy Optimization (GRPO) treats the motion generator as a policy and samples a group of motion-token sequences from the old policy for each instruction. After decoding, each candidate receives tracking and alignment rewards. Their weighted sum is normalized using the group's reward mean and standard deviation to form relative advantages. Candidates that better combine executability with semantic correspondence receive more favorable updates, without a separate value network. These within-group comparisons concern complete candidate motions for the same instruction, rather than the absolute difficulty of different instructions.

The following reward relation is an explanatory restatement based only on the prose definitions. Equations in the cached text are corrupted, so this is not presented as a verbatim reconstruction of the authors' equation:

\[ R = \lambda_{\mathrm{track}}\,\mathbf{1}[\mathrm{success}] - \lambda_{\mathrm{align}}\,\lVert E_l(l)-E_m(\hat m)\rVert_2^2. \]

The first term is binary feedback from successful execution by the student tracker; the second penalizes semantic distance between the instruction and generated motion. Policy-ratio clipping limits changes relative to the old policy, while KL regularization against the pretrained reference model discourages excessive drift. Semantic verification and KL regularization are not interchangeable: the former checks correspondence to the current instruction, whereas the latter constrains overall policy deviation. The paper places reward weights and several training details in an appendix absent from this cache, so specific weights and group size are not supplied here.

4. Hierarchical Alternating Training: expand the controller's capability boundary as well

In plain RLPF, the tracker is a fixed evaluator. A challenging motion can repeatedly fail because the evaluator lacks the necessary skill, encouraging the generator to avoid it. Hierarchical RLPF defines an iteration as two steps: first update the generator through RLPF using feedback from the current tracker, then train the tracker on motions produced by the current generator. The generator gains control capability better matched to its own output distribution, while the tracker is no longer restricted to a fixed offline motion library.

The key is to reuse on-policy motions already sampled during reinforcement learning, exposing the low-level learner to trajectories the high-level model generates in practice instead of creating an entirely separate data-collection pipeline. This is alternating optimization, not differentiable back-propagation through the physics simulator or online controller parameter updates during deployment. Since the components might adapt to one another without generalizing, the paper also evaluates each generator with an independent frozen tracker and each tracker on unseen ground-truth reference motions, checking whether gains extend beyond the jointly trained pairing.

A Worked Example¶

Consider the forward-jump instruction discussed in the paper. The base model may produce excessive jump amplitude and forward displacement. Even after human-to-robot retargeting, the student controller may lose balance, yielding failure feedback. A candidate with more moderate amplitude but the same intended action can receive a higher joint reward. This is a mechanism illustration, not an additional measurement of jumping distance.

If a third candidate merely stands still, its physical score may be high, but the semantic encoder separates it from the jumping instruction, preventing it from winning on one metric alone. GRPO changes generation probabilities through within-group comparisons; the hierarchical variant then uses generated trajectories to continue training the tracker. At deployment, the system directly generates and executes a reference motion without rerunning this training loop for every jump.

Loss & Training¶

Motion-generator pretraining uses motion-token negative log-likelihood; the low-level teacher uses PPO; the student minimizes MSE against teacher actions; and the semantic verifier uses a contrastive objective over matched and mismatched text-motion pairs. Generator post-training then uses GRPO with clipping and KL regularization, treating nondifferentiable simulation outcomes as rewards rather than back-propagating a differentiable loss through the controller.

The paper tests plain RLPF with a frozen tracker on AMASS and the hierarchical variant on the relatively lower-quality walking and jumping subsets of MotionX. Training simulation uses IsaacGym, followed by zero-shot transfer to MuJoCo and real execution demonstrations on Unitree G1. The main text does not provide complete split sizes, hardware resources, training duration, or all hyperparameters. The method should therefore not be interpreted as lightweight post-processing that needs neither a pretrained controller nor substantial simulation computation.

Key Experimental Results¶

Main Results¶

Table 1 excerpts the AMASS results from the paper's Table 1. Base Model, SFT-Filter, the adapted RobotMDM, and RLPF share the LLaMA2-7B generator backbone. PhysDiff uses a different diffusion backbone, so its entire performance gap cannot be attributed to the reward mechanism. SFT-Filter applies supervised fine-tuning only to successfully tracked samples, while RobotMDM incorporates a feasibility critic. The comparisons follow the main text's test setting, but exact split sizes from the appendix are unavailable in the cache.

Lower FID indicates closer generated and ground-truth motion-feature distributions. R@1 is top-ranked retrieval accuracy for paired text and motion, where higher is better. MM-D measures paired text-motion embedding distance, where lower is better. SR is the proportion of complete rollouts without falls or severe tracking failure. An SR of 0.95 means 95%, not a joint-position error or a real-robot test success rate.

Method	FID ↓	R@1 ↑	MM-D ↓	IsaacGym SR ↑	MuJoCo SR ↑
Base Model	0.29	0.47	3.06	0.83	0.85
SFT-Filter	0.25	0.48	3.02	0.85	0.87
RobotMDM	0.27	0.48	3.10	0.86	0.88
PhysDiff	0.40	0.35	3.32	0.84	0.82
RLPF	0.26	0.50	3.02	0.95	0.94

Relative to Base Model, RLPF improves IsaacGym and MuJoCo SR by 12 and 9 percentage points, respectively, while raising R@1 from 0.47 to 0.50. Its FID is not the best in the table: SFT-Filter's 0.25 is slightly lower than 0.26. The paper also reports LangWBC SR values of 0.81 and 0.79 for the two simulators. That method directly outputs control actions, so corresponding high-level generation metrics do not apply. The real-robot evidence consists of qualitative Unitree G1 demonstrations; the main text provides no large-scale real-robot success rate comparable to the simulation table.

Ablation Study¶

Table 2 corresponds to the paper's Table 3 under the AMASS ablation setting. MPJPE specifically denotes joint-angle error here, measured in rad; MPKPE is keypoint-position error, measured in m. Lower is better for both. MPJPE should not be confused with the commonly used millimeter-scale joint-position metric.

Config	FID ↓	R@1 ↑	MM-D ↓	SR ↑	MPJPE (rad) ↓	MPKPE (m) ↓
RLPF-Full	0.26	0.50	3.02	0.95	0.20	0.06
RLPF-w/o-T: no tracking reward	0.23	0.50	2.98	0.50	0.31	0.21
RLPF-w/o-A: no semantic verification	41.97	0.07	7.57	0.99	0.18	0.07

Table 3 excerpts the hierarchical iteration results from the paper's Table 2. MotionX-W is the walking subset, and MotionX-J is the jumping subset. SR denotes the reported low-level executability metric; the table does not separately identify MuJoCo, so these values should not be presented as jointly achieved across both simulators or on the real robot.

Iteration	MotionX-W SR ↑	MotionX-W R@1 ↑	MotionX-J SR ↑	MotionX-J R@1 ↑
Iter0	0.56	0.41	0.36	0.39
Iter1	0.75	0.40	0.73	0.39
Iter2	0.85	0.38	0.76	0.39
Iter3	0.93	0.39	0.84	0.39

Key Findings¶

Removing the tracking reward lowers SR from 0.95 to 0.50. Removing semantic verification raises SR to 0.99 but worsens FID to 41.97. Both rewards are necessary, and SR alone cannot establish task completion.
After three alternating iterations, walking and jumping SR increase by 37 and 48 percentage points. Walking R@1 changes from 0.41 to 0.39, so semantic quality is largely maintained rather than uniformly improved without loss.
In the paper's Table 4, generator SR under an independent frozen tracker rises from 0.72 and 0.31 for walking and jumping to 0.84 and 0.57. Tracker SR on unseen ground-truth motions rises from 0.66 and 0.21 to 0.86 and 0.69. This supports transferable gains on both sides, but does not establish generalization to arbitrary new robots.

Highlights & Insights¶

The physical reward is grounded in control execution, incorporating the target robot and controller's capabilities into the generation distribution. This is closer to the deployment interface than merely removing penetration or skating.
The ablation that achieves 0.99 success by standing exposes the risk of substituting a metric for the task. Generated motions must remain both trackable and semantically appropriate for high success rates to be meaningful.
In the hierarchical variant, generated samples serve both high-level policy exploration and low-level training. A transferable lesson is to ask whether the executor can also learn from the generator's distribution, rather than aligning only the generator in tasks with an execution component.

Limitations & Future Work¶

The authors explicitly note that the discrete tokenizer's finite codebook constrains RL exploration, and propose extending the framework to diffusion-based text-to-motion models with continuous representations.
This note's assessment: binary success feedback is sparse and depends on tracker capability and failure thresholds. Hierarchical training mitigates but does not eliminate this dependence. Decomposing failure causes would be more informative than treating every failure as infeasibility.
This note's assessment: the semantic verifier provides an embedding-space proxy, which does not guarantee fine-grained instruction fidelity, rare-action coverage, or reliability in complex environments. Current hierarchical evidence focuses on walking and jumping.
Evidence boundary: the cache contains the main paper and references, but not the cited appendix or supplementary videos. Exact split details, complete reward weights, compute budgets, and real-robot repetition counts cannot be checked. Some equations are corrupted in text extraction, and this note does not reconstruct the exact GRPO objective from them.

vs MotionGPT / T2M-GPT: These methods also discretize motion into tokens. This paper focuses on execution feedback after pretraining, rather than presenting a unified language-motion representation as its main novelty.
vs PhysDiff / RobotMDM: The former introduces physical projection during diffusion, and the latter learns a feasibility critic. RLPF instead receives direct execution feedback from a deployment-relevant tracking policy while explicitly preventing semantic collapse.
vs DeepMimic / BumbleBee: These lines primarily improve low-level imitation and tracking of reference motions. This paper reverses the direction by having a tracker evaluate and improve upstream references, then couples updates to both components in the hierarchical variant.
vs LangWBC: LangWBC learns language-directed whole-body control. RLPF retains the text-to-motion-to-control interface and uses a pretrained large motion model's semantic capability. Their capacities and output representations differ, so this is not a comparison that changes only one reward function.

Rating¶

Novelty: 4/5. Execution-grounded rewards, semantic preservation, and alternating generator-controller training form a well-motivated combination, while the underlying algorithms largely reuse existing techniques.
Experimental Thoroughness: 4/5. Same-backbone comparisons, reward ablations, cross-simulator evaluation, and independent-tracker analyses are included, but quantitative real-robot evidence and broad task coverage remain limited.
Writing Quality: 4/5. The pipeline and central degeneration case are clear; corrupted equations are a limitation of the cached reading material, not necessarily the paper's typesetting.
Value: 4/5. The framework offers a concrete route from large motion models to deployable controllers, although reliable deployment still requires fuller real-robot statistics and safety validation.