IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/HumanMLLM/IRG-MotionLLM
Area: Human Understanding
Keywords: text-to-motion generation, motion assessment, motion refinement, interleaved reasoning, GRPO

TL;DR

IRG-MotionLLM places generated motion inside an interleaved generation, assessment, and refinement loop, using three-stage training to raise HumanML3D R-Precision Top-1 from 0.496 for its MotionLLM baseline to 0.535, although FID does not improve alongside alignment in the final reinforcement-learned model.

Background & Motivation

Text-to-motion generation must translate body parts, action order, and linguistic details into continuous human movement, rather than merely produce a broadly relevant action category. Diffusion, masked motion modeling, and autoregressive models can generate natural movement, but naturalness does not guarantee faithful execution of a description. For example, a model may correctly produce jumping while omitting the heel kicks required during each jump. Motion-aware large language models (LLMs), such as MotionGPT and MotionChain, place motion captioning and generation in one model so that understanding and generation can share representations. However, if training and invocation still treat them as independent tasks, the ability to describe motion does not imply the ability to inspect and correct a newly generated motion.

The paper therefore asks how understanding can become useful feedback for generation, rather than simply adding another task name. Analyzing text only before generation cannot reveal omissions, incorrect body parts, or extra actions in the actual output. Useful feedback must consider both the goal text and the current motion, identify their differences, and produce actionable refinement instructions. This introduces training challenges: obtaining motions and assessments at different error levels, learning when to continue refining, and avoiding supervision that teaches incorrect motions as valid generation targets.

The paper uses motion assessment and refinement to connect existing capabilities, then employs explicit interleaved trajectories and reinforcement learning to let the model organize those steps itself. This changes the training and inference protocol rather than redesigning the motion decoder or introducing a physics simulator. Core Idea: let the same motion-language model inspect its generated motion and use the assessment in the next generation, turning motion understanding into feedback within the generation process.

Method

Overall Architecture

The inputs are a goal motion description and a reasoning instruction; the output is a final human motion sequence, with intermediate motions, assessments, and refinement instructions also available. The backbone retains the combination of a Motion VQVAE and an LLM, with Gemma2-2B used in the experiments. Motion VQVAE encodes continuous motion into discrete codebook indices, and the LLM vocabulary receives \(K+2\) additional entries, where \(K\) is the motion codebook size and the remaining entries are motion boundary tokens. <Motion> and </Motion> mark motion spans, allowing text and motion tokens to alternate within one autoregressive sequence. The final motion is extracted using these boundaries and reconstructed by the motion decoder; the model does not merely output a textual account of movement.

Training proceeds through a graded-feedback data engine, atomic-task initialization, incorrect-masked interleaved learning, and alignment-reward reinforcement. The models from the three training stages are called S1, S2, and S3; the data engine prepares their supervision rather than acting as an online assessment service during inference. S1 requires externally sequenced prompts for generation, assessment, and refinement, whereas S2 and S3 can autonomously advance the structured interleaved process after a single input.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Training text-motion pairs"] --> Data["Graded-feedback<br/>data engine"]
    Data --> Atomic["Atomic-task<br/>initialization"]
    Atomic --> CoT["Incorrect-masked<br/>interleaved learning"]
    CoT --> RL["Alignment-reward<br/>reinforcement"]
    RL --> Runtime["Inference: analyze, generate, assess"]
    Goal["Goal text and reasoning instruction"] --> Runtime
    Runtime -->|Regenerate using refinement instructions| Runtime
    Runtime -->|Assessment finds no refinement needed| Output["Final motion"]

Key Designs

1. Graded-feedback data engine: include both obvious errors and nearly correct motions

Random negative sampling usually produces motions far from the goal, encouraging coarse rejection rather than sensitivity to detailed errors. The paper first represents dataset motions with a pretrained TMR motion encoder and ranks other motions by embedding distance from an anchor motion. Candidates are ordered from distant to close; each iteration samples one negative motion from the top \(p\) fraction with the largest distances in the current pool, then removes that fraction. Repeating this procedure yields negatives at different proximity levels instead of repeatedly drawing unrelated movements. Proximity here is an encoder-space ranking, not a manually measured absolute semantic correctness score.

The engine uses GPT-4o to analyze the requirements in each goal description, then asks an LLM to produce alignment assessments and refinement instructions from the anchor and negative text descriptions. Synthetic feedback therefore relies mainly on existing dataset descriptions; it does not mean GPT-4o directly watches and checks every motion. For instance, if the goal requires lowering both hands and then waving the right hand while the negative caption describes raising hands and turning, the assessment should identify missing and extra components and suggest replacements. The authors also manually check and filter failures such as empty responses and instructions to modify already correct motions. These annotations can become individual tasks or progressively improving interleaved trajectories, avoiding separate full data collection for every training stage.

2. Atomic-task initialization: first learn to identify differences and execute edits

S1 supervised fine-tuning includes four basic tasks: motion captioning, direct motion generation, goal analysis, and analysis-guided motion generation. The first two establish bidirectional text-motion mapping, while the latter two make language analysis identify constraints that generation must preserve. Four improving tasks are added: text-motion alignment evaluation, motion refinement instructing, direct motion refinement, and instruction-guided motion refinement. Direct refinement receives the current motion and goal; instruction-guided refinement additionally receives explicit editing requirements, teaching both problem localization and execution of existing feedback. Unlike captioning, evaluation must compare the motion against a goal; unlike fresh sampling, refinement must use the current motion and diagnostic information.

These atomic tasks use task-specific instructions for supervised fine-tuning rather than requiring autonomous long trajectories from the outset. S1 can already form a loop through externally sequenced task calls, but this manually connected process should not be described as fully autonomous reasoning. In Table 1, basic tasks alone reach Top-1 of 0.500, compared with 0.504 for S1 with improving tasks, indicating a real but modest single-step gain. The stronger evidence is Table 3: the same S1 improves from 0.504 at initial generation to 0.522 at the end of the loop. This separates the effect of obtaining a better model through multitask training from the effect of using understanding-based feedback to further improve a motion.

3. Incorrect-masked interleaved learning: retain errors as context without teaching them as answers

S2 uses an IRMoGen-CoT template that places goal analysis, motion generation, alignment assessment, refinement instructions, and next-step plans in one sequence. [Plan] describes the next step, [Analysis] contains goal analysis, [Generation] contains motion, and [Assessment] contains evaluation and refinement requirements. Refinement is not a separate continuous optimizer: the model generates motion tokens again after considering previous motion and feedback. When assessment indicates no refinement is necessary, the model plans to stop thinking; for a trajectory with \(n\) generations, the first \(n-1\) motions are intermediate and the last is final. Training trajectories order negative examples by progressively improving alignment and end with the ground-truth motion paired with the goal.

The danger is that ordinary next-token prediction over the whole sequence also rewards producing intermediate motions already known to be misaligned. Ignore Incorrect keeps those motions as inputs to subsequent assessment but masks their corresponding loss terms, avoiding direct supervision of incorrect generation. The following equivalent masking notation explains the strategy; it is not a transcription of a numbered equation in the paper:

\[ \mathcal{L}_{\mathrm{S2}}=-\sum_{\ell\notin\mathcal{I}_{\mathrm{wrong}}}\log p_\theta(y_\ell\mid q,y_{<\ell}). \]

Here \(q\) is the input, \(y\) is the interleaved trajectory, and \(\mathcal{I}_{\mathrm{wrong}}\) contains the positions of incorrect intermediate motion tokens. These positions remain in context: masking target terms neither removes the motion from the sequence nor freezes shared model parameters. Analysis, plans, assessments, and the final correct motion still provide supervision, teaching the model how to continue when confronted with such errors. The clearest evidence is Table 1: removing masking reduces Top-1 to 0.448 and increases FID to 0.592. However, learning the trajectory format does not guarantee repeated refinement; the main text reports that S2 usually stops after its first generation.

4. Alignment-reward reinforcement: make further refinement potentially worthwhile

Starting from S2, S3 uses GRPO to sample a group of complete interleaved trajectories for the same goal, score their final motions, and compare rewards within the group. A format reward encourages compliance with the template so that motions, assessments, and stopping points can be identified. A text-motion alignment reward uses paired text and motion encoders to encourage proximity between the final motion and the goal description. A motion-motion alignment reward compares the final motion with the paired ground-truth motion; this term is set to 0 for text-only training examples without ground-truth motion. Each reward is normalized within the group to the same scale and then summed for standard GRPO optimization.

The cached reward equations on page 8 have missing operators and damaged case extraction, so their exact expressions cannot be reliably recovered; only components and comparison targets clearly supported by the prose are retained here. The reward encoders come from the official HumanML3D evaluation framework; they serve a different role from the TMR encoder used to rank negatives in the data engine. Rewards concern the final motion rather than explicitly rewarding reasoning length, so longer trajectories should be understood as an observed behavioral change after training. Figure 5 shows more multi-generation trajectories after reinforcement learning, but this does not establish that every additional round improves the result. Likewise, embedding-alignment rewards are not physical constraints and do not guarantee better FID or body-contact quality.

A Worked Example

The goal in Figure 6 is a person jumping up and down while kicking her heels in the air. The initial motion already contains jumping in place but omits the heel kicks; text analysis only before generation would not directly inspect this omission in the actual output. The model assesses the partial match, identifies the absent heel motion during jumping, and requests a clear heel kick in each jump. The next generation incorporates that detail rather than merely emitting text claiming that the error has been fixed. This is a qualitative example from the paper, without added confidence values, aggregate refinement counts, or success rates.

Loss & Training

S1 applies supervised fine-tuning to eight atomic tasks, S2 supervises trajectories while masking incorrect-motion losses, and S3 applies GRPO primarily through final-motion rewards. HumanML3D starts from official pretrained MotionLLM weights; the KIT-ML baseline is trained following its official implementation, so the two should not be described as using the same pretrained checkpoint directly. The default HumanML3D S3 setting uses approximately 9k examples and 900 GRPO steps. Data come from the original training set and InstructMotion-expanded descriptions; KIT-ML similarly uses original and GPT-4o-augmented descriptions. The authors exclude prompts identical to test prompts, which does not establish removal of all semantic near-duplicates. The scaled variant increases data to a scale similar to UniMo and trains for 2,600 steps; its results should not be mixed with the default 900-step setting. The supplied cache contains the main paper and references but no appendix, so learning rates, group size, sampling parameters, and appendix ablation details are not filled in.

Key Experimental Results

Main Results

The following selection is from Table 4, page 13, under standard evaluation settings for each dataset; evaluations are repeated 20 times, and only means are shown here, while the original table also gives 95% confidence intervals. R-Precision Top-1 measures cross-modal retrieval matching accuracy, with higher better; FID measures the difference between generated and real motion feature distributions, with lower better; MM-Dist is paired text-motion embedding distance, with lower better. The HumanML3D baseline here is the re-evaluated official checkpoint actually used by the authors, not the separate literature-reported MotionLLM row in the original table.

Dataset	Method	Top-1 ↑	FID ↓	MM-Dist ↓
HumanML3D	MotionLLM, official checkpoint used	0.496	0.198	3.029
HumanML3D	S1 + IRMoGen	0.522	0.135	2.906
HumanML3D	S2 + IRMoGen	0.526	0.111	2.885
HumanML3D	Motion-R1	0.515	0.201	2.854
HumanML3D	S3 + IRMoGen	0.535	0.242	2.785
KIT-ML	MotionLLM, trained from implementation	0.416	0.560	3.001
KIT-ML	S2 + IRMoGen	0.419	0.389	2.908
KIT-ML	S3 + IRMoGen	0.445	0.432	2.740

Default S3 gains 3.9 percentage points in HumanML3D Top-1 over the baseline used, but its FID of 0.242 is worse than the baseline's 0.198 and S2's 0.111. KIT-ML also shows improved alignment but worse FID from S2 to S3, supporting an alignment-enhancement claim rather than universal superiority across quality dimensions.

Ablation Study

The following rows come from Table 1, page 11, on HumanML3D; S1 rows use single-step generation, whereas S2/S3 use their post-training inference settings, so cross-stage comparisons are not fixed-inference-budget experiments.

Config	Top-1 ↑	FID ↓	MM-Dist ↓
S1, T2M task only	0.495	0.185	3.072
S1, basic tasks only	0.500	0.123	3.086
S1, all eight tasks	0.504	0.141	3.018
S2, no assessment/refinement trajectories	0.503	0.125	3.023
S2, no Ignore Incorrect	0.448	0.592	3.519
S2, full model	0.526	0.111	2.885
S3, full model	0.535	0.242	2.785

Removing Ignore Incorrect reduces Top-1 by 7.8 percentage points relative to full S2, showing that supervising incorrect motions substantially harms alignment rather than only affecting output format. The following selection from Table 3, page 12, directly compares initial and final motions within each stage; S1 uses externally sequenced prompts for its loop.

Stage	Motion position	Top-1 ↑	FID ↓	MM-Dist ↓
S1	Initial	0.504	0.141	3.018
S1	Final	0.522	0.135	2.906
S2	Initial	0.497	0.136	3.052
S2	Final	0.526	0.111	2.885
S3	Initial	0.503	0.221	3.006
S3	Final	0.535	0.242	2.785

Key Findings

• The loop effect is clearer than an improvement in the very first generation: S3 starts at Top-1 of 0.503 and ends at 0.535, while S1 already starts at 0.504.
• Alignment gains persist under another evaluator: Table 6, page 14, reports Top-1 of 0.470 for the baseline used and 0.509 for S3 under the MARDM evaluator, but their FIDs are 0.240 and 0.259.
• Scaling the training budget helps: Table 5, page 13, moves from Top-1 of 0.535 and FID of 0.242 at the default 900 steps to 0.564 and 0.208 with expanded data and 2,600 steps; data volume and step count change together, preventing separate attribution.
• Understanding also benefits: Table 2, page 11, reports CIDEr of 48.94 for basic tasks alone and 52.4 for full S1, although this cannot replace direct evaluation of feedback factual correctness.

Highlights & Insights

• Intermediate errors are teaching context, not imitation targets. Keeping incorrect motion enables diagnostic learning, while masking its loss protects generation ability; the same principle may transfer to other generative training with failure trajectories.
• Assessment and refinement provide a concrete interface. Captioning answers what happened, whereas goal-conditioned assessment answers what is wrong, giving understanding a specific route to influence the next generation.
• Within-stage initial/final comparisons locate the source of gains. They test the loop more directly than final leaderboards alone, although compute-matched repeated-sampling baselines are still needed to isolate the effect of extra computation.

Limitations & Future Work

• Rewards and quality are not fully aligned. S3 repeatedly improves alignment while worsening FID; adding smoothness, contact, and physical-consistency constraints is a reader suggestion, not a validated improvement in this paper.
• Synthetic assessments depend on caption coverage. If negative captions omit body details, feedback synthesized from text differences may also miss errors; independent human assessment directly grounded in motion is needed.
• Inference cost remains unclear. Repeated motion tokens and textual feedback increase generation volume, and the main paper lacks sufficient latency, memory, or fixed-budget comparisons to support real-time deployment advantages.
• Source boundaries should be preserved. Page 8 states that more than 70% of S2 trajectories generate only once, while page 12 states more than 75%; these are retained as their respective lower bounds rather than converted into an invented exact percentage.
• Supplementary evidence is unverified. The authors place user studies, cross-model reward transfer, MotionFix editing experiments, and further limitations in the appendix; these parts are absent from the current cache, so their numerical outcomes are not reported here.

Related Work & Insights

• vs MotionGPT / MotionChain: These works establish multitask or conversational motion interfaces; this paper connects tasks specifically to inspect current generation and repeatedly refine it, rather than only extending conversational functionality.
• vs Motion-R1: According to this paper's related-work discussion, Motion-R1 performs motion-free textual reasoning before generation, whereas IRG-MotionLLM includes generated motion in subsequent reasoning context; this distinction matters more than whether GRPO is used.
• vs UniMo: Table 5 demonstrates the importance of reinforcement-learning data volume and training steps, so unified motion-model comparisons should report budgets rather than attributing every difference to the interleaved template.
• Research direction: Compare random resampling, assessment-based selection, and assessment-guided refinement at equal motion-generation counts, then manually annotate omission-repair rates to distinguish feedback content from additional computation.

Rating

• Novelty: 4/5, makes motion an inspectable object within interleaved reasoning and links existing tasks through assessment and refinement.
• Experimental Thoroughness: 4/5, includes two datasets, two evaluators, and key training ablations, but inference-budget evidence and available supplementary evidence remain limited.
• Writing Quality: 4/5, presents a clear three-stage argument, although percentage wording and reward-equation extraction boundaries require attention.
• Value: 4/5, offers a reusable feedback-training approach for motion generation while exposing a trade-off between semantic alignment and distributional quality.