Reconstruction-Anchored Diffusion Model for Text-to-Motion Generation

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://feifeifeiliu.github.io/RAM
Area: Human Understanding
Keywords: text-to-motion generation, diffusion models, motion representation learning, reconstructive error guidance, classifier-free guidance

TL;DR

To resolve the representation gap of pre-trained text encoders lacking kinematic dynamics and error propagation during iterative denoising, RAM anchors diffusion within a motion reconstruction latent space using self-regularization and asymmetric alignment, complemented by Reconstructive Error Guidance (REG) at inference time to achieve a state-of-the-art 0.032 FID on HumanML3D in just 20 sampling steps.

Background & Motivation

Synthesizing realistic, kinematically plausible 3D human motion sequences from natural language descriptions is fundamental for virtual reality, animation in gaming, and embodied robotics. However, natural language is inherently abstract and multi-interpretable, whereas human motion is continuous, high-dimensional, and governed by strict physical and kinematic constraints. Establishing robust many-to-many mappings across these modalities remains challenging. Current approaches have largely bifurcated into discrete VQ-VAE pipelines and continuous diffusion models. While diffusion-based methods offer unparalleled flexibility and conditioning control, their motion realism and distribution fidelity (measured by FID) have consistently lagged behind top VQ-VAE models.

This performance bottleneck stems from two primary predicaments. First, standard motion diffusion frameworks rely directly on pre-trained text backbones (such as CLIP or DistilBERT) that were optimized on static image-text pairs or text corpora. These encoders naturally lack temporal continuity and kinematic motion dynamics, forcing diffusion decoders to bridge an unnecessarily wide representational chasm. Second, diffusion models suffer acutely from error propagation across iterative denoising steps. Early sampling steps must reconstruct global motion structures out of pure Gaussian noise, where inaccuracies and off-manifold drift easily emerge. In subsequent reverse steps, these artifacts cascade and compound, leading to severe motion distortions, unnatural drifting, or semantic collapse.

To surmount both hurdles, this work proposes anchoring diffusion generation directly onto an expressive motion reconstruction manifold while activating the inherent error-correction capacity of the denoiser during sampling. Core idea: build a dual-stream reconstruction-generation pipeline where self-regularization and motion-centric asymmetric alignment map text onto an expressive motion latent manifold, while Reconstructive Error Guidance (REG) during inference uses biased reconstructions of prior estimates as negative references to steer sampling away from error patterns toward the clean data manifold.

Method

Overall Architecture

RAM consists of a two-stream representation learning pipeline during training and a dual-guidance sampling mechanism at inference time. In the training phase, the model utilizes a shared Transformer-based diffusion denoiser (MDM backbone) across two branches: a motion reconstruction branch where a Transformer motion encoder \(E_m\) extracts a global motion latent \(z_m\) to supervise reconstruction, and a text-to-motion generation branch where a Transformer text encoder \(E_t\) transforms token embeddings into a text latent \(z_t\). To refine the latent landscape, self-regularization (\(L_{sr}\)) maximizes discriminability within the motion latent space, while motion-centric alignment (\(L_{latent}\)) pulls text latents toward motion latents with an asymmetric gradient stop. At test time, RAM encodes the previous step's estimate to generate a biased reconstruction that captures prior error patterns, utilizing residual amplification to correct errors step-by-step.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    subgraph S1["Training: Dual-Stream Motion-Centric Latent Learning"]
        direction TB
        M_IN["Ground-Truth Motion x0"] --> M_ENC["Motion Encoder Em"] --> Z_M["Motion Latent zm"]
        T_IN["Text Description t"] --> T_ENC["Text Encoder Et"] --> Z_T["Text Latent zt"]
        Z_M -->|Inter-sample Repulsion| L_SR["Self-Regularization Lsr"]
        Z_T -->|Motion-Centric Alignment| L_LAT["Latent Alignment Llatent<br/>(Stop Gradient on Em: beta=0.01)"]
        Z_M -.->|Intermediate Anchor| L_LAT
        Z_M & Z_T --> DIFF_TRAIN["Shared Diffusion Denoiser D(xt, t, z)"]
    end

    subgraph S2["Inference: Reconstructive Error Guidance & Sampling"]
        direction TB
        PREV_X["Previous Estimate xt+1,s"] --> PREV_ENC["Motion Encoder Em"] --> Z_PREV["Error Latent zm,t+1"]
        CURR_T["Text Input t"] --> CURR_ENC["Text Encoder Et"] --> Z_CURR["Current Text Latent zt"]
        DIFF_SAMPLE["Multi-Condition Reverse Denoising"]
        Z_PREV -->|Negative Reference Reconstruction| DIFF_SAMPLE
        Z_CURR -->|Conditional Prediction| DIFF_SAMPLE
        NULL_IN["Null Token ∅"] -->|Unconditional Prediction| DIFF_SAMPLE
        DIFF_SAMPLE --> REG_CALC["Reconstructive Error Guidance (REG)<br/>Amplify Residual Over Error Reconstruction"]
        DIFF_SAMPLE --> CFG_CALC["Classifier-Free Guidance (CFG)<br/>Amplify Text Conditional Residual"]
        REG_CALC & CFG_CALC --> OUT_STEP["Final Step Output xt,s"]
    end

    S1 ==> S2

Key Designs

1. Motion-Centric Dual-Stream Latent Space: Dynamic Manifold Intermediate Supervision To bridge the representational gap where pre-trained text models lack kinematic cues, RAM rejects the classical symmetric joint language-motion embedding paradigm, which often dilutes high-frequency motion dynamics in favor of coarse semantic alignment. Instead, RAM establishes the motion reconstruction space as the definitive anchor. The motion branch processes raw motion sequences \(x_0\) through a 6-layer Transformer motion encoder \(E_m\) alongside a special token \(s_m\) to output a holistic latent vector \(z_m = E_m(s_m, x_0)\). Simultaneously, the text branch employs a 6-layer Transformer text encoder \(E_t\) initialized with DistilBERT token features to produce text latent \(z_t = E_t(s_t, f_t)\). The diffusion denoiser \(D\) shares weights across both streams, forcing text-driven generation to navigate the rich kinematic dynamics intrinsic to the motion reconstruction manifold.

2. Latent Self-Regularization: Contrastive Repulsion for Semantic Discriminability Unconstrained motion latent spaces often suffer from manifold collapse where distinct motions cluster densely onto low-dimensional manifolds, hampering semantic resolution. RAM introduces a self-regularization objective (\(L_{sr}\)) acting as a batch-wise categorical cross-entropy loss over normalized motion latents \(\tilde{z}_m\). Defining the cosine similarity as \(\text{sim}(\tilde{z}_m^i, \tilde{z}_m^j) = (\tilde{z}_m^i)^\top \tilde{z}_m^j\), only identical motion sequences serve as positive pairs while all other intra-batch samples serve as negatives: $\(L_{sr} = \frac{1}{B} \sum_{i=1}^B -\log \frac{\exp(\text{sim}(\tilde{z}_m^i, \tilde{z}_m^i)/\tau)}{\sum_{j=1}^B \exp(\text{sim}(\tilde{z}_m^i, \tilde{z}_m^j)/\tau)}\)$ By pushing distinct motion representations apart on the unit sphere, this objective expands the geometric volume of the motion manifold, substantially boosting semantic discriminability and providing fine-grained targets for text-to-motion mapping.

3. Motion-Centric Latent Alignment: Asymmetric Gradient Gating To map text latents \(z_t\) accurately onto the motion manifold without distorting its learned kinematic structure, RAM designs an asymmetric latent alignment objective (\(L_{latent}\)): $\(L_{latent} = \mathbb{E}_{z_m, z_t} \left[ \| z_t - ((1 - \beta)\operatorname{sg}(z_m) + \beta z_m) \|_2 \right]\)$ Here \(\operatorname{sg}(\cdot)\) denotes the stop-gradient operator. The parameter \(\beta\) is fixed at a tiny value (\(\beta = 0.01\)). This architectural constraint ensures that 99% of the alignment gradient forces the text encoder \(E_t\) to adapt to the motion manifold. The residual 1% gradient injection preserves stable end-to-end joint convergence, avoiding optimization stagnation while strictly preserving motion dynamics.

4. Reconstructive Error Guidance (REG): Leveraging Historical Artifacts as Negative References To suppress off-manifold drift caused by error propagation during reverse sampling, RAM devises Reconstructive Error Guidance (REG). While diffusion networks possess an innate capacity to denoise corrupted inputs, they lack an explicit negative gradient to suppress accumulating error patterns. At step \(t\), REG encodes the previous step's estimate \(\hat{x}_{t+1, s}\) via the motion encoder into \(z_{m, t+1} = E_m(s_m, \hat{x}_{t+1, s})\). Because \(\hat{x}_{t+1, s}\) retains earlier sampling artifacts, the reconstruction \(D(x_t, t, z_{m, t+1})\) reproduces these error signatures. Treating it as a negative reference, REG amplifies the improvement residual between the current text prediction and this corrupted reconstruction: $\(\hat{x}_{t,s} = D(x_t, t, z_t) + w_1 \left( D(x_t, t, z_t) - D(x_t, t, z_{m, t+1}) \right) + w_2 \left( D(x_t, t, z_t) - D(x_t, t, \varnothing) \right)\)$ Here \(w_1\) regulates the REG residual amplification strength to penalize historical error patterns, while \(w_2\) controls standard classifier-free guidance (CFG). This dual-guidance formulation simultaneously guarantees physical realism and linguistic fidelity.

Loss & Training

The overall training loss is a weighted sum of four objectives: $\(L_{overall} = L_{rec} + L_{gen} + w_{sr} L_{sr} + w_{latent} L_{latent}\)$ where \(L_{rec}\) and \(L_{gen}\) denote the MSE denoising losses for motion reconstruction and text-driven generation, respectively. Hyperparameters are configured with \(w_{sr} = 1.0\), \(w_{latent} = 0.5\), temperature \(\tau = 1.0\), and gradient coefficient \(\beta = 0.01\). The model is optimized using AdamW with a learning rate of \(10^{-4}\) and batch size 64 across 450K training steps on HumanML3D (400K on KIT-ML). Total training diffusion steps \(T = 50\), with 10% null-token dropout. At inference, sampling takes only 20 linearly spaced steps with guidance coefficients \(w_1 = 5.0\) and \(w_2 = 1.5\).

Key Experimental Results

Main Results

On HumanML3D and KIT-ML, RAM achieves state-of-the-art FID among all diffusion models and matches or surpasses leading discrete VQ-VAE methods with only 20 denoising steps.

Dataset	Model Category	Method	FID↓	R-Precision (Top 1)↑	R-Precision (Top 3)↑	Multimodal-Dist↓	Diversity
HumanML3D	Real Data	Ground Truth	0.002±.000	0.511±.003	0.797±.002	2.974±.008	9.503±.065
HumanML3D	VQ-VAE-based	T2M-GPT (CVPR 2023)	0.116±.004	0.491±.003	0.775±.002	3.118±.011	9.761±.081
HumanML3D	VQ-VAE-based	MoMask (CVPR 2024)	0.045±.002	0.521±.002	0.807±.002	2.958±.008	-
HumanML3D	VQ-VAE-based	LaMP (ICLR 2025)	0.032±.002	0.557±.003	0.843±.001	2.759±.007	9.571±.069
HumanML3D	Diffusion-based	MDM (ICLR 2023)	0.489±.025	0.418±.005	0.707±.004	3.360±.023	9.450±.066
HumanML3D	Diffusion-based	MLD (CVPR 2023)	0.473±.013	0.481±.003	0.772±.002	3.196±.010	9.724±.082
HumanML3D	Diffusion-based	ReMoDiffuse (ICCV 2023)	0.103±.004	0.510±.005	0.795±.004	2.974±.016	9.018±.075
HumanML3D	Diffusion-based	Salad (CVPR 2025)	0.076±.002	0.581±.003	0.857±.002	2.649±.009	9.696±.096
HumanML3D	Diffusion-based	RAM (Ours)	0.032±.002	0.561±.003	0.839±.002	2.716±.007	9.487±.084
KIT-ML	Diffusion-based	ReMoDiffuse (ICCV 2023)	0.155±.006	0.427±.014	0.765±.055	2.814±.012	10.80±.105
KIT-ML	Diffusion-based	Salad (CVPR 2025)	0.296±.012	0.477±.006	0.828±.005	2.585±.016	11.10±.095
KIT-ML	Diffusion-based	RAM (Ours)	0.172±.010	0.464±.006	0.803±.005	2.653±.024	11.15±.101

Ablation Study

1. Progressive Component Ablation (HumanML3D, 20 Steps)

\(E_m\) Stream	\(L_{latent}\)	\(L_{sr}\)	REG Guidance	FID↓	R-Precision Top 1↑	R-Precision Top 3↑	MM Dist↓	Note
-	-	-	-	0.786±.016	0.417±.002	0.729±.003	3.433±.012	Text-only diffusion baseline at 20 steps
✓	-	-	-	0.624±.013	0.493±.004	0.800±.002	3.045±.013	Add dual-stream motion reconstruction
✓	✓	-	-	0.187±.005	0.530±.002	0.814±.002	2.885±.008	Add motion-centric alignment (\(L_{latent}\))
✓	✓	✓	-	0.132±.005	0.561±.002	0.838±.002	2.744±.006	Add latent self-regularization (\(L_{sr}\))
✓	✓	✓	✓	0.032±.002	0.561±.003	0.839±.002	2.716±.007	Full model (REG drives FID to 0.032)

2. Disentanglement of Guidance Strategies (\(w_1\) for REG, \(w_2\) for CFG)

\(w_1\) (REG)	\(w_2\) (CFG)	FID↓	R-Precision Top 1↑	R-Precision Top 3↑	MM Dist↓	Empirical Insights
0.0	0.0	0.297±.010	0.529±.002	0.812±.002	2.912±.007	Baseline sampling without guidance
3.0	0.0	0.088±.005	0.542±.002	0.825±.002	2.808±.007	Enabling REG alone slashes FID from 0.297 to 0.088
5.0	0.0	0.128±.006	0.537±.002	0.820±.001	2.837±.008	Moderate REG strength provides optimal realism
0.0	1.5	0.132±.005	0.561±.002	0.838±.002	2.744±.006	CFG exclusively boosts R-Precision (0.529 → 0.561)
0.0	4.5	0.095±.003	0.562±.002	0.839±.002	2.734±.007	Increasing CFG saturates without reaching 0.032 FID
5.0	1.5	0.032±.002	0.561±.003	0.839±.002	2.716±.007	Orthogonal synergy: CFG anchors semantics, REG purifies realism

Key Findings

• REG Eradicates Reverse-Denoising Distribution Drift: Incorporating REG during inference (\(w_1=5.0, w_2=1.5\)) slashes FID from 0.132 to 0.032 (a 75.8% relative reduction) on identical model checkpoints. This confirms that off-manifold error accumulation is the primary driver of poor FID in motion diffusion, and contrasting predictions against prior erroneous reconstructions effectively cancels drift.
• Orthogonal Synergy Between CFG and REG: Experiments reveal distinct specializations: CFG primarily enforces text-motion alignment (driving R-Precision Top 1 from 0.529 to 0.561), whereas REG governs distributional realism and physical consistency (driving FID down to 0.032).
• Superiority of Motion-Centric Latent Structuring: In the latent space strategy comparison (Tab. 5), bidirectional alignment (TEMOS) and cross-modal contrastive learning (TMR) degrade FID to 0.776 and 0.708 respectively. By contrast, RAM's asymmetric stop-gradient alignment (\(\beta=0.01\)) strictly shields motion dynamics, achieving 0.132 FID before guidance and outperforming TMR's 0.218.

Highlights & Insights

• Dual-Purpose Reconstruction Stream: The motion reconstruction branch serves as a kinematic supervisory anchor during training and effortlessly pivots to an error-pattern extractor during inference, requiring zero auxiliary discriminators.
• Negative Residual Amplification Without Extra Models: Unlike Perturbed-Attention Guidance (PAG) or classifier guidance which alter attention or require separate networks, REG exploits the historical state of the diffusion model itself as a negative reference.
• Inference Efficiency and Acceleration: Operating at 20 steps with full REG achieves an average inference latency of 0.398 seconds (surpassing 50-step MDM at 0.490 s). Furthermore, restricting REG to only the first 6 denoising steps incurs just 26% computational overhead while capturing 71% of the FID reduction (0.132 → 0.038).

Limitations & Future Work

• Global Token Compression: The model collapses entire text descriptions into a single sequence-level latent token \(z_t\), limiting fine-grained compositional fidelity over long-horizon actions.
• Sensitivity to Early-Step Artifacts: Extreme noise in early steps might produce distorted \(\hat{x}_{t+1, s}\); excessively large \(w_1\) without balanced CFG could induce over-correction.
• Future Directions: Extending the framework to multi-token cross-attention architectures for compositional long motions, and incorporating REG principles into few-step consistency distillation.

Related Work & Insights

• vs MDM / MotionDiffuse: Direct diffusion in raw pose space lacks high-level manifold regularization and suffers from unchecked error compounding. RAM integrates motion-centric manifold learning and REG, reducing FID from 0.489 to 0.032.
• vs TEMOS / TMR: Prior two-stream methods force bidirectional language-motion symmetry, sacrificing high-frequency kinematic details. RAM adopts asymmetric gradient gating (\(\beta=0.01\)) to anchor the space firmly on motion dynamics.
• vs MoMask / LaMP (VQ-VAE Pipelines): While VQ-VAE frameworks historically dominated FID benchmarks through discrete codebook quantization, RAM demonstrates that continuous diffusion models with explicit error correction can match their distribution fidelity (0.032 FID) while retaining continuous generative flexibility.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Anchoring diffusion in a motion reconstruction space and repurposing it as a negative error reference is conceptually novel and rigorous]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Exhaustive ablations across component progressions, orthogonal guidance dynamics, and five distinct latent space paradigms]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulations, coherent motivations, and disciplined prose without marketing fluff]
• Value: ⭐⭐⭐⭐⭐ [Decisively closes the historic performance gap between diffusion models and discrete VQ-VAE architectures in 3D motion synthesis]