ReMoMask: Retrieval-Augmented Masked Motion Generation¶

Conference: ECCV2026
Paper: ECCV Paper
Area: Human Understanding
Keywords: text-to-motion generation, retrieval-augmented generation, hierarchical alignment, topology structured masking, asymmetric attention
Reading Date: 2026-09-17

TL;DR¶

ReMoMask gives motion retrieval, training masks, and generative evidence fusion a shared body structure through HBM, TSM, and SSTA, achieving FID 0.026 and R-Precision Top-1 0.566 in the main HumanML3D experiment.

Background & Motivation¶

Text-to-motion generation must produce smooth skeletal trajectories while respecting the body parts and action order specified by a prompt. Methods such as MoMask discretize motion and recover masked tokens in parallel; retrieval-augmented methods additionally supply similar motions as references for less common descriptions. However, an overall similar motion need not match the requested left/right limbs, local actions, or transitions.

ReMoMask addresses two consecutive bottlenecks. Aligning only complete texts and motions compresses away local differences during retrieval; even useful references can lose their body-part and temporal relationships when simply concatenated into a 1D generative sequence. In a controlled MoMask experiment, combining 2D latents with cross-attention improves HumanML3D FID from 0.057 to 0.036 and Top-1 from 0.511 to 0.536 relative to 1D latents with concatenation. This comparison changes both representation and fusion, so its full gain cannot be attributed to 2D representation alone.

The paper therefore extends the retriever's part semantics into masking and controls the pathways through which retrieved information enters attention. Core Idea: make the body structure learned during retrieval consistent with what generation preserves, reconstructs, and reads from reference motions.

Method¶

Overall Architecture¶

The input is a motion description, and the output is a time-varying human motion sequence. A pretrained 2D residual vector-quantized variational autoencoder (2D-RVQ-VAE) supplies discrete motion representations. The generator predicts codes on a latent grid retaining joint and temporal indices, after which the codes are decoded into motion.

HBM first learns a retrieval space connecting text, complete motions, and body parts, allowing a query to retrieve relevant motions and texts. During generator training, TSM uses part relevance to determine which motion tokens are more likely to be masked. SSTA routes the query, retrieved text, and retrieved motion through different attention pathways so masked regions can be reconstructed using semantic and motion references.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    prompt["Input text and motion database"] --> hbm["Hierarchical Bidirectional<br/>Momentum (HBM)"]
    codes["2D motion codes<br/>Ground truth during training"] --> tsm["Topology Structured<br/>Masking (TSM)"]
    hbm -->|Part relevance during training| tsm
    hbm -->|Retrieved text and motion| ssta["Semantic Spatial-Temporal<br/>Attention (SSTA)"]
    tsm -->|Masked motion codes during training| ssta
    prompt -->|Query text| ssta
    ssta --> output["Motion code prediction and decoding"]

TSM in this diagram is a training strategy, not an additional requirement for users to provide ground-truth motion at inference. The paper describes masked reconstruction and retrieval conditioning but does not sufficiently specify inference initialization, per-round remasking, or stopping conditions. This note does not invent a complete sampling algorithm.

Key Designs¶

1. Hierarchical Bidirectional Momentum: distinguish overall motion and local body semantics

HBM divides human motion into 6 semantic parts: the left and right arms, left and right legs, backbone, and root. A text encoder produces the query representation, a whole-motion encoder provides a global representation, and part encoders provide local representations, all in a shared space. The important step is not simply merging part features afterward: text receives explicit cross-modal supervision against both global and part representations, rather than training retrieval only through an aggregated embedding.

Matching runs in both directions at each level: text finds its motion counterpart, and motion finds its text counterpart. Positive examples pair text with the corresponding global or part motion representation; negatives come from a queue of momentum-encoded representations. Cosine similarity measures compatibility, with a temperature controlling contrastive discrimination. The global objective constrains the complete action, while part supervision retains distinctions such as left/right limbs. This aligns whole-text embeddings with body-part embeddings; the paper does not establish explicit word-level or clause-level supervision.

Momentum encoders change slowly with the online encoders, supplying relatively stable historical representations for a large negative set and reducing rapid feature drift within the queue. The prose explicitly identifies an exponential moving average, but operators are missing from cached equation 2, preventing exact recovery of the update. This note retains the mechanism and reported settings without substituting a formula inferred from standard implementations.

2. Topology Structured Masking: preserve semantic anchors while learning to coordinate other parts

TSM first computes cosine similarities between the query text and each body-part motion embedding, then applies SoftMax across parts to obtain relative semantic relevance. The explicit explanation in Section 4.3 says better-aligned parts are more likely to be preserved, while less relevant parts are masked more aggressively. The generator thus completes other motion regions while key body semantics remain visible, rather than reconstructing those semantics after their evidence has been entirely removed.

The resulting training task concerns body coordination. When a description primarily constrains the arms, preserved arm information can anchor reconstruction of the torso and lower body. Part-specific masking probabilities propagate to the 2D joint-time grid through skeletal partitions, with joints in the same part sharing a probability. Sharing a probability does not establish that all those joints must be masked together; the prose does not confirm such groupwise sampling. Selected tokens become learnable mask tokens, and the generator reconstructs the original motion codes.

Cached equation 8 does not preserve the complete transformation from relevance to masking probability, and equation 9 does not identify the exact reconstruction loss. The probability mapping, clipping or normalization details, and a specific cross-entropy or norm-based loss therefore remain unresolved. Phrases about focusing on critical parts in the abstract and local summaries could misleadingly suggest masking those parts more heavily. This note follows the explicit explanation of preserving semantic anchors and completing other parts, while retaining this wording caveat.

3. Semantic Spatial-Temporal Attention: use semantics to select and motion to supply synthesis content

SSTA is asymmetric because Query, Key, and Value receive different information. Query comes only from the current motion latents, allowing the generation state to determine what to read. Key uses the current motion state and a compact semantic token produced by passing concatenated query text, retrieved text, and retrieved motion through an MLP. Semantic evidence therefore primarily influences attention weights instead of directly becoming the values used to synthesize motion.

Value combines current motion latents with retrieved motion while deliberately excluding query text and retrieved text. Retrieved motion supplies concrete dynamics, and current motion preserves continuity with the generation state: text helps select, while motion provides synthesis content. This does not mean all retrieved information enters only Key. Retrieved motion participates in both Key's semantic context and Value's motion content; only text is restricted to the semantic pathway.

For attention computation, the 2D joint-time grid is flattened into a sequence, and the output is reshaped back into the grid. A 2D representation therefore does not mean never flattening: it retains recoverable spatial and temporal organization for structured modeling. The prose does not fully specify broadcasting, concatenation axes, or sequence-length matching between the global semantic token and motion sequences. The conceptual diagram is not an executable tensor specification, nor does the paper establish that this design guarantees physical feasibility.

A Worked Example¶

Consider the type of motion shown in Figure 4: stepping backward, sitting down, and standing up again. HBM must do more than identify a candidate containing sitting. Its global and part representations help distinguish simply stepping backward, sitting on the floor, and stepping backward before sitting and rising. Retrieved motion serves as a reference rather than being copied directly as the final output.

During training, given the ground-truth motion and text, TSM uses the relative relevance of 6 body parts to preserve semantic anchors and mask other positions. The encoder determines the actual weights; this example does not assign a made-up probability to either leg. SSTA then uses the query and retrieved text to guide selection, retrieved motion to supply dynamics, and current motion codes to complete reconstruction.

At generation time, text and retrieved evidence continue to condition the model, but the ground-truth motion used during training should not be presented as a required input. This example explains module-level information flow without claiming that the model explicitly outputs a three-step program for stepping backward, sitting, and rising. Figure 4 alone also cannot establish correct execution of every complex action order.

Loss & Training¶

Training includes retrieval alignment and masked motion modeling. HBM combines instance-level and part-level bidirectional contrastive supervision, with the part weight set to 1 in the experiments; TSM requires reconstruction at masked motion-code positions. Besides equations 2, 8, and 9, several other cached expressions contain formatting damage, so this note does not replace the implementation with speculative mathematical expressions.

Motion representation uses a pretrained 2D-RVQ-VAE with 6 quantization layers, each containing 512 codes of dimension 512. Each body-part retrieval encoder has 4 Transformer layers, with both part and instance embeddings of dimension 512. The text encoder uses frozen CLIP ViT-B/32 with 1 additional trainable Transformer layer.

The retriever is trained for 200 epochs with batch size 128, a negative queue of 65,536, and momentum 0.999 on a single Tesla A800. TSM uses a base masking ratio of 0.5; this does not mean every body's part-specific final probability equals 0.5.

SSTA has 6 layers, 8 heads, and latent dimension 512. The masked models train on 8 Tesla A800 GPUs for up to 2,000 epochs with batch size 64. The paper does not clearly state whether this batch size is global or per GPU, so an effective batch size cannot be inferred.

Key Experimental Results¶

Main Results¶

Evaluation covers HumanML3D, KIT-ML, and SnapMoGen. Lower FID indicates closer generated and real motion feature distributions; higher R-Precision Top-1 indicates better identification of the correct text for generated motion; lower MM Dist indicates a smaller text-motion embedding distance. These generation metrics are distinct from R@1 in standalone retrieval.

The table below selects results from the paper's Table 4, preserving means and 95% confidence intervals over 20 evaluation repetitions. FID values from different datasets should not be directly compared as a ranking of task performance.

Dataset	Method	FID, lower is better	Top-1, higher is better	MM Dist, lower is better
HumanML3D	MoMask	0.045 +/- 0.002	0.521 +/- 0.002	2.958 +/- 0.008
HumanML3D	LaMP	0.032 +/- 0.002	0.557 +/- 0.003	2.759 +/- 0.007
HumanML3D	ReMoMask	0.026 +/- 0.002	0.566 +/- 0.004	2.867 +/- 0.010
KIT-ML	MoMask	0.204 +/- 0.011	0.433 +/- 0.007	2.779 +/- 0.022
KIT-ML	LaMP	0.141 +/- 0.013	0.479 +/- 0.006	2.704 +/- 0.018
KIT-ML	ReMoMask	0.131 +/- 0.004	0.483 +/- 0.006	2.882 +/- 0.010
SnapMoGen	MoMask	17.404 +/- 0.051	0.777 +/- 0.002	3.812 +/- 0.008
SnapMoGen	MoMask++	15.060 +/- 0.065	0.802 +/- 0.001	3.784 +/- 0.006
SnapMoGen	ReMoMask	13.509 +/- 0.041	0.811 +/- 0.004	3.459 +/- 0.007

Standalone retrieval appears in Table 3: HumanML3D text-to-motion R@1 increases from 11.00 for ReMoGPT to 18.49 for HBM, expressed as percentages, a gain of 7.49 percentage points. ReMoGPT is an author reproduction in that table, not a direct comparison using its official implementation. For KIT-ML motion-to-text R@1, HBM scores 12.14 against MotionPatches at 13.61, so the method does not rank first in every direction and metric.

Ablation Study¶

The following results reproduce Table 5's incremental component additions on HumanML3D, retaining its row labels. They are not three independent removal experiments. The cached text does not clearly specify the full Baseline configuration, so it should not be equated directly with MoMask in the main table.

Config	FID, lower is better	Top-1, higher is better	MM Dist, lower is better
Baseline	0.054 +/- 0.013	0.536 +/- 0.003	2.946 +/- 0.004
+ HBM	0.037 +/- 0.021	0.547 +/- 0.003	2.862 +/- 0.009
+ SSTA	0.031 +/- 0.006	0.558 +/- 0.004	2.879 +/- 0.004
+ TSM	0.026 +/- 0.007	0.565 +/- 0.006	2.837 +/- 0.008

Key Findings¶

HBM corresponds to the largest adjacent FID reduction in Table 5, from 0.054 to 0.037. SSTA and TSM further improve FID and Top-1, but MM Dist does not decrease at every step, precluding a claim of monotonic improvement across all metrics.
Hierarchical retrieval in Table 6 scores R@1 14.87 without momentum and 18.34 with momentum. The latter is a separate table entry from Table 3's 18.49 and must not be merged into a single experimental result.
Table 8 compares cosine scheduling with TSM: FID is 0.034 versus 0.025, and Top-1 is 0.556 versus 0.567. These results likewise should not replace the final row of Table 5.
Increasing database coverage from 10% to 100% reduces FID from 0.073 to 0.026 and raises Top-1 from 0.502 to 0.566, showing substantial dependence on retrieval-resource coverage.

Highlights & Insights¶

Part semantics learned for retrieval are reused to define the masking distribution rather than only a candidate ranking. Retrieval representations thus directly change the generator's training problem for body coordination.
SSTA separates the basis for attention selection from the content being aggregated: text helps select, while motion supplies dynamics. This principle may transfer to other retrieval-augmented structured generation tasks, but such gains require separate experiments.
The paper examines representation structure, retrieval hierarchy, momentum, and masking rather than reporting only a full-system comparison. However, numerical and configuration differences across tables require preserving the boundaries of each experiment.

Limitations & Future Work¶

The authors explicitly acknowledge dependence on retrieval database quality and coverage, supported by the coverage experiment. Beyond enlarging databases, robustness should be tested under incorrect references, rare body-part combinations, and out-of-domain motion libraries.
The main table does not support universal superiority: MM Dist is worse than LaMP on HumanML3D and KIT-ML. The KIT-ML Top-1 difference is small with overlapping confidence intervals, and no paired significance test is provided.
Cached equations 2, 8, and 9 cannot be recovered precisely, and several other expressions are damaged. The masking-probability function, reconstruction-loss form, and attention tensor matching remain reproduction gaps; this note does not guess operators or invent implementation details.
References 5 and 13 contain mismatched authors, titles, and identifiers, so they cannot justify external links or citation corrections. Only the official paper entry is linked here. This does not invalidate clearly labeled experimental tables, but it limits bibliographic reliability.
Inference sampling, the number of retrieved examples, and database isolation are insufficiently specified in the available full text. Training/test separation, failure cases, and the complete implementation require further verification; unreported safeguards should not be assumed.

vs MoMask / MoGenTS: The former supplies the masked motion generation paradigm, while the latter provides the background of spatial-temporal motion modeling. ReMoMask's central addition connects structured retrieval, masking, and fusion rather than merely using discrete motion codes.
vs ReMoDiffuse / ReMoGPT: ReMoMask emphasizes explicit global and part-level bidirectional alignment and SSTA-based evidence injection. In Table 7, replacing only the retriever with HBM changes FID from 0.103 to 0.091 and from 0.205 to 0.165, respectively, supporting some transferability of its retrieval module.
vs TMR: Part supervision and momentum negatives extend global retrieval alignment to distinguish similar motions. Dedicated left/right swaps, action-order swaps, and incorrect-reference tests could check whether gains reflect genuinely fine-grained understanding; these are suggestions from this note, not experiments completed in the paper.

Rating¶

Novelty: 4/5. The contribution lies mainly in structural consistency across retrieval, masking, and fusion, rather than separately inventing the underlying operators.
Experimental Thoroughness: 4/5. Results cover 3 datasets and multiple ablations, but reproduction details, failure analysis, and significance evidence remain limited.
Writing Quality: 3/5. Mechanism descriptions are clear, but damaged equations, mismatched references, and differing table settings complicate interpretation; the damage cannot be attributed confidently to typesetting versus text extraction.
Value: 4/5. Useful for structured retrieval-augmented motion generation research, with exact reproduction still requiring intact equations and implementation verification.