Skip to content

Language-Guided Transformer Tokenizer for Human Motion Generation

Conference: ECCV 2026
Paper: ECCV page
Code: https://eanson023.github.io/LG-Tok/
Area: Human Understanding / Motion Generation
Keywords: motion tokenization, vector quantization, language guidance, text-to-motion generation, Transformer

TL;DR

This paper proposes LG-Tok, which moves language guidance from the generation stage into the tokenization stage: a set of learnable latent tokens with a fully Transformer-based tokenizer/detokenizer injects text semantics into the motion representation before quantization, and a language-drop training scheme prevents shortcut learning while enabling language-free guidance decoding in motion space, cutting FID on HumanML3D from MARDM's 0.114 to 0.057.

Background & Motivation

Text-driven human motion generation is currently dominated by a two-stage paradigm: a VQ-VAE-style motion tokenizer first compresses continuous motion into discrete tokens, after which a GPT- or MaskGIT-style generative model performs autoregressive or masked modeling over the token sequence. The appeal of this pipeline is that it reuses the training objectives and sampling algorithms that language models have already refined, with almost no modification; the price is that the properties of the tokenizer directly determine what the generative model can learn, since discrete tokens are the only view of motion the generator ever sees. Existing tokenizers are almost uniformly convolutional: a 196-frame motion is downsampled 4× by 1D convolutions into 49 latent tokens, which are then quantized (T2M-GPT, MoMask, MMM all follow this recipe).

This pipeline contains a trade-off that is repeatedly observed but never directly addressed: more tokens improve reconstruction but make generation harder. The paper lays it out with a set of measurements — as the token count grows from 104 to 160 and 236, reconstruction rFID drops from 0.143 to 0.110 and 0.090, while generation gFID instead rises from 0.230 to 0.205 and 0.257. The most obvious route, simply adding tokens, therefore has a ceiling, because a longer token sequence is intrinsically harder for an autoregressive model to learn. The authors' observation is that language descriptions already carry high-level semantic abstraction — a single phrase such as "a person walks forward" summarizes the intent of an entire motion — yet prior methods use text only at generation time and never during tokenization. Discrete tokens are thus forced to encode high-level semantics such as "what this motion roughly is" as well, wasting capacity on what the text already states; on top of that, the local receptive field of a convolutional tokenizer cannot support global language guidance, because the semantics of one sentence must act on every token of the whole motion, which local convolutions cannot deliver.

This paper moves language guidance earlier, into the tokenization stage, and replaces the entire tokenizer architecture to make that possible: a set of learnable latent tokens serves as queries, and text embeddings from a frozen LLaMA are concatenated with linearly projected motion into a single sequence for self-attention, so that the pre-quantization latent representation is itself computed under the sentence. Tokens can then concentrate on the fine-grained motion details that language does not fully convey, while reconstruction can lean on the text for semantics in return. Core idea: shape the discrete representation of motion before quantization with learnable latent tokens and text embeddings, and use a random text-dropping scheme to force those tokens into self-sufficient motion representations — yielding both easier generative learning and language-free guidance directly in motion space.

Method

Overall Architecture

The input to LG-Tok is a 196-frame motion sequence together with its natural language description, and the output is a set of discrete motion tokens that a generative model can model (during training, those tokens must also be decodable back into motion). The pipeline keeps the three-stage structure of tokenize–quantize–detokenize, but all three stages are rewritten: a frozen LLaMA-3.2-1B produces text embeddings; the Transformer tokenizer uses a set of learnable latent tokens to run self-attention over the joint sequence of text and motion, emitting compact semantic latent representations; these are turned into 104 / 160 / 236 discrete tokens by interpolated multi-scale residual quantization (the MoSa quantizer with 10 scales); and the detokenizer reconstructs the 196-frame motion from a set of learnable mask tokens that read the dequantized embeddings and the text embeddings through two separate cross-attention paths. At generation time, MoSa's scale-wise autoregressive model sits on top of the tokens, followed by one language-free guidance pass at the detokenizer end in motion space.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["motion sequence + text description"] -->|text| T["Language-Guided Tokenization<br/>frozen LLaMA embeddings into both ends"]
    A -->|motion| B["Transformer Tokenizer<br/>learnable latent tokens aggregate semantics"]
    T --> B
    T --> D
    B --> C["Multi-Scale Residual Quantization<br/>interpolated downsampling + 10 scales"]
    C --> D["Transformer Detokenizer<br/>mask tokens via two cross-attentions"]
    C --> E["Generative Model<br/>scale-wise autoregressive sampling"]
    E --> D
    D --> F["Language-Drop Scheme<br/>p=0.1 text dropping · motion-space guidance"]
    F --> G["generated motion"]

Two data flows must be distinguished in the figure: training follows the reconstruction path A→B→C→D, with text entering both the tokenizer and the detokenizer, whereas inference follows the generation path of "generative model samples tokens → detokenizer → guided extrapolation"; both paths share the same detokenizer. The quantizer and the generative model are components inherited from prior work (MoSa), so this paper's changes are concentrated in the tokenizer, the detokenizer, and the way text is injected.

Key Designs

1. Language-Guided Tokenization: text embeddings injected at both the tokenizing and detokenizing ends

In prior methods text appears only at generation time (as a condition of the generative model), and tokenization never sees the language at all. Here, text embeddings extracted by a frozen LLaMA-3.2-1B (up to 77 tokens) are concatenated at the tokenizer input with the learnable latent tokens and the linearly projected motion into a single sequence, and the detokenizer additionally lets its mask tokens read the same embeddings through one cross-attention path:

\[z = \mathcal{E}([t; z_l; m]), \qquad \hat{m} = \mathcal{D}(\hat{m}_l, \hat{z}, t)\]

where \(z_l\) are the learnable latent tokens, \(m\) the motion, \(t\) the text embeddings, \(\hat{z}\) the dequantized latent representation, and \(\hat{m}_l\) the detokenizer's mask tokens. Notably this is neither conditional quantization (the codebook does not change with text) nor an extra cross-modal alignment loss — the only training objective is motion reconstruction, with no text reconstruction at all, and the language acts purely through the architectural fact that the representation is computed with the text present before quantization.

The benefit holds in both directions: on the encoding side the pre-quantization representation has already absorbed sentence-level semantics, so tokens need not re-encode "what this motion is" and can focus on fine-grained details that language leaves unstated; on the decoding side the text is an extra source of information that helps the mask tokens complete the motion. The ablation (Table 5e) shows that injecting into only one end is not enough — tokenizer-only gives reconstruction FID 0.063 / MPJPE 57.5 and detokenizer-only 0.055 / 58.7, while injecting into both reaches 0.049 / 56.1, indicating that "shaping the representation while encoding" and "assisting reconstruction while decoding" are complementary concerns.

2. Transformer Tokenizer: learnable latent tokens replace convolutional downsampling

The fundamental problem with a convolutional tokenizer is not its performance but the form of its representation, which cannot carry global language guidance: a 196-frame motion is 4× downsampled by 1D convolutions into 49 latent tokens, and each token covers only a fixed 4-frame neighborhood, so sentence-level semantics have no global seat to sit in. This paper instead predefines a set of learnable latent tokens \(z_l\) of length \(T\), concatenates them with the linearly projected motion, feeds the result into a 9-layer self-attention encoder, and keeps only the outputs at the latent-token positions as the tokenization result. The token count \(T\) is thus fully decoupled from the frame count: 196 frames can be compressed into 25 tokens (LG-Tok-mini), and each token aggregates information from the entire motion through attention. The paper deliberately avoids patchify as well, since a 1D sequence of 196 frames costs far less than 2D images and chunking would only lose information for nothing.

Architecturally, both the tokenizer and the detokenizer follow LLaMA's design: RMSNorm, SwiGLU activations, and RoPE (base set to 100 to suit this roughly 10-second short-sequence task), plus UNet-like long skip connections for higher reconstruction fidelity. Compared with M2DM, which also adopts a Transformer VQ-VAE, the difference is that M2DM keeps a 1:1 frame-to-latent alignment and trains on 64-frame short crops, whereas LG-Tok encodes 196 frames with \(N\) queries decoupled from the frame count, so its tokens carry sentence-level rather than local per-frame semantics. Quantization follows MoSa's interpolated multi-scale residual quantization: latent tokens are interpolated and downsampled to different granularities and quantized residually across 10 scales (quantizers), giving 104 / 160 / 236 discrete tokens for the three variants (25 / 36 / 49 latent tokens). The paper writes the scale sequences as \(S=(1,2,\dots,25)\), \((2,4,\dots,36)\) and \((3,6,\dots,49)\); ⚠️ the exact per-scale values and their correspondence to \(N=10\) should be checked against the original paper and its appendix, and the codebook size, denoted \(V\), is not given numerically.

3. Transformer Detokenizer: mask tokens reconstruct motion through two cross-attention paths

The detokenizer faces an information asymmetry: it must restore 196 continuous frames while holding only the dequantized embeddings of a few dozen discrete tokens, leaving almost no buffer for reconstruction pressure. The solution is to prepare a set of learnable mask tokens \(\hat{m}_l\) (matching the frame count) that actively fetch information rather than waiting for convolutional upsampling: the mask tokens have one independent cross-attention path to the dequantized embeddings and another to the text embeddings, the former supplying fine-grained motion detail and the latter high-level semantics, and a final linear layer regresses into motion space. The design clearly borrows from object queries in object detection and MAE/DeiT-style decoders — a set of query tokens separates "how many output positions to produce" from "where to read information from."

The asymmetry between the two interaction modes is the crux. The tokenizer uses in-context conditioning (text, latent tokens and motion concatenated into one sequence for joint self-attention), while the detokenizer uses cross-attention. The ablation (Table 5g) tries both combinations: in-context at both ends gives 0.053 / 66.1, switching the tokenizer to cross-attention collapses to 1.120 / 111.2, and switching the detokenizer to in-context collapses further to 2.049 / 127.7. The reason is that the encoding end needs text and motion to shape each other within one sequence and jointly determine the pre-quantization representation, whereas the decoding end needs a set of query slots that repeatedly and separately fetch the two kinds of information.

4. Language-Drop Scheme: randomly dropping text for both shortcut resistance and motion-space guidance

Feeding text straight into the tokenizer and detokenizer invites shortcut learning: the model quickly discovers that the text embeddings already contain the answer, and reconstructs by pasting textual semantics back onto the motion, so the discrete tokens themselves learn little fine-grained motion detail. The evidence is telling — without dropping, reconstruction Top-1 and MM-Dist are actually slightly better, yet reconstruction FID degrades markedly (HumanML3D 0.022→0.039, Motion-X 0.041→0.057). This is the signature of a shortcut: textual priors leak into the reconstructed motion and inflate the alignment score in the joint text-motion embedding space, while a purely distributional measure such as FID honestly exposes the loss of fidelity. The remedy is minimal: during training the text input is set to empty (\(t=\varnothing\)) with probability \(p=0.1\), forcing both the tokenizer and the detokenizer to work without language so that discrete tokens become self-sufficient motion representations.

The scheme buys one extra capability. Since the detokenizer has seen both "with text" and "without text" modes, inference can apply classifier-free guidance (CFG) directly — and in motion space rather than in logits or noise space. Two forward passes, conditional and unconditional, yield \(m_c\) and \(m_u\), extrapolated by the guidance scale \(g\):

\[\hat{m} = (1+g)\times m_c - g\times m_u\]

This provides a complementary, essentially free guidance mechanism alongside existing methods that operate in logits space, and sweeping \(g\) consistently helps on all three datasets (best at \(g=2.0\) for HumanML3D and KIT-ML, \(g=1.0\) for Motion-X).

A Worked Example

Take LG-Tok processing "a person walks forward and turns left" on HumanML3D. The training-time reconstruction path is: the 196-frame motion is linearly projected into 196 vectors of dimension 256, concatenated with 49 learnable latent tokens and the LLaMA text embeddings (≤77 tokens) into a single sequence fed to the 9-layer tokenizer; only the outputs at the 49 latent-token positions are kept, giving a 49×256 representation — the semantic summary of this motion under this sentence. These 49 vectors are then expanded into 236 discrete tokens by the 10 residual quantization scales; dequantizing them yields \(\hat{z}\), which enters the detokenizer together with the text embeddings: 196 mask tokens separately query \(\hat{z}\) and the text, and the output is regressed by a linear layer into 196 frames of motion, compared against the original with a smooth L1 loss.

Inference replaces the quantizer in the middle with the generative model: MoSa samples 236 tokens scale by scale conditioned on the text (a few tokens at scale 1 first, then more at scale 2, and so on, with multiple tokens emitted in parallel at each step), and PAD masks are no longer needed at this stage. After dequantization, the detokenizer runs two forward passes — one with text and one without — producing the conditional motion \(m_c\) and the unconditional motion \(m_u\), which are finally extrapolated in motion space with \(g=2.0\). Across the whole process the text appears three times: shaping the tokens, assisting reconstruction, and driving the guided extrapolation.

Loss & Training

The only training objective is motion reconstruction: a smooth L1 loss between reconstructed and original motion, with no text reconstruction and no velocity loss (the authors consider the meng representation compact enough that a velocity term is unnecessary). Optimization uses AdamW with batch size 128 for 200 epochs, the learning rate dropping from \(2\times10^{-4}\) to \(2\times10^{-5}\) at epoch 180, and gradient clipping at 0.01; the language-drop probability is \(p=0.1\).

Structurally, the tokenizer and detokenizer each stack 9 layers with 4 attention heads, latent dimension 256 and SwiGLU dimension 1024; the text encoder is a frozen LLaMA-3.2-1B with a maximum text length of 77 and truncation beyond that. The generative model keeps MoSa's default configuration (unchanged except that PAD masks are no longer required), while the generative model and the retrieval metrics used for evaluation rely on CLIP-ViT-B/32 text embeddings to stay consistent with all baselines — LLaMA features serve only LG-Tok's own tokenizer and detokenizer. Training uses mixed precision and flash attention, and fits on a single RTX 4090.

Key Experimental Results

Three benchmarks are used: HumanML3D (14,616 motions / 44,970 text descriptions), KIT-ML (3,911 / 6,278) and the larger Motion-X (filtered to motion-text pairs of at most 200 frames, leaving 37,751 motion sequences / 61,637 captions), split 80% / 5% / 15%. All datasets adopt the meng (64–67 dim) representation, with Motion-X's whole-body representation also converted to meng, and finger and facial information deliberately ignored to avoid a modal mismatch with the textual descriptions. Metrics are: FID (distributional distance between high-level features of generated and real motion, measuring overall action quality), R-Precision (Top-1/2/3 text-to-motion retrieval) together with MultiModal Distance (MM-Dist, the distance between text and generated motion in the joint embedding space, measuring semantic consistency), Multimodality (diversity across samples from the same text), and CLIP-score (compatibility of motion-text pairs).

Main Results

Table 1: main results on the HumanML3D test set (± denotes a 95% confidence interval; Real motions is the reference upper bound).

Method Venue #Tokens R-Prec Top-1↑ FID↓ MM-Dist↓ Multimodality↑ CLIP↑
Real motions - - 0.501 0.000 3.251 3.179 0.639
T2M-GPT CVPR'23 49 0.470 0.335 3.505 2.018 0.607
MMM CVPR'24 49 0.487 0.132 3.359 1.241 0.635
MoMask CVPR'24 294 0.490 0.116 3.353 1.263 0.637
StableMoFusion ACM MM'24 - 0.510 0.177 3.182 1.969 0.654
MARDM CVPR'25 - 0.500 0.114 3.270 2.231 0.642
MoSa arXiv'25 236 0.518 0.064 3.150 1.789 0.657
LG-Tok-mini - 104 0.521 0.085 3.113 1.728 0.655
LG-Tok-mid - 160 0.537 0.109 3.061 1.674 0.664
LG-Tok - 236 0.542 0.057 2.997 1.540 0.669

Table 2: main results on the Motion-X test set (the authors' reproduction on the meng representation).

Method Venue #Tokens R-Prec Top-1↑ FID↓ MM-Dist↓ Multimodality↑ CLIP↑
Real motions - - 0.595 0.000 3.717 - 0.672
T2M-GPT CVPR'23 49 0.470 1.085 5.488 12.807 0.622
MMM CVPR'24 49 0.424 2.918 6.098 2.342 0.607
MoMask CVPR'24 294 0.502 0.247 4.832 2.715 0.644
MARDM CVPR'25 - 0.528 0.147 4.433 3.077 0.643
MoSa arXiv'25 236 0.513 0.210 4.783 3.167 0.654
LG-Tok-mini - 104 0.588 0.071 3.835 2.235 0.681
LG-Tok-mid - 160 0.591 0.076 3.758 2.245 0.682
LG-Tok - 236 0.582 0.088 3.844 2.294 0.682

Table 3: comparison of discrete tokenizers across reconstruction (FID / Top-1 / MPJPE) and generation (FID / MM-Dist).

Method Recon. FID↓ Recon. Top-1↑ Recon. MPJPE↓ Gen. FID↓ Gen. MM-Dist↓
T2M-GPT (HumanML3D) 0.081 0.483 72.6 0.335 3.505
MoMask (HumanML3D) 0.029 0.497 31.5 0.116 3.353
MoSa (HumanML3D) 0.023 0.496 43.0 0.064 3.150
LG-Tok (HumanML3D) 0.022 0.502 39.0 0.057 2.997
w/o text guidance (HumanML3D) 0.025 0.494 39.0 0.062 3.129
MoMask (Motion-X) 0.394 0.554 24.9 0.247 4.832
MoMask (LG-Tok) (Motion-X) 0.076 0.580 23.0 0.157 4.259
MoSa (Motion-X) 0.072 0.558 39.0 0.210 4.783
LG-Tok (Motion-X) 0.041 0.577 31.0 0.088 3.844
w/o text guidance (Motion-X) 0.090 0.568 33.0 0.257 4.274

Ablation Study

Table 4: effect of the language-drop scheme and the guidance scale (reconstruction and generation each report FID↓ / Top-1↑ / MM-Dist↓; ⚠️ this table spans two columns in the original PDF, so the grouping of reconstruction versus generation numbers should be checked against the paper).

Config Dataset Recon. FID Recon. Top-1 Recon. MM-Dist Gen. FID Gen. Top-1 Gen. MM-Dist
LG-Tok (\(g\)=2.0) HumanML3D 0.022 0.502 3.250 0.057 0.542 2.997
LG-Tok (\(g\)=0.0) HumanML3D - - - 0.061 0.534 3.032
w/o lang-drop HumanML3D 0.039 0.505 3.233 0.132 0.533 3.068
LG-Tok (\(g\)=1.0) Motion-X 0.041 0.577 3.890 0.088 0.582 3.844
LG-Tok (\(g\)=0.0) Motion-X - - - 0.139 0.576 3.969
w/o lang-drop Motion-X 0.057 0.580 3.846 0.136 0.574 3.918

Table 5: ablations on architecture, guidance location and interaction mode, run on a smaller LG-Tok-tiny (3-layer Transformers, 5,000 training samples; the default is RMSNorm + SwiGLU + skip connections + RoPE(base=100) + LLaMA text encoder + in-context tokenizer / cross-attention detokenizer).

Axis Config Recon. FID↓ MPJPE↓
Normalization LayerNorm → RMSNorm 0.050 → 0.049 58.7 → 56.1
Activation GeLU → SwiGLU 0.050 → 0.049 57.4 → 56.1
Skip connections w/o → with 0.059 → 0.049 61.7 → 56.1
Positional encoding Learnable / RoPE(10) / RoPE(100) / RoPE(1000) / RoPE(10000) 0.065 / 0.061 / 0.049 / 0.042 / 0.052 54.7 / 54.8 / 56.1 / 56.3 / 56.6
Guidance location None / tokenizer only / detokenizer only / both 0.064 / 0.063 / 0.055 / 0.049 64.1 / 57.5 / 58.7 / 56.1
Text encoder CLIP / T5 / BERT / LLaMA 0.051 / 0.053 / 0.055 / 0.049 59.1 / 58.9 / 58.9 / 56.1
Interaction (tokenizer / detokenizer) in-context / cross-attention (default) 0.049 56.1
Interaction (tokenizer / detokenizer) in-context / in-context 0.053 66.1
Interaction (tokenizer / detokenizer) cross-attention / cross-attention 1.120 111.2
Interaction (tokenizer / detokenizer) cross-attention / in-context 2.049 127.7

Table 6: reconstruction–generation trade-off (three LG-Tok variants; rFID / MPJPE for reconstruction, gFID / Top-1 / MM-Dist / CLIP for generation).

Dataset Variant #Tokens rFID↓ MPJPE↓ gFID↓ Top-1↑ MM-Dist↓ CLIP↑
HumanML3D LG-Tok-mini 104 0.094 50.2 0.085 0.521 3.113 0.655
HumanML3D LG-Tok-mid 160 0.052 40.6 0.109 0.537 3.061 0.664
HumanML3D LG-Tok 236 0.022 39.3 0.057 0.542 2.997 0.669
Motion-X LG-Tok-mini 104 0.094 36.4 0.071 0.588 3.835 0.681
Motion-X LG-Tok-mid 160 0.051 34.2 0.076 0.591 3.758 0.682
Motion-X LG-Tok 236 0.041 31.7 0.088 0.582 3.844 0.682

⚠️ In the original paper, LG-Tok's MPJPE on HumanML3D is listed as 39.0 in Table 3 and 39.3 in Table 6; the two tables may differ slightly in protocol — refer to the original paper.

Key Findings

  • Language guidance pays off on both the reconstruction and the generation side. In the tokenizer comparison (Table 3), removing the text guidance (keeping only the Transformer tokenization architecture) gives a HumanML3D reconstruction FID of 0.025 and generation FID of 0.062; with language guidance these become 0.022 and 0.057, and the gap on Motion-X is much wider (0.090→0.041 and 0.257→0.088).
  • Where guidance is injected matters more than whether it is injected. Both ends (0.049 / 56.1) clearly beats one end only (0.063 / 57.5 for tokenizer-only, 0.055 / 58.7 for detokenizer-only), while no injection at all hurts both quality and accuracy (0.064 / 64.1) — MPJPE degrades from 56.1 to 64.1, showing the text genuinely helps localize joints rather than merely improving "does it look right".
  • The interaction mode is the most brittle switch. Changing the tokenizer from in-context to cross-attention collapses reconstruction FID from 0.049 to 1.120 (MPJPE 111.2), and changing the detokenizer to in-context collapses it to 2.049 (MPJPE 127.7). The encoding end needs text and motion to see each other within one sequence, and the decoding end needs queries that fetch separately; the two are not interchangeable.
  • Language-drop is demonstrably double-edged. Without dropping, reconstruction Top-1 (HumanML3D 0.505 vs 0.502, Motion-X 0.580 vs 0.577) and reconstruction MM-Dist are slightly better, but reconstruction FID degrades markedly and generation FID rises from 0.057 to 0.132 on HumanML3D and from 0.088 to 0.136 on Motion-X. That is exactly the face of shortcut learning: textual priors in the reconstructed motion inflate the alignment score in the joint embedding space, while distributional metrics expose the fidelity loss.
  • The optimal guidance scale follows the data distribution: \(g=2.0\) for HumanML3D and KIT-ML, \(g=1.0\) for Motion-X. Within a dataset, \(g=2.0\) versus \(g=0.0\) lowers generation FID from 0.061 to 0.057 on HumanML3D and from 0.139 to 0.088 on Motion-X. On KIT-ML the \(g=2.0\) versus \(g=0.0\) comparison is 0.185 / 0.401 / 3.270 against 0.241 / 0.390 / 3.331, the largest gain of the three (the remaining KIT-ML numbers are in the original Table 4; ⚠️ column grouping as in the paper).
  • Generalization is validated by swapping the generative model: attaching LG-Tok's tokenizer to another representative framework, MoMask (MoMask (LG-Tok)), improves both reconstruction and generation over the original MoMask on both datasets — on Motion-X, reconstruction FID 0.394→0.076, generation FID 0.247→0.157 and generation MM-Dist 4.832→4.259.
  • The interaction between token count and data scale decides the best configuration. The reconstruction–generation trade-off (Table 6) shows that on the smaller HumanML3D (14k sequences) going from 104 to 236 tokens nearly quadruples the reduction in reconstruction error (rFID 0.094→0.022), which outweighs the cost of a longer sequence, so the full LG-Tok wins on every metric; on the larger Motion-X (37k) the reconstruction gain is only about twofold (0.094→0.041), and beyond 160 latent tokens generation degrades (gFID 0.071→0.088), leaving LG-Tok-mid as the sweet spot.
  • The learned representation also looks healthier: on t-SNE the dequantized embeddings of LG-Tok form well-structured clusters with clear boundaries, and codebook usage frequencies are more uniform than MoSa's across all scales, indicating fuller coverage of motion space and less codebook redundancy (⚠️ this claim rests on qualitative visualizations with no corresponding numeric table). The semantic representation also makes the generative model easier to learn: perplexity drops from 146.5 to 103.1 on Motion-X and from 160.6 to 155.9 on HumanML3D.

Highlights & Insights

  • Moving language guidance from the generation stage into the tokenization stage is an elegant relocation of difficulty: high-level semantics are carried by the text, tokens concentrate on fine-grained detail, and the token sequence becomes both shorter and more information-dense. The recipe transfers directly to other "text–sequence" two-stage modalities such as audio, trajectories or gestures — all it requires is that the pre-quantization representation explicitly sees the conditioning text.
  • Language-drop is the cheapest yet most practical design in the paper: a single \(p=0.1\) random blanking both cures shortcut learning and hands over a motion-space CFG for free. Upgrading condition dropping from a training trick to an inference-time guidance interface is a move worth copying — image and video tokenizers could apply the same motion/pixel-space extrapolation on their decoders.
  • The evidence used to diagnose the shortcut (retrieval metrics improving while FID worsens) is itself a reusable diagnostic: when retrieval-style and distribution-style metrics disagree, trust the distributional one, because retrieval metrics are easily polluted by priors that paste the condition's semantics back into the output.
  • The asymmetric conclusion that in-context suits encoding while cross-attention suits decoding is counter-intuitive (neither is intrinsically better at inference) and is instructive for any "multimodal queries + reconstruction" architecture: the encoding end wants mutual shaping, the decoding end wants divided information fetching.

Limitations & Future Work

  • The tokenizer remains small by modern standards: 9 layers, 4 heads, 256 dimensions, trainable on a single RTX 4090. The paper does not report how the tokenizer behaves as layers, width or latent-token count scale up, so it is hard to tell how much of the gain comes from the architecture itself rather than from capacity.
  • The method depends on a frozen external text encoder (LLaMA-3.2-1B). Table 5f shows reconstruction FID rising to 0.051–0.055 with CLIP / T5 / BERT, so the method is sensitive to the semantic quality of the text encoder; moreover the evaluation side still uses CLIP-ViT-B/32, so two sets of text embeddings coexist, adding memory and engineering cost in deployment.
  • Language guidance presupposes paired text. On motion data without captions (or when a user gives no description at inference), the method degenerates into a plain Transformer tokenizer, where Motion-X reconstruction FID is 0.090 and generation FID 0.257 — considerably worse than the text-guided version, meaning part of the reconstruction quality is staked on text availability.
  • The ablations are mostly run on the shrunken LG-Tok-tiny (3 layers, 5,000 samples), so architectural conclusions (e.g. RoPE base=1000 giving reconstruction FID 0.042 versus base=100's 0.049, at the cost of slightly worse MPJPE) are not verified on the full model or all datasets.
  • The language-drop probability is only tested at \(p=0.1\), with no sweep; whether tokens can still exploit language at large \(p\), or whether the shortcut returns at small \(p\), remains open. Guidance is also applied only at the detokenizer end; combining it with logits-space guidance inside the generative model is worth trying.
  • One extension worth exploring: generalizing language-drop from "drop everything" to partial or field-level dropping (for example dropping only direction or speed words), which could yield finer-grained controllability and make motion-space guidance extrapolation more directional.
  • vs TxtTok: Both share the high-level motivation of using language to guide tokenization in exchange for higher compression, but the technical routes differ. TxtTok naively conditions on text, whereas this paper argues that this invites representation collapse through shortcut learning, and therefore introduces language-drop, which TxtTok lacks, and further turns it into inference-time motion-space CFG; this paper is also fully Transformer-based, while the TxtTok line leans toward 1D CNNs.
  • vs MoSa: MoSa supplies the interpolated multi-scale residual quantization and the scale-wise autoregressive generative model adopted here, forming the base for quantization and generation; this paper changes the tokenizer in front of it and how text is injected on both sides. Under the same generative model, LG-Tok improves HumanML3D generation FID from 0.064 to 0.057 and MM-Dist from 3.150 to 2.997. Conversely, applying this paper's language-drop on top of MoSa-style tokenizers would test whether the scheme is decoupled from the specific quantization method — which is precisely the motivation for the MoMask experiment.
  • vs MoMask: MoMask is the representative masked non-autoregressive generator with 294 tokens (more than LG-Tok's 236). Plugging LG-Tok's tokenizer into MoMask improves every metric (HumanML3D generation FID 0.180→0.111, Motion-X 0.247→0.157), showing the gain comes from the tokenizer rather than from any particular generation paradigm.
  • vs MARDM: MARDM works with masked autoregressive diffusion over deterministic AE latents without discrete quantization, and its HumanML3D FID of 0.114 is the previous state of the art this paper compares against. The two routes differ (continuous latent space versus discrete tokens); this paper reaches FID 0.057 on the discrete route.
  • vs M2DM: Also a Transformer-structured VQ-VAE, but M2DM keeps a 1:1 frame-to-latent alignment and trains on 64-frame short crops, so its token semantics stay local; LG-Tok encodes 196 frames with \(N\) queries decoupled from the frame count, letting each token carry sentence-level semantics — a more thorough take within the "Transformer tokenizer" line.

Rating

  • Novelty: ⭐⭐⭐⭐ The combination of language-guided tokenization, a fully Transformer tokenizer and language-drop is a clean shift of paradigm earlier in the pipeline, though each ingredient (multimodal conditioning, attention-based tokenization, CFG) has precedents in the image domain.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Three datasets, two generation backbones (MoSa / MoMask), and metrics on both reconstruction and generation sides, with ablations over architecture, guidance location, text encoder and interaction mode; the weakness is that the core ablations run on a shrunken model with 5,000 samples, and neither \(p\) nor tokenizer scale is swept.
  • Writing Quality: ⭐⭐⭐⭐ The motivation chain (reconstruction–generation trade-off → language carries high-level semantics → convolutions cannot support global guidance) is clearly laid out and the figures match the conclusions; some tables span columns awkwardly, making the reconstruction/generation split easy to misread, and the KIT-ML details need checking against the original.
  • Value: ⭐⭐⭐⭐⭐ The tokenizer is the upstream bottleneck of two-stage motion generation, and here half the tokens already match or beat the previous state of the art, with a scheme that is decoupled from the generative model and trainable on a single GPU — high practical value.