Skip to content

GeoMotionGPT: Geometry-Aligned Motion Understanding with Large Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: motion understanding, discrete motion tokenization, geometric alignment, orthogonality regularization, large language models

TL;DR

GeoMotionGPT addresses the geometric disconnect between discrete motion tokenization and large language model (LLM) vocabulary expansion by introducing an approximate orthogonality prior shared across both stages via a decoder-only Gumbel-Softmax quantizer, a parameter-free sparse isometric initialization that exactly preserves the codebook Gram matrix, and two-stage soft orthogonality regularization.

Background & Motivation

Human motion understanding is fundamental for embodied agents operating in and interacting with the physical world. A prevailing paradigm in recent multimodal research converts continuous human skeletal motion into discrete token sequences using vector-quantized autoencoders, subsequently expanding the vocabulary of an LLM with these motion indices to perform text-conditioned motion generation, cross-modal retrieval, and motion captioning. While this tokenization interface elegantly casts a continuous temporal physical signal into discrete sequential items that language models can naturally digest, conventional two-stage motion-language pipelines connect the vector quantizer and the downstream LLM solely through integer token IDs.

This standard handoff leaves a critical geometric disconnect between the representation spaces. During the tokenizer pre-training stage, vector quantization learns a codebook whose code vectors form specific geometric relationships, pairwise angles, and inner products in the latent space. However, when initializing the corresponding new motion-token embeddings in the language model, these pairwise geometric relationships are completely discarded. Existing systems typically resort to random embedding initialization or unconstrained learned linear projections, forcing the LLM to recover the relational topology among motion tokens purely from the sparse supervision of downstream tasks. Furthermore, standard VQ-VAE codebooks suffer from dead codes and skewed utilization distributions, where a small subset of codes monopolizes assignments and produces an ill-conditioned embedding space.

If the motion codebook can be regularized to follow a well-conditioned geometric structure and that structure can be transferred losslessly into the LLM's initial embedding space, the downstream multi-modal adaptation burden can be greatly alleviated without requiring paired cross-modal contrastive pre-training. Core idea: establish a shared approximate orthogonality prior across both stages by training a decoder-only Gumbel-Softmax quantizer with utilization entropy regularization, transferring the learned codebook Gram matrix exactly to LLM motion-token embeddings via a fixed sparse isometric projection at initialization, and applying soft orthogonality penalties during both stages to prevent directional redundancy.

Method

Overall Architecture

The pipeline of GeoMotionGPT consists of two sequential yet structurally bridged training stages: Stage-1 decoder-only vector quantization (DVQ) and Stage-2 LLM motion-token fine-tuning. Continuous 3D skeletal motion sequences are sampled at 20 FPS, z-normalized, and temporally downsampled by a factor of 8 using 1D convolutional residual blocks. The quantizer directly predicts categorical logits over \(K=512\) codes, applying Gumbel-Softmax relaxation and a straight-through estimator (STE) to jointly optimize reconstruction loss, codebook soft orthogonality, and utilization entropy regularization. At the stage handoff, a fixed parameter-free sparse projection matrix maps the \(D=512\) dimensional code vectors into the higher-dimensional LLM embedding space (\(D'=768\) for GPT-2) as an isometry, strictly preserving all pairwise inner products and the codebook Gram matrix at step zero. In Stage 2, original text embeddings remain frozen, while newly introduced motion-token embeddings and LLM backbone parameters are fine-tuned under autoregressive captioning loss combined with a soft orthogonality penalty over the motion-token block.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input motion skeletal sequence<br/>20 FPS temporal downsampling"] --> B["Decoder-only vector quantization and balanced utilization<br/>DVQ + Gumbel-Softmax + STE"]
    B --> C["Structure-preserving sparse isometric projection<br/>Fixed P matrix transferring Gram matrix"]
    C --> D["Two-stage orthogonality regularization & LLM adaptation<br/>Frozen text vocabulary + soft orthogonal penalty"]
    D --> E["Output descriptive motion caption<br/>Autoregressive sequence generation"]

Key Designs

1. Decoder-only vector quantization and balanced utilization: differentiable relaxation for unified geometric optimization Standard VQ-VAE frameworks rely on non-differentiable nearest-neighbor vector quantization and heuristic moving average or commitment losses. This prevents the codebook from being optimized directly via gradient descent alongside explicit geometric constraints within a single unified computation graph. GeoMotionGPT introduces Decoder-Only Vector Quantization (DVQ), where a 1D convolutional encoder maps motion sequences directly to assignment logits \(z = Q(x) \in \mathbb{R}^K\). To enable end-to-end backpropagation, categorical sampling is relaxed via the Gumbel-Softmax distribution: $\(y_{\text{soft}} = \text{softmax}\left(\frac{z + g}{\tau}\right), \quad y_{\text{hard}} = \text{one-hot}\left(\arg\max_k y_{\text{soft},k}\right)\)$ Crucially, the decoder network exclusively receives the discrete quantized embeddings \(h = y_{\text{hard}}^\top C\) via the straight-through estimator (STE). Because no continuous latent path bypasses the discrete bottleneck, reconstruction gradients directly shape the codebook vectors. To counter token collapse and codebook under-utilization, the model incorporates a mini-batch code utilization self-entropy maximization objective: $\(L_{\text{util}} = -H(q) = \sum_{k=1}^K q_k \log q_k\)$ By transitioning the computation of assignment statistics \(q\) from soft probabilities to hard assignments across epochs, DVQ ensures that all 512 codes remain active while curbing heavy-tailed frequency skew.

2. Structure-preserving sparse isometric projection: parameter-free exact Gram matrix transfer Following Stage 1, the learned codebook \(C = \{c_1, \dots, c_K\} \subset \mathbb{R}^D\) approximates an orthonormal set \(\langle c_i, c_j \rangle \approx \delta_{ij}\) in \(\mathbb{R}^D\) (\(K=D=512\)). However, language model embedding dimensions are typically larger (\(D' \gg D\), e.g., \(D'=768\) in GPT-2). A learned dense projection layer introduces optimization stochasticity and destroys the pre-trained pairwise geometric relations at initialization. GeoMotionGPT introduces a fixed sparse projection matrix \(P \in \{0, 1\}^{D' \times D}\) constructed by selecting \(D\) unique row indices \(I \subset \{1, \dots, D'\}\) to act as identity mappings while setting all remaining rows to zero. Because each column of \(P\) contains exactly one '1' at a distinct row index and zeros elsewhere, the columns are orthonormal, guaranteeing \(P^\top P = I_D\). For any pair of code vectors \(c_i, c_j\), the projected embeddings \(e_i = P c_i\) and \(e_j = P c_j\) satisfy: $\(e_i^\top e_j = (P c_i)^\top (P c_j) = c_i^\top (P^\top P) c_j = c_i^\top I_D c_j = c_i^\top c_j\)$ Consequently, the projection is a linear isometry that transfers all pairwise inner products and preserves the exact Gram matrix: \(\|\hat{E}\hat{E}^\top - I_K\|_F = \|\hat{C}\hat{C}^\top - I_K\|_F\). This zero-cost initialization ensures that the language model's newly added motion vocabulary starts in an identically well-conditioned geometric state.

3. Two-stage orthogonality regularization & LLM adaptation: directional anti-collapse during downstream tuning Orthogonality operates as a shared structural inductive bias across both learning phases. In Stage 1, an explicit Gram matrix penalty is applied to the row-normalized codebook \(\hat{C}\): $\(L_{\text{ortho}} = \|G - I_K\|_F^2 = \|\hat{C}\hat{C}^\top - I_K\|_F^2\)$ In Stage 2, the LLM vocabulary is expanded with \(K+3\) special tokens (512 <motion_id_k> tokens plus motion boundary and mask tokens). Original text embeddings remain frozen, while the motion embedding block \(E \in \mathbb{R}^{K \times D'}\) and LLM parameters are fine-tuned on the motion-to-text task. To prevent downstream gradient updates from collapsing motion embeddings into a small cluster of redundant directions, a soft orthogonality penalty is maintained on the row-normalized motion block \(\hat{E}\): $\(L_{\text{tuning}} = L_{\text{task}} + \lambda'_{\text{orth}} \|\hat{E}\hat{E}^\top - I_K\|_F^2\)$ This regularizer acts as a soft directional prior rather than a rigid constraint. It does not force motion tokens into fixed semantic locations or require paired multimodal contrastive objectives; rather, it maintains an isotropic, well-spread representational capacity, allowing the LLM's self-attention layers to independently develop compositional motion semantics.

Loss & Training

The Stage-1 composite loss is \(L_{\text{DVQ}} = L_{\text{rec}} + \lambda_{\text{ortho}} L_{\text{ortho}} + \lambda_{\text{util}} L_{\text{util}}\), with reconstruction error \(L_{\text{rec}} = \|x - \hat{x}\|_2^2\) and coefficients \(\lambda_{\text{ortho}} = 10^{-2}, \lambda_{\text{util}} = 10^{-2}\). Trained for 500 epochs (batch size 512, AdamW, base learning rate \(2 \times 10^{-4}\) with 3% linear warmup and cosine decay; quantizer learning rate \(10^{-4}\); Gumbel temperature \(\tau\) fixed at 0.4 for 300 epochs, exponentially annealed to 0.01 over 100 epochs). Stage-2 instruction tuning runs for 100 epochs (batch size 320, learning rate \(10^{-4}\), weight decay \(10^{-2}\), \(\lambda'_{\text{orth}} = 10^{-2}\)).

Key Experimental Results

Main Results

Evaluated on the standard HumanML3D and KIT-ML benchmarks under the GPT-2 backbone setting, GeoMotionGPT achieves superior motion captioning performance compared to competitive motion-language baselines. On HumanML3D, the descriptive aggregate Average reaches 53.48, outperforming the previous best system MotionGPT3 by 22.4%, with prominent gains in text generation metrics (CIDEr improved by 107.9%, BLEU-4 by 33.3%, BERTScore by 39.2%).

Approach (HumanML3D / GPT-2) R@1 โ†‘ R@2 โ†‘ R@3 โ†‘ MMDist โ†“ BLEU-1 โ†‘ BLEU-4 โ†‘ R-L โ†‘ CIDEr โ†‘ BERTScore โ†‘ Avg. โ†‘
Real Ground Truth 0.523 0.725 0.828 2.901 - - - - - -
TM2T (Guo et al., 2022b) 0.516 - 0.823 2.935 48.90 7.00 38.10 16.80 32.20 34.99
MotionGPT (Jiang et al., 2023) 0.543 - 0.827 2.821 48.20 12.50 37.40 29.20 32.40 38.03
LaMPM2T (Li et al., 2024) 0.547 - 0.831 2.808 47.80 13.04 37.10 28.90 32.70 38.07
MoTe (Wu et al., 2024b) 0.577 - 0.871 2.649 46.70 11.15 37.40 31.50 30.30 38.24
MotionGPT3 (Zhu et al., 2025) 0.573 0.773 0.864 2.430 59.08 19.41 46.18 28.72 35.23 43.71
GeoMotionGPT (Ours) 0.533 0.729 0.817 2.680 65.65 25.88 51.32 59.71 49.03 53.48

Note: Avg. is a descriptive composite aggregate combining scaled R-Precision and generation metrics. While MoTe and MotionGPT3 retain slight advantages on retrieval metrics (R@1 and MMDist), GeoMotionGPT demonstrates dominant captioning generation fidelity.

Ablation Study

Ablation experiments conducted on HumanML3D under the GPT-2 backbone isolate the contribution of tokenizer choice, projection strategy, regularization type, and regularization weight:

Study / Setting R@1 โ†‘ R@2 โ†‘ R@3 โ†‘ MMDist โ†“ BLEU-1 โ†‘ BLEU-4 โ†‘ R-L โ†‘ CIDEr โ†‘ BERTScore โ†‘ Avg. โ†‘
Baseline (Init โœ—, \(R=0\)) 0.488 0.668 0.712 2.943 58.70 20.41 46.52 44.80 41.90 45.77
Tokenizer: PQ-VAE (Hong et al., 2025) 0.534 0.716 0.804 2.920 60.03 20.97 46.35 44.06 41.14 46.84
Projection: Linear Projection 0.486 0.665 0.754 4.200 54.34 16.73 40.32 33.54 30.19 39.77
Regularization: Pairwise Cosine Dist. 0.513 0.691 0.781 3.070 59.34 20.75 45.73 43.71 41.53 46.20
Ortho. Weight: Init โœ“, \(R = 0\) 0.525 0.721 0.812 2.820 57.30 19.90 47.40 47.40 42.90 47.23
Ortho. Weight: Init โœ—, \(R = 10^{-2}\) 0.509 0.699 0.804 2.800 63.66 24.29 47.14 50.69 41.49 49.06
Full Model (DVQ + Sparse + \(R=10^{-2}\)) 0.533 0.729 0.817 2.680 65.65 25.88 51.32 59.71 49.03 53.48

Key Findings

  • Crucial role of sparse isometric projection: Replacing the parameter-free sparse projection with a standard learned linear layer causes the descriptive Average to plummet from 53.48 to 39.77 (-34.5%), with MMDist degrading to 4.200 and CIDEr collapsing to 33.54. This highlights that an unconstrained linear map disrupts codebook geometry during early training, whereas isometric initialization provides an optimal starting point.
  • Empirical validation of geometric orthogonality (PCS): In diagnostic checks measuring mean absolute pairwise cosine similarity (PCS), the DVQ codebook achieves a PCS of \(4.51 \times 10^{-4}\) (over two orders of magnitude lower than VQ-VAE's \(9.62 \times 10^{-2}\)). After Stage-2 tuning, motion-token embeddings retain a PCS of \(2.24 \times 10^{-3}\), well below GPT-2's surrounding text embeddings (\(2.68 \times 10^{-1}\)), confirming persistent directional conditioning.
  • Straight-through estimator is vital for training stability: Ablating the STE in Stage 1 causes final validation reconstruction loss to surge from 0.3589 to 0.9908, with 323 of 512 codes dying out (63.1% dead codes) and the most frequent code assigned 50,482 times. With STE, 100% of the codebook remains active with a maximum code count of only 6,300.
  • Cross-backbone transfer: Under identical LoRA adaptation settings on Qwen3 0.6B, GeoMotionGPT elevates the descriptive Average from 29.96 to 45.55 (+52.0% relative improvement), proving that benefits generalize beyond full fine-tuning on GPT-2.

Highlights & Insights

  • Parameter-free isometric structural bridge: Demonstrates that selecting an orthonormal coordinate slice via a fixed sparse binary matrix \(P\) transfers codebook Gram matrices to higher-dimensional spaces in exact closed form, avoiding dense adapter instability without introducing learnable parameters.
  • Decoupled geometric conditioning and semantic alignment: The framework shows that newly added modality tokens do not need paired cross-modal contrastive pre-training to establish well-behaved embedding spaces; providing an isotropic orthogonality bias allows standard language autoregressive objectives to efficiently organize semantic representations.
  • Broad relevance for discrete multimodal tokenizers: The three-step paradigm (soft-orthogonal vector quantization, sparse isometric projection, and soft-regularized fine-tuning) offers a drop-in architectural blueprint for expanding LLM vocabularies with robotics, audio, or tactile discrete tokens.

Limitations & Future Work

  • Author-admitted limitations: Backbone comparisons are partially confounded by adaptation budget: GPT-2 was fully fine-tuned, while Qwen3 0.6B and Llama 3.2 1B were trained with LoRA due to compute constraints. Additionally, orthogonality is purely a geometric condition and does not enforce semantic alignment with text; DVQ also trades a slight increase in reconstruction loss (0.3597 vs 0.2421) for improved codebook conditioning.
  • Scope limitations: Experiments evaluate motion-to-text understanding and captioning but do not explore reverse text-to-motion generation; the impact of DVQ's slightly higher reconstruction error on motion synthesis fidelity remains an open question.
  • Future directions: Investigating weak cross-modal semantic guidance alongside the soft orthogonality prior to encourage semantically related motion clusters while preserving directional diversity.
  • vs MotionGPT / TM2T: Traditional motion-language systems connect the tokenizer to the LLM purely via discrete index IDs, discarding all codebook relational geometry; GeoMotionGPT preserves the codebook Gram matrix across stages using a sparse isometric projection.
  • vs MoTe / MotionGPT3: Recent baselines deploy heavy diffusion models or large-scale multi-task pre-training. In contrast, GeoMotionGPT achieves superior captioning quality (CIDEr 59.71 vs 28.72) using standard GPT-2 purely through geometric regularization and clean stage initialization.

Rating

  • Novelty: โญโญโญโญโ˜† Elegant parameter-free sparse projection bridge combining two-stage orthogonality regularization to resolve discrete multi-modal handoffs.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluations across HumanML3D and KIT-ML, detailed component ablations, geometric diagnostics (PCS, codebook skew), and cross-LLM validation.
  • Writing Quality: โญโญโญโญโญ Rigorous mathematical formulation, clear narrative, and honest exposition of metric trade-offs and compute boundaries.
  • Value: โญโญโญโญโ˜† Highly actionable, computationally lightweight blueprint for incorporating new discrete token modalities into pre-trained LLMs.