Skip to content

SFM: Taming State Space Models for Text-to-Motion via Spatial-Frequency Modeling

Conference: ECCV 2026
Paper: ECCV Official
Area: Human Understanding
Keywords: text-to-motion generation, state space models, frequency modeling, bidirectional spatial modeling, diffusion model

TL;DR

Addressing the quadratic complexity of Transformers and the restricted receptive fields of CNNs, SFM introduces a dual-axis state-space diffusion framework that decouples spatial and frequency modeling within a UNet denoiser, attaining state-of-the-art motion fidelity and linguistic alignment in linear time.

Background & Motivation

Text-to-motion generation aims to synthesize realistic, physically viable 3D human motion sequences from natural language descriptions. In downstream scenarios such as virtual reality, character animation, embodied robotics, and autonomous driving simulation, a practical motion synthesis system must concurrently fulfill three rigorous criteria: fine-grained semantic alignment with textual prompts, long-horizon temporal consistency without physical artifacts, and low-latency inference. However, existing diffusion-based motion denoisers inevitably face design compromises. Transformer-based architectures model long-range global dependencies effectively, but their self-attention computation scales quadratically with sequence length. Conversely, CNN-based UNets are computationally lightweight and adaptable, yet fundamentally constrained by local receptive fields, often losing long-term temporal coherence and missing fine-grained directional or sequential nuances.

This inherent tension between computational efficiency and representation capacity becomes particularly acute when generating multi-stage actions or handling directional instructions. For instance, generating "A person flips to the left side" demands strict spatial awareness of directional keywords alongside high-energy body dynamics, whereas "A man crouched down and then rolled forward" requires multi-step sequential coordination with seamless phase transitions and accurate floor-contact cues. While recently emerging selective state space models (SSMs), such as Mamba, achieve linear-time sequence scaling via data-dependent state transitions and hardware-aware parallel scanning, directly transferring a vanilla Mamba backbone to motion diffusion remains inadequate. Conventional SSMs lack task-tailored mechanisms to disentangle multi-scale motion dynamics into structural and fine-grained components, and fail to provide explicit cross-modal semantic aggregation for motion-text synthesis.

The angle of attack in this work is to decompose motion sequence modeling across two orthogonal dimensions: the frequency domain, where global gross motion and fine limb articulations naturally separate on an orthogonal basis, and the spatial time-domain, where bidirectional state transitions remedy the blind spots of unidirectional causal scanning. Core idea: propose SFM, a dual-axis state-space diffusion framework integrating a Frequency State-Space Module (F-SSM) with text-gated adaptive modulation and a bidirectional Spatial State-Space Module (S-SSM) into a UNet denoiser, capturing long-range dependencies with linear complexity while achieving precise text alignment across both global body postures and high-frequency articulations.

Method

Overall Architecture

SFM is structured as a UNet-style diffusion denoiser consisting of three main stages: shallow motion feature Extraction, multi-scale Spatial-Frequency State Space Groups (SFSSGs), and a direct motion Prediction Head. Given a noisy motion sequence \(\mathbf{M}_t \in \mathbb{R}^{N \times D}\) perturbed at diffusion timestep \(t\) from clean motion \(\mathbf{M}_0 \in \mathbb{R}^{N \times D}\) (where \(N\) is the number of frames and \(D\) is the per-frame motion parameter dimension), the Extraction stage maps \(\mathbf{M}_t\) into shallow feature embeddings \(\mathbf{F} \in \mathbb{R}^{N \times D'}\) via two 1D convolutional layers with GroupNorm and Mish activations. The deep feature extraction stage then employs stacked SFSSGs with Scale-down (1D convolution) and Scale-up (transposed convolution) blocks for multi-resolution denoising. Within each group, cross-modal semantic interaction is reinforced by a Semantic Attention layer. Finally, the Prediction Head directly predicts the clean motion sequence \(\mathbf{M}'_0\) using a single 1D convolutional layer.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Noisy Motion Mt and Text Condition ctext"] --> B["Shallow Feature Extraction<br/>MishNormConv1D ×2"]
    B --> C["Conditional Feature Modulation<br/>Scale-Shift FiLM"]
    C --> D["Dual-Axis State-Space Decoupling"]
    subgraph D["Dual-Axis State-Space Decoupling"]
        direction TB
        D1["Frequency State-Space Modeling (F-SSM)<br/>DCT Transformation and Frequency SSM"]
        D2["Text-Gated Frequency Modulation (Text-Gated)<br/>CrossAttn Soft Mask M Weighting"]
        D3["Spatial State-Space Modeling (S-SSM)<br/>Bidirectional SSM Parallel Scan"]
        D1 --> D2
    end
    D --> E["Dual-Branch Summation and Residual Norm<br/>Flsf = Norm(Ff + Fs)"]
    E --> F["Semantic Attention Fusion<br/>Text as K/V, Motion as Q"]
    F --> G["Prediction Head Regression<br/>1D Conv output M'0"]

Key Designs

1. Spatial-Frequency State-Space Block: Dual-Axis Parallel Decoupling

At each UNet layer, the fundamental building block is the Spatial-Frequency State Space Block (SFSSB). Given the per-frame pose embedding \(\mathbf{p}^{(i)}\), the model first applies conditional feature-wise linear modulation (FiLM) driven by the diffusion timestep \(t\) and text embedding \(\mathbf{c}_{\text{text}}\) through MLPs to compute adaptive scale and shift parameters. The modulated feature \(\mathbf{F}^l\) is subsequently branched into two parallel pathways: the Frequency State-Space Module (F-SSM), which captures multi-scale frequency dynamics conditioned on semantic text, and the Spatial State-Space Module (S-SSM), which establishes long-horizon spatial context across frames. The outputs from both streams are fused via element-wise summation and LayerNorm normalization to form an integrated spatio-frequency representation \(\mathbf{F}_{sf}^l\). This design avoids the quadratic complexity of full spatio-temporal attention while structurally ensuring cross-domain feature synergy.

2. Frequency State-Space Module with Text-Gated Adaptive Modulation

Standard spatial-only modeling struggles to isolate macroscopic body trajectory shifts from high-frequency joint vibrations. F-SSM addresses this by projecting the input feature into two embeddings, \(\mathbf{x}_m\) and \(\mathbf{z}_m\), processed with 1D convolutions to capture local continuity. The feature \(\mathbf{x}'_m\) is then mapped to the real-valued frequency domain using the Discrete Cosine Transform (DCT), providing an ordered, compact energy representation without the numerical instability of complex-valued FFTs. A selective SSM layer then processes these frequency tokens into spectrum representation \(\mathbf{y}_f\).

To avoid the information loss incurred by hard-threshold frequency filtering, F-SSM incorporates a soft text-gated mechanism. Using auxiliary motion feature \(\mathbf{z}'_m\) as Query and text embedding \(\mathbf{c}_{\text{text}}\) as Key and Value, cross-attention generates text-fused features, from which an MLP with Sigmoid activation outputs an adaptive frequency modulation mask \(\mathbf{M} \in [0, 1]^{L \times 1}\): $$ \mathbf{M} = \text{Sigmoid}\left(\text{MLP}\left(\text{CrossAttn}\left(\mathbf{z}'m, \mathbf{c}\right)\right)\right) $$ The mask modulates the frequency feature through element-wise multiplication }\(\mathbf{y}'_f = \mathbf{M} \odot \mathbf{y}_f\), adaptively emphasizing semantic-relevant frequency components. Finally, an Inverse DCT (IDCT) converts \(\mathbf{y}'_f\) back into the spatial time domain, where it is concatenated with \(\mathbf{z}'_m\) and projected linearly to produce \(\mathbf{F}_f^l\).

3. Spatial State-Space Module: Bidirectional Sequence-Level Modeling

Standard causal SSMs process tokens unidirectionally, which is suboptimal for non-autoregressive motion diffusion where subsequent motion frames exert physical constraints and contextual cues on earlier postures. The S-SSM projects input features into representations \(\mathbf{x}_m\) and gating stream \(\mathbf{z}_m\). The sequence is processed by 1D convolution to yield \(\mathbf{x}'_o\), which parameterizes the selective discretization parameters \((\mathbf{B}_o, \mathbf{C}_o, \boldsymbol{\Delta}_o)\). The module executes parallel selective scans in both forward and backward temporal directions, producing directional features \(\mathbf{y}_{\text{forward}}\) and \(\mathbf{y}_{\text{backward}}\): $$ \mathbf{z}'m = \text{SiLU}(\text{Conv1D}(\mathbf{z}_m)), \quad \mathbf{F}_s^l = \text{Linear}\left(\left[\mathbf{y}'}} \odot \mathbf{zm, \, \mathbf{y}'_m\right]\right) $$ This bidirectional formulation provides full-sequence receptive fields, ensuring smooth acceleration profiles, seamless transitions between poses, and coherent foot-ground contacts throughout the motion horizon.}} \odot \mathbf{z

4. Group-Level Semantic Cross-Attention: Preventing Semantic Drift

At the tail of each SFSSG, SFM incorporates a Semantic Attention layer to prevent textual conditioning from degrading during multi-resolution downsampling and upsampling operations. The fused spatio-frequency feature \(\mathbf{F}_{sf}^l\) serves as Query, while the text prompt embedding \(\mathbf{c}_{\text{text}}\) serves as Key and Value. Equipped with dropout regularization, this attention mechanism reinforces global text guidance across multiple architectural levels.

Loss & Training

The network is trained to directly predict the clean motion sequence using an unconstrained mean squared error (MSE) objective: $$ \mathcal{L}{\text{simple}} = \mathbb{E}0, \boldsymbol{\epsilon}}\left[|\mathbf{M}_0 - G\theta(\mathbf{M}t, t, \mathbf{c})|_2^2\right] $$ SFM is trained using the AdamW optimizer with a learning rate of }\(1 \times 10^{-4}\) and a batch size of 64 for 200,000 steps. The forward diffusion schedule uses \(T = 1000\) steps with a linear \(\beta\) schedule. Classifier-Free Guidance (CFG) is enabled during training with an unconditional text dropout rate of \(p = 0.1\). During inference, the DPM-Solver++ sampler executes rapid sampling in just 10 denoising steps with a CFG guidance scale of \(\omega = 3\).

Key Experimental Results

Main Results

Quantitative evaluations were performed on two standard benchmarks, HumanML3D and KIT-ML, comparing SFM with leading Transformer architectures, CNN baselines, and a vanilla Mamba variant (all metrics report averages over 20 independent runs with 95% confidence intervals):

Dataset Method Architecture R-Precision (Top-1) ↑ R-Precision (Top-2) ↑ R-Precision (Top-3) ↑ FID ↓ MM-Dist ↓
HumanML3D Ground Truth (GT) - 0.511 ± 0.003 0.703 ± 0.003 0.797 ± 0.002 0.002 ± 0.000 2.958 ± 0.008
HumanML3D MDM Transformer 0.320 ± 0.005 0.498 ± 0.004 0.611 ± 0.007 0.544 ± 0.044 5.556 ± 0.027
HumanML3D MotionDiffuse Transformer 0.491 ± 0.001 0.681 ± 0.001 0.782 ± 0.001 0.630 ± 0.001 3.113 ± 0.001
HumanML3D MoMask Transformer 0.521 ± 0.002 0.713 ± 0.002 0.807 ± 0.002 0.045 ± 0.002 2.958 ± 0.008
HumanML3D StableMoFusion CNN 0.499 ± 0.004 0.680 ± 0.006 0.779 ± 0.007 0.189 ± 0.003 -
HumanML3D Motion Mamba SSM 0.502 ± 0.003 0.693 ± 0.002 0.792 ± 0.002 0.281 ± 0.009 3.060 ± 0.058
HumanML3D SFM (Ours) SSM 0.561 ± 0.004 0.749 ± 0.003 0.845 ± 0.003 0.066 ± 0.003 2.758 ± 0.086
KIT-ML MoMask Transformer 0.433 ± 0.007 0.656 ± 0.005 0.781 ± 0.005 0.204 ± 0.011 2.779 ± 0.022
KIT-ML StableMoFusion CNN 0.445 ± 0.006 0.660 ± 0.005 0.782 ± 0.004 0.258 ± 0.029 2.800 ± 0.018
KIT-ML Motion Mamba SSM 0.419 ± 0.006 0.645 ± 0.005 0.765 ± 0.006 0.307 ± 0.041 3.021 ± 0.025
KIT-ML SFM (Ours) SSM 0.453 ± 0.005 0.680 ± 0.005 0.798 ± 0.004 0.228 ± 0.024 2.548 ± 0.034

In terms of computational efficiency, SFM achieves an average inference time per sentence (AITS) of 0.19 seconds on an NVIDIA A100 GPU, drastically outperforming Transformer models such as MDM (15.33 s) and MoMask (0.25 s), while substantially surpassing the CNN-based StableMoFusion (0.13 s) in generation quality and semantic alignment. On the HumanML3D-LS long-sequence benchmark (>190 frames), SFM achieves a Top-3 R-Precision of 0.801, significantly ahead of Motion Mamba (0.713) and StableMoFusion (0.665).

Ablation Study

Ablation experiments conducted on HumanML3D validate the necessity of each architectural component, frequency filtering strategy, and spectral transform:

Ablation Category Configuration Top-1 ↑ Top-2 ↑ Top-3 ↑ FID ↓ Note
Core Components Full Model 0.561 ± 0.004 0.749 ± 0.003 0.845 ± 0.003 0.066 ± 0.003 Full dual-axis model
Core Components w/o F-SSM 0.462 ± 0.006 0.641 ± 0.005 0.733 ± 0.005 0.157 ± 0.003 Without frequency SSM; large quality drop
Core Components w/o S-SSM 0.512 ± 0.002 0.708 ± 0.002 0.779 ± 0.003 0.104 ± 0.004 Without bidirectional spatial SSM
Core Components w/o Semantic Attention 0.490 ± 0.005 0.670 ± 0.005 0.758 ± 0.005 0.116 ± 0.003 Without group-level cross-attention
Core Components w/o Text-Gated 0.528 ± 0.007 0.727 ± 0.006 0.804 ± 0.006 0.093 ± 0.004 Without soft text-gated modulation
Frequency Filtering Hard Selection k=32 0.201 ± 0.005 0.290 ± 0.005 0.397 ± 0.005 12.520 ± 0.004 Low-frequency cutoff causes collapse
Frequency Filtering Hard Selection k=128 0.430 ± 0.005 0.646 ± 0.004 0.724 ± 0.005 1.642 ± 0.004 Hard thresholding misses semantic bands
Frequency Filtering Hard Selection k=Full (208) 0.528 ± 0.007 0.727 ± 0.006 0.804 ± 0.006 0.093 ± 0.004 All frequencies without text gating
Frequency Transforms STFT 0.517 ± 0.005 0.704 ± 0.004 0.796 ± 0.004 0.112 ± 0.004 Sensitive to window hyperparameters
Frequency Transforms FFT 0.541 ± 0.005 0.718 ± 0.004 0.815 ± 0.004 0.103 ± 0.003 Complex representation complicates optimization
Frequency Transforms DWT 0.552 ± 0.004 0.736 ± 0.004 0.834 ± 0.003 0.085 ± 0.003 Discrete wavelet transform close to DCT
Frequency Transforms DCT (Ours) 0.561 ± 0.004 0.749 ± 0.003 0.845 ± 0.003 0.066 ± 0.003 Real-valued, compact energy orthogonal basis

Key Findings

  • Frequency modeling is the primary performance driver: Removing F-SSM results in a precipitous drop in Top-3 R-Precision from 0.845 to 0.733 and worsens FID from 0.066 to 0.157, verifying that frequency-domain multi-scale modeling is indispensable for capturing fine-grained human motions.
  • Adaptive soft gating dramatically outperforms hard frequency truncation: Preserving only \(k=32\) low frequencies causes model collapse (FID 12.520). Even retaining all frequencies (\(k=208\)), lacking text-driven modulation yields a lower Top-3 of 0.804 compared to 0.845 with soft gating, demonstrating that frequency importance must be dynamically dictated by linguistic semantics.
  • Denoising exhibits coarse-to-fine spectral evolution: Frequency spectrum analysis shows that in early diffusion steps (high noise), the network dedicates ~60% of spectral energy to low-frequency structural components to establish macro body poses. As denoising approaches convergence, the low-frequency energy ratio stabilizes near 40%, shifting representational capacity toward high frequencies to refine ground contacts and joint articulations.

Highlights & Insights

  • Dual-axis state-space decoupling: Addresses the vanilla Mamba shortcoming in capturing localized multi-scale dynamics by coupling real-valued DCT spectral modeling with spatial bidirectional scanning, reconciling linear scalability with fine-grained motion precision.
  • Text-gated spectral mask: Instead of employing fixed low-pass or band-pass filters, cross-attention generates input-dependent soft spectral masks, allowing prompt-specific energy distribution for both high-energy actions (e.g., flips, runs) and low-energy postures.
  • Spectral dynamics in diffusion: Unveils the progressive transition from macro-structure formation to micro-detail refinement across diffusion steps in the frequency domain, providing a grounded theoretical intuition for future frequency-guided generative models.

Limitations & Future Work

  • Lack of explicit physical contact simulation: Although high-frequency joint modeling improves limb motion, the absence of an explicit rigid-body dynamics solver occasionally permits subtle foot-sliding or slight interpenetration during aggressive ground impacts.
  • Global temporal scope of the frequency mask: The text-gated frequency mask operates globally across the sequence length, which may be sub-optimal for complex compound prompts with shifting tempos (e.g., "sprint forward, stop, and gently wave").
  • Future directions: Integrating spatial-frequency SSM modules with skeletal kinematic graph structures or physics-guided simulation layers could further boost physical fidelity while retaining ultra-fast sampling speed.
  • vs Motion Mamba [53]: Motion Mamba demonstrated the feasibility of Mamba in motion generation but relied on standard unidirectional spatial blocks without frequency decomposition or adaptive multimodal aggregation. SFM achieves substantially higher Top-3 R-Precision (0.845 vs. 0.792) and lower FID (0.066 vs. 0.281) on HumanML3D.
  • vs StableMoFusion [18]: StableMoFusion adopts a CNN-based denoiser for speed, but local receptive fields limit semantic coherence in extended sequences. SFM matches its practical inference efficiency (0.19 s vs. 0.13 s) while drastically improving prompt alignment and long-sequence stability (Top-3 0.845 vs. 0.779).
  • vs MoMask [12]: MoMask uses a discrete hierarchical Transformer tokenizer to obtain high motion quality (FID 0.045), but relies on multi-stage iterative token masking and incurs quadratic attention overhead. SFM provides superior text alignment (Top-1/3 R-Precision 0.561/0.845 vs. 0.521/0.807) and lower multimodal distance (2.758 vs. 2.958) at linear scaling.

Rating

  • Novelty: ⭐⭐⭐⭐☆ [Novel dual-axis spatial-frequency decomposition that successfully adapts Mamba for text-driven motion diffusion]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive comparisons on two benchmarks, long-sequence subsets, runtime profiling, user study, and four distinct ablation setups]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural organization, thorough mathematical formulation, and insightful frequency-domain diffusion visualization]
  • Value: ⭐⭐⭐⭐☆ [Establishes a solid, efficient state-space baseline for long-horizon 3D human motion generation]