SIGNER: Temporally Grounded Sign Language Generation via Time-Resolved Conditioning

Conference: ECCV 2026
arXiv: 2506.07460
Code: None
Area: Human Understanding
Keywords: Sign Language Generation, Temporal Gloss Conditioning, Local Temporal Fusion, Discrete Diffusion, Motion Generation

TL;DR

SIGNER proposes a time-resolved conditioning framework. By constructing a time-resolved Gloss conditioning sequence and injecting it via Local Temporal Fusion (LTF) during diffusion denoising, it explicitly preserves the temporal grounding of downstream sign language segments. This solves the core issues of disordered sign order and inaccurate semantics in existing sign language generation methods, significantly outperforming prior SOTA on CSL-Daily and Phoenix-2014T.

Background & Motivation

Sign language generation (text-to-sign generation) aims to synthesize sign language motion sequences from spoken text and is a crucial technology connecting deaf and hearing populations. Unlike general generation tasks such as text-to-image, sign language generation is constrained by two rigid linguistic requirements: first, sign language expresses meaning through a sequence of gestures aligned with Glosses (word-level semantic units), which must appear in the correct lexical order to convey the original meaning; second, each gesture must faithfully reflect the semantics of its corresponding Gloss. These two constraints collectively require the generation process to maintain temporal grounding—that is, the semantic signals of each Gloss must be strictly aligned with the corresponding gesture segments along the temporal dimension.

Limitations of Prior Work: Existing sign language generation methods generally adopt global conditioning strategies (such as cross-attention sharing conditioning signals across the entire sequence), which leads to Gloss semantics being blended along the temporal dimension, thus weakening temporal grounding. Specifically, the model does not know which Gloss should be expressed at a specific point in time, resulting in disordered sign order and blurred semantics. Although some works (e.g., G2P-DDM, NAT-EA) have introduced Gloss sequence information to reflect lexical structures, they still inject conditions via global cross-attention, meaning the temporal grounding problem is only partially mitigated rather than fundamentally solved.

Key Challenge: Sign language generation requires strict temporal Gloss-gesture alignment, but the global condition injection mechanism of mainstream diffusion/autoregressive models naturally treats the time dimension as freely mixable, creating a fundamental conflict between the two.

Key Insight: Shift the conditioning signal from "globally shared" to "time-resolved"—explicitly laying out Gloss semantics along the timeline and restricting the conditioning injection range with a local window during denoising.

Core Idea: Replace global cross-attention with time-resolved Gloss conditioning + Local Temporal Fusion (LTF) to force each timestep to only observe the semantics of the current and neighboring Glosses, thereby naturally preserving the temporal grounding and lexical order of the downstream gestures.

Method

Overall Architecture

The objective of SIGNER is to explicitly preserve the temporal grounding between Gloss semantics and gesture segments during discrete diffusion generation. The entire pipeline consists of three stages: (1) constructing time-resolved Gloss conditions from the input text—converting text into a Gloss sequence, estimating the duration in frames for each Gloss, and repeating/concatenating Gloss embeddings according to their duration to form a time-aligned condition sequence; (2) encoding the motion sequence into discrete tokens for four body parts (body, left hand, right hand, and face) using part-aware PVQ-VAEs; (3) in the discrete diffusion denoiser, passing through Self-Attention, Inter-Part Attention (IPA), and Local Temporal Fusion (LTF) in sequence at each layer, where LTF serves as the core module to guarantee temporal grounding.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Input Text"] --> B["Time-Resolved Gloss Condition Construction<br/>Gloss Extraction → Duration Estimation → Repeat & Concatenate"]
    C["Motion Sequence"] --> D["Part-Aware Motion Latent Space Learning<br/>PVQ-VAE (body/lhand/rhand/face)"]
    D --> E["Discrete Token Sequence<br/>k_body, k_lhand, k_rhand, k_face"]
    B --> F["Denoiser ×14 Layers"]
    E --> F
    subgraph Layer["Inside a Single Denoiser Layer"]
        direction TB
        L1["Self-Attention + AdaLN"]
        L2["Inter-Part Attention (IPA)"]
        L3["Local Temporal Fusion (LTF)"]
        L1 --> L2 --> L3
    end
    F --> G["PVQ-VAE Decoder"]
    G --> H["Generated Sign Language Sequence"]

Key Designs

1. Time-Resolved Gloss Condition Construction: Explicitly expanding Gloss semantics along the time axis

Text order determines sign order, but directly using text embeddings as global conditions fails to indicate what Gloss should be expressed at each moment. SIGNER's solution is to translate the input text into a Gloss sequence \(g = \{g_j\}\) using an off-the-shelf text-to-gloss model, extract embeddings \(c_j \in \mathbb{R}^{D_{\text{cond}}}\) for each Gloss via a pre-trained mBART, and predict the token-level duration \(l_j\) in frames for each Gloss through a learnable duration estimator (a 2-layer MLP). With the target generation length set to \(L_g = \sum_j l_j\), each \(c_j\) is repeated \(l_j\) times and concatenated in the original order of the Glosses to yield the intermediate sequence \(\bar{S} \in \mathbb{R}^{L_g \times D_{\text{cond}}}\). Finally, this is linearly projected to the denoiser feature dimension to obtain the final time-resolved condition \(S = \Phi(\bar{S}) \in \mathbb{R}^{L_g \times D_{\text{feat}}}\).

The key to this design is that the \(t\)-th timestep of \(S\) naturally corresponds to the semantic embedding of the current Gloss, rather than a mixture of all Glosses. This provides a precise frame-by-frame conditioning signal for subsequent LTF local injection, acting as the "temporal anchor" of the entire framework.

2. Local Temporal Fusion (LTF): Injecting conditions with a local window instead of global attention

This is the core contribution of SIGNER. Given the time-resolved Gloss condition \(S\) and the intermediate denoiser features \(\mathbf{X}_{\text{IPA}}^{\text{part}}\) (the IPA outputs of each part), LTF generates token-wise scale and shift parameters \(u, v \in \mathbb{R}^{L_g \times D_{\text{feat}}}\) from \(S\) using two MLPs, and then performs adaptive normalization coupled with a local 1D convolution:

\[\mathbf{X}_{\text{LTF}}^{\text{part}} = \text{Conv1D}\!\left(\text{LN}(\mathbf{X}_{\text{IPA}}^{\text{part}}) \odot (1+u) + v\right)\]

Where AdaLN ensures that the Gloss condition at each token index is precisely aligned with the corresponding spatial position (the \(t\)-th token uses \(u_t, v_t\) from the \(t\)-th Gloss), while the 1D convolution (kernel size=3, stride=1) aggregates adjacent context within a local window to yield smooth transitions. In contrast to global cross-attention where each motion token can attend to all Gloss tokens, LTF restricts condition fusion to a local neighborhood. This both strengthens the semantic guidance of the current Gloss (ensuring correct sign order) and prevents over-mixing of semantics across segments (ensuring semantic accuracy).

In ablation studies, replacing the 1D convolution with a fully connected layer (AdaLN + FC) led to a consistent drop in performance, demonstrating that local temporal aggregation is crucial for incorporating neighboring contexts and rendering smooth transitions. When replacing the entire LTF module with global cross-attention, performance degraded drastically (WER increased from 55.05 to 84.90), directly verifying the necessity of local conditioning injection for temporal grounding.

3. Inter-Part Attention (IPA): Coordinating body-hand-face correlation

Since SIGNER's PVQ-VAEs encode and decode the four body parts (body, left hand, right hand, face) independently, the denoising process might produce uncoordinated motions across different parts. IPA addresses this issue via cross-part attention: the body features serve as queries to attend to the concatenated hand and face features, while the hands and face each perform cross-attention with body features as keys/values, forming three attentional flows: B\(\rightarrow\)H, H\(\rightarrow\)B, and B\(\rightarrow\)F. Ablation studies show that enabling all three flows concurrently yields the best results. Activating either B\(\rightarrow\)H or H\(\rightarrow\)B alone already provides stable improvements, and incorporating the face attentional flow further supplements facial expressions, validating the importance of multi-part coordination for generation quality.

4. Part-Aware Motion Latent Space Learning: Enhancing hand and face expressiveness with PVQ-VAEs

Design Motivation: SIGNER adopts part-aware VQ-VAEs to compress motion sequences into discrete tokens. The input motion \(\mathbf{X}\) is decomposed into four parts: body, left hand, right hand, and face. Each part is equipped with an independent 1D convolutional encoder \(E^{\text{part}}\), decoder \(G^{\text{part}}\), and codebook \(\mathcal{Z}^{\text{part}}\). During training, the sum of reconstruction losses of the four PVQ-VAEs is minimized. Hand and face movements are far more critical in expressing semantics than the torso; however, in a global VQ-VAE, large-scale torso movements easily dominate codebook allocation. The part-aware design allows each part's codebook to focus on its own motion patterns, thereby improving the reconstruction precision of key signing parts.

Loss & Training

The standard discrete diffusion objective from VQ-Diffusion is adopted. Given the clean token sequence \(\mathbf{k}_0\) and the corrupted \(\mathbf{k}_t\), the denoiser predicts the posterior distribution of the original tokens \(p_\theta(\mathbf{k}_0 \mid \mathbf{k}_t, t, S)\) conditioned on the time-resolved Gloss condition \(S\) and diffusion step \(t\). The training objective is the negative log-likelihood:

\[\mathcal{L}_{\text{diff}} = \mathbb{E}_{t, \mathbf{k}_0, \mathbf{k}_t}\big[-\log p_\theta(\mathbf{k}_0 \mid \mathbf{k}_t, t, S)\big]\]

The diffusion steps are set to 50, and the denoiser consists of 14 layers with a feature dimension of 512 and 16 attention heads. Both the PVQ-VAEs and the denoiser are optimized using AdamW (lr=\(1\times 10^{-4}\), weight decay=\(4.5\times 10^{-2}\)), trained for 30K iterations each with a batch size of 56 on a single RTX 4090 GPU.

Key Experimental Results

Main Results

Method	CSL-Daily WER↓	CSL-Daily BLEU-4↑	CSL-Daily ROUGE↑	Phoenix WER↓	Phoenix BLEU-4↑	Phoenix ROUGE↑
MotionGPT (gloss-free)	94.92	1.79	15.48	99.60	0.81	7.07
MDM (gloss-free)	95.54	2.78	16.29	99.83	0.97	7.40
MoMask (gloss-free)	91.51	3.79	20.12	92.01	5.15	22.80
SOKE (gloss-free)	85.69	4.09	21.70	87.74	5.52	19.69
G2P-DDM (gloss-based)	74.28	6.44	25.31	88.00	5.17	16.78
NAT-EA (gloss-based)	67.80	7.87	29.17	75.65	8.79	23.04
SIGNER (Ours)	55.05	15.60	39.52	67.16	11.46	29.39

Under a unified evaluation setup (where all methods are re-evaluated using the same motion representation and the same back-translation evaluation model), SIGNER leads consistently across both datasets. Gloss-free methods perform the worst due to the lack of lexical order guidance. Though gloss-based methods show improvement, NAT-EA (which also utilizes time-resolved Gloss conditioning) still falls substantially behind SIGNER (CSL-Daily WER 67.80 vs 55.05). This suggests that having the Gloss condition alone is insufficient; rather, the key lies in how it is injected—where the local fusion mechanism of LTF stands as the ultimate guarantee for temporal grounding.

Ablation Study

Configuration	CSL-Daily WER↓	CSL-Daily BLEU-4↑	CSL-Daily ROUGE↑	Analysis
Cross-Attention (Global Fusion)	84.90	5.81	24.87	Global mixing disrupts temporal grounding
AdaLN + FC	72.52	7.85	27.90	Lack of local context aggregation leads to unnatural transitions
AdaLN + 1D Conv (LTF)	55.05	15.60	39.52	Frame-by-frame alignment + local smoothing yields the best overall performance
w/o IPA (All Off)	61.31	13.27	36.42	Parts are denoised independently, lacking synergy
IPA Full (B\(\rightarrow\)H+H\(\rightarrow\)B+B\(\rightarrow\)F)	55.05	15.60	39.52	Three-way attention flow synergy yields the best results

The LTF ablation shows that both global cross-attention and AdaLN+FC perform significantly worse than LTF, confirming the indispensable role of the "local window constraint" in maintaining temporal grounding. The IPA ablation demonstrates that while activating any signal flow alone yields gains, having all three active achieves the best performance, proving that the body, hand, and facial expressions provide complementary coordination.

Motion Smoothness Analysis

Model	Peak Velocity↓	Peak Jerk↓
NSA	0.75±0.20	1.75±0.59
MoMask	0.44±0.13	0.63±0.31
NAT-EA	0.43±0.10	0.69±0.31
SIGNER (Ours)	0.38±0.10	0.35±0.12

SIGNER achieves the lowest peak velocity and peak acceleration (jerk), indicating that the generated motions transition most smoothly between different Gloss segments without experiencing the abrupt segment boundary jumps commonly seen in retrieve-and-stitch approaches (e.g., Spoken2Sign).

Key Findings

• LTF is the most critical module of the framework: substituting LTF with global cross-attention causes the WER to spike from 55.05 to 84.90. This performance drop is far more severe than other ablations, validating the core hypothesis that "disrupting temporal grounding results in disordered sign order."
• Gloss duration is robust to perturbations: imposing a ±4 frame (approx. ±25%) manual increment/decrement or random perturbation on the predicted duration on CSL-Daily only alters the WER from 55.05 to 56.60~60.74, indicating that the model does not strictly rely on dead-on duration accuracy to preserve temporal grounding.
• The comparison with NAT-EA is highly illustrative: while both models utilize time-resolved Gloss conditions, NAT-EA's global cross-attention injection preserves temporal grounding only partially. SIGNER's LTF-based local injection slashes the WER by 12.75 points, proving that the injection method is more crucial than the conditioning information itself.

Highlights & Insights

• The design of LTF is essentially a "pixel-wise local" restructuring of condition injection in diffusion models: unlike mainstream global cross-attention (where conditioning signals completely mix along the temporal dimension), LTF uses AdaLN for token-wise index alignment + 1D convolution for local smoothing. This simultaneously ensures both hard lexical ordering and neighborhood continuity. This concept can be easily generalized to other conditional sequence generation tasks requiring keyframe/temporal alignment (such as text-to-motion generation, or speech-to-gesture generation).
• The insight that "the structure/format of conditioning injection is more critical than the sheer volume of information" is highly generalizable: NAT-EA holds almost the identical Gloss conditioning info as SIGNER, yet the difference in structural arrangement (global vs. local) makes a night-and-day difference in performance. This suggests we should prioritize asking "with what structure should conditions be injected" rather than "how much information should they convey."
• The part-aware PVQ-VAE design is straightforward yet highly effective: in scenarios with starkly unequal information density across different body parts (hands/face are far more important than the torso), global codebooks tend to be dominated by the torso's high-amplitude yet low-information movements. Partitioning the codebook solves this "codebook capacity allocation" issue and is directly transferable to whole-body pose estimation, gestures, etc.

Limitations & Future Work

• Reliance on external text-to-gloss models: Although experiments show that SIGNER is robust to the choice of different text-to-gloss models (Table 3a), moving to a new language dataset requires training a text-to-gloss model for that target language first. Future work could explore end-to-end joint optimization of text-to-gloss and sign language generation.
• The local window size of LTF is fixed (1D convolution kernel=3): The duration of sign segments corresponding to different Glosses can vary wildly. A fixed window may lead to under-smoothing or over-smoothing for extremely short or long Glosses. An adaptive window size (e.g., dynamically adjusting the kernel size based on Gloss duration) might bring further improvements.
• Evaluation depends on back-translation pipelines: Since back-translation itself introduces errors, metrics like WER, BLEU, or ROUGE may not perfectly reflect sign language quality. The field still lacks direct sign semantic fidelity metrics (such as subjective evaluations from native signers).
• It has only been evaluated on CSL-Daily (Chinese Sign Language) and Phoenix-2014T (German Sign Language), both of which are relatively small-scale (~20K and ~8K samples). Generalizability on larger-scale, multilingual datasets remains to be explored.

Related Work & Insights

• vs NAT-EA: Both utilize time-resolved Gloss conditioning, but NAT-EA injects it via global cross-attention, meaning temporal grounding is only partially preserved. SIGNER’s LTF enforces local window constraints, fundamentally addressing the temporal misalignment issue. This comparison between the "same information but different injection structures" reveals a universal design principle: the spatial-temporal structure of condition injection is at least as important as the conditional content itself.
• vs MoMask / SOKE: These approaches inject text features across the entire sequence via global cross-attention, completely ignoring the lexical ordering constraints of sign language. They inherently treat sign language generation (SLG) as general motion generation. SIGNER demonstrates that tailoring the injection method to domain-specific structural constraints yields massive benchmarks gains.
• vs Spoken2Sign: The retrieve-and-stitch paradigm naturally guarantees lexical order, yet suffers from jerky transitions at segment boundaries (peak jerk of 1.70 vs. SIGNER's 0.35). SIGNER blends the seamlessness of generative approaches with explicit lexical ordering, offering an elegant compromise between the two worlds.

Rating

• Novelty: ⭐⭐⭐⭐ Explicitly modeling temporal grounding—a fundamental linguistic constraint in sign language—as a spatial-temporal structure for conditioning injection. The design of LTF (AdaLN alignment + local convolution) is clean, effective, and avoids trivial module stacking.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ The main results cover two datasets and 7 baselines. The ablation study covers four dimensions (LTF, IPA, text-to-gloss choices, and duration perturbations), complemented by motion smoothness analysis and qualitative visualizations.
• Writing Quality: ⭐⭐⭐⭐ The core conflict (global fusion vs. temporal grounding) is clearly articulated. The conceptual diagrams in Figure 2 and Figure 5 intuitively convey structural design motivations, with precise mathematical formulations.
• Value: ⭐⭐⭐⭐ For the first time in the SLG domain, this work systematically demonstrates that the spatial-temporal structure of conditioning injection is critical for lexical order and semantic accuracy. As a lightweight injection paradigm, LTF has high potential to extend to other temporally-conditioned generation tasks.

Related Papers

• [ECCV 2026] SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translation
• [ECCV 2026] BackTranslation2.0 -- A Linguistically Motivated Metric to Assess Sign Language Production
• [CVPR 2026] SignPR: A Progressive Vector-Quantized Diffusion Framework for Sign Language Production
• [CVPR 2026] Learning Effective Sign Features without Text for Gloss-free Sign Language Translation
• [CVPR 2026] Focal–General Diffusion Model with Semantic Consistent Guidance for Sign Language Production