SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translation¶

Conference: ECCV 2026
arXiv: 2606.28626
Code: TBD (Project page https://cogvis-cvssp.github.io/papers/signet)
Area: Human Understanding
Keywords: Sign Language Translation, Cross-Language Transfer, Motion Knowledge Transfer, Mixture-of-Experts, Skeleton Graph Convolution

TL;DR¶

SIGNET proposes treating multiple sign language skeleton backbones pre-trained on large corpora as frozen core motion-visual experts. By utilizing hand-prior-driven attention aggregation and learnable gating fusion, it achieves cross-language motion-level knowledge transfer. With only \(1.5\text{M}\) trainable parameters, it delivers state-of-the-art (SOTA) or comparable results on four translation benchmarks and the WLASL recognition benchmark.

Background & Motivation¶

Sign Language Translation (SLT) aims to directly translate continuous sign language videos into spoken text. Although gloss-free methods eliminate the dependency on costly intermediate annotations, they face severe challenges in cross-language generalization: almost all existing methods are trained independently for a single dataset/language. Whenever transferring to a new language, they either start from scratch or undergo full fine-tuning, directly discarding previously learned representations. This practice overlooks a crucial commonality across different sign languages—although ASL, BSL, DGS, and CSL belong to different language families with vast grammar and vocabulary differences, the underlying motion structures of articulatory organs (hand shape changes, arm trajectories, facial non-manual signals) are shared. The transferability of motion-visual patterns implies that the "motion vocabulary" captured by a backbone trained on one language could be equally valuable for another. However, there is currently no mechanism to exploit this cross-language motion knowledge transfer, leaving each new language as an isolated "training island" starting from scratch.

The core challenge of this dilemma lies in the discrepancy between data scale and model size: individual sign language datasets are too small (Phoenix14T has only 11 hours, CSL-Daily only 23 hours) to train robust general representations. Conversely, models pre-trained on large-scale corpora (YT-ASL with 984 hours, CSL-News with 1985 hours) are directly discarded when adapting to a new language. An ideal solution should reuse the complementary motion knowledge already learned by multiple pre-trained models, dynamically select the most relevant combination of expertise based on the visual features of the new language input, and do so without touching the full model parameters to avoid catastrophic forgetting. SIGNET breakthroughs by leveraging the inherent characteristics of sign language itself—since the fingers and hands are the core carriers of sign language vocabulary, the differences in hand motion patterns across languages are highly effective at distinguishing the boundaries of experts' specialties. Using this as a routing signal naturally guides the fusion system to make accurate expert selections.

The core idea of this paper is to treat multiple skeleton backbones pre-trained on different languages as frozen core motion-visual experts, extract hand spatiotemporal features from within each expert as domain-specific routing prior, generate input-dependent global expert descriptors through attention aggregation, and dynamically fuse the most useful expert outputs using an asymmetric gating network (trained via Gumbel-Softmax and inferred with Top-\(k\)). This achieves efficient cross-language motion-level knowledge transfer with only \(1.5\text{M}\) trainable parameters.

Method¶

Overall Architecture¶

SIGNET constructs a three-stage training pipeline with an inference architecture of frozen experts and lightweight adaptation. The input video is first processed by RTMPose to extract 133 full-body 2D skeleton keypoints, which are divided into four anatomical regions: body, left hand, right hand, and face. Three ST-GCN skeleton backbones, end-to-end pre-trained on large corpora (CSL-News, YT-ASL, BSL), process these regional keypoints in parallel, serving as frozen core motion-visual experts. The output embeddings of each expert are lightweight domain-fine-tuned via LoRA-style low-rank adapters. Simultaneously, the hand temporal features extracted from each expert's hand ST-GCN stream are fused into input-dependent global expert descriptors via an attention aggregation module, which are then passed to a lightweight MLP router. During the training phase, Gumbel-Softmax sampling is used to generate random fusion weights to explore the experts' specialty boundaries, while the inference phase replaces this with deterministic Top-\(k\) selection, activating only the \(k\) experts with the largest logits for weighted fusion. Finally, the fused embeddings are sent to an mT5-base encoder-decoder to autoregressively generate the target translation text.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Video"] --> B["RTMPose<br/>2D Skeleton Extraction"]
    B --> C["Four-Region Keypoints<br/>Body, Left Hand, Right Hand, Face"]
    C --> D1["CSL Expert (Frozen)<br/>ST-GCN"]
    C --> D2["ASL Expert (Frozen)<br/>ST-GCN"]
    C --> D3["BSL Expert (Frozen)<br/>ST-GCN"]
    D1 --> E1["LoRA Adapter"]
    D2 --> E2["LoRA Adapter"]
    D3 --> E3["LoRA Adapter"]
    D1 --> F["Hand Prior<br/>Temporal Average Pooling"]
    D2 --> F
    D3 --> F
    F --> G["Attention Aggregation<br/>Global Expert Descriptor"]
    G --> H["MLP Router<br/>Gumbel-Softmax → Top-k"]
    H --> I["Weighted Fused Embeddings"]
    E1 --> I
    E2 --> I
    E3 --> I
    I --> J["mT5-base<br/>Encoder-Decoder"]
    J --> K["Translation Output"]

Key Designs¶

1. Frozen Visual Experts and Lightweight LoRA Adaptation: Treating Multilingual Pre-trained Backbones as Reusable Expert Pools

The core insight of SIGNET is that although different sign languages vary in vocabulary and grammar, pre-trained ST-GCN skeleton backbones capture the underlying motion patterns of the articulatory organs—such as hand shape changes and arm trajectories—which are shared across datasets of different languages. The paper completely freezes three skeleton backbones end-to-end pre-trained on CSL-News (1985 hours), YT-ASL (984 hours), and BSL corpora (1666 hours), treating them as a heterogeneous visual expert pool. Each expert contains a four-stream ST-GCN encoding the spatiotemporal features of the body, left hand, right hand, and face, respectively. The features of each part are concatenated and projected linearly into the embedding space of mT5, trained end-to-end with language modeling loss. To compensate for the representation bias of frozen experts on the target domain, the output embeddings of each expert are followed by a LoRA-style lightweight adapter: performing layer normalization first, and then learning residual offsets through low-rank matrices (\(A: 768 \rightarrow 16\), \(B: 16 \rightarrow 768\)), which are initialized to zero to preserve the pre-trained features. The entire video encoder requires only \(1.5\text{M}\) trainable parameters, while avoiding the catastrophic forgetting caused by full fine-tuning.

2. Hand-Prior-Driven Attention Aggregation: Encoding Sign Language Domain Knowledge as Routing Signals

Standard MoEs typically use input features themselves as gating signals. In sign language translation, however, different experts excel at capturing motion patterns of different languages, and general visual features alone make it difficult to determine which expert is most relevant to the current input. The key mechanism of SIGNET is utilizing "hand priors" as routing signals—extracting temporal average pooled features from the hand ST-GCN stream within each expert, and concatenating left and right hand features to form a compact hand descriptor. The domain intuition of this design is that the hands are the primary carriers of sign language vocabulary and semantic content. Hand motions across different languages present both commonalities (basic hand shapes like fist or extended fingers) and specificities (specific hand shape configurations of vocabulary), rendering them the highest in semantic density and the most effective clue for distinguishing expert specialties. Ablation studies confirm that using only hand features as aggregation input yields the best performance, whereas adding body and face features introduces noise and degrades translation quality.

The aggregation module adopts a data-dependent attention mechanism: the hand priors of all experts are first average-pooled and passed through a two-layer MLP to generate an input-dependent query vector. A scaled dot-product attention is then calculated between this query and each individual hand prior, producing a weighted, fused global expert descriptor. This descriptor dynamically encodes which experts are most needed for the current input—if the input gesture resembles the hand motion patterns of ASL, the hand prior of the ASL expert receives a higher attention weight, guiding the gating network to prioritize selecting that expert.

3. Asymmetric Gating Fusion with Gumbel-Softmax Training and Top-\(k\) Inference: Exploring Combinatorial Diversity during Training and Ensuring Deterministic Selection during Inference

Once the global expert descriptor is obtained, a lightweight two-layer MLP router maps it to \(N\)-dimensional logits. The paper adopts different strategies during the training and inference phases: during training, Gumbel-Softmax reparameterization sampling is applied to generate random soft weights. This allows the router to randomly explore different expert combinations during the early stages (even if an expert has lower logits temporarily, it still has a chance to be selected), thereby learning the specialty boundaries of each expert without falling into local optima. During inference, this is replaced by a deterministic Top-\(k\) selection, activating only the \(k\) experts with the highest logits, and performing weighted fusion after Softmax normalization. The experiments show that \(k=2\) is optimal: single-expert information is insufficient to cover multilingual motion patterns, while active experts introduce redundancy and over-smoothing. \(k=2\) strikes a balance between sparsity and diversity. Compared with uniform average fusion and simple concatenation fusion, this choice yields gains of \(+3.66\) and \(+3.48\) BLEU-4, respectively, indicating that not all experts contribute equally to every input, and dynamic selection is crucial.

Loss & Training¶

Three-stage training: Stage I trains the backbones and LLM end-to-end on each large corpus for 20 epochs (the LLM is mT5-base with ~580M parameters) using language modeling loss. Stage II performs a 20-epoch symmetric InfoNCE contrastive alignment on downstream corpora for "unseen languages during pre-training," aligning the visual embeddings with the text embedding space of the target language, while keeping the LLM decoder frozen to preserve the language prior. Stage III fine-tunes the model on the downstream datasets for 40 epochs, updating the adapters, attention aggregation, gating, and the LLM encoder-decoder. The optimizer is AdamW, with a learning rate of \(3 \times 10^{-4}\), cosine scheduling, batch size of 8, gradient accumulation of 4, and weight decay of \(1 \times 10^{-4}\). Data augmentation includes Gaussian joint noise (\(\sigma = 0.01\)) and \(15\%\) random frame dropping.

Key Experimental Results¶

Main Results¶

Dataset	Metric	SIGNET	Prev. SOTA	Gain
Phoenix14T Test	BLEU-4	27.82	PGG-SLT 27.32	+0.50
CSL-Daily Test	BLEU-4	28.51	Geo-Sign 27.42	+1.09
How2Sign Test	BLEU-4	15.4	SSVP-SLT-LSP 15.5	Comparable (does not rely on pre-training)
MeineDGS Test	BLEU-4	2.75	Spotter+GPT 0.64	+2.11
WLASL2000	P-I / P-C	64.87 / 62.07	Geo-Sign 63.64 / 61.89	+1.23 / +0.18

SIGNET achieves SOTA among gloss-free methods on all four translation benchmarks, using only the skeleton modality. It also outperforms existing methods on the WLASL2000 isolated sign recognition benchmark.

Ablation Study¶

Configuration	Phoenix14T B-4	CSL-Daily B-4	Description
Full (3 experts, \(k=2\))	27.78	28.27	Full model
Uniform average fusion	24.12 (-3.66)	25.08 (-3.19)	Without gating, equivalent to ignoring the input
Simple concatenation fusion	24.30 (-3.48)	25.75 (-2.52)	High-dimensional concatenation leads to overfitting
w/o Contrastive (Stage II)	27.63	28.27	New language (Phx) requires alignment; same language (CSL) is unaffected
1 Backbone (no gating)	24.05 (-3.73)	25.56 (-2.71)	Insufficient single-expert capability
Leave-one-out: w/o CSL expert	24.68 (-3.10)	21.71 (-6.56)	CSL expert is most critical on CSL-Daily
Leave-one-out: w/o ASL expert	23.04 (-4.74)	26.79 (-1.48)	ASL expert is most critical on Phoenix14T (DGS)

Key Findings¶

Language-Matching Expert Dominance: The leave-one-out experiments clearly demonstrate the pattern of "the language-matching expert contributes the most." Removing the CSL expert on CSL-Daily leads to a drop of \(6.56\) BLEU-4, and removing the ASL expert on How2Sign drops by \(5.6\). For DGS (an unseen language during pre-training), the contribution of each expert is more balanced (removing ASL/BSL/CSL drops performance by \(4.74/3.76/3.10\), respectively), indicating that the framework automatically discovers complementary expertise.
Hand Priors Outperform Full-Body Priors: Using only hand features as routing inputs yields the best performance (Phoenix14T B-4 \(27.78\)). Adding body features drops performance to \(26.23\), and further adding facial features drops it to \(25.12\), indicating that high-dimensional noise dilutes the semantic density of the routing signal.
\(k=2\) is Optimal: For three backbones, \(k=2\) outperforms \(k=1\) (insufficient information) and \(k=3\) (redundancy and over-smoothing). Comparing this with the gating removal configuration indicates that the routing mechanism itself contributes a significant gain.
Pre-training Scale is Far From Saturated: Performance continuously drops when reducing the pre-training data from \(100\%\) to \(25\%\), and no saturation is observed at \(100\%\), indicating that the bottleneck lies in the coverage of the pre-training datasets rather than the capacity of the adapter module.

Highlights & Insights¶

Cross-Language Transfer Paradigm of Heterogeneous Experts + Dynamic Fusion: Treating multiple language-specific pre-trained models as a frozen core expert pool, without imposing uniform structural assumptions, theoretically allows for seamless extension to more languages (French Sign Language, Japanese Sign Language, Arabic Sign Language, etc.). This provides a scalable path for cross-language SLT without the need for full retraining.
Domain-Knowledge-Driven Gating Signal Design: Routing signals are designed based on the domain knowledge that "hands are the core semantic carriers of sign language," rather than using generic features. The ablation studies strongly support the necessity of this design from both positive and negative perspectives, making it an excellent example of domain-guided architecture design.
Dual Verification of Skeleton Advantages and Efficiency: Using only 2D skeletons and \(1.5\text{M}\) trainable parameters, the model matches or even surpasses RGB-based methods that require tens of thousands of A100 GPU hours (e.g., SSVP-SLT-LSP requires \(21,504\) A100 hours to achieve \(15.5\) B-4 on How2Sign, while SIGNET achieves \(15.4\) B-4 with only about \(576\) RTX 3090 hours of pre-training \(+\) \(24\) hours of fine-tuning), improving efficiency by nearly two orders of magnitude.
Asymmetric Training/Inference Gating: The strategy of random exploration via Gumbel-Softmax during training and deterministic Top-\(k\) selection during inference is highly generalizable, making it applicable to any scenario requiring dynamic selection from multiple pre-trained models (such as general action recognition, medical image analysis, etc.).

Limitations & Future Work¶

One-off Cost of Backbone Construction: Stage I requires training dedicated backbones on three large sign language corpora, taking approximately \(576\) RTX 3090 GPU hours each. Although this is a one-time investment, currently there are only three language experts (CSL, ASL, BSL), posing a barrier to expanding backbones for new languages for resource-constrained teams.
Extremely Low-Resource Scenarios Remain Far From Practical: On MeineDGS (\(50\) hours, ~10,000 unique glosses, where DGS is unseen in pre-training), the model achieves only \(2.75\) BLEU-4. Although this drastically outperforms the baseline (Spotter+Transformer \(1.08\)), it remains orders of magnitude away from practical applicability. The transfer effect may degrade further when the target language is distant from the pre-trained language families.
Information Bottleneck of 2D Skeletons: Pure 2D skeletons cannot capture fine-grained information such as subtle finger poses, palm orientations, and wrist rotations, which can be critical distinguishing features for certain sign vocabulary. Future work could incorporate 3D hand pose estimation or local RGB crops to supplement this.
Applicability of Evaluation Metrics: N-gram metrics like BLEU originate from machine translation and are less accurate for evaluating cross-modal, cross-lingual SLT (such as reflecting differences in word order between sign language and spoken language). Although BLEURT mitigates this by addressing the semantic dimension, the quantitative evaluation of semantic fidelity remains an open problem.

vs GFSLT-VLP / Sign2GPT / SignLLM: These RGB-based methods are trained or fine-tuned on a single language, representing "island training." SIGNET achieves cross-language transfer with a lightweight skeleton representation by employing frozen experts and dynamic fusion. While this approach performs best when the target language matches a pre-trained language, it also significantly outperforms training from scratch in cross-language scenarios.
vs Uni-Sign / Geo-Sign: While both are skeleton-based methods, they fine-tune the entire model from scratch on new languages. SIGNET's freeze-and-adapt strategy not only achieves superior performance in cross-language scenarios (CSL-Daily B-4 \(28.27\) vs. Geo-Sign \(27.42\)), but is also much more computationally efficient (forward pass of \(320\text{ms}\) per batch vs. Geo-Sign's \(2550\text{ms}\)).
vs Standard MoE: Experts in classical MoEs are typically homogeneous and trained from random initialization or a single starting point. SIGNET's "experts" are intrinsically heterogeneous—each pre-trained on a different language, making each expert a native specialist. The router's task is not to train the experts but rather to discover and schedule existing expertise, which drastically lowers the training requirements for MoE.
vs SSVP-SLT-LSP: This RGB-based method utilizes both YT-ASL and How2Sign for pre-training and requires \(64\) A100 GPUs for two weeks. Without using How2Sign for pre-training and using only ~1/64 of the resources, SIGNET achieves comparable performance (\(15.4\) vs. \(15.5\) BLEU-4. Its design philosophy—pre-training on multiple languages and performing lightweight adaptation on the target language—is more sustainable than "training on a single larger dataset."

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Creatively combines a frozen heterogeneous expert MoE with sign language domain knowledge (hand-prior routing). The cross-language transfer has paradigm-shifting significance in the SLT field.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated on 4 translation benchmarks + 1 recognition benchmark, featuring comprehensive ablations (components, number of experts, \(k\), pre-training scale, leave-one-out, and fusion strategies) along with detailed efficiency comparison data.
Writing Quality: ⭐⭐⭐⭐ Strong motivation, self-consistent logic, and deep ablation analysis. However, the methodology section is somewhat symbol-dense; many equations could be simplified into narrative to improve readability.
Value: ⭐⭐⭐⭐⭐ Provides a pragmatic and efficient framework for cross-language scalability in sign language translation. Its computational efficiency advantage (~2 orders of magnitude) gives it practical deployment potential, and its design philosophy can be migrated to other motion analysis fields.