SignNet-1M: Large-Scale Multilingual Sign Language Video Dataset with Downstream Benchmarks¶

Conference: ECCV2026
arXiv: 2606.24361
Code: https://signnet.chatsign.ai/
Area: Human Understanding
Keywords: Sign Language Translation, Data Augmentation, 3D Gaussian Splatting, Novel View Synthesis, Distribution Shift Robustness

TL;DR¶

SignNet-1M leverages 3DGS novel view synthesis, diffusion-based scene/identity editing, and post-rendering augmentation to expand 7 public sign language corpora into approximately 1 million multilingual sign language video clips (ASL/CSL/DGS). It introduces a unified evaluation protocol (Orig/Zero-shot/Trained) to expose and mitigate the robustness vulnerabilities of existing models under view, background, and identity variations.

Background & Motivation¶

Significant progress has been made in Sign Language Translation (SLT) and Sign Language Recognition (SLR) on standard benchmarks such as Phoenix14T, How2Sign, and CSL-Daily. However, these datasets are almost exclusively recorded under fixed conditions: front-facing close-up cameras, monotonous studio backgrounds, and a very limited number of signers. State-of-the-art models like SpaMo and UniSign perform exceptionally under such i.i.d. evaluations, but their performance drops drastically when deployed in real-world scenarios where the camera views the signer from the side, the background is a cluttered living room, or the signer is an unfamiliar individual. This blind spot is quantitatively evaluated: SpaMo achieves a BLEU-4 of 22.49 on the original Phoenix14T dataset, but its zero-shot performance plunges to 7.81 with minor view perturbations, representing a drop of 14.68 points. Traditional pixel-level augmentations (cropping, flipping, color jittering) fail to synthesize topologically consistent 3D view changes and struggle to replace scenes and characters while preserving fine-grained hand movements.

Recent advances in 3D Gaussian Splatting (3DGS) and diffusion-based video editing make "structured augmentation" feasible. 3DGS supports explicit camera control for novel view synthesis, while diffusion models (e.g., FlowPortal, IC-Light) enable temporally consistent scene replacement and relighting. The key insight of this paper is that by combining these two generative techniques into a sign-language-specific augmentation pipeline, controllable variations can be systematically generated along three axes: view, background, and identity. This is achieved while keeping hand poses and linguistic annotations intact, thereby revealing uncovered robustness vulnerabilities in existing datasets and mitigating them through training on augmented data. Core Idea: Construct a three-stage generative augmentation pipeline—diffusion-based background replacement, 3DGS novel view rendering with cross-identity reenactment, and post-rendering jittering—to scale 7 sign language corpora into a million-scale multilingual dataset, SignNet-1M. Accompanied by a unified Orig/Zero-shot/Trained evaluation protocol, the trained models improve BLEU-4 by 4–15 points under distribution shifts without compromising performance on the original distribution.

Method¶

Overall Architecture¶

The augmentation pipeline of SignNet-1M consists of three sequential stages, each generating controllable variations along an independent distribution shift axis. The input is the original sign language video clip from the source corpus, and the output is the augmented clip preserving the original linguistic annotations (gloss/translation). Stage 1 uses diffusion models for background replacement and lighting editing to create scene diversity. Stage 2 serves as the core: the signer is first tracked into SMPL-X/FLAME parametric representations using EHM-Tracker, and then GUAVA (a 3DGS-based animatable avatar renderer) is utilized to perform novel view synthesis (static/dynamic cameras) and cross-identity reenactment. Stage 3 applies lightweight post-rendering augmentations (video-consistent spatial transformations and subtle temporal resampling) to further simulate geometric, photometric, compression, and temporal degradation from real-world capture.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Original Sign Language Clip"] --> B["Stage 1: Background Replacement<br/>FlowPortal + IC-Light"]
    B --> C["Stage 2: Parametric Tracking &<br/>3DGS Augmented Rendering"]
    C --> D["Sub-pipeline 2a: Novel View Synthesis<br/>(Yaw/Pitch/Zoom/Dynamic)"]
    C --> E["Sub-pipeline 2b: Cross-Identity Reenactment<br/>(Signer Appearance Mimicry)"]
    D --> F["Stage 3: Post-Rendering Augmentations<br/>(Geometry/Photometry/Degradation/Temporal)"]
    E --> F
    F --> G["SignNet-1M<br/>~1M Augmented Clips"]
    G --> H["Downstream Evaluation:<br/>Orig / Zero-shot / Trained"]

Key Designs¶

1. 3D Gaussian Splatting Novel View Synthesis: Synthesizing Continuous View Changes under Geometric Consistency

One of the fundamental vulnerabilities of current sign language models is their extreme sensitivity to camera viewpoint shifts—models trained with frontal "vase" data fail completely when deployed with side or high-angle cameras. Traditional cropping and perspective transforms cannot synthesize 3D-consistent view changes because pixel-level operations lack knowledge of the subject's 3D structure. To solve this, SignNet-1M fits the signer frame-by-frame using EHM-Tracker into a parametric SMPL-X (body) + FLAME (face) model, decoupling each clip into identity parameters \((\beta, \Delta\mathbf{j}, s_h, s_w, \gamma)\) that remain constant across clips, and time-varying motion parameters \((\psi_t, \mathbf{e}_t, \mathbf{j}_t, \mathbf{c}^{\text{src}}_t)\). The motion parameters \(\psi_t\) then drive GUAVA—an animatable avatar renderer based on 3DGS—to render the signer under preset camera angles, executing an alpha-blend onto the background. The paper defines \(K=10\) "LookAt"-centered spherical camera presets (yaw \(\phi \in [0^\circ, 30^\circ]\), pitch \(\theta \in [0^\circ, 25^\circ]\), and zoom \(r \in [0.6, 1.5]\)) indexed L1–L10 in ascending perturbation severity. Besides static views, it supports dynamic camera trajectories (linear interpolation between two endpoints) to simulate natural view variations as the signer moves while signing. The ingenuity of this design lies in using parametric tracking to embed the 3D geometric prior of the signer into the augmentation pipeline, ensuring all viewpoint transformations occur in a geometrically consistent representation, free from the shape distortion and hand fragmentation common in traditional pixel-level augmentations.

2. Diffusion-Based Background Replacement and Lighting Editing: Generating Scene Diversity While Preserving Sign Semantics

Beyond viewpoint shifts, background variation is another massive robustness vulnerability—models trained on plain studio backgrounds suffer severe degradation under cluttered indoor scenes or natural outdoor lighting. However, augmenting along this dimension is challenging: crudely matting the signer and pasting them onto a new background creates artifacts like harsh edges, mismatched lighting, and missing shadows, which become new cues for model overfitting. SignNet-1M addresses this by first estimating a temporally consistent soft alpha mask \(\alpha^{\text{bg}}\) via video matting. It then uses IC-Light to generate light-aware reference images, followed by FlowPortal (a diffusion-based temporally consistent video editing framework) to perform flow-guided editing with residual correction in latent space. The edited results are blended with the original hand regions using \(\alpha^{\text{bg}}\) to ensure that fine hand actions and facial expressions are completely untouched by background editing. To quantitatively evaluate the impact of background changes on the model, the paper defines a signer-region lighting shift score \(s_{\text{light}}\): it computes the luminance difference \(\Delta_L\) and chrominance difference \(\Delta_{ab}\) of the signer area in Lab color space before and after editing, normalizes it to \([0,1]\) via weighted correlation, and bins the values into L1–L10 severity levels. This enables the evaluation to pinpoint exactly at what level of lighting variation the model begins to degrade, providing fine-grained diagnosis rather than a generic "background change works/does not work" conclusion.

3. Cross-Identity Reenactment: Decoupling Identity and Motion for Signer Replacement

The diversity of signer identity is another neglected robustness dimension—if a model has only seen two or three signers, it fails to recognize the same gestures done by an unfamiliar person. Traditional approaches either collect massive volumes of sign videos with diverse signers (prohibitively expensive) or employ simple facial swapping (disrupting structural motion coherence). SignNet-1M exploits the core advantage of parametric decoupling: extracting identity parameters \(\Theta^{\text{id}}\) from a source image and motion parameters \(\Theta^{\text{mot}}\) from a target clip, recombining them into \(\tilde{\theta}_t = (\Theta^{\text{id}}, \Theta^{\text{mot}}_t)\), and rendering them directly through the GUAVA renderer and background blending pipeline. This implies the appearance of any signer in the source image can be "mapped" onto the motion sequence of the target clip, while keeping hand poses, body gestures, and facial expressions unchanged. According to human evaluations, clips generated via cross-identity reenactment were judged semantically correct by 14 native ASL signers at a rate of 94.8% (compared to >99% for novel view/background replacement). Although it is the most challenging augmentation type, it yields the highest robustness gain. Ablation studies show that identity editing is the single component associated with either the largest performance drop or the highest gain under Zero-shot and Trained protocols (Phoenix14T Zero-shot 13.06 vs Full 7.81, indicating that although identity changes are difficult to fully align, learning them yields the best model generalization).

4. Post-Rendering Augmentation: A Low-Cost Diversity Layer Mimicking Real-World Capture Degradations

Following the three generative staging phases, SignNet-1M applies a lightweight post-processing layer to simulate typical degradations in real-world capturing, encoding, and transmission. This includes geometric transforms (cropping/rotation/perspective), photometric jittering (brightness/saturation/gamma), degradation simulation (blurring/noise/compression artifacts), and temporal resampling (framerate variations/slight speed jittering/frame dropping). Although this stage has lower "technological novelty," it exhibits indispensable complementarity in ablation: depending solely on the three generative stages results in a Zero-shot BLEU-4 of 5.37 (Phoenix14T), which recovers to the system's full 7.81 with post-rendering additions. This suggests that while generative augmentations address structured distribution shifts (views, scenes, identities), real-world videos still contain abundant scale-agnostic "unstructured" degradations (low bitrate, poor lighting, dirty lens) that require a cheap and simple layer for comprehensive coverage.

A Complete Example¶

Taking a DGS sign language video from Phoenix14T (approximately 4 seconds, frontal close-up, studio white background, 1 signer) as an example. The pipeline first enters Stage 1: replacing the background with a sunlit outdoor garden scene (synthesized via FlowPortal + IC-Light), resulting in a lighting shift of approximately L4. In Stage 2, EHM-Tracker fits the SMPL-X parameters, and GUAVA renders the clip from a yaw of \(+15^\circ\) and zoom of \(0.8\) (camera preset L5). Concurrently, a source signer image is selected from another ASL corpus, How2Sign, injecting the identity into the DGS motion sequence. This generates a cross-identity reenactment version depicting "a new person performing DGS sign language in a garden." Stage 3 further applies lightweight Gaussian blur and temporal downsampling of \(0.9\times\). Ultimately, a single original clip passes through the entire pipeline to yield 3 augmented variants (novel view + dynamic camera blend / background replacement / identity reenactment), all maintaining the original gloss annotations.

Loss & Training¶

Naturally, the native training configuration of the downstream backbones remains unchanged during evaluation: SpaMo utilizes Flan-T5-XL + LoRA (\(r=16\), \(\alpha=32\)), optimized via AdamW (\(\text{lr}=6\times 10^{-4}\), cosine schedule) up to 500 epochs with early stopping. UniSign employs an ST-GCN encoder + mT5-Base decoder, optimized via DeepSpeed ZeRO-2 and AdamW (\(\text{lr}=10^{-3}\)). Ablation studies control all training settings strictly identical except for the data, ensuring that the improvements originate from the data diversity of SignNet-1M rather than additional optimization schedules.

Key Experimental Results¶

Main Results¶

Table 3 presents the SLT BLEU-4 evaluation on four sub-datasets across Orig / Zero-shot / Trained settings, and Table 4 shows the corresponding SLR WER% results.

Sub-dataset	Method	A (Orig)	B (Zero-shot)	C (Trained)	Gain (C−B)	D (Trained \(\rightarrow\) Orig)
Phoenix14T	SpaMo	22.49	7.81	18.98	+11.17	27.78
How2Sign	SpaMo	10.11	6.25	10.36	+4.11	18.48
OpenASL	UniSign	22.67	8.12	22.83	+14.71	—
CSL-Daily	SpaMo	20.55	12.70	21.32	+8.62	20.93

Sub-dataset	Method	Orig WER%	Zero-shot WER%	Trained WER%	Gain
Phoenix14T	Online-CSLR	22.21	51.45	28.19	+23.26%
CSL-Daily	UniSign	28.20	56.90	30.74	+26.16%

Ablation Study¶

Table 5 shows the ablation of individual augmentation components (Phoenix14T, SpaMo):

Configuration	Zero-shot BLEU-4	Trained BLEU-4	Description
Novel View (Single View)	7.10	18.85	Only baseline novel view rendering
+ yaw	8.17	18.59	Stable gain along yaw direction
+ pitch	6.98	18.36	Pitch is the hardest to learn
+ zoom	8.89	18.98	Zoom improvement is the most consistent
+ dynamic camera	6.87	17.13	Dynamic camera is the most unstable
Scene editing	4.91	12.53	Performance drops most severely with only background replacement
Identity editing	13.06	22.04	Identity editing yields the best Zero-shot performance
Post-rendering aug.	5.37	18.16	Post-processing alone has limited effect
Full SignNet-1M	7.81	18.98	Complete pipeline achieves the best performance

Key Findings¶

Identity editing is a double-edged sword but yields the highest gain: In Table 5, Identity editing alone achieves a Zero-shot BLEU-4 of 13.06, far exceeding the Full model (7.81). This is because identity ablation evaluates only a single shift corridor, whereas the Full model confronts concurrent multi-modal shifts across views, scenes, and identities. However, identity editing exhibits the most dramatic post-training boost (Trained 22.04 vs Full 18.98), underscoring cross-identity augmentation as the most valuable individual signal for training.
Gains scale with view/light shift severity: The severity-stratified analysis in Figure 4 demonstrates that Zero-shot BLEU-4 drops from 8.31 at L1 to 6.94 at L10 under view perturbations. Under lighting changes, it drops even more dramatically (7.02 \(\rightarrow\) 2.96). Crucially, the training gains are maximized at L10 (+9.92 for view, +6.85 for lighting), showing that SignNet-1M delivers the highest utility under the most severe perturbations.
Scaling experiments demonstrate diversity overrides data volume: Under matched-compute (GPU hours) scenarios, the performance curves for \(K=2, 5, 10\) overlap heavily (Figure 6). This demonstrates that the gains of SignNet-1M do not merely come from "viewing the same data multiple times," but rather from the expanded diversity introduced by larger-scale augmentations.
Human evaluation verifies semantic preservation: Evaluations conducted by 14 native ASL signers over 27,000 synthetic video pairs show acceptance rates \(>99\%\) for novel view, background, and post-processing augmentations. Even the most challenging cross-identity reenactments achieved a \(94.8\%\) acceptance rate.

Highlights & Insights¶

The paradigm of "structured augmentation" is more transferable than the concrete methods: The paper does not invent new generative models, but instead ingeniously packages existing techniques (3DGS, diffusion video editing, parametric tracking) into a cohesive sign-language-specific augmentation pipeline. This "disassemble by shift axis + reassemble" paradigm can readily migrate to other human motion understanding tasks demanding robustness (e.g., dance recognition, motion capture, social signal analysis).
The three-state Orig/Zero-shot/Trained protocol is an elegant tool for evaluating robustness: It concurrently gauges "how fragile existing models are" (Gap = B\(-\)A) and "how powerful the augmented data is" (Gain = C\(-\)B), coupled with Setting D to verify whether the augmentations degrade performance on the original distribution. This three-state design is vastly more informative than simple "with vs. without augmentation" baselines and can be directly adopted by other fine-grained video understanding tasks.
The high gains of identity editing are driven by "combinatorial explosion": Pairing \(N\) source signer identities with \(M\) target motion clips theoretically yields \(N \times M\) unique signer-motion combinations. SignNet-1M reports approximately 10,000 signer identities (including synthesized ones). Replicating this diversity through manual acquisition would be orders of magnitude more expensive.
The paper open-sources all step-by-step configuration files and evaluation scripts, rather than just the final dataset. This allows researchers to replicate the parameter settings for each augmentation axis or regenerate them under modified terms, showing an exemplary level of reproducibility for data papers.

Limitations & Future Work¶

Reconstruction quality in GUAVA depends on a single image: Under vast viewpoint changes, rendering quality for fine hand details and loose clothing degrades. For sign language, which relies heavily on microscopic finger movements, any structural artifacts in the hands can alter the linguistic meaning. Future work could incorporate multi-view inputs or multi-frame fusion to improve rendering fidelity.
Only three sign languages (ASL/DGS/CSL) are covered: There are over 140 sign languages globally, meaning SignNet-1M's linguistic scale remains far from complete. Furthermore, the visual features of different sign languages vary dramatically (e.g., ASL using single-handed alphabet fingerspellings vs. British Sign Language (BSL) using two-handed ones), implying that the augmentation pipeline parameters likely require language-specific tuning.
Source images for identity reenactment originate from public datasets: This bounds the upper limit of synthesized identity diversity—if the source datasets have limited demographic variety, the augmentation will struggle to introduce entirely new ethnicities, ages, or clothing distributions. Integrating StyleGAN or diffusion models to directly generate virtual identity images could overcome this upper bound in the future.
The computational overhead of augmentation is non-trivial: Constructing the complete SignNet-1M dataset requires approximately 12K GPU hours. Facing such expensive synthesis limits, constructing million-scale augments might remain impractical for labs with restricted compute resources. On-demand rendering or few-shot augmentation strategies represent valuable future directions.

vs Phoenix14T / How2Sign / CSL-Daily: These are monolingual sign language datasets captured under fixed configurations (only 9–10 signers, near-frontal views) with no controllable distribution shifts. SignNet-1M takes them as source corpora and augments them along three controllable axes rather than collecting raw data from scratch, scaling the training sets from thousands/tens of thousands of clips to the million scale.
vs RandAugment / AugMix / CutMix: Generic visual augmentation methods work phenomenally on image/video classification, but they are incapable of synthesizing 3D-consistent viewpoint changes or semantically aligned background replacements—the exact gap SignNet-1M bridges using 3DGS and diffusion models.
vs SignBT / cross-modality SLT: These methods alleviate data sparsity via back-translation or multi-modal fusion, but they do not alter the visual distribution itself. SignNet-1M acts as a direct patch toward visual distribution shifts, making them complementary to each other.
vs other 3DGS augmentation works (e.g., GUAVA itself): GUAVA is a generic animatable avatar renderer. SignNet-1M packages it as a single component of a sign language augmentation pipeline, adding sign-specific EHM-Tracker/SMPL-X tracking, background editing, and unified evaluation protocols. Rather than a simple engineering duplicate, it represents domain-specialized assembly and systematic evaluation.

Rating¶

Novelty: ⭐⭐⭐⭐ [Assembling 3DGS + diffusion-based editing into a sign-language-specific augmentation pipeline represents an innovative system-level design, though the individual components are existing techniques with limited core algorithmic innovations]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Covers 3 languages, 4 sub-datasets, 2 tasks, 3 evaluation protocols, severity-stratified analysis, component ablations, scaling tests, visual quality assessments, human evaluations, and matched-compute comparisons—representing one of the most systematic sign language robustness studies in recent years]
Writing Quality: ⭐⭐⭐⭐⭐ [The motivation is exceptionally clear (demonstrating the gap quantitatively first to persuade the reader), with complete methodological details (including hyper-parameters of each stage in the appendix), and progressively structured experiments]
Value: ⭐⭐⭐⭐⭐ [Releasing a million-scale augmented multilingual sign language dataset, complete pipeline code, and unified evaluation protocols—making it highly likely to become a standard benchmark for sign language robustness research]