SignRefine: Adapting Foundational Video Models for Sign Language Generation¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://cogvis-cvssp.github.io/papers/signrefine/
Area: Human Understanding
Keywords: sign language video generation, foundational video diffusion models, local adapters, spatial grounding, NVSign dataset

TL;DR¶

Addressing severe articulation blur and anatomical collapse in hand and facial regions when using general video diffusion models for sign language, SignRefine introduces decoupled local spatial adapters with CoordConv positional grounding and sink-token attention masking into a frozen DiT backbone, enabling high-fidelity and comprehensible sign language video generation.

Background & Motivation¶

Sign languages are the primary communicative medium for Deaf communities worldwide. Because most sign languages lack a widely adopted written script, high-fidelity video generation is indispensable for digital accessibility, content creation, and autonomous translation. In recent years, foundational Video Diffusion Models (VDMs) built on Diffusion Transformers (DiTs) have made breathtaking strides in synthesizing realistic, temporally continuous human motion. However, directly applying these foundational video models to sign language generation consistently fails: sign language demands extreme anatomical precision across rapid finger articulations and nuanced non-manual facial cues (such as lip shapes and eyebrow movements). In standard video latents, these critical articulatory regions occupy less than 15% of the frame and undergo aggressive spatial compression within VAE patchification, causing fine structural details to be overwhelmed by the global mean-squared-error objective. Consequently, even with full-body skeleton conditioning, off-the-shelf models produce distorted, blurred, or missing hands and frozen facial expressions that render the generated signs completely unintelligible to Deaf viewers.

The field has long faced a sharp dichotomy between specialized and generalized video models. Specialized sign language synthesis models—predominantly trained from scratch using GANs or small-scale diffusion networks—are strictly confined to small, sterile datasets such as studio interpreter broadcasts. They suffer from low resolution, overfit heavily to fixed signer appearances, and fail catastrophically when generalizing across diverse environments, novel signers, or dynamic camera setups. Conversely, foundational human video animation models possess vast visual priors and high generalization capabilities, but their pretraining on natural conversational or spoken-language footage provides insufficient prior for the rapid, intricate articulatory dynamics of continuous signing. Compounding this challenge, nearly all legacy benchmarks feature interpreted signing rather than native discourse, lacking authentic multi-signer interactions, rich conversational turn-taking, and complex non-manual markers.

The angle of attack in this work is that foundational video models need not be discarded or fine-tuned end-to-end; rather, their strong generalizable prior should be preserved while explicitly disentangling articulatory channels through high-resolution, spatially grounded local adapters. Core idea: disentangle manual (hands) and non-manual (face) articulators into independent high-resolution condition streams, integrate absolute frame-level geometric coordinates via CoordConv, and inject targeted features into a frozen DiT backbone via masked cross-attention with learned sink tokens.

Method¶

Overall Architecture¶

SignRefine takes a reference image (defining signer identity and background appearance) and a driving 2D full-body skeleton sequence, feeding them into the frozen backbone of Wan2.1-1.3B-Fun-Control. To rectify anatomical distortions in the hands and face, the system extracts high-resolution ($256 \times 256$) regional crops around the face, left hand, and right hand from the driving skeleton sequence. Three independent condition encoders process these crops into spatiotemporal motion tokens. To resolve spatial ambiguity caused by local cropping, normalized absolute coordinates from CoordConv are concatenated directly into the input channels. The extracted regional motion tokens are then injected into selected DiT transformer blocks via localized cross-attention adapters. During feature injection, a spatial latent mask combined with a trainable sink token routes background queries away from the articulatory features, preventing high-frequency hand and face textures from bleeding into the background or torso. The entire generation process achieves fine-grained anatomical grounding while leaving the backbone's open-world generative priors intact.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Data<br/>Reference Image + 2D Full-Body Skeleton"] --> B["Decoupled Local Condition Encoding<br/>Face/Hands Crops + CoordConv Coordinate Injection"]
    B --> C["Spatially Grounded Cross-Attention<br/>Staggered DiT Injection + Spatial Masking & Sink Token"]
    C --> D["Regionally Weighted Diffusion Reconstruction<br/>Elevated Hand/Face Loss on Frozen Backbone"]
    D --> E["Synthesized Video<br/>Anatomically Accurate & Comprehensible Sign Language"]

Key Designs¶

1. Decoupled Local Condition Encoding with CoordConv Spatial Awareness: resolving resolution bottlenecks and local coordinate ambiguity In sign language linguistics, manual channels (hands, characterized by rapid, high-frequency structural transitions) and non-manual channels (face and head, characterized by continuous deformations like mouth morphemes and eyebrow shifts) operate as separate articulatory streams. Processing them with a monolithic encoder causes mutual feature interference. SignRefine employs three separate ResNet-style condition encoders for the face, left hand, and right hand. Each encoder downsamples a $256 \times 256$ crop into a $16 \times 16$ spatial grid, and applies stacked causal 1D temporal convolutions to compress $T$ frames into $T/4$, exactly matching the temporal downsampling factor of the Wan DiT latent space to produce motion tokens $M \in \mathbb{R}^{(T/4) \times S \times d}$. However, cropping strips away the absolute frame-level coordinates needed to project features back into the full composition. To restore global positioning, the model appends two CoordConv channels mapping each crop pixel $(i, j)$ to its normalized coordinate in $[-1, 1]$ relative to the full frame: $$x_{i,j} = \frac{2(x_1 + (x_2 - x_1) \cdot j / W)}{W_{\text{orig}}} - 1, \quad y_{i,j} = \frac{2(y_1 + (y_2 - y_1) \cdot i / H)}{H_{\text{orig}}} - 1$$ where $(x_1, y_1, x_2, y_2)$ denote the bounding box coordinates and $W_{\text{orig}}, H_{\text{orig}}$ denote original frame dimensions. This simple geometric injection ensures that the condition encoders maintain unambiguous awareness of spatial trajectory across the visual field.

2. Localized Feature Injection via Masked Cross-Attention and Sink Tokens: targeted steering without background corruption The high-resolution regional features must be integrated into the DiT backbone without degrading torso consistency or background stability. In the 30-block DiT architecture, SignRefine attaches the face adapter cross-attention at blocks $\{0, 6, 12, 18, 24\}$ and the hand adapter at blocks $\{2, 8, 14, 20, 26\}$. This staggered arrangement prevents any individual transformer block from receiving simultaneous multi-adapter residual shocks, balancing conditioning loads across depth. In addition, left-hand and right-hand conditions are concatenated before injection to facilitate modeling inter-hand occlusions and coordination. To enforce strict spatial localization in latent space, the bounding boxes are projected onto the coarse latent grid to form a binary mask $m \in \{0, 1\}^S$. A trainable sink token $s \in \mathbb{R}^{1 \times d}$ is appended to the motion sequence $M$. The cross-attention weights are computed via: $$A_{i,j} = \begin{cases} \text{softmax}\left(\frac{\mathbf{q}_i \cdot \mathbf{k}_j}{\sqrt{d}}\right), & \text{if } m_i = 1 \text{ and } j \le N \text{ (foreground region)} \\ \text{softmax}\left(\frac{\mathbf{q}_i \cdot \mathbf{k}_{\text{sink}}}{\sqrt{d}}\right), & \text{if } m_i = 0 \text{ (background query to sink)} \end{cases}$$ Foreground latent tokens ($m_i = 1$) attend strictly to the $N$ extracted regional motion tokens, whereas background tokens ($m_i = 0$) are diverted to the sink token. The sink token acts as a zero-gradient absorption buffer, preventing fine-grained hand or facial details from leaking into unrelated image regions.

3. Regionally Weighted Diffusion Loss: counteracting area imbalance and gradient starvation During denoising diffusion training, standard uniform MSE distributes gradients proportionally to pixel area. Because hands and face together occupy only about 15% of the frame latent space, they receive negligible gradient under uniform weighting. SignRefine partitions the latent elements of each frame into hands, face, and background regions (with hand masks taking precedence over face in overlapping zones), computing a weighted MSE objective: $$\mathcal{L} = \frac{w_h N_h \mathcal{L}_h + w_f N_f \mathcal{L}_f + w_b N_b \mathcal{L}_b}{w_h N_h + w_f N_f + w_b N_b}$$ Setting $w_h = w_f = 10$ and $w_b = 1$ shifts the optimization focus toward fine articulatory geometry while still allocating approximately one-third of total gradient mass to the background to maintain global body identity and temporal coherence. Output projection layers in the adapters are zero-initialized, ensuring stable cold-start training on top of the frozen backbone.

Loss & Training¶

The entire Wan2.1-1.3B-Fun-Control DiT backbone (T5 text encoder, CLIP image encoder, 3D VAE, and 30 DiT blocks) remains frozen throughout training. Only the three condition encoders, cross-attention projection layers, and sink tokens are trained. Latents for a generic text prompt are cached offline to eliminate repetitive text encoding overhead. The model is trained on approximately 40,000 video clips from the NVSign training set for 30,000 iterations using the Adam optimizer with a learning rate of $1 \times 10^{-4}$, gradient accumulation over 8 steps, per-GPU batch size of 1, and mixed-precision computation across 4 NVIDIA GH200 GPUs.

Key Experimental Results¶

Main Results¶

Quantitative evaluations are conducted across three diverse benchmarks: NVSign (in-the-wild native signing, appearance-independent test split), Phoenix14T (broadcast German sign language), and BSL Corpus (studio conversational British sign language). Joint accuracy is measured via MPJPE using 133 keypoints extracted by RTMPose-X, alongside perceptual metrics LPIPS and SSIM.

Dataset	Method	Face MPJPE↓	Face LPIPS↓	RH MPJPE↓	RH SSIM↑	LH MPJPE↓	LH SSIM↑
NVSign	SignGAN	10.424	0.564	24.774	0.388	21.852	0.384
	SignViP	10.576	0.534	23.957	0.428	24.383	0.424
	Wan-VACE	4.067	0.422	15.904	0.380	15.431	0.407
	Wan-FC	1.835	0.197	9.565	0.583	9.341	0.597
	Ours	1.770	0.194	6.602	0.629	6.638	0.640
Phoenix14T	SignGAN	5.457	0.585	28.507	0.469	25.373	0.470
	SignViP	7.194	0.456	10.345	0.425	14.405	0.376
	Wan-VACE	2.162	0.383	11.717	0.367	13.214	0.299
	Wan-FC	1.151	0.223	5.754	0.629	5.719	0.642
	Ours	0.916	0.214	4.582	0.660	5.101	0.668
BSL Corpus	SignGAN	9.415	0.634	22.726	0.406	16.858	0.390
	SignViP	28.222	0.626	23.085	0.381	22.111	0.372
	Wan-VACE	3.172	0.436	17.524	0.351	15.385	0.338
	Wan-FC	1.790	0.221	8.232	0.559	7.002	0.563
	Ours	1.524	0.222	6.314	0.608	5.406	0.607

In a blind comprehension study conducted with 6 fluent BSL signers evaluating 10 clips from YouTube-SL-25, SignRefine dramatically outperformed the best baseline Wan-FC: Keyword Recall rose from 0.30 to 0.54, BLEURT improved from 0.19 to 0.31, LLM Adequacy increased from 1.50 to 2.07, and the clip Failure Rate dropped from 0.27 to 0.17.

Ablation Study¶

Config	Face MPJPE↓	Face LPIPS↓	LH MPJPE↓	LH LPIPS↓	RH MPJPE↓	RH LPIPS↓	Note
Sparse conditions (full model)	1.770	0.194	6.638	0.321	6.602	0.330	Default 2D sparse keypoint crops
Dense conditions (mesh & normals)	1.692	0.192	6.665	0.319	6.713	0.329	3D MANO + surface normals; no benefit on hands
w/o attention masking	1.949	0.214	7.309	0.353	7.270	0.363	Dropping masks & sink tokens increases hand error by ~10.1%
w/o regional coords	1.847	0.202	6.926	0.335	6.888	0.344	Without CoordConv, error consistently degrades by ~4.3%

Key Findings¶

Spatial masking and sink tokens prevent cross-region feature contamination: Removing the attention mask and sink token degrades hand MPJPE by over 10.1%, demonstrating that unconstrained cross-attention allows high-frequency local tokens to pollute background latents, whereas sink tokens safely absorb irrelevant queries.
High-resolution localized cropping dominates over 3D dense priors: Dense 3D conditioning (MANO hand meshes and face normals) offers a negligible gain on the face and no improvement on hands compared to sparse 2D keypoints. The pre-trained DiT backbone already possesses strong internal structural priors; spatial resolution magnification ($256 \times 256$ crop) is the primary driver of articulation quality, rendering costly 3D mesh reconstruction redundant.
Severe domain collapse in specialized baselines: SignViP, trained only on Phoenix14T, experiences a catastrophic doubling of hand error when transferred to in-the-wild NVSign data (10.345 to 23.957 MPJPE). In contrast, SignRefine preserves the broad visual distribution of the base video foundation model, generalizing smoothly across all three benchmarks.

Highlights & Insights¶

Surgical micro-structure editing via CoordConv and Sink Tokens: By pairing absolute normalized coordinate encoding with an attention sink token, SignRefine establishes an elegant paradigm for refining tiny, high-frequency regions without fine-tuning foundational video weights. This modular architecture can be directly transferred to other fine-grained video control tasks such as portrait animation, hand-object tool manipulation, and micro-expression synthesis.
Pioneering native conversational sign language benchmark (NVSign): Moving past decades of sterile, broadcast-interpreter corpora, NVSign introduces 78.1 hours of native BSL signing across 2,184 signers, featuring 40% multi-party conversational interaction and dynamic camera framing, providing a transformative resource for sign language processing and multi-signer conversational AI.
Rigorous human-centered comprehension validation: Going beyond standard pixel-level reconstruction heuristics, the authors paired geometric pose estimation with blind Deaf user rankings and free-text semantic recovery, demonstrating over 80% head-to-head preference and proving that micro-geometric improvements translate into genuine semantic intelligibility.

Limitations & Future Work¶

High diffusion latency precludes real-time deployment: Generating an 81-frame video takes minutes on high-end GPUs, preventing real-time assistive translation; future work should explore consistency distillation or flow matching for accelerated sampling.
Artifact inheritance from training motion blur: Filmed at standard 25 fps, rapid co-articulation transitions in the training data exhibit physical motion blur, leading to occasional hand smearing during high-velocity signs.
Cascading sensitivity to keypoint estimation errors: The conditioning pipeline relies on off-the-shelf detectors like RTMPose and WiLoR; severe tracking noise or self-occlusions in the input skeletons propagate directly into distorted video generations.

vs SignGAN / SignViP: Traditional specialized models train from scratch on small studio datasets, resulting in severe overfitting and catastrophic generalization drops in the wild. SignRefine freezes a 1.3B foundational video DiT and employs lightweight adapters to retain open-world appearance and background priors.
vs Animate Anyone / UniAnimate: Foundational human image animation models focus on whole-body dance and pose tracking, where coarse full-frame latent tokens compress hands and face into blurry blobs. SignRefine decouples manual and non-manual channels with dedicated $256 \times 256$ regional adapters to achieve linguistic-grade articulation.

Rating¶

Novelty: ⭐⭐⭐⭐☆ The combination of decoupled condition encoders, CoordConv spatial grounding, and sink-token attention routing provides a clean, principled solution to latent video compression bottlenecks.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across three benchmarks, comprehensive component ablations, and extensive Deaf user preference and translation comprehension studies.
Writing Quality: ⭐⭐⭐⭐⭐ Exemplary clarity, lucid linguistic motivation, and precise technical descriptions.
Value: ⭐⭐⭐⭐⭐ Bridges foundational generative AI with sign language accessibility, providing both an effective methodology and the groundbreaking NVSign dataset.