Video-Text Alignment Model for Sign Language Translation

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/junyi2005/vtamo
Area: Human Understanding
Keywords: Continuous Sign Language Translation, Gloss-free SLT, Cross-Modal Alignment, Optimal Transport, Pseudo-Gloss

TL;DR

Addressing the severe temporal discrepancy between sign gesture order and spoken word order in gloss-free sign language translation, VTaMo introduces an explicit multi-granularity alignment framework that reorders visual features via entropy-regularized optimal transport with a learnable null token, establishes global geometric calibration through an orthogonal transformation, and enforces token-level grounding via position-aligned contrastive learning, establishing new state-of-the-art benchmarks across four datasets.

Background & Motivation

Continuous sign language translation (SLT) aims to translate continuous sign language videos into fluent spoken language text, serving as a critical artificial intelligence technology to bridge the communication gap between Deaf and hard-of-hearing signers and hearing communities. In recent years, gloss-free translation paradigms have emerged as a dominant research trend by eliminating the prohibitive expense of manual sign-by-sign (gloss) annotations. These methods leverage pre-trained visual backbones to encode continuous video streams and directly decode spoken target sentences end-to-end via autoregressive large sequence-to-sequence language models such as Flan-T5. However, without explicit temporal segmentation and gloss-level alignment supervision, models are burdened with navigating extreme cross-modal semantic gaps and complex word order variations.

The fundamental tension stems from the inherent structural clash between sign language grammar and spoken syntax, exacerbated by the vast granularity mismatch between continuous visual frames and discrete linguistic tokens. Natural sign language typically exhibits grammatical orders distinct from spoken languages; signers frequently produce salient content gestures first and introduce grammatical relations or spatial modulations later. Consequently, the chronological progression of video gestures departs markedly from the target spoken word order. Furthermore, spoken sentences contain numerous functional words (e.g., determiners, prepositions, auxiliary particles) that lack direct manual realization, while continuous sign videos abound with non-semantic transitional gestures, pauses, and co-articulation. Existing gloss-free frameworks (such as SpaMo, Uni-Sign, and ShuBERT) feed visual tokens in raw chronological order directly into the decoder, relying solely on implicit cross-attention to resolve the latent permutation. This implicit coupling leads to diffuse attention weights, severe alignment hallucinations, and compromised translation quality.

To break this reliance on the decoder's implicit discovery of permutation, VTaMo's angle of attack is to explicitly extract fine-grained correspondences and supply the autoregressive decoder with a semantically reordered visual sequence during training. The framework strips away untranslated function words to construct content-word pseudo-glosses, solves an optimal transport matching problem to align frames with tokens while mapping transitional frames into a dedicated null token, and offloads spoken syntax reordering at test time to a lightweight text-only recovery model. Core idea: introduce local entropy-regularized optimal transport with a learnable null token to explicitly establish frame-to-token correspondences and reorder training features, combined with norm-preserving global orthogonal calibration and position-aligned contrastive learning, decoupling lexical cross-modal alignment from target syntax restoration.

Method

Overall Architecture

The VTaMo pipeline integrates multi-scale visual encoding, temporal attention downsampling, multi-granularity explicit alignment, windowed feature reordering, and decoupled two-stage inference. Given a raw sign video, frame-level visual tokens are extracted using a frozen CLIP-ViT-Large backbone and aggregated through an attention-based temporal encoder to achieve 4× temporal downsampling before being projected into the language model's latent dimension. On the textual side, target spoken sentences are processed offline with a frozen part-of-speech tagger to retain content words and discard sign-absent function words, producing a pseudo-gloss sequence. During training, VTaMo computes soft frame-to-token correspondences via local Sinkhorn optimal transport, reorders visual temporal windows into the target token sequence for the LoRA-adapted Flan-T5 decoder, aligns global sentence geometry via orthogonal Earth Mover's Distance over a memory queue, and enforces token-level grounding via position-aligned contrastive learning. At inference time, visual features in chronological order are decoded directly into video-order pseudo-glosses and subsequently restored into fluent spoken sentences by an offline text-only recovery model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Continuous Sign Video Frames"] --> B["Attention-Based Temporal Encoding<br/>CLIP-ViT + 4× attention aggregation"]
    C["Ground-Truth Target Spoken Text"] --> D["Pseudo-Gloss Extraction<br/>Keep content tokens, drop function words"]

    subgraph S1 ["Explicit Cross-Modal Alignment Pipeline"]
        direction TB
        B --> E["Local Optimal Transport Alignment<br/>Sinkhorn solver + learnable null token"]
        D --> E
        B --> F["Global Orthogonal Calibration<br/>Attention pooling + memory queue EMD"]
        D --> F
        E --> G["Window Feature Reordering & Contrastive Learning<br/>Multi-frame window sorting & InfoNCE"]
        D --> G
    end

    G --> H["LoRA Flan-T5 Decoder<br/>Trained on reordered features / Direct test decoding"]
    H --> I["Text-Only Recovery Model<br/>Reconstructs syntax and function words"]
    I --> J["Final Fluent Spoken Translation"]

Key Designs

1. Local Optimal Transport Alignment: resolving transitional gestures and non-injective mapping via a learnable null token

To address the issue where continuous signing contains numerous co-articulation phases, transitional movements, and reset gestures that do not correspond to any spoken token—where forcing a strict one-to-one matching inevitably injects severe semantic noise—VTaMo formulates frame-to-token matching as an entropy-regularized optimal transport problem augmented with a learnable null token. A dedicated null vector \(\mathbf{e}_\varnothing \in \mathbb{R}^{d_t}\) is prepended to the frozen target pseudo-gloss sequence \(\mathbf{E} \in \mathbb{R}^{U \times d_t}\), producing an augmented text matrix \(\tilde{\mathbf{E}} \in \mathbb{R}^{K \times d_t}\) where \(K = U + 1\). Given the projected visual features \(\mathbf{S} \in \mathbb{R}^{M \times d_t}\), the pairwise cost matrix is constructed from cosine distances with a learnable scalar bias \(b_\varnothing\) modulating affinity toward the null token:

\[C_{m,k} = \begin{cases} 1 - \bar{\mathbf{s}}_m^\top \bar{\mathbf{e}}_k + b_\varnothing, & \text{if } k = 0 \text{ (null)}, \\ 1 - \bar{\mathbf{s}}_m^\top \bar{\mathbf{e}}_k, & \text{otherwise} \end{cases}\]

where \(\bar{\mathbf{s}}_m\) and \(\bar{\mathbf{e}}_k\) denote \(\ell_2\)-normalized representations. The transport plan \(\mathbf{A}^* \in \Pi(\mathbf{a}, \mathbf{b})\) is iteratively solved via the log-domain Sinkhorn algorithm. To prevent premature convergence to sub-optimal local minima caused by random initialization, an annealing schedule gradually relaxes the entropic parameter \(\varepsilon\) from \(0.12\) down to \(0.03\), steering the transport matrix smoothly from diffuse exploration to sharp, confident segment assignments.

2. Global Orthogonal Calibration: eliminating cross-modal orientation mismatch while preserving latent geometry

Because vision and text backbones are pre-trained independently across heterogeneous data distributions, paired sign and text sentence representations suffer from severe coordinate and angular orientation offsets, even when their underlying semantic semantics match. To align sentence-level distributions without distorting internal intra-modal distances, VTaMo applies a strictly constrained orthogonal transformation. Frame-level visual features and text embeddings are compressed into sentence-level vectors \(\mathbf{z}^\text{sign}, \mathbf{z}^\text{text} \in \mathbb{R}^{d_t}\) via attention pooling with a mean query vector. A learnable transformation matrix \(\mathbf{T} \in \mathbb{R}^{d_t \times d_t}\) rotates the visual sentence embedding:

\[\tilde{\mathbf{z}}^\text{sign}_n = \frac{\mathbf{z}^\text{sign}_n \mathbf{T}}{\|\mathbf{z}^\text{sign}_n \mathbf{T}\|_2}\]

To overcome the paucity of negative pairs under small single-GPU batch regimes, a FIFO memory queue stores the \(256\) most recent sentence vector pairs. An Earth Mover's Distance (EMD) loss is minimized over the queue pairs, subject to an explicit orthogonality penalty \(\mathcal{L}_\text{orth} = \|\mathbf{T}^\top \mathbf{T} - \mathbf{I}\|_F^2\). Restricting \(\mathbf{T}\) to a near-rigid rotation ensures that norms, cosine distances, and pairwise geometry remain uncorrupted, preventing arbitrary affine collapses.

3. Window Feature Reordering and Position-Aligned Contrastive Learning: aligning training sequence while grounding discrete token semantics

The autoregressive cross-entropy loss of the sequence-to-sequence decoder naturally expects input context to match target left-to-right token order; feeding unaligned temporal gestures forces the gradients of translation and alignment to cancel each other out. VTaMo partitions the \(M\) visual frames into \(N_w = U + 2\) overlapping temporal windows, aggregates the transport mass \(\pi_w = \sum_{m=s_w}^{e_w-1} A^*_{m,:}\), and assigns each window to the non-null content token with maximal assigned probability. Reordering the assigned windows by token index and concatenating their frames in chronological sub-sequences produces a reordered visual sequence \(\mathbf{R}\) tailored for the LoRA-adapted Flan-T5 decoder.

Because this reordering cannot be performed during inference when the target token sequence is unknown, VTaMo incorporates position-aligned InfoNCE contrastive learning to enforce token-level discriminative grounding. Visual frames assigned to token \(u\) are mean-pooled into a token representation \(\mathbf{r}_u\), mapped through a 2-layer MLP projection head, and contrasted against the corresponding text token embedding \(\mathbf{e}_u\):

\[\mathcal{L}_\text{contra} = - \frac{1}{|\mathcal{V}|} \sum_{i \in \mathcal{V}} \log \frac{\exp(\text{sim}(\hat{\mathbf{r}}_i, \hat{\mathbf{e}}_i)/\tau_c)}{\sum_{j \in \mathcal{V}} \exp(\text{sim}(\hat{\mathbf{r}}_i, \hat{\mathbf{e}}_j)/\tau_c)}\]

Gradients are allowed to update only the visual branch while blocking backpropagation into the text representations, anchoring visual features directly onto fixed semantic landmarks so that each visual token remains robustly decodable even when arriving out of order.

4. Text-Only Recovery Model: decoupling lexical video grounding from target syntax generation

During inference, target text order is completely unavailable, and the visual sequence is fed into the decoder in its original chronological order. Benefiting from the robust token-level grounding established by the contrastive objective, the decoder reliably outputs pseudo-gloss tokens in sign gesture order. The final translation is synthesized by a lightweight text-only recovery model, which restores fluent spoken word order and re-inserts missing function words. Crucially, this recovery model is trained strictly on monolingual text data without viewing any sign videos: sentences are stripped to pseudo-glosses, their content words are artificially permuted, and the model is trained to reconstruct the original fluent sentence. This clean decoupling liberates the multimodal vision-language model from balancing complex grammatical permutations alongside visual perception.

Loss & Training

VTaMo employs a two-phase training protocol. In Phase 1, the language model backbone is kept frozen while only the visual temporal encoder, projection layers, and alignment heads are trained to project visual features into the text embedding space. In Phase 2, the full system is jointly fine-tuned alongside Flan-T5 LoRA adapters. The overall training objective is formulated as:

\[\mathcal{L} = \mathcal{L}_\text{T5} + \lambda_\text{local} \mathcal{L}_\text{local} + \lambda_g(t) \mathcal{L}_\text{EMD} + \beta_\text{orth} \mathcal{L}_\text{orth} + \beta_\text{contra} \mathcal{L}_\text{contra}\]

The local objective \(\mathcal{L}_\text{local} = \beta_\text{local} \mathcal{L}_\text{trans} + \beta_\text{tv} \mathcal{L}_\text{tv} + \beta_\text{null} \mathcal{L}_\text{null}\) incorporates the transport cost \(\mathcal{L}_\text{trans}\), a total variation regularization \(\mathcal{L}_\text{tv}\) penalizing uncoordinated token switching across adjacent frames, and a null budget penalty \(\mathcal{L}_\text{null}\) preventing the learnable bias \(b_\varnothing\) from absorbing valid semantic frames. The global EMD weight \(\lambda_g(t)\) ramps up linearly across the first 4,000 steps to ensure that global distribution matching only engages once local alignment plans stabilize.

Key Experimental Results

Main Results

VTaMo was comprehensively evaluated on the official test sets of four benchmark datasets: Phoenix-2014T (German weather domain), CSL-Daily (Chinese daily life communication), How2Sign (large-scale American sign language instructional videos), and OpenASL (open-domain ASL news and web videos). VTaMo operates solely on RGB input without any task-specific visual pre-training, consistently outperforming previous gloss-free and gloss-based state-of-the-art models:

Dataset	Setting / Method	Modality	Visual Ft.	BLEU-1	BLEU-4	ROUGE-L	BLEURT
Phoenix-2014T	TS-SLT (SOTA Gloss-based)	RGB	Yes	54.90	28.95	53.48	—
Phoenix-2014T	SpaMo (Prev. SOTA Gloss-free)	RGB	No	49.80	24.32	46.57	—
Phoenix-2014T	VTaMo (Ours)	RGB	No	61.32	28.86	60.24	—
CSL-Daily	Uni-Sign (Prev. SOTA Gloss-free)	Pose+RGB	Yes	55.08	26.36	56.51	—
CSL-Daily	SpaMo	RGB	No	48.90	20.55	47.46	—
CSL-Daily	VTaMo (Ours)	RGB	No	57.64	27.16	57.28	—
How2Sign	Uni-Sign	Pose+RGB	Yes	40.20	14.90	36.00	49.40
How2Sign	SpaMo	RGB	No	33.41	10.11	30.56	42.23
How2Sign	VTaMo (Ours)	RGB	No	46.67	18.47	37.14	50.26
OpenASL	Uni-Sign	Pose+RGB	Yes	49.35	23.14	43.22	60.40
OpenASL	SpaMo	RGB	No	46.28	21.25	41.04	57.82
OpenASL	VTaMo (Ours)	RGB	No	53.58	25.94	46.85	62.48

Ablation Study

Ablations on the Phoenix-2014T test set demonstrate that each component of the multi-granularity alignment framework is critical and non-redundant:

Configuration	BLEU-1	BLEU-4	ROUGE-L	Note
Full model	61.32	28.86	60.24	Complete multi-granularity alignment
w/o \(\mathcal{L}_\text{contra}\)	51.23	22.56	50.14	Largest drop (-6.30 B4); loss of token grounding
w/o \(\mathcal{L}_\text{local}\)	52.62	23.77	51.29	Severe drop (-5.09 B4); invalidates window reordering
w/o \(\mathcal{L}_\text{EMD}\)	57.46	26.21	57.92	Uncalibrated space orientation (-2.65 B4)
w/o null token	—	27.60*	—	Forced matching; assignment entropy worsens to 0.46
w/o memory queue	—	27.35	—	Weak batch diversity; peak epoch slows from 36 to 68
w/o orthogonality	—	28.13	—	Unconstrained projection distorts distance metric

(Note: On the dev set, removing the null token degraded assignment peak probability from 0.76 to 0.61 and increased segment change rate from 0.12 to 0.33)

Key Findings

• Complementarity between local OT and contrastive grounding: While local optimal transport produces sharp segment boundaries and orders the sequence for autoregressive generation, position-aligned contrastive learning \(\mathcal{L}_\text{contra}\) delivers the largest single performance contribution (removing it drops BLEU-4 by 6.30 points). It ensures each visual segment firmly grounds into the correct lexical embedding even when presenting out-of-order gestures at inference time.
• Amplified performance advantages on open-domain datasets: While VTaMo matches the best gloss-based model on Phoenix-2014T (28.86 vs 28.95), its margins over prior methods expand dramatically on open-vocabulary benchmarks (OpenASL and How2Sign), outpacing previous RGB-based SOTA SpaMo by +4.69 and +8.36 BLEU-4, respectively.
• Invariance to background and signer appearance: In controlled synthetic experiments on How2Sign using ControlNet to preserve body/hand kinematics while altering scene backgrounds (outdoor/kitchen) and signer appearance (clothing/synthetic identity), translation performance degraded by at most 0.42 BLEU-4, verifying that explicit alignment targets gesture kinematics rather than background shortcuts.

Highlights & Insights

• Learnable Null Token resolves continuous sequence alignment: Traditional sequence alignment methods (such as DTW or unconstrained OT) force every visual frame onto a text token, creating severe distortion during pauses or transitional motions. Adding a learnable null token allows the model to naturally filter non-semantic intervals, leading to clean block-diagonal transport plans.
• Orthogonal mapping calibrates spaces without distance distortion: Unconstrained linear mappings often collapse high-dimensional geometry into degenerate subspaces. Enforcing an explicit orthogonality penalty keeps the transformation near a rigid rotation, preserving intra-modal cosine distances and accelerating convergence by nearly 2×.
• Training reordering coupled with test-time direct decoding: The framework reconciles the conflicting demands of autoregressive training and sequential inference by decoupling lexical alignment from syntactic reordering, an insight that naturally generalizes to other unaligned sequence generation domains (such as video step grounding and unsegmented speech translation).

Limitations & Future Work

• Cascading dependency on the text recovery model: Because syntactic word order is restored downstream by a purely text-trained recovery model, any lexical omissions or errors from the visual decoder can trigger compounding syntactic hallucination in the final output.
• Offline setup and compute requirements: The current pipeline relies on a heavy Flan-T5-XL backbone and multi-scale ViT features, designed primarily for offline translation fidelity rather than real-time on-device streaming.
• Future directions: Integrating lightweight skeleton structural priors directly into the local cost matrix, and exploring end-to-end unified architectures that seamlessly merge video decoding and text re-ordering without external two-stage pipelines.

Related Work & Insights

• vs SpaMo (NAACL 2025): SpaMo applies batch-level contrastive learning across visual and linguistic modalities but leaves sentence-internal frame-to-token correspondence implicit. VTaMo introduces fine-grained, sentence-internal optimal transport and window reordering, yielding an 8.36 BLEU-4 margin on How2Sign.
• vs Uni-Sign / SHuBERT (ICLR 2025 / ACL 2025): Both Uni-Sign and SHuBERT require extensive self-supervised sign video pre-training and multi-modal pose+RGB inputs. In contrast, VTaMo demonstrates that explicit multi-granularity alignment on top of a frozen generic CLIP encoder can surpass heavily pre-trained multimodal systems using only RGB frames.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant combination of null-regularized optimal transport, orthogonal calibration, and decoupled inference for gloss-free sign language translation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across four multilingual benchmarks with comprehensive ablations, transport quality metrics, and ControlNet synthetic noise robustness analysis.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulation, clear conceptual motivation, and well-structured exposition.
• Value: ⭐⭐⭐⭐⭐ Sets a new state of the art for gloss-free sign language translation and provides a transferable blueprint for weakly-aligned cross-modal sequence learning.

Related Papers

• [ECCV 2026] SignBind-LLM: Multi-Stage Modality Fusion for Sign Language Translation
• [CVPR 2026] Learning Effective Sign Features without Text for Gloss-free Sign Language Translation
• [ECCV 2026] SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translation
• [CVPR 2025] Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues
• [CVPR 2026] BoostSLT: Boosting Sign Language Translation via a Plug-and-Play Diffusion-Based Semantic Enhancer