Skip to content

BackTranslation2.0 -- A Linguistically Motivated Metric to Assess Sign Language Production

Conference: ECCV 2026
arXiv: 2606.28673
Code: cogvis-cvssp.github.io/BackTranslation2/
Area: Human Understanding
Keywords: Sign Language Production Evaluation, Backtranslation Metrics, Linguistically Motivated Evaluation, Multi-dimensional Scoring, LLM Agents

TL;DR

Proposes BackTranslation2.0, a linguistically motivated text-to-sign language translation evaluation metric. Utilizing a two-phase agent framework (10 dedicated tools for evidence extraction + 4 LLM cross-comparison modules), it produces deterministic scores across four dimensions: grammatical correctness, phonological accuracy, motion fluency, and generation fidelity. It achieves a Pearson \(r=1.00\) / Spearman \(\rho=1.00\) correlation with human judgment on known corruptions and synthetic data, significantly outperforming existing backtranslation and motion similarity baselines.

Background & Motivation

Sign Language Production (SLP) is a core frontier in accessibility technology for the deaf, but reliably assessing the quality of generated sign language remains an unresolved bottleneck. Existing automatic evaluations mainly rely on two paradigms: backtranslation, which translates the generated sign language sequence back to text using a sign language recognition model and compares it with the source sentence using text metrics like BLEU-4; and motion-space reference comparison, which directly compares the distance between the generated and reference sequences in the pose/embedding space using DTW, SiBLEU, or SignCLIP.

These two types of methods suffer from fundamental flaws. The problem with backtranslation is that sign-to-text translation is itself a largely unsolved problem—using poor recognition/translation technology to evaluate generative technology that is also in development creates a circular dependency, and text metrics like BLEU are completely insensitive to sign-specific grammatical structures, spatial referencing, and directional verb agreement. The motion reference paradigm requires paired ground-truth sign sequences, which are expensive to collect, and reduces the rich linguistic expressions of sign language (grammar, vocabulary, phonetic variations) to deviations from a single reference trajectory, failing to model the fundamental sign language characteristic where "multiple expressions can be correct."

In contrast, human evaluation—especially by deaf native signers—simultaneously examines multiple linguistic dimensions: handshape accuracy, place of articulation, use of signing space and directionality, non-manual features (facial expressions/body poses), syntactic word order, and motion naturalness. However, expert annotation is expensive and not scalable.

The Key Challenge emerges: automatic metrics are cheap but inaccurate (poor correlation with human judgment), while human evaluation is accurate but not scalable (costly, time-consuming, and limited in agreement). The Core Idea of this paper is to avoid relying on a single end-to-end model for evaluation. Instead, a linguistically motivated, multi-dimensional agent framework is constructed: a suite of specialized tools extracts structured evidence across different modalities, which an LLM then cross-checks for consistency as a reasoning layer, finally calculating reproducible scores via deterministic weighted formulas—approaching the multi-dimensional granularity of human evaluation while maintaining full automation.

Method

Overall Architecture

BackTranslation2.0 (BT2) is a deterministic agent framework. It takes a source sentence \(s\) and a generated sign output \(\hat{y}\) (which can be RGB video, pose sequence, mesh animation, or hybrid rendered formats) as inputs, and outputs structured evidence for each tool, scores for four dimensions, an overall comprehensibility score, and an evidence-based natural language evaluation report.

The entire pipeline is split into two fixed phases. All samples follow the exact same tool invocation sequence without dynamic tool selection to ensure reproducibility and cross-sample comparability. First, multimodal feature extraction constructs a structured sample \(\mathcal{X} = \Phi(s, \hat{y}) = \{x^{(1)}, \dots, x^{(M)}, x^{\text{seq}}\}\), where \(x^{(m)}\) represents the sign features of the \(m\)-th temporal segment (body, hand, face, spatial location, full-body keypoints), and \(x^{\text{seq}}\) represents full-sequence-level features. Subsequently, 10 base tools (Phase 1) execute to extract evidence, followed by 4 comparison tools (Phase 2) for cross-comparison. All intermediate results are written to a shared memory trajectory \(\mathcal{M}\). Finally, deterministic weighted formulas calculate four dimensional scores \(\mathbf{d} = [d_{\text{gram}}, d_{\text{phon}}, d_{\text{flu}}, d_{\text{fid}}]\) (each on a 0-4 scale), with the overall score \(u = \frac{1}{4}(d_{\text{gram}} + d_{\text{phon}} + d_{\text{flu}} + d_{\text{fid}})\) as the arithmetic mean. In the final step, the LLM is solely responsible for writing a natural language explanation report based on the complete memory trajectory, without modifying any numerical scores.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Source Sentence s + Generated Sign ŷ"] --> B["Multimodal Feature Extraction<br/>Segment-by-segment body/hand/face/location/full-body features"]
    B --> C["Phase 1: Dedicated Linguistic Tool Suite<br/>10 base tools extracting evidence in parallel"]
    C --> D["Shared Memory Trajectory M<br/>Stores scores/confidence/evidence"]
    D --> E["Phase 2: LLM Cross-Comparison<br/>spot-gloss / manual / NMF / directionality"]
    E --> F["Deterministic Dimension Scores<br/>grammar / phonology / fluency / fidelity → 0-4"]
    F --> G["Overall Comprehensibility u = 4-dimension mean"]
    F --> H["Final LLM Natural Language Evaluation Report"]
    D --> H

Key Designs

1. Linguistically Motivated Four-Dimensional Scoring System: Deconstructing "sign language quality" into four independently measurable dimensions aligned with human evaluation

Traditional backtranslation yields only a single BLEU score, conflating grammatical errors, inaccurate handshapes, stiff movements, and low video quality. This is uninterpretable and fails to diagnose where point deductions occur. BT2 decomposes evaluation into four dimensions: Grammatical Correctness, Phonological Accuracy, Motion Fluency, and Generation Fidelity. Each dimension directly corresponds to linguistic attributes investigated by human evaluators in structured studies. The grammatical dimension covers use of signing space, pronominal indexing, and directional verb agreement (directionality). The phonological dimension covers the correctness of sub-lexical components such as handshape, movement, location, orientation, and non-manual features (NMF). The fluency dimension measures the naturalness and temporal smoothness of signing performance, and the fidelity dimension checks the visual quality and anatomical plausibility of the generated output. The selection of these four dimensions is not arbitrary; rather, it directly aligns with five core attributes defined in the SLP Tier framework (an automated grading standard for sign language systems developed with deaf community participation), ensuring evaluation content matches real-world application needs.

2. Phase 1 Dedicated Linguistic Tool Suite: Designing specialized extractors for each linguistic attribute instead of making black-box LLM predictions

Phase 1 of BT2 deploys 10 base tools, each targeting specific linguistic or visual evidence and writing them to the shared memory trajectory for consumption in Phase 2. Grammatical and spatial evidence is covered by four tools: the Pseudo-Gloss generator uses an LLM to map the source sentence to a candidate gloss sequence \(g^\star\), serving as a reference of 'what signs are expected'; Sign Spotting embeds each signing segment and performs cosine retrieval against a dictionary to return Top-K candidate gloss matches; the Directionality tool determines directionality by combining SL-GCN index prediction and wrist trajectory DTW matching; and the Topographic Scorer uses FastDTW to compare wrist path directions. Phonological evidence is covered by three tools: the Handshape classifier uses a Transformer to predict handshapes separately for the left and right hands, taking the higher-confidence side as the dominant hand; the Location classifier rates location using deep features after SMPL-X re-optimization coupled with contact proximity cues; and the Non-Manuals classifier aggregates frame-level Action Units via semantic mapping to segment-level non-manual label sets. Motion and fidelity evidence is covered by three tools: the Fluency tool calculates the log-likelihood of U-Net bottleneck features from a conditional flow matching model under a reference Gaussian distribution and calibrates it linearly to a 0-4 scale; Visual Metrics combine four deterministic kinematic metrics—high-frequency energy, smoothness (jerk), motion amplitude, and dominant frequency plausibility—with fixed weights; and Hand Fidelity maps the mean logit of a binary classifier (real hand vs. synthetic hand crop) to 0-4.

3. Phase 2 LLM Cross-Comparison: Utilizing LLMs as a reasoning layer to cross-verify consistency of tool outputs instead of simply averaging individual tool scores

This is the fundamental difference between BT2 and a simple ensemble scheme that runs various tools and averages them. Phase 2 contains four fixed comparison tools. Each tool receives two types of inputs—the 'detected' evidence from Phase 1 and the 'expected' evidence from pseudo-glosses/dictionaries—which the LLM then compares for consistency under linguistic constraints. For Spot-gloss comparison, deterministic pre-filtering first handles obvious matching pairs, and ambiguous semantic correspondences are handed over to the LLM for semantic matching; Manual-features comparison follows dictionary mapping paths, falling back to the LLM when confidence is low; Non-manual-features comparison can option-select inferring expected non-manual markings via the LLM or using deterministic rules; and Directionality comparison is completely deterministic. Ablation experiments show that replacing Phase 2's LLM with deterministic fallback rules causes the comprehensibility \(\rho\) on synthetic data to plunge from 0.60 to 0.09 (an 85% drop), proving that the performance gain stems from reasoning capability rather than engineering assembly.

4. Fully Auditable Deterministic Scoring Pipeline: Every step from low-level observations to final scores is traceable

The score quantification of BT2 is entirely executed by fixed weighted formulas, and the LLM does not participate in numerical calculations. The subscore of each dimension comes from a weighted combination of corresponding tool outputs after re-normalization. All intermediate results—ranging from individual Phase 1 tool scores/confidence/evidence to Phase 2 comparison labels and dimension-level aggregations—are retained in the shared memory trajectory \(\mathcal{M}\). This enables any final score to be traced down to the specific output of a specific tool, solving the fundamental problem of end-to-end evaluation metrics where one 'only knows the score is low but has no idea why'. The final LLM stage only generates natural language evaluation reports without modifying any numerical scores, ensuring that the scores are reproducible.

Loss & Training

BT2 itself is an evaluation metric rather than a generative model and does not involve end-to-end training. However, multiple specialized tools in Phase 1 (handshape classifier, location classifier, non-manual classifier, hand fidelity classifier, and flow-matching fluency model) are independently trained on BSL corpora, with training details and calibration settings provided in the supplementary material. All tools have fixed weights during inference and do not undergo online learning or adaptation.

Key Experimental Results

Main Results

The paper constructs a dedicated BSL evaluation benchmark comprising four complementary subsets: Human SRT (6 signers x 6 sentences, covering different proficiency levels), Known Corruptions (three linguistic corruptions of the same 6 sentences: missing non-manuals, incorrect word order, incorrect handshape), Synthetic Artifacts (6 generative systems, including SL-specific methods like SignGAN/SignSplat/SignStream/SignSparK and general human motion generators like GUAVA/Wan2.1), and a Directionality/Topographic Placement dedicated set (directional verbs, pronominal indexing, spatial placement). Concurrently, a user study with 20 participants (deaf and hearing) was conducted to collect 5-point Likert ratings and anomaly annotations.

The table below compares the directionally normalized Pearson \(r\) / Spearman \(\rho\) of BT2 against reference-based baselines (requiring reference sequences) and reference-free baselines (requiring only source text) across subsets.

Method Known Corruptions Under. \(r/\rho\) Synthetic Under. \(r/\rho\) Synthetic Qual. \(r/\rho\) Human SRT Under. \(r/\rho\)
DTW-MPJPE (Reference-based) 0.91/0.50 -0.14/-0.03 -0.33/-0.37 0.04/0.00
SiBLEU (Reference-based) -0.58/-0.50 0.03/0.31 -0.05/0.31 0.96/0.90
SignCLIP p2p (Reference-based) -0.97/-1.00 0.85/0.90 0.82/0.70 0.69/0.80
SVAE (Reference-based) 0.95/1.00 0.43/0.26 0.54/0.31 0.81/0.80
BT1 BLEU-4 (Reference-free) -0.19/0.00 -0.38/-0.44 -0.15/-0.10 0.00/0.00
SignCLIP p2t (Reference-free) 0.89/0.50 -0.64/-0.60 -0.41/-0.30 -0.07/-0.10
BT2 Overall (Ours) 1.00/1.00 0.42/0.60 0.52/0.66 -0.43/-0.30
BT2 Phonology 0.28/0.50 0.48/0.26 0.35/-0.03 0.87/0.90
BT2 Fidelity 0.80/1.00 0.74/0.94 0.83/0.94 -0.81/-0.90
Human Inter-rater Agreement Gwet's AC2 0.69 0.69 0.68 0.69

Key Findings: BT2 Overall outperforms all reference-free baselines on known corruptions (\(r/\rho = 1.00/1.00\)) and synthetic data, matching or surpassing the strongest reference-based methods. On the Human SRT subset, BT2 Overall exhibits negative correlation. This is due to a severe ceiling effect in this subset (74% of ratings \(\ge 4\)), inter-rater ICC close to 0, and a signer mean span of only 0.55 points—making rank-based evaluation extremely challenging for any metric within such a highly compressed rating range. However, BT2 Phonology reaches \(\rho=0.90\) on the same SRT subset, demonstrating that BT2 remains effective in dimensions where true phonological variations exist.

Ablation Study

Condition KC Under. \(r/\rho\) Synthetic Under. \(r/\rho\) Synthetic Qual. \(r/\rho\) SRT Under. \(r/\rho\)
No LLM (Lexical heuristic fallback) 0.73(↓27%)/0.50(↓50%) 0.21(↓50%)/0.37(↓38%) 0.43(↓17%)/0.54(↓18%) -0.68(↓58%)/-0.40(↓33%)
Phase 1 only 0.73(↓27%)/0.50(↓50%) 0.21(↓50%)/0.37(↓38%) 0.43(↓17%)/0.54(↓18%) -0.68(↓58%)/-0.40(↓33%)
Phase 1 + Deterministic Phase 2 0.08(↓92%)/0.50(↓50%) 0.22(↓48%)/0.09(↓85%) 0.42(↓19%)/0.37(↓44%) -0.63(↓47%)/-0.50(↓67%)
Full - spot-gloss 0.73(↓27%)/0.50(↓50%) 0.31(↓26%)/0.37(↓38%) 0.41(↓21%)/0.49(↓26%) -0.68(↓58%)/-0.40(↓33%)
Full - manual feats. 0.56(↓44%)/0.50(↓50%) 0.26(↓38%)/0.37(↓38%) 0.34(↓35%)/0.49(↓26%) -0.66(↓53%)/-0.70(↓133%)
Full - non-manual 0.35(↓65%)/0.50(↓50%) 0.50(↑19%)/0.54(↓10%) 0.54(↑4%)/0.60(↓9%) -0.09(↑79%)/-0.10(↑67%)
BT2 Full (Ours) 1.00/1.00 0.42/0.60 0.52/0.66 -0.43/-0.30

Each ablation variant results in a drop in \(\rho\) from 1.00 on known corruptions. Replacing the Phase 2 LLM with deterministic fallback rules causes the synthetic comprehensibility \(\rho\) to plunge by 85%, proving that the core gain of Phase 2 derives from the reasoning capability of the LLM. Although some ablations show marginal gains on certain columns (e.g., removing non-manual comparison slightly increases synthetic Qual. performance), no single variant beats the full system across all categories. The negative correlation on Human SRT persists across all ablations, confirming this is a structural limitation of the subset rather than a defect of a specific component.

Key Findings

  • LLM reasoning in Phase 2 is irreplaceable: Replacing LLM reasoning with deterministic fallbacks drops \(\rho\) from 0.60 to 0.09 (an 85% reduction), far exceeding the impact of removing any single tool, which signifies that semantic reasoning during cross-comparison is the core value of the overall system.
  • The four dimensions complement rather than duplicate each other: Fidelity correlates at \(\rho=0.94\) on synthetic quality, whereas Phonology scores \(\rho=0.90\) on Human SRT—different dimensions show strengths in different scenarios, validating the necessity of a multi-dimensional decomposed design.
  • BT2 accurately localizes corruption types in corruption-sensitivity experiments: On a large-scale synthetic corpus (900 matched pairs), the handshape dimension achieves a 79-88% win rate under handshape corruptions, and the location dimension achieves a 73-98% win rate under location corruptions. Meanwhile, fluency and fidelity regularly display reverse (wrong-way) scoring behavior under these corruptions, proving that BT2 precisely attributes each degradation to its target dimension instead of subtracting points globally.
  • Grammatical evaluation remains a bottleneck: Correlation for the grammatical dimension is significantly weaker than the phonological dimension. This is because natural sign language permits far more flexible word order variations than strictly controlled settings, and inter-rater agreement on word-order anomalies is extremely low (\(\alpha = 0.01\) on synthetic data and \(-0.01\) on Human SRT).

Highlights & Insights

  • Using an LLM as a reasoning layer rather than a direct generator or recognizer: Instead of passing raw sign videos directly to an LLM for black-box, uninterpretable scoring, BT2 feeds structured evidence produced by specialized tools into the LLM to inspect consistency and perform cross-comparisons under linguistic rules. This paradigm—"LLM as a reasoning layer over structured tool outputs"—is highly transferable to other domains requiring multi-dimensional expert evaluations, such as medical image report quality assessment and multi-dimensional evaluations of code generation.
  • Deterministic pipeline + auditable memory trajectory design: Many LLM agent systems suffer from non-reproducible scoring (varying across calls). By using a fixed tool call sequence + deterministic weighted formulas + complete intermediate state retention, BT2 maintains LLM reasoning flexibility while ensuring score reproducibility and auditability. Designing the shared memory trajectory \(\mathcal{M}\) as an "audit trail" is highly practical.
  • "Wrong-way" signals in corruption-sensitivity experiments validate metric fidelity: When handshape is corrupted, fluency and fidelity scores systematically increase rather than decrease (as handshape alterations do not degrade motion smoothness or image generation quality). This selective degradation behavior ("what should drop drops, what shouldn't doesn't") is far more convincing than aggregate scores in proving that each dimension measures exactly what it claims to measure.
  • Clever design of pseudo-glosses as an "expected anchor": Utilizing an LLM to generate the expected gloss sequence \(g^\star\) from the source sentence, and then allowing Phase 2 comparison tools to cross-check detected signs against this sequence. This bypasses the need for hard ground-truth sign translations while providing a linguistically meaningful comparison standard—essentially transferring "how signs should be performed" knowledge from training datasets to the LLM's language knowledge base.

Limitations & Future Work

  • Grammatical assessment fails in natural scenarios: BT2's grammatical dimension correlates perfectly on known corruptions (\(\rho=1.00\)) but yields a strong negative correlation on natural sign performances from Human SRT (\(\rho=-0.60\)). The authors attribute this to flexible word orders in natural sign language and extremely low inter-rater agreement. A more fundamental issue is that current gloss-level "word-order matching" may be too rigorous for languages like BSL that allow topicalization, ellipsis, and spatial synthesis—future improvements could replace sequential word-order matching with dependency parsing or semantic role labeling.
  • Suboptimal overall performance on the Human SRT subset reflects a methodological challenge: When inter-rater agreement is near random (\(ICC \approx 0\)), no automated metric can perform well. Rather than seeing this as a deficit of BT2, it underscores that the task of "fine-grained differentiation within high-quality sign language intervals" may inherently require more precise rating scales and more diverse stimulus materials.
  • Robustness risks of depending on multiple pre-trained specialized tools: If any Phase 1 tool (such as handshape, location, or non-manual classifiers) fails on out-of-distribution data, errors will propagate to Phase 2, where they may be amplified or misconstrued during LLM cross-comparison. Currently, there is no systematic cascading error analysis for these tools.
  • BSL-specificity vs. cross-lingual generalization: All experiments were conducted on BSL. Syntax and phonology vary significantly across different sign languages (e.g., ASL and CSL feature markedly different syntactic structures); hence, translating BT2 to other sign languages will require re-training or adapting the pseudo-gloss generator, handshape classifiers, and dictionary resources.
  • vs. Traditional Backtranslation (BT1, Saunders et al. 2020): BT1 translates generated signs back into text using a recognition model and benchmarks it via BLEU, essentially "evaluating one inaccurate model with another inaccurate one." BT2 completely bypasses the sign recognition channel and directly extracts linguistic evidence from the sign outputs, avoiding circular dependency.
  • vs. Embedding Space Methods like SignCLIP / SVAE: These methods perform well in the presence of reference sequences (e.g., SVAE reaches \(\rho=1.00\) on known corruptions) but require paired ground-truth, essentially measuring "how similar the output is to the reference" rather than "how well the sign language is performed." BT2's evaluation is reference-free (no paired ground-truth needed) and assesses the linguistic properties themselves rather than similarity to a single realization.
  • vs. LLM-as-a-Judge (Zheng et al. 2024): The LLM-as-a-Judge paradigm deploys the LLM directly as a rater. In contrast, BT2 utilizes the LLM as a "reasoning layer"—the LLM ingests structured tool outputs to perform cross-comparisons in linguistically constrained contexts, and the final scores are calculated via deterministic formulas rather than generated by the LLM. This hybrid "LLM-reasoning + deterministic aggregation" architecture is more reproducible and auditable than pure LLM scoring.
  • vs. SLP Tier Framework (Bowden et al. 2025): SLP Tier is a taxonomy rather than an evaluation metric, defining the automation tiers of sign language systems. The four evaluation dimensions of BT2 directly map to the five core attributes of SLP Tier, making them complementary: Tier defines "what a system is capable of doing," and BT2 quantifies "how well the system performs it."

Rating

  • Novelty: ⭐⭐⭐⭐⭐ The first to introduce an agent-based framework to sign language evaluation. The three-tier architecture ("specialized tool extraction + LLM reasoning layer cross-comparison + deterministic aggregation") is completely novel in the sign language domain, and the "LLM as a reasoning layer over tool outputs" paradigm holds cross-domain transfer value.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Solid experimental depth and coverage, featuring four complementary benchmark subsets, a 20-person user study, seven ablation variants, a large-scale synthetic corruption-sensitivity analysis (900 pairs), and qualitative tool-behavior visualizations.
  • Writing Quality: ⭐⭐⭐⭐ Complete formulas and clear tool definitions in the methodology section, although some training and calibration details for Phase 1 tools were relegated to the supplementary material, which reviewers might have struggled to evaluate fully during the review process.
  • Value: ⭐⭐⭐⭐⭐ Sign language production evaluation is a historically neglected yet critical fundamental problem. BT2 provides a scalable, interpretable, and deaf-community-aligned evaluation paradigm, while its auditable memory trajectory design addresses the common pain point of reproducibility in LLM agent systems.