DiscoSign: Discourse-Aware Text to Sign Language Gloss Translation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Human Understanding
Keywords: Sign Language Translation, Discourse Modeling, Spatial Coreference, Pseudocleft Clauses, Large Language Models
TL;DR¶
DiscoSign is the first discourse-aware computational framework for text-to-sign language gloss translation, using explicit state registries and deterministic post-processing in a single LLM call to enforce spatial coreference, question-answer clause restructuring, and concept-gloss consistency across multi-sentence discourse.
Background & Motivation¶
Sign languages rely on a sophisticated three-dimensional spatial reference system and distinct rhetorical structures that span entire multi-sentence narratives. Despite these linguistic characteristics, computational text-to-sign translation and production pipelines have predominantly operated at the isolated sentence level. Treating each sentence independently strips the model of cross-sentence context, preventing it from tracking referents in signing space, coordinating information packaging, and maintaining vocabulary consistency across discourse. As a result, generated gloss sequences often suffer from referential collapse and confusing lexical drift, impairing comprehension for Deaf users and causing severe downstream errors in signing avatars.
This failure of sentence-level modeling manifests in three fundamental linguistic tensions. First, spatial coreference resolution breaks down across sentence boundaries. In American Sign Language (ASL), entities are mapped to specific spatial coordinates (e.g., using pointing signs such as IX-3p:i and IX-3p:j), and subsequent pronominal references or directional verbs (e.g., i:GIVE:j) must adhere strictly to these coordinates. Without cross-sentence state tracking, a referent established at location i in one sentence is frequently reassigned to a different location in the next, confusing narrator and actor identities. Second, Question-Answer Clauses (QACs)โpseudocleft structures with silent copulasโare critical topic-comment constructions used to convey causal relationships, topicalization, and emphasis (e.g., translating "He agreed because he would take the bus" into "IX-3p:j AGREE WHY? IX-3p:j TAKE BUS"). Sentence-level systems lack the multi-clause context required to determine when QAC topicalization is stylistically natural, while also risking erroneous QAC insertions at discourse openings. Third, concept-gloss inconsistency arises when English synonyms or paraphrastic mentions of the same entity (such as "car" and "vehicle") alternate unpredictably between CAR, VEHICLE, and MACHINE, creating visual ambiguity and unnecessary cognitive load.
Providing raw discourse context alone is insufficient to resolve these interdependent phenomena because standard language models lack explicit mechanisms to guarantee spatial and lexical persistence. Core idea: formalize discourse-level linguistic phenomena into explicit state registries for spatial indices, question-answer clause decisions, and concept-gloss mappings, enforcing them as hard prompt constraints alongside deterministic string verification within a single LLM call per sentence.
Method¶
Overall Architecture¶
The DiscoSign framework processes multi-sentence source text sequentially in a causal, left-to-right fashion. When translating sentence \(s_t\), the system accumulates discourse history across preceding sentences \(s_1, \dots, s_{t-1}\) into a unified state representation \(U = (S, L, Q)\). This unified state, along with a configurable context window of prior sentence-translation pairs, is embedded into a structured instruction prompt as explicit hard constraints. A large language model produces a JSON payload containing the target ASL gloss translation accompanied by structured metadata. A deterministic verification layer then cross-checks the generation against the active registries, rectifying any remaining violations via direct string replacement without additional LLM calls, and updates the registries to constrain the subsequent sentence \(s_{t+1}\).
The architecture coordinates three dedicated modules: the Spatial Coreference Module (SCM), the Question-Answer Clause Module (QACM), and the Concept-Gloss Consistency Module (CGCM). Rather than spawning separate sub-agents or multi-pass model invocations, all three modules operate within a unified prompt structure and post-processing routine, guaranteeing referential coherence and lexicon compliance with minimal latency.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input sentence s_t<br/>with sliding window previous context"] --> B["Retrieve discourse registries U=(S, L, Q)<br/>and construct constrained prompt"]
B --> C["Single LLM inference step<br/>Generate ASL gloss and structured metadata"]
C --> D["Spatial coreference registration<br/>SCM: Enforce coordinate consistency"]
D --> E["Concept-gloss consistency mapping<br/>CGCM: Align synonyms to canonical glosses"]
E --> F["Discourse position and context check<br/>QACM: Validate pseudocleft appropriateness"]
F --> G["Deterministic post-processing & registry update<br/>Output verified gloss g_t and pass state to s_{t+1}"]
Key Designs¶
1. Spatial Coreference Registration: Maintaining Persistent Spatial Coordinates Spatial indexing in signed languages serves as the visual counterpart of morphological agreement and pronominal binding. The Spatial Coreference Module (SCM) tracks entity-coordinate assignments across discourse using a state registry: $\(S = \{(e, i, t) : e \in E, i \in I, t \in T\}\)$ where \(e\) denotes an entity mention, \(i\) is a spatial coordinate index (e.g., IX-3p:i, IX-3p:j), and \(t\) indexes the sentence where the assignment was established. The SCM enforces two programmatic constraints: first, Spatial Consistency, dictating that if \((e, i, t) \in S\), all subsequent mentions of \(e\) in sentences \(t' > t\) must strictly utilize index \(i\); second, Coreference Correspondence, requiring that any spatial reference claiming entity \(e\) at index \(i\) must correspond to a valid mention within the English coreference chain \(C_e\). For directional verbs (\(i\text{:VERB:}j\)), both source and target indices are evaluated and constrained independently. If the LLM generates a conflicting index for an established entity, the post-processor deterministically applies string replacement across the translation, ensuring that visual pointing remains anchored to the exact same spatial locus throughout the discourse.
2. Discourse Position and Context Check: Regulating Pseudocleft Restructuring Question-Answer Clauses (QACs) are topic-comment structures composed of a rhetorical question constituent followed by an answer constituent. In ASL, they emphasize salient information or clarify causal motivations, but misplacing them disrupts narrative flow. The Question-Answer Clause Module (QACM) maintains a registry of historical restructuring choices: $\(Q = \{(t, q) : t \in T, q \in \{0, 1\}\}\)$ where \(q\) indicates whether a QAC was deployed at sentence \(t\). The module regulates QAC usage along two dimensions: Contextual Appropriateness, which permits QAC insertion only when the source clause conveys explicit discourse triggers such as causal explanations (e.g., replacing "because", "so that", "in order to" with "WHY?"), contrastive elements, or focal emphasis; and Discourse Position, which strictly forbids QAC constructions at discourse beginnings (\(t=0\)) or when directly answering genuine questions. The discourse position constraint is baked into the prompt for \(t=0\) via pre-processing, with a deterministic fallback in metadata verification.
3. Concept-Gloss Consistency Mapping: Enforcing Lexical Stability and Vocabulary Compliance
In spoken and written English, authors frequently alternate between synonyms and hypernyms to avoid repetitive phrasing. In sign language, however, arbitrary lexical switching introduces ambiguity. The Concept-Gloss Consistency Module (CGCM) maintains an active registry of established noun-to-sign correspondences:
$\(L = \{(c, g, t) : c \in C, g \in V, t \in T\}\)$
where \(c\) is an English concept, \(g\) is an ASL gloss from vocabulary \(V\) (drawn from the 2,859-gloss ASLLRP SignBank lexicon), and \(t\) is the inception sentence. The CGCM enforces Mapping Consistency, which requires that any concept \(c\) previously registered in \(L\) must be mapped to gloss \(g\) on all subsequent appearances, and Vocabulary Compliance, ensuring that all produced glosses exist in \(V\). If a concept lacks a lexical sign in \(V\), the module mandates fingerspelling fallback (e.g., fs-W-E-A-S-E-L). During post-processing, if a recognized concept deviates from its registered gloss, the canonical sign is substituted deterministically throughout the translation string.
A Worked Example¶
Consider the opening sentences of an Aesop fable describing a bat and a weasel:
- Source English sentences:
- S1: "A Bat fell to the ground and was caught by a Weasel, and was just going to be killed and eaten when it begged to be let go."
- S2: "The Weasel said he couldnโt do that because he was an enemy of all birds on principle."
- Sentence-level baseline behavior:
- Translating S1 produces FOX IX-3p:j for the weasel, erroneously swapping lexical identity.
- Translating S2 independently fails to link the referent, rendering weasel as fingerspelled FS-W-E-A-S-E-L but erroneously reassigning it to IX-3p:i (the location originally occupied by the bat), while omitting the causal QAC structure.
- DiscoSign execution:
- In S1, SCM registers (bat, IX-3p:i, 1) and (weasel, IX-3p:j, 1); CGCM registers ("weasel", fs-W-E-A-S-E-L, 1).
- In S2, these registries enter the prompt as hard constraints. The model correctly applies fs-W-E-A-S-E-L at IX-3p:j, uses IX-3p:i to refer back to the bat, and QACM detects the causal connective "because", synthesizing the natural ASL pseudocleft: FS-W-E-A-S-E-L IX-3p:j SAY CANNOT FREE IX-3p:i WHY? IX-3p:j OPPOSITE+AGENT ALL BIRD RULE.
Key Experimental Results¶
Main Results¶
Evaluation was conducted on the multi-sentence Aesop's Fables dataset (284 stories, ~5 sentences per story) for discourse phenomena, and two sentence-level benchmarks (ASL STEM Wiki and a licensed dataset with expert Deaf annotations). The evaluation benchmarks both standard machine translation metrics (chrF, COMET) and three newly formulated discourse metrics: Spatial Coreference Accuracy (SCA), Concept-Gloss Consistency (CGC), and Question-Answer Clause Appropriateness (QACAp).
| Backbone | System Approach | Spatial Coref. (SCA) โ | Concept Consist. (CGC) โ | QAC Approp. (QACAp) โ | chrF โ | COMET โ |
|---|---|---|---|---|---|---|
| Gemini 2.5 Pro | Sentence-Level Baseline | 0.29 | 0.59 | 0.70 | 40.9 | 0.77 |
| Gemini 2.5 Pro | Context-Aware Baseline | 0.48 | 0.68 | 0.72 | 39.4 | 0.76 |
| Gemini 2.5 Pro | DiscoSign (Proposed) | 0.84 | 0.97 | 0.76 | 41.7 | 0.78 |
| Qwen3.6-35B-A3B | Sentence-Level Baseline | 0.12 | 0.73 | 0.73 | 40.1 | 0.73 |
| Qwen3.6-35B-A3B | Context-Aware Baseline | 0.19 | 0.83 | 0.72 | 41.2 | 0.74 |
| Qwen3.6-35B-A3B | DiscoSign (Proposed) | 0.61 | 0.86 | 0.75 | 40.3 | 0.74 |
On isolated sentence datasets, DiscoSign verified that discourse capabilities do not degrade local semantics: on ASL STEM Wiki, back-translation chrF rose from 41.9 to 54.8 (COMET: 0.73 โ 0.81); on the licensed dataset, back-translation chrF reached 67.2 vs. 65.2 (COMET: 0.90 vs. 0.88).
Ablation Study¶
A component-wise ablation on Aesop's Fables using Gemini 2.5 Pro demonstrates the precise isolation and impact of each module:
| Configuration | SCM | CGCM | QACM | SCA โ | CGC โ | QACAp โ | chrF | COMET | Note |
|---|---|---|---|---|---|---|---|---|---|
| Spatial only | โ | โ | โ | 0.81 | 0.64 | 0.71 | 41.8 | 0.78 | Lacks concept tracking; CGC drops to 0.64 |
| Concept only | โ | โ | โ | 0.71 | 0.95 | 0.72 | 42.5 | 0.79 | Lacks spatial tracking; SCA drops to 0.71 |
| Spatial + Concept | โ | โ | โ | 0.84 | 0.97 | 0.72 | 42.5 | 0.77 | Preserves coreference and vocabulary, lower QACAp |
| Full Framework | โ | โ | โ | 0.84 | 0.97 | 0.76 | 41.7 | 0.78 | Balanced performance across all discourse metrics |
Key Findings¶
- Traditional MT metrics are blind to discourse-level coherence: Across all conditions, string overlap (chrF ~39โ42) and neural semantic similarity (COMET ~0.74โ0.78) barely shifted despite severe spatial coordinate swapping and entity hallucinations. In fact, on Qwen, the context-aware baseline scored the highest chrF while exhibiting a low SCA of 0.19, underscoring that lexical overlap metrics cannot evaluate sign language spatial cohesion.
- Explicit state tracking decisively outperforms raw context access: Providing prior sentences without registries (Context-Aware) yielded only modest improvements in SCA (0.29 โ 0.48 on Gemini). By contrast, DiscoSign achieved an SCA of 0.84. In context-window scaling experiments, DiscoSign with a 1-sentence window (SCA: 0.57) outperformed the context-aware baseline supplied with the entire story (SCA: 0.48).
- Human ratings validate automated discourse metrics: Double-blind ratings by two ASL-fluent experts across 32 stories (152 sentences) showed statistically significant story-level correlation with SCA (\(\rho = 0.52, p = 0.003\) combined) and CGC (\(\rho = 0.48, p = 0.007\)). QACAp displayed rater subjectivity, as signers found pseudocleft topicalization acceptable in wider contexts than rule-based criteria alone.
Highlights & Insights¶
- Dual-layer guarantee combining prompt constraints with deterministic repair: Passing registries as hard constraints guides the LLM to generate valid structured outputs, while deterministic string replacement corrects residual coordinate slips without incurring the latency or monetary cost of re-prompting.
- First systematic formalization and evaluation suite for sign language discourse: Establishing formal definitions and automated metrics for spatial coreference accuracy (SCA), concept-gloss consistency (CGC), and pseudocleft appropriateness (QACAp) fills a critical void in sign language natural language processing.
- Model-agnostic generalization to open-weight architectures: Demonstrating large discourse improvements on both Gemini 2.5 Pro and the open-weight MoE model Qwen3.6-35B-A3B (+0.49 SCA gain) verifies that the framework is architecture-agnostic and viable for local or privacy-sensitive deployment.
Limitations & Future Work¶
- Dependency on structured JSON output capability: The framework relies on the backbone LLM's capacity to follow strict instruction formats and output valid JSON metadata; less capable base models may suffer syntax parsing failures.
- Rule-based rigidity in QAC triggers: The QACM uses deterministic discourse connectives (e.g., causal conjunctions) to trigger pseudoclefts, missing stylistic register variations where native signers use QACs without explicit lexical markers. Replacing trigger rules with a learned, probabilistic scoring model is a natural next step.
- Exclusion of non-manual markers (NMMs): Glosses capture manual sign labels but omit critical facial expressions, head tilts, and eyebrow movements (e.g., raised eyebrows during the question constituent of a QAC). However, DiscoSign's registries provide exactly the structured state needed to condition downstream visual synthesis engines.
Related Work & Insights¶
- vs. Sentence-level and Gloss-free Sign Translation (Gong et al., 2024; Wong et al., 2024): Previous neural models targeted direct end-to-end video mapping or sentence-level gloss decoding without tracking entity reference frames. DiscoSign establishes cross-sentence continuity as a first-class citizen in text-to-gloss pipelines.
- vs. Linguistic Coreference & Spatial Studies (Yin et al., 2021a; Caponigro & Davidson, 2011): Prior linguistic research identified the theoretical importance of spatial deictic loci and topic-comment structures, but lacked an integrated, deployable computational system. DiscoSign translates these formal linguistic principles into programmatic registries and operational constraints.
Rating¶
- Novelty: โญโญโญโญโญ [Pioneers discourse-aware sign language gloss translation with novel formalizations and metrics]
- Experimental Thoroughness: โญโญโญโญโญ [Extensive validation across proprietary/open-weight LLMs, multi-sentence fables, sentence benchmarks, and Deaf expert evaluations]
- Writing Quality: โญโญโญโญโญ [Clear linguistic grounding, elegant formulation of constraints, and thorough analysis]
- Value: โญโญโญโญโญ [Provides an essential foundation for linguistically authentic sign language generation and virtual signing avatars]