Skip to content

CWoMP: Morpheme Representation Learning for Interlinear Glossing

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://cwomp.github.io
Area: Self-Supervised & Representation Learning
Keywords: Interlinear Glossed Text (IGT), Morpheme Representation Learning, Contrastive Learning, Mutable Lexicon Retrieval, Low-Resource NLP

TL;DR

CWoMP reformulates interlinear glossing from unconstrained character sequence generation into generative retrieval over a mutable lexicon of atomic form-meaning morpheme units, eliminating morpheme hallucinations, enforcing strict 1:1 segmentation-gloss alignment, reducing inference FLOPs by 8.5x, and allowing zero-retraining inference-time vocabulary expansion.

Background & Motivation

In endangered language documentation, language revitalization, and documentary linguistic scholarship, Interlinear Glossed Text (IGT) serves as the primary and most linguistically informative format for structural annotation. Following the Leipzig Glossing Rules, standard IGT presents raw transcriptions aligned with a morphological segmentation tier (dividing stems and affixes with hyphens) and an interlinear gloss tier (specifying uppercase grammatical category tags and lowercase lexical meanings). Despite its foundational role in linguistics, producing accurate IGT manually requires deep domain expertise and is notoriously labor-intensive. This has spurred significant interest in automating IGT generation within computational linguistics, exemplified by the SIGMORPHON shared tasks. Recent state-of-the-art approaches, such as GlossLM and finetuned ByT5 architectures, approach the task as unconstrained character- or byte-level sequence-to-sequence (seq2seq) generation, relying on massively multilingual pretraining to directly decode gloss string sequences.

However, treating linguistic glosses as unconstrained continuous text strings fundamentally contradicts the discrete, compositional nature of morphology. This paradigm mismatch induces critical failure modes in real-world documentation. First, seq2seq models suffer severely from morpheme hallucinations—generating spurious grammatical tags, garbled suffixes, or non-existent lexical roots; on extremely low-resource languages such as Gitksan, GlossLM exhibits a staggering morpheme hallucination rate of 44.84%, which poses a catastrophic risk of corrupting irreplaceable endangered language archives. Second, because segmentation and glossing tiers are either generated disjointly or the segmentation is bypassed altogether, models frequently output mismatched numbers of segments and gloss tags, producing unusable alignment errors. Third, autoregressive decoding over raw character or byte sequences scales linearly with output byte length, demanding heavy compute budgets (over 210B FLOPs per sentence). Most critically, documentary field linguistics invariably encounters out-of-vocabulary (OOV) morphemes in the field, yet traditional black-box neural generators cannot assimilate newly discovered stems or affixes without costly full-parameter retraining.

To bridge this disconnect between computational research and the empirical workflows of documentary linguists, this paper reframes automated glossing around morphological compositionality and discrete lexical grounding. Core idea: treat morphemes as atomic, indivisible form-meaning units, pretrain a shared dual encoder via multi-positive contrastive learning to align contextual words with constituent morpheme embeddings in a discrete codebook, and employ a lightweight autoregressive decoder that retrieves codebook entries step-by-step—guaranteeing zero morpheme hallucinations, strictly synchronized segmentation-to-gloss alignment, and zero-retraining inference-time lexicon expansion.

Method

Overall Architecture

The CWoMP pipeline operates in two sequential stages: contrastive representation learning followed by generative codebook retrieval. In the first stage, the Bag-of-Morphemes (BoM) Encoder—initialized from a multilingual LaBSE backbone—learns to project words in sentential context and candidate morpheme form-meaning units into a shared metric embedding space. Once trained, the BoM encoder is frozen and executed offline across the target language's training vocabulary to construct a precomputed matrix of discrete morpheme representations (\(V_{\text{emb}}\)). In the second stage, an autoregressive IGT Decoder takes the dense embedding of an unsegmented input word as conditioning prefix, generates continuous hidden representations across sequential morphological steps, and matches each state to its nearest neighbor in the discrete lexicon via dot-product similarity, directly producing the joint segmentation and gloss sequence.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input: target word w in context + translation<br/>+ candidate morpheme pool M = {(s_i, g_i)}"]
    In --> Des1["Multi-Positive Contrastive BoM Encoding<br/>Forced character spacing + shared LaBSE encoder + multi-positive InfoNCE"]
    Des1 --> Codebook["Discrete Morpheme Codebook V_emb<br/>Offline precomputed embeddings of all (segment, gloss) pairs + [eos]"]
    Des1 --> Des2["Discrete Lexicon Retrieval Autoregressive Decoding<br/>4-layer Transformer conditioned on word embedding; dot-product codebook matching"]
    Codebook --> Des2
    Des2 --> Des3["Inference-Time Mutable Lexicon Expansion<br/>Interactive injection of novel morpheme embeddings; zero-retraining OOV coverage"]
    Des3 --> Out["Output: strictly 1:1 aligned<br/>morphological segmentation and gloss sequence G_w"]

Key Designs

1. Multi-Positive Contrastive BoM Encoding: Resolving Morpheme Reuse and Form-Meaning Polysemy When mapping natural language sentences into morphological constituent sets, identical affixes and grammatical markers (e.g., verbal agreement prefixes or nominal case suffixes) repeatedly co-occur across distinct target words within the same training mini-batch. In standard InfoNCE contrastive training, all other in-batch morphemes are penalized as negative instances, inadvertently penalizing the model for recognizing valid shared morphological building blocks across different lexical items. To resolve this, CWoMP models each contextual word as a bag-of-morphemes (\(B_w = \{m_1, \dots, m_k\}\)), where each constituent morpheme is an atomic tuple \(m_i = (s_i, g_i)\) pairing its surface/canonical segment string \(s_i\) with its semantic/grammatical gloss \(g_i\). The text is explicitly processed with inter-character spacing to force character-level tokenization—circumventing cross-lingual subword tokenization fragmentation—and concatenated with sentential context and translation strings. The word-in-context and morpheme representations are generated by shared Transformer parameters \(\theta\) and \(L_2\)-normalized to yield \(p_\theta(w)\) and \(q_\theta(m)\). To handle valid intra-batch co-occurrences, the model identifies the full set of valid positive pairs \(\mathcal{P}_i\) for each query \(w_i\) and averages their log-probabilities in a multi-positive InfoNCE formulation: $\(\mathcal{L}(\theta) = -\frac{1}{B} \sum_{i=1}^B \frac{1}{|\mathcal{P}_i|} \sum_{p \in \mathcal{P}_i} \log \frac{\exp\left(S(w_i, m_p) / \tau\right)}{\sum_{j=1}^B \exp\left(S(w_i, m_j) / \tau\right)}\)$ where \(S(w_i, m_j) = p_\theta(w_i) \cdot q_\theta(m_j)\) denotes cosine similarity and \(\tau\) is a learnable temperature. This contrastive objective induces a geometrically regular space supporting linear morphological vector arithmetic (achieving mean pairwise cosine similarities of 0.70–0.79 in analogy tests) while disambiguating homographic morphemes via sentential context.

2. Discrete Lexicon Retrieval Autoregressive Decoding: Eradicating Hallucinations and Enforcing Structural Alignment Unconstrained autoregressive generation on characters or bytes inherently lacks discrete structural bounds, resulting in invalid morpheme abbreviations, non-existent roots, and severe mismatches between segmentation hyphens and gloss tags. CWoMP resolves this by decoupling representation learning from sequence generation. Once the BoM encoder is frozen, an offline embedding codebook \(V_{\text{emb}}\) is compiled containing representations for every unique \((s_i, g_i)\) pair in the lexicon, alongside a terminating [eos] token. The downstream IGT decoder is a lightweight 4-layer Transformer (512 hidden dimension, 4 attention heads). The contextual word embedding \(q_\theta(w)\) is assigned to position 0, followed by [bos] and previously retrieved morpheme embeddings. At each autoregressive step \(j\), the decoder's hidden state \(h_j\) is scored directly against all entries in the frozen codebook \(V_{\text{emb}}\): $\(\text{logits}_j = \frac{h_j^\top V_{\text{emb}}}{\kappa}\)$ where \(\kappa\) is a learnable scaling temperature. Decoding is executed via beam search with a width of 5. Because prediction is mathematically constrained to discrete lexicon entries, the model achieves a 0% hallucination rate by design. Furthermore, because each retrieved entry simultaneously retrieves both the segmentation string and its gloss label, the number of morpheme segments and glosses is guaranteed to be identical, completely eliminating segmentation-gloss misalignment. Additionally, decoding at the morpheme level (typically 2–4 steps per word) bypasses the computational bottleneck of decoding hundreds of byte tokens.

3. Inference-Time Mutable Lexicon Expansion: Overcoming Low-Resource OOV Bottlenecks Without Retraining In documentary field linguistics, extreme data scarcity guarantees that evaluation corpora and newly collected fieldwork sessions will contain out-of-vocabulary (OOV) morphemes unobserved during training. For standard parametric seq2seq models, adapting to new morphemes requires full-model backpropagation, which is impractical for field linguists operating on laptops without specialized compute or machine learning expertise. CWoMP leverages its generative retrieval architecture to provide an inference-time mutable lexicon. When a linguist identifies an unrecorded morpheme \((s_{\text{new}}, g_{\text{new}})\), they do not need to update or fine-tune any Transformer weights; they simply pass the new tuple through the frozen BoM encoder to compute its embedding vector and append it as a new column in \(V_{\text{emb}}\). The downstream decoder immediately incorporates the new morpheme into its beam search candidate space. This mechanism seamlessly integrates into human-in-the-loop interfaces such as GlossAssist, allowing linguist verification decisions to dynamically expand the system's operational vocabulary in real time.

A Worked Example

The concrete inference trajectory for the Tsez word ragäƛin demonstrates the generative retrieval workflow: - Input Encoding: For the utterance T’ay rił łu ragäƛin? ("Who licked the butter out of it?"), target word ragäƛin and its sentential context are encoded by the frozen BoM encoder into a single dense vector \(q_\theta(w)\). - Step 1 (Prefix Retrieval): Conditioned on \(q_\theta(w)\) at index 0 and [bos], the 4-layer IGT decoder projects hidden state \(h_1\) against the codebook, retrieving nearest neighbor noun-class prefix \(m_1 = (\text{r-}, \text{IV})\). - Step 2 (Stem Retrieval): Embedding \(m_1\) is fed back into the decoder. At step 2, \(h_2\) queries \(V_{\text{emb}}\) and matches the lexical verbal stem \(m_2 = (\text{agi}, \text{lick})\). - Step 3 (Interrogative Suffix): Feeding \(m_2\) forward, the decoder projects \(h_3\) to retrieve the question particle \(m_3 = (\text{-a}, \text{Q})\). - Step 4 (Quotative Suffix): The decoder predicts \(h_4\), matching the quotative enclitic \(m_4 = (\text{-ƛin}, \text{QUOT})\). - Step 5 (Termination): The decoder matches [eos], completing the beam search. - Assembly: Concatenating retrieved entries yields the segmentation r-agi-a-ƛin and the gloss IV-lick-Q-QUOT, achieving exact ground-truth agreement with strictly aligned morphological tiers.

Loss & Training

  • BoM Encoder Pretraining: Initialized from multilingual LaBSE (768-dim embeddings). Trained with multi-positive InfoNCE using Adam optimizer, learning rate \(2 \times 10^{-5}\), 100 warmup steps, batch size 128, for 100 epochs (1000 epochs for Gitksan).
  • IGT Decoder Training: 4 Transformer decoder blocks (hidden dim 512, 4 heads), optimized via teacher-forced cross-entropy over codebook indices. Batch size 32, learning rate \(1 \times 10^{-4}\), weight decay 0.01, gradient clipping norm 1.0, dropout 0.1, trained for 100 epochs (1000 for Gitksan).
  • Inference Decoding: Standard beam search with a beam width of 5 over precomputed codebook embeddings.

Key Experimental Results

Main Results

CWoMP is evaluated on seven typologically diverse endangered and low-resource languages from the SIGMORPHON 2023 shared task: Arapaho (arp, 39,501 train sentences), Uspanteko (usp, 9,774 sentences), Tsez (ddo, 3,558 sentences), Nyangbo (nyb, 2,100 sentences), Natugu (ntu, 791 sentences), Lezgi (lez, 701 sentences), and Gitksan (git, 31 sentences). Evaluations are conducted under both the Train Lexicon (strictly closed vocabulary) and Extended Lexicon (oracle expansion with evaluation set ground-truth morphemes) settings.

The gloss generation performance (Morpheme Accuracy and Word Accuracy) comparing CWoMP against the strongest baseline GlossLM (+ft) is shown below (quoted directly from paper Table 5):

Language Train Sentences Morpheme Acc (GlossLM) Morpheme Acc (CWoMP Train) Morpheme Acc (CWoMP Ext.) Word Acc (GlossLM) Word Acc (CWoMP Train) Word Acc (CWoMP Ext.)
arp (Arapaho) 39,501 82.1% 80.6% 84.1% 81.5% 83.7% 87.9%
usp (Uspanteko) 9,774 78.6% 69.8% 72.9% 81.0% 75.3% 79.9%
ddo (Tsez) 3,558 83.6% 82.8% 82.9% 87.3% 86.9% 87.4%
nyb (Nyangbo) 2,100 87.4% 88.2% 89.1% 86.2% 86.8% 87.8%
ntu (Natugu) 791 62.8% 66.6% 67.8% 78.9% 82.4% 86.1%
lez (Lezgi) 701 57.3% 66.0% 69.3% 64.9% 77.3% 81.9%
git (Gitksan) 31 10.1% 11.1% 14.8% 28.4% 24.7% 38.3%

On the joint morphological segmentation task, CWoMP is benchmarked against a finetuned ByT5 baseline using Morpheme Error Rate (MER \(\downarrow\)), as reported below (quoted directly from paper Table 3):

Model Setting arp usp ddo git lez ntu nyb
ByT5 Baseline 0.08 0.15 0.04 0.98 0.27 0.09 0.03
CWoMP (Train Lexicon) 0.11 0.21 0.03 0.71 0.16 0.14 0.04
CWoMP (Extended Lexicon) 0.07 0.17 0.02 0.57 0.11 0.10 0.02

Ablation Study

The authors ablate key architectural components across four languages (Tsez, Gitksan, Natugu, Nyangbo), isolating the effects of the encoder backbone, contrastive loss formulation, and prompt formatting (validation set averages, quoted from paper Table 8):

Encoder Backbone Loss Formulation Prompt Format NDCG@10 \(\uparrow\) mAP@100 \(\uparrow\) P@1 \(\uparrow\) R@10 \(\uparrow\)
CANINE-S Standard InfoNCE Word (W) .837 .774 .761 .894
CANINE-S Standard InfoNCE W + Transcript (Tr) .838 .777 .766 .933
CANINE-S Standard InfoNCE W + Translation (Ts) .827 .765 .758 .931
CANINE-S Standard InfoNCE W + Tr + Ts .847 .791 .776 .937
LaBSE Standard InfoNCE Word (W) .832 .792 .773 .908
LaBSE Standard InfoNCE W + Tr .846 .805 .795 .916
LaBSE Standard InfoNCE W + Ts .866 .823 .816 .938
LaBSE Standard InfoNCE W + Tr + Ts .881 .844 .840 .942
LaBSE Standard InfoNCE W + Tr + Ts + Spacing (Sp) .890 .846 .821 .970
LaBSE Multi-Positive InfoNCE W + Tr + Ts + Sp (Full System) .897 .856 .826 .972

Key Findings

  • Crucial Role of Multi-Positive Loss and Forced Character Spacing: Transitioning from standard InfoNCE to multi-positive InfoNCE boosts NDCG@10 from .890 to .897 and mAP from .846 to .856. Enforcing character-level tokenization via spaces raises R@10 from .942 to .970, preventing catastrophic subword tokenization fragmentation.
  • Superiority in Extremely Low-Resource Scenarios: CWoMP substantially outperforms GlossLM on mid-to-low resource languages. On Lezgi (701 sentences), whole-word accuracy increases from 64.9% to 77.3% (81.9% extended). On Gitksan (31 sentences), segmentation MER drops dramatically from 0.98 (ByT5) to 0.71 (0.57 extended).
  • Proportionality Between OOV Rate and Recoverable Gain: The proportion of OOV morphemes (pOOV) directly explains the performance delta between the Train and Extended lexicon settings (\(\Delta\)MER in Table 2: Gitksan pOOV .446 vs \(\Delta\)MER .133; Lezgi pOOV .064 vs \(\Delta\)MER .047).
  • Dramatic Compute Footprint Reduction: CWoMP requires 485M parameters compared to GlossLM's 582M. More importantly, while GlossLM requires 211.7B FLOPs per sentence (dominated by ByT5's byte-level autoregressive decoder at 123.9B FLOPs), CWoMP absorbs 88% of its compute in a single batched pass through the BoM encoder (21.8B FLOPs), with its lightweight morpheme decoder requiring just 3.0B FLOPs—reducing total inference FLOPs to 24.8B (an ~8.5x efficiency gain).
  • Elimination of Hallucinations: While GlossLM exhibits a 4.27% micro-average rate of generating illegal, unobserved morpheme types (spiking to 44.84% on Gitksan), CWoMP inherently enforces a 0% hallucination rate by constraining beam search to attested codebook entries.

Highlights & Insights

  • Morphology Grounded in Generative Retrieval: By framing morphological generation as step-wise retrieval over discrete form-meaning representations rather than free-form text synthesis, the model eliminates hallucinations while preserving full compositional flexibility.
  • Zero-Retraining Lexical Extensibility: Decoupling the parametric sequence decoder from the morpheme codebook allows documentary linguists to expand vocabulary on-the-fly by projecting new morpheme pairs into the vector space, creating an effective human-in-the-loop workflow.
  • Geometric Regularity of Morphological Algebra: The learned metric space captures genuine linguistic structure, exhibiting word2vec-style linear vector offsets for inflectional transformations across stems with mean cosine similarities up to 0.79.

Limitations & Future Work

  • Sensitivity to Inconsistent Spelling and Surface Variants: CWoMP underperforms GlossLM on Uspanteko because the dataset features heavy dialectal and transcription spelling variation (e.g., tiit' vs. tit'). Because CWoMP relies on discrete codebook matching, it cannot smoothly generalize across unobserved surface spelling variants.
  • Restriction to Concatenative Morphotactics: The architecture assumes concatenative morphology with hyphen-delimited segments and 1:1 segment-to-gloss correspondence, limiting immediate application to non-concatenative phenomena (such as Semitic root-and-pattern templatic morphology, infixation, and reduplication).
  • Future Directions: Integrating fuzzy phonetic/orthographic string matching to handle noisy field transcriptions, adopting Generalized Glossing Guidelines for non-concatenative morphology, and extending the retrieval framework to biological sequence annotation and Universal Dependencies morphosyntactic feature tagging.
  • vs GlossLM (Ginn et al., 2024c): GlossLM relies on massively multilingual ByT5 pretraining to perform unconstrained byte-level sequence generation; CWoMP trains solely on monolingual target IGT data, achieves 0% hallucinations, eliminates tier misalignments, slashes inference FLOPs by 8.5x, and supports inference-time lexicon expansion.
  • vs Tü-cl / Girrbach (2023): Girrbach utilizes stochastic hard attention with straight-through gradient estimation to predict segmentation and glosses jointly; CWoMP instead establishes a metric embedding space via multi-positive contrastive learning, yielding more stable autoregressive codebook retrieval.
  • vs Traditional RAG & Generative Retrieval (SimCIT, Pointer Networks): Unlike standard RAG (which retrieves documents to condition downstream open text generation) or generative retrieval (which generates composite item IDs), CWoMP integrates retrieval directly as the generative step, independently selecting discrete form-meaning linguistic entities from a mutable external codebook.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant shift from unconstrained seq2seq text generation to discrete generative retrieval over form-meaning representations.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 7 typologically diverse endangered languages, measuring MER, accuracy, FLOPs, hallucination rates, and vector analogy geometry.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, disciplined problem formulation, and insightful connections to linguistic field requirements.
  • Value: ⭐⭐⭐⭐⭐ Directly addresses the critical pain points of documentary linguistics with high reliability, low computational requirements, and seamless human-in-the-loop integration.