Skip to content

Lost Between Scripts: Recovering Arabic Words from Noisy Learner Orthographies

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/CAMeL-Lab/ArabScribe
Area: LLM Reasoning
Keywords: Cross-Script Retrieval, Intended-Word Recovery, Arabic NLP, Form-Meaning Alignment, LLM Reasoning

TL;DR

Addressing the acute challenge where second-language learners produce distorted spellings across Arabic script and Romanized Arabizi, this paper formulates gloss-grounded intended-word recovery, demonstrating that combining noisy surface forms with English semantic glosses closes cross-script performance disparities and allows leading LLMs to approach human expert retrieval accuracy.

Background & Motivation

Language learners and cross-lingual search users frequently attempt to look up words with only fragmented recollections of their phonetic pronunciation and semantic definitions, lacking precise knowledge of standard orthography. In conventional computational linguistics, spelling correction and phonetic search systems concentrate almost exclusively on surface orthographic forms, whereas reverse dictionaries retrieve lexical entries purely from definitions. In realistic reference lookup scenarios, however, both phonetic and semantic cues are concurrently accessible yet fraught with uncertainty. Formulating how intelligent language models can jointly reason over these noisy, multi-faceted cues is an essential challenge connecting theoretical lexical retrieval with practical assistive language interfaces.

This challenge is uniquely magnified in the Arabic writing system. Written Modern Standard Arabic (MSA) is primarily an abjad that routinely omits short vowels and consonant gemination markers while possessing a rich inventory of uvular, pharyngeal, and emphatic phonemes that frequently confound non-native learners. Furthermore, learners routinely resort to Roman-script transcriptions or informal Arabizi (such as substituting numerals like 3 for \(\varepsilon\) and 7 for \(\hbar\)), where orthographic mappings lack standardized one-to-one conventions. Consequently, a single target word spawns dozens of wildly disparate transcriptions across scripts, causing surface-only string matching to drown in orthographic ambiguity while definition-only retrieval suffers from an overly broad semantic search space.

Relying on either distorted surface forms or isolated semantic glosses fundamentally caps retrieval reliability under heavy learner noise. Large language models present an unprecedented opportunity to bridge this gap through their latent cross-lingual representations and joint inference capabilities. Core idea: formulate gloss-grounded intended-word recovery as a ranked candidate generation problem, leveraging LLM cross-lingual reasoning to co-align distorted multi-script surface forms with semantic gloss constraints, thereby systematically bridging cross-script ambiguities and learner proficiency chasms.

Method

Overall Architecture

The recovery pipeline models word retrieval as a constrained, ranked generation problem. Given a learner's noisy transcription in either Arabic script or Roman script, paired optionally with an English meaning gloss, a prompted LLM must infer and return exactly ten unique, ranked diacritized Arabic word candidates. The system comprises three foundational phases: enriching multi-script learner transcriptions from ArabScribe with comprehensive morphological glosses from SAMA, executing structured-prompting multi-cue inference across controlled experimental conditions, and evaluating ranked candidate lists under multi-tier normalization schemes.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Noisy Learner Transcription<br/>(Arabic Script / Roman Arabizi)"] --> B["Morpho-Semantic Gloss Enrichment<br/>SAMA database lookups preserving multi-gloss senses"]
    C["English Semantic Meaning (Gloss)"] --> B
    B --> D["Structured Prompting & Multi-Cue Reasoning<br/>Word / Gloss / Word+Gloss experimental conditions"]
    D --> E["Multi-Tier Normalization & Candidate Reranking<br/>Acc@1 / Acc@10 / MRR evaluation"]
    E --> F["Output: Top-10 Ranked Standard Arabic Candidates"]

Key Designs

1. Morpho-Semantic Gloss Enrichment: Preserving Polysemy Boundaries via SAMA

The original ArabScribe benchmark focused purely on form-level transcription matching, leaving the semantic dimension unrepresented. To authentically replicate the lookup context where learners recall what a word means even if its orthography is uncertain, this work augments gold target words using the Standard Arabic Morphological Analyzer (SAMA). Crucially, the authors avoid artificially simplifying lexical ambiguity: when SAMA returns multiple analyses or glosses for a given diacritized gold target (such as qam~aH generating pay in installments; apportion), all candidate glosses are retained. This provides realistic, dictionary-grade semantic grounding without biasing the model with hand-curated disambiguation.

2. Structured Prompting & Multi-Cue Reasoning: Enforcing JSON Schemas across Orthogonal Cue Conditions

To benchmark how models process conflicting and complementary evidence, the framework structures model queries via native API JSON schema constraints, requiring an ordered array of exactly ten unique diacritized Arabic lemmas. Three controlled prompting conditions are tested: Word (presenting only the learner's noisy transcription to evaluate raw phonological and orthographic correction), Gloss (presenting solely the English definition as a reverse-dictionary control), and Word+Gloss (supplying both cues to emulate realistic dictionary lookups). By maintaining identical prompt scaffolds across conditions, the setup cleanly isolates the marginal utility of semantic conditioning against orthographic distortion.

3. Multi-Tier Normalization & Candidate Reranking: Decoupling Surface Variations from Lexical Retrieval

Arabic learners and native speakers frequently introduce non-distinctive surface orthographic variations, particularly regarding Alif variants (ุฃ / ุฅ / ุข / ุง), Ya versus Alif Maqsura (ูŠ / ู‰), and Ta-Marbuta versus Ha (ุฉ / ู‡), alongside arbitrary vowel diacritic omissions. To rigorously measure genuine lexical recovery rather than superficial diacritization matching, the evaluation pipeline defines three assessment protocols: Full Normalization (folding spelling variants and stripping diacritics), Undiacritized matching, and Exact Match. Full Normalization serves as the primary benchmark metric for Acc@1, Acc@10, and Mean Reciprocal Rank (MRR), ensuring that candidates which correctly recover the target lemma are not penalized for surface-level spelling conventions.

Benchmark & Evaluation Protocol

Experiments are conducted on the ArabScribe test partition (comprising 654 Arabic-script and 1,342 Roman-script transcriptions produced by 103 participants spanning six Arabic proficiency levels from complete beginners to native speakers). Four prominent LLMs representing diverse architectures are evaluated under zero-shot prompting: Gemini-3.1-Pro, GPT-5.4, Llama-4-Scout (17B active / 109B total MoE), and the Arabic-focused Fanar-2 (27B). In addition, two expert native Arabic NLP annotators (specializing in Egyptian and Lebanese Arabic) independently annotated 200 sampled test instances under identical condition protocols, establishing an empirical upper bound for human cognitive recovery.

Key Experimental Results

Main Results

Under primary zero-shot evaluation with full normalization, model performance across scripts and cue conditions is reported below:

Model Cue Condition Arabic Script Acc@1 Arabic Script Acc@10 Arabic Script MRR Roman Script Acc@1 Roman Script Acc@10 Roman Script MRR
Gemini-3.1-Pro Word 46.3 59.9 0.51 38.2 62.0 0.46
Gemini-3.1-Pro Gloss 43.1 69.4 0.52 42.8 73.5 0.52
Gemini-3.1-Pro Word+Gloss 76.0 93.1 0.83 82.3 93.7 0.87
GPT-5.4 Word 41.7 58.0 0.47 25.3 49.9 0.33
GPT-5.4 Gloss 36.9 57.6 0.43 33.8 57.9 0.43
GPT-5.4 Word+Gloss 67.1 80.4 0.72 70.4 84.4 0.75
Llama-4-Scout Word 37.3 45.9 0.40 19.1 33.0 0.24
Llama-4-Scout Gloss 22.3 36.5 0.27 16.8 34.6 0.22
Llama-4-Scout Word+Gloss 49.2 66.2 0.55 42.0 59.1 0.48
Fanar-2 Word 34.7 43.6 0.38 15.9 27.4 0.20
Fanar-2 Gloss 12.2 21.4 0.16 12.2 26.4 0.18
Fanar-2 Word+Gloss 40.5 54.4 0.45 33.8 47.5 0.39

Ablation & Fine-Grained Analysis

The table below contrasts human expert performance with the average zero-shot Acc@1 of all four LLMs stratified across normalized edit distance bins and learner proficiency tiers:

Dimension / Stratification Arabic Word Arabic Word+Gloss Roman Word Roman Word+Gloss Empirical Observation & Insight
Human Expert 1 43.0 77.0 40.0 71.0 Gloss lifts expert Acc@1 by over 30 points
Human Expert 2 43.0 67.0 38.0 70.0 Inter-expert agreement climbs from 57% to 75%
Edit Distance [0, 0] 88.3 79.5 59.6 71.7 High accuracy when learner transcription has zero distortion
Edit Distance (0, 0.25] 25.2 54.9 34.2 63.4 Word-alone drops sharply; dual-cues provide robust gains
Edit Distance (0.25, 0.5] 4.9 48.1 22.0 57.9 High noise causes Word-only collapse; Gloss anchors recovery
Edit Distance (0.5, 1.0] 0.6 24.7 7.7 44.2 Word-only fails completely; semantics recovers up to 44.2%
Beginner Learners 32.9 55.6 22.7 56.1 Experience \(\le\) 2 yrs; glosses narrow proficiency gap
Advanced Learners 67.6 68.1 35.8 63.2 Standard spellings afford high Word-alone baseline

Key Findings

  • Profound Cue Complementarity: Word+Gloss overwhelmingly exceeds single-cue performance across all models and scripts. Averaged over all four LLMs, Arabic-script Acc@1 increases from 40.0 (Word) to 58.2 (Word+Gloss), while Roman-script Acc@1 jumps from 24.6 to 57.1. In Roman script, 21.9% of all test items are recovered exclusively when both cues are provided (failing under Word-alone and Gloss-alone).
  • Closing the Cross-Script Performance Gap: In isolation, Roman-script transcriptions are significantly more difficult than Arabic-script input, lagging by 15.4 Acc@1 points (24.6 vs 40.0) due to phonetic ambiguities and unstandardized Arabizi substitutions. Conditioning on semantic glosses collapses this disparity to just 1.1 points (57.1 vs 58.2), proving semantic grounding offsets cross-script entropy.
  • Human Expert Alignment: Even professional Arabic linguists achieve modest Acc@1 scores (38%โ€“43%) when examining transcriptions alone, reflecting inherent phonetic ambiguity. Supplying English glosses elevates expert accuracy to 67%โ€“77% and inter-annotator agreement from 57% to 75%. Notably, Gemini-3.1-Pro attains 76% (Arabic) and 78% (Roman) on this human evaluation subset, matching or marginally outperforming individual human experts.
  • Robustness Across Distortion and Proficiency: Under extreme distortion (edit distance \(> 0.5\)), Word-alone accuracy collapses to 0.6% on Arabic script, whereas Word+Gloss rescues accuracy to 24.7% (and 44.2% on Roman script). Similarly, the proficiency gap between beginners and advanced learners under Arabic Word input shrinks from 34.7 points to 12.5 points when glosses are added.
  • Bottlenecks in Rare Lexical Targets: An analysis of 62 hard cases unresolved by any model shows they are concentrated in only 28 unique target words. Infrequent lexical items like Dah.l ('small quantity') and qam~aH ('pay in installments') are consistently missed because their transcriptions closely resemble common roots (baH.r 'sea', nah.r 'river') or trigger frequent semantic substitutes (qas~aTa), highlighting an inherent LLM bias toward high-frequency lexical priors.

Highlights & Insights

  • The formulation bridges the historical divide between spelling correction (form-centric) and reverse dictionaries (meaning-centric), establishing a unified framework for multi-modal, cross-script lexical recovery.
  • Rigorous human expert benchmarking demonstrates that semantic augmentation is not merely an artificial LLM crutch, but an indispensable cognitive anchor required even by native lexicographers when deciphering noisy cross-script transcriptions.
  • Error analysis provides a valuable insight for LLM reasoning: retrieval failures in specialized domains are rarely caused by complete semantic incomprehension, but rather by competition from high-frequency lexical distractors when observed forms deviate from standard orthography.

Limitations & Future Work

  • The study focuses on isolated Modern Standard Arabic (MSA) word recovery, without evaluating dialectal Arabic variations (which feature distinct phonology and vocabulary) or sentence-level syntactic contexts that naturally constrain lexical ambiguity in practice.
  • The evaluation employs pristine, curated dictionary glosses from SAMA, representing an idealized upper bound of semantic support. Real-world learners typically supply informal, incomplete, or partially erroneous definitions.
  • All experiments rely on prompting off-the-shelf models without supervised task-specific fine-tuning, constrained decoding against verified lexical inventories, or interactive user-in-the-loop validation studies.
  • vs MultiScript Phonetic Search (Zhang et al., 2017): The original ArabScribe baseline performed dictionary lookup via edit-distance mapping over a fixed lexicon and could only take surface transcriptions; this work expands the task to open-vocabulary LLM generation while integrating cross-lingual semantic conditioning.
  • vs Reverse Dictionaries (WantWords, Qi et al., 2020; Hill et al., 2016): Traditional reverse dictionaries map definitions to words but provide poor ranking precision for near-synonyms; this paper demonstrates that even severely corrupted surface phonetic fragments provide critical pruning power for semantic queries.

Rating

  • Novelty: โญโญโญโญ Elegantly conceptualizes gloss-grounded intended-word recovery, bridging phonetic cross-script ambiguity with cross-lingual semantic grounding.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across 4 state-of-the-art LLMs, 3 cue conditions, 3 normalization tiers, multi-dimensional stratification, and a solid human expert baseline.
  • Writing Quality: โญโญโญโญโญ Exceptional clarity, rigorous linguistic foundations, and insightful empirical decompositions.
  • Value: โญโญโญโญ Provides foundational insights and an extensible benchmark for assistive language tools, cross-script phonetic search, and multilingual LLM evaluation.