Skip to content

When Similar Means Different: Evaluating LLMs on Arabic–Hebrew Cognates

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/mbzuai-nlp/SemCog
Area: Multilingual & Machine Translation
Keywords: Cognate Identification, False Friends, Semantic Disambiguation, Semitic Languages, LLM Evaluation

TL;DR

This paper introduces SemCog Bench, a curated benchmark of 1,858 Arabic–Hebrew word pairs and 2,222 contextual sentence MCQs across true cognates, false friends, and loanwords, revealing that LLMs heavily rely on surface-form similarity rather than genuine cross-lingual semantic understanding.

Background & Motivation

In cross-lingual natural language processing and historical linguistics, lexical overlap between closely related languages serves as both a primary driver of cross-lingual transfer and a major source of semantic ambiguity. Arabic and Hebrew, two major Semitic languages, share extensive lexical overlap rooted in a non-concatenative root-and-pattern morphology where triconsonantal roots are combined with vocalic patterns. In modern written communication, short vowel diacritics—Arabic tashkil and Hebrew nikkud—are almost universally omitted, leaving consonantal skeletons that look remarkably similar across languages. Crucially, this surface similarity conflates three linguistically distinct relationships: true cognates inheriting both common Semitic ancestry and overlapping core meanings (e.g., Arabic وردة wrdℏ and Hebrew ורד wrd, both meaning 'rose'); false friends preserving strong phonological or orthographic resemblance despite radical semantic divergence over historical development (e.g., Arabic لحم lHm 'meat' vs. Hebrew לחם lxm 'bread', or Arabic خريف xryf 'autumn' vs. Hebrew חורף xwrp 'winter'); and modern loanwords borrowed in parallel from third non-Semitic languages (e.g., סושי / سوشي swšy 'sushi').

Computational modeling of cognates and cross-lingual lexical semantics has historically relied on surface string distance, phonetic class alignment (such as LingPy's sound-class alignment), or static bilingual word embeddings (such as FastText and multilingual Sentence-BERT). While these methods capture phonetic and semantic distances, they fail to disentangle inherited cognates from parallel borrowings, nor can they evaluate whether a word is pragmatically valid within dynamic sentence contexts. Recent multilingual LLM benchmarks have begun addressing cross-lingual lexical semantics, but prior efforts such as RoBoCoP remain restricted to context-free classification in Indo-European Romance languages, while StingrayBench evaluates contextual false friends across Indo-European and Austronesian pairs without distinguishing borrowing from genealogical inheritance. Crucially, no benchmark has investigated these form–meaning tensions in Semitic languages, where rich morphology and script differences create severe orthographic ambiguity.

Consequently, it remains unknown whether current multilingual and regional LLMs perform genuine semantic reasoning when encountering cross-lingual cognates, or whether they merely exploit superficial orthographic and phonological heuristics. To address this fundamental gap, the authors construct SemCog Bench, a rigorously curated benchmark with expert linguistic validation at both the word and sentence level, testing models across word-only and contextual settings under four orthographic and phonetic representations. Core idea: evaluate and decouple the reliance of multilingual LLMs on surface-form similarity versus genuine semantic reasoning by probing true cognates, false friends, and loanwords across orthogonal script representations, contextual environments, and model scales.

Method

Overall Architecture

The SemCog Bench framework comprises four core stages: lexical candidate aggregation from specialized Semitic lexicons and loanword inventories followed by part-of-speech filtering; rigorous two-stage human expert validation and arbitration; LLM-assisted contextual sentence generation refined through human naturalness auditing and minimal-contrast perturbation; and multi-level probing protocol execution across four input representations (undiacritized, fully diacritized, IPA transcription, and universal Romanization) under two benchmark tasks: Cognate Identification (CI) and Semantic Disambiguation (SD).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Linguistic Candidate Aggregation<br/>Lexicons + Loanwords + POS Filter"] --> B["Word-Level Four-Stage Curation and Expert Arbitration"]
    B --> C["Sentence-Level Minimal Contrast Generation and Naturalness Auditing"]
    C --> D["Four-Fold Representation and Multi-Level Probing Protocol"]
    D --> E["Cross-Model Diagnostic Evaluation<br/>CI Accuracy + SD Disambiguation F1 + Confusion Matrices"]

Key Designs

1. Word-Level Four-Stage Curation and Expert Arbitration: noisy automatic lexicons -> multi-source aggregation with AC1 double-blind arbitration -> 1,858 validated tri-category pairs
Automated extraction from raw bilingual dictionaries introduces archaic vocabulary, dialectal slang, proper names, and misleading semantic annotations. To build an uncompromised gold standard, candidate pairs for true cognates and false friends were aggregated from a comparative Semitic lexicon and the bilingual Palisra dictionary, while loanwords were sourced from the Academy of the Hebrew Language, Wiktionary, and specialized studies on Modern Hebrew and Modern Standard Arabic (MSA) loanword phonology. The dataset was restricted to content words (nouns, verbs, adjectives, and adverbs), filtering out function words to yield 1,016 true cognate, 446 false friend, and 724 loanword candidates. Two native-speaking linguistic experts (holding an MA in Semitic languages and a PhD in linguistics, respectively) evaluated the candidate pairs. A pair was validated as a true cognate if it shared at least one core meaning derived from a common root, as a false friend if meanings diverged completely, and as a loanword if both borrowed from the same non-Semitic source language. Inter-annotator reliability was verified on a random 10% sample using Gwet's AC1 statistic to avoid the well-known high-agreement/low-kappa paradox caused by skewed marginal distributions; raw agreement reached 85% with a Gwet's AC1 of 0.80 (compared to paradoxically deflated Cohen's \(\kappa\) and Krippendorff's \(\alpha\) of 0.32). The final gold set contains 1,858 word pairs: 858 true cognates, 364 false friends, and 636 loanwords.

2. Sentence-Level Minimal Contrast Generation and Naturalness Auditing: static word pairs miss dynamic polysemy -> prompt-based generation with adversarial perturbation -> 2,222 controlled disambiguation MCQs
Evaluating isolated word pairs allows models to rely on surface co-occurrence or dictionary memorization rather than contextual semantic understanding. To evaluate dynamic usage, the authors used Gemini-3.1-pro-preview to generate 3,716 contextual sentences (one Arabic and one Hebrew sentence per pair, each 8–20 words long). A human annotator vetted every sentence for native fluency and idiomatic precision: 3,549 sentences (95.5%) were retained without modification, 151 (4.1%) received minor lexical adjustments, and only 16 (0.4%) required substantial rewriting. For the Semantic Disambiguation (SD) task, true cognates and loanwords retain their validated sentences where both languages are semantically appropriate, making "Both sentences are appropriate" the correct option. For the 364 false-friend pairs, controlled adversarial perturbations were synthesized: each validated sentence was translated into the counterpart language, and the correct translation was swapped with its false-friend counterpart while strictly preserving morphosyntactic inflection and gender/number agreement. This produced grammatically flawless but semantically absurd sentences (e.g., swapping "bread" for "meat" in Hebrew, or "autumn" for "winter" in Arabic), yielding two directional minimal-contrast instances per false-friend pair and resulting in a total of 2,222 MCQ items.

3. Four-Fold Representation and Multi-Level Probing Protocol: intertwined orthographic-phonetic cues -> orthogonal script transformations -> isolating superficial character heuristics
Because Semitic orthographies omit short vowels and obscure phonological divergence under visually distinct scripts (Arabic vs. Hebrew alphabets), evaluating raw text alone cannot determine whether LLMs fail due to tokenization fragmentation, missing phonetic information, or defective semantic reasoning. The benchmark evaluates four parallel input representations: the raw undiacritized native script reflecting common digital corpora; fully diacritized text restoring short vowels and gemination via CAMeL Tools (for Arabic tashkil) and Phonikud (for Hebrew nikkud); phonetic transcriptions using the International Phonetic Alphabet (IPA) to project both languages onto a unified phonetic space; and Latin-based transliteration via Universal Romanization (Uroman). These representations are evaluated across both binary (TC vs. FF) and three-class (TC vs. FF vs. LW) Cognate Identification (CI) with and without sentence-level context, providing a rigorous test matrix that isolates surface orthographic heuristics from true semantic inference.

Evaluation Benchmark and Multi-Level Probing Protocol

All LLMs were tested under zero-shot prompting via LM Evaluation Harness and vLLM using greedy decoding (\(t=0\), maximum generation length of 20 tokens). API-based proprietary models used version-controlled snapshots (gpt-5.4-2026-03-05, gpt-4o-2024-11-20, deepseek-v4-pro, and qwen3.6-plus-2026-04-02). Choice positions were randomly permuted to eliminate position bias. - Cognate Identification (CI): Given \((W_{Ar}, W_{He})\)—or optionally sentence pairs \((S_{Ar}, S_{He})\) in the contextual setting—the model must select from TRUE_COGNATE, FALSE_FRIEND, or LOANWORD. Baselines include LingPy Sound-Class-Based Phonetic Alignment (SCA) on IPA strings, and cosine distances computed over aligned FastText word embeddings and multilingual Sentence-BERT (distiluse-base-multilingual-cased-v2) representations evaluated on 1,222 TC/FF pairs. - Semantic Disambiguation (SD): Given \((W_{Ar}, W_{He})\) and contextual sentences \((S_{Ar}, S_{He})\), the model must select whether (A) only the Arabic sentence is appropriate, (B) only the Hebrew sentence is appropriate, or (C) both sentences are appropriate.

Key Experimental Results

Main Results

The primary evaluation of 16 LLMs on three-class CI and SD under raw undiacritized native script is detailed below:

Model Category Evaluated Model CI Acc (Overall) CI AccTC (Cognates) CI AccFF (False Friends) CI AccLW (Loanwords) CI Macro F1 SD Acc (Overall) SD AccTC SD AccFF SD AccLW SD Macro F1
Proprietary GPT-4o 73.9 62.5 71.4 90.7 72.5 74.8 94.6 35.6 93.1 73.7
Proprietary GPT-5.4 81.9 85.9 47.0 96.4 76.9 79.0 88.7 57.7 90.2 78.9
Proprietary DeepSeek-v4-pro 71.1 91.6 1.1 83.5 55.8 28.5 17.0 50.1 19.3 24.0
Proprietary Qwen3.6-Plus 82.1 87.3 47.8 94.8 77.3 79.0 91.5 50.8 94.3 78.9
Open-source Qwen2.5-7B-Instruct 51.2 64.9 64.6 25.0 47.9 41.4 43.1 40.7 40.1 36.6
Open-source Qwen3-8B 51.4 75.2 52.7 18.6 45.1 60.0 87.8 6.3 84.1 58.6
Open-source Llama-3.1-8B-Instruct 44.7 91.7 12.1 0.0 27.1 34.1 31.9 30.2 41.3 29.8
Open-source Llama-3.3-70B-Instruct 76.9 87.9 23.1 92.9 67.5 57.2 80.1 8.1 82.7 56.5
Open-source GLM-4-9B-Chat 55.5 88.0 24.7 29.2 47.1 36.3 26.7 52.9 30.3 26.2
Open-source Gemma-2-9B-IT 64.6 80.7 23.9 66.2 57.1 31.0 38.1 13.9 41.2 22.5
Open-source Gemma-4-31B-IT 78.5 84.0 38.7 93.9 72.5 79.6 90.7 58.0 89.3 79.5
Arabic-centric Jais-Adapted-7B-Chat 41.9 62.4 2.2 36.9 31.0 32.9 33.7 30.6 34.6 18.9
Arabic-centric Jais-2-8B-Chat 46.0 96.2 8.0 0.0 25.2 32.3 33.8 29.7 33.3 16.4
Hebrew-capable DictaLM-2.0-Instruct 31.6 33.3 82.4 0.2 26.4 32.7 34.2 29.5 34.4 16.4
Multilingual Aya-23-8B 41.4 6.3 39.0 90.3 34.7 36.2 38.0 26.7 44.6 25.2
Open-weights GPT-OSS-120B 69.7 66.6 54.4 82.7 69.8 35.7 47.0 12.2 47.3 46.9

On binary cognate identification across the 1,222 true-cognate and false-friend (TC/FF) pairs, the comparison between baseline methods and the top LLMs is summarized below:

Method / Model Architecture / Mechanism AccTC (Cognates) AccFF (False Friends) Overall Acc Macro F1
LingPy (SCA) Phonetic Sound-Class Alignment 79.1 23.4 62.5 50.9
FastText Bilingual Aligned Word Embeddings 61.8 61.5 61.7 59.1
SBERT Multilingual Sentence Embeddings 54.9 61.5 56.9 55.0
Qwen3.6-Plus Proprietary LLM (Zero-shot) 94.1 35.7 76.7 66.3
Gemma-4-31B-IT Open-source LLM (Zero-shot) 88.9 43.1 75.3 68.4

Ablation Study: Impact of Input Representation & Context

Changes in accuracy (\(\Delta\) Acc) across alternative representations relative to undiacritized native input evaluated via two-sided approximate randomization tests (10,000 trials, * indicates \(p < 0.05\)):

Model CI Diacritized \(\Delta\)Acc CI IPA \(\Delta\)Acc CI Roman \(\Delta\)Acc SD Diacritized \(\Delta\)Acc SD IPA \(\Delta\)Acc SD Roman \(\Delta\)Acc
GPT-4o +1.7 -15.1* -32.1* -29.9* -45.7* -45.5*
GPT-5.4 -1.8* -11.3* -20.1* -45.9* -43.8* -54.7*
DeepSeek-v4-pro -3.0* -10.8* -17.5* -9.4* -10.5* +3.7*
Qwen3.6-Plus -2.7 -16.0 -30.8* -44.1 -42.7 -53.8*
Llama-3.3-70B-Instruct -7.1* -32.2* -33.0* -2.0* -5.4* -17.8*
Gemma-4-31B-IT -0.7 -11.9* -21.1* -43.1* -46.9* -54.7*
Aya-23-8B +0.2 -7.8* -7.5* +3.5* -3.5* +16.3*

Accuracy improvements (\(\Delta\) Acc and \(\Delta\) F1) gained by providing sentence-level context in Cognate Identification:

Model Undiacritized \(\Delta\)Acc Diacritized \(\Delta\)Acc IPA \(\Delta\)Acc Romanized \(\Delta\)Acc
GPT-4o +15.1* (+15.6* F1) +14.2* (+15.5* F1) +18.2* (+16.5* F1) +17.9* (+17.9* F1)
GPT-5.4 +10.1* (+13.5* F1) +9.9* (+13.8* F1) +16.1* (+17.1* F1) +18.4* (+20.5* F1)
Qwen3.6-Plus +7.5* (+10.8* F1) +9.0* (+13.4* F1) +10.4* (+9.0* F1) +14.3* (+12.8* F1)
Gemma-4-31B-IT +12.5* (+17.4* F1) +12.3* (+19.0* F1) +13.3* (+15.0* F1) +12.2* (+14.8* F1)
Qwen2.5-7B-Instruct +13.0* (+14.2* F1) +17.4* (+22.0* F1) +21.3* (+24.6* F1) +18.4* (+21.3* F1)
DictaLM-2.0-Instruct +11.1* (+3.4* F1) +12.8* (-7.7* F1) +19.9* (+15.1* F1) +6.4* (+9.5* F1)
Aya-23-8B -13.1* (-11.6* F1) -6.8* (-2.4 F1) -12.9* (-16.8* F1) -6.1* (-0.2 F1)

Key Findings

  • Severe over-reliance on surface-form similarity: While top LLMs achieve 84%–96% accuracy on true cognates (TC), their performance on false friends (FF) collapses dramatically (e.g., DeepSeek-v4-pro attains 91.6% on TC but only 1.1% on FF; Jais-2-8B-Chat reaches 96.2% on TC and only 8.0% on FF). Confusion matrices demonstrate that false friends are overwhelmingly misclassified as true cognates.
  • Dense semantic baselines outperform LLMs on false friends: On the binary TC/FF task, static embedding baselines (FastText and SBERT at 61.5% FF accuracy) substantially outperform Qwen3.6-Plus (35.7%) and Gemma-4-31B-IT (43.1%), indicating that dense embedding spaces are less prone to being misled by surface orthographic cues than auto-regressive next-token predictors.
  • Cross-task performance inversion: High false-friend accuracy in word-level CI does not translate to sentence-level SD. GPT-4o drops from 71.4% FF accuracy in CI to 35.6% in SD; DictaLM-2.0 drops from 82.4% to 29.5%. Models struggle to transfer static lexical recognition to semantic coherence verification in context.
  • Representation degradation: Shifting from original scripts to IPA or Romanization severely harms top models, with Llama-3.3-70B dropping 32.2 accuracy points on CI under IPA, and GPT-5.4, Qwen3.6-Plus, and Gemma-4-31B-IT dropping over 40–54 points on SD. Models are heavily anchored to standard script tokenizations and struggle with phonetic abstractions.
  • Asymmetric scaling and context benefits: Context consistently boosts CI across most models (+7 to +18 points), and model scaling improves CI up to 14B/31B parameters. In contrast, SD scaling exhibits erratic behavior (e.g., Qwen3 peaks at 8B and degrades at 14B), demonstrating that deep semantic disambiguation does not emerge reliably from parameter scaling alone.

Highlights & Insights

  • First systematic tri-category Semitic benchmark: By formalizing the structural boundaries between true cognates, false friends, and loanwords across Semitic root-and-pattern morphology, the benchmark provides an essential diagnostic tool for cross-lingual NLP.
  • Exposing superficial heuristics in multilingual LLMs: The findings demonstrate that multilingual language models exhibit a pseudo-multilingual shortcut—conflating orthographic and phonological similarity with semantic equivalence—which leads to hallucinations when encountering false friends.
  • Phonetic grounding as an open frontier: The dramatic performance collapse when transitioning from standard script to IPA or Romanization indicates that pre-training objectives must incorporate phonetic and morphological grounding rather than relying solely on subword BPE tokenization.

Limitations & Future Work

  • Language pair coverage: The study focuses strictly on Modern Standard Arabic and Modern Hebrew; findings may not fully generalize to spoken colloquial dialects (e.g., Levantine Arabic) or other Semitic branches (e.g., Amharic, Tigrinya).
  • Multiple-choice constraints: The benchmark utilizes closed-form multiple-choice questions to ensure rigorous control, which may not capture open-ended generation failures such as machine translation hallucinations or post-editing errors.
  • Synthetic context artifacts: Contextual sentences were initially generated with Gemini-3.1-pro-preview prior to expert review, which could introduce subtle stylistic or syntactic biases.
  • vs. RoBoCoP (Dinu et al., 2023): RoBoCoP pioneered cognate and borrowing classification in Romance languages without contextual sentences; SemCog Bench extends evaluation to non-concatenative Semitic morphology and introduces sentence-level semantic disambiguation.
  • vs. StingrayBench (Cahyawijaya et al., 2025): StingrayBench evaluated false-friend disambiguation across four language pairs without separating borrowings from inherited cognates; SemCog Bench models true cognates, false friends, and loanwords in a unified framework.
  • vs. Traditional Alignment (LingPy, FastText): The paper shows that while phonetic alignment methods (LingPy) mirror LLM biases toward surface similarity, dense embedding models (FastText, SBERT) provide more balanced discrimination between true and false cognates.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First systematic evaluation of LLMs on Arabic–Hebrew cognate and false-friend disambiguation under Semitic morphology.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 16 LLMs, four input representations, context variations, model scales, and permutation tests.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptional clarity in linguistic motivation, rigorous statistical reporting, and insightful error analysis.
  • Value: ⭐⭐⭐⭐⭐ Critical resource for cross-lingual NLP, machine translation robustness, and mitigating semantic hallucinations in multilingual models.