Skip to content

ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Andrian0s/any-lang-embed-eval
Area: Multilingual & Translation
Keywords: Multilingual Text Embeddings, Cross-Lingual Minimal Pairs, Abstract Meaning Representation, Fine-Grained Evaluation, Low-Resource Languages

TL;DR

By applying controlled semantic perturbations on the English side of parallel corpora via Abstract Meaning Representation (AMR) with bidirectional NLI filtering, ALEE establishes a dynamic minimal-pair diagnostic benchmark across 275+ languages, exposing systematic cross-lingual alignment blind spots in structural and lexical semantics.

Background & Motivation

Multilingual semantic text embeddings are foundational across modern information retrieval, cross-lingual semantic matching, and bitext mining pipelines. However, prevailing evaluation methodologies remain predominantly confined to static benchmarks and coarse-grained similarity judgments (e.g., X-STS, bitext mining tasks). These traditional benchmarks suffer from three chronic weaknesses: first, they are heavily biased toward high-resource languages and largely blind to low-resource languages and minor regional dialects; second, their static nature makes them inherently susceptible to test-set leakage and overfitting in massive web-scale pretraining; third, holistic similarity judgments are routinely conflated with surface lexical overlap and topical coincidence, failing to probe whether models truly distinguish polarity, thematic role relations, or lexical abstraction.

In monolingual English contexts, minimal-pair testing provides a powerful methodology to isolate targeted linguistic properties without lexical confounders. Yet extending minimal pairs across diverse languages has historically demanded prohibitive manual linguistic annotation, preventing scale across hundreds of languages. Furthermore, attempting to synthesize adversarial perturbations directly in target low-resource languages via black-box LLMs introduces severe grammatical corruption, unfaithful hallucinations, and uncontrollable quality degradation.

To resolve this bottleneck, the authors introduce a decoupled strategy: perform surgical semantic perturbations on the English pivotโ€”where formal semantic parsing and generation are matureโ€”and project diagnostic probes onto target languages via parallel sentences. Core idea: construct ALEE, a scalable cross-lingual minimal-pair evaluation framework that applies four rule-based AMR graph manipulations (polarity negation, role swaps, antonymy, and hypernymy) on English parallel text, validates semantic divergence through bidirectional NLI, and iteratively extends to paragraph contexts across 275+ languages.

Method

Overall Architecture

ALEE evaluates embedding spaces through triplet relative preference testing: given an unchanged target-language sentence \(target_{\text{orig}}\), paired with the original English counterpart \(en_{\text{orig}}\) and an edited English foil \(en_{\text{foil}}\), the model must assign higher cosine similarity to the true pair \((en_{\text{orig}}, target_{\text{orig}})\) than to the foil pair \((en_{\text{foil}}, target_{\text{orig}})\). The construction workflow comprises single-sentence AMR semantic graph editing, bidirectional natural language inference (NLI) divergence filtering, greedy iterative paragraph injection, and multilingual Triplet Accuracy (TACC) diagnosis.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Parallel Corpora<br/>(FLORES-200 / WMT24++ / BOUQuET)"] --> B["AMR Semantic Graph Manipulation<br/>(Polarity / Role Swap / Antonym / Hypernym)"]
    B --> C["Bidirectional NLI Divergence Validation<br/>Entailment probability filtering & false-positive pruning"]
    C --> D["Iterative Paragraph Insertion Pipeline<br/>Sentence-level edit validation & context-level rollback"]
    D --> E["Cross-Lingual Triplet Evaluation<br/>Macro-TACC scoring across languages & phenomena"]

Key Designs

1. AMR Semantic Graph Manipulation: Controlled Formal Semantic Operators

To ensure perturbations target precise grammatical and logical constructs rather than surface bag-of-words alterations, ALEE formalizes four orthogonal graph operations grounded in Abstract Meaning Representation: - Polarity Negation (PN): Attaches a :polarity - attribute to the matrix predicate, producing contradictory propositions (e.g., syntactic negation or morphological reversal) to probe truth-functional logic; - Role Swap (RS): Permutes the core argument edges :ARG0 (Agent) and :ARG1 (Patient) of a target predicate, altering the core event semantics ("who did what to whom") while leaving all lexical tokens intact, directly measuring structural sensitivity; - Antonym Replacement (AR): Queries WordNet lexical taxonomies to swap conceptual nodes with lexical opposites, evaluating cross-lingual sensitivity to polar lexical opposition without negation markers; - Hypernym Substitution (HS): Replaces specific concepts with their superordinates (e.g., penguin \(\to\) bird), breaking bidirectional entailment and testing whether representations track lexical hierarchical abstraction.

2. Bidirectional NLI Divergence Validation: Automated Foil Sanitization

Text surface generation from perturbed AMR graphs can occasionally neutralize the target semantic modification or yield unfaithful artifacts. To guarantee that every candidate foil introduces a definitive semantic shift, ALEE deploys an automated bidirectional NLI filtering filter: Given original sentence \(s\) and generated foil \(f\), the system evaluates \(P(\text{entailment} \mid s \to f)\) and \(P(\text{entailment} \mid f \to s)\). A candidate is rejected if either: (i) both directions are classified as entailment, or (ii) the entailment probability in either direction exceeds 0.8. This dual constraint eliminates false positives where the generated text remains semantically synonymous or weakly divergent from the source text.

3. Iterative Paragraph Insertion Pipeline: Scalable Contextual Perturbation

Because AMR parsers degrade over long document boundaries, ALEE develops an iterative first-success procedure for paragraph-level parallel text (such as WMT24++ and BOUQuET): The paragraph is first decomposed into ordered sentences; the algorithm attempts surgical AMR edits and sentence-level NLI validation starting from sentence \(i=0\); once an edit passes sentence validation, the candidate is re-embedded into the whole paragraph, and a full paragraph-level bidirectional NLI check is executed; if the document-level entailment fails to break, the edit is reverted and the procedure recursively advances to sentence \(i+1\). This guarantees that paragraph-level foils introduce a verified, localized semantic divergence embedded within natural running context.

4. Cross-Lingual Triplet Evaluation: Zero-Threshold Metric Protocol

Rather than relying on calibrated absolute cosine thresholds that vary arbitrarily across models and languages, ALEE evaluates discriminative alignment via Triplet Accuracy (TACC). For target vector \(\mathbf{e}_t\), original English vector \(\mathbf{e}_{en}\), and foil vector \(\mathbf{e}_{\text{foil}}\), accuracy over dataset \(D\) is defined as:

\[\text{TACC} = \frac{1}{|D|} \sum_{(en, \text{foil}, t) \in D} \mathbb{I}\Big(\cos(\mathbf{e}_{en}, \mathbf{e}_t) > \cos(\mathbf{e}_{\text{foil}}, \mathbf{e}_t)\Big)\]

Macro-TACC averages TACC uniformly across the four semantic perturbation categories, with random guessing baseline at 0.5. The metric directly measures a model's capacity to favor the true semantic equivalent over an adversarial, semantically shifted distractor.

Key Experimental Results

Main Results

Evaluations span three diverse parallel corpora: ALEE-F200 (professional translations across 200 languages), ALEE-MT61 (sentence- and paragraph-level texts across 61 languages including Romansh varieties), and ALEE-BQ275 (275 languages translated from multi-register human originals). Evaluated models include leading encoder models (LaBSE, Multilingual E5, BGE-M3, mGTE, Jina) and decoder-based models (Qwen3-Embedding).

Model Family / Model Name Parameter Count ALEE-F200 (Macro) ALEE-MT61 (Macro) ALEE-BQ275 (Macro) Three-Dataset Average
Bitext Encoder LaBSE ~470M 0.896 0.917 0.823 0.879
Instruct Encoder mE5-large-instruct ~560M 0.905 0.899 0.842 0.882
Dense Distilled BAAI_bge_m3 ~567M 0.887 0.885 0.782 0.851
Long-Context mGTE-base ~305M 0.818 0.819 0.760 0.799
Standard Encoder mE5-base ~278M 0.876 0.842 0.768 0.829
Decoder Embedding Qwen3-Embedding-8B 8B 0.842 0.839 0.723 0.801
Decoder Embedding Qwen3-Embedding-4B 4B 0.777 0.799 0.680 0.752
Decoder Embedding Qwen3-Embedding-0.6B 0.6B 0.696 0.735 0.644 0.691

Fine-Grained Linguistic Breakdown

Performance differs markedly across semantic manipulation categories. The table below reports fine-grained TACC across perturbation types for top-ranking representative models:

Model Dataset Polarity Neg. (PN) Role Swap (RS) Antonym Repl. (AR) Hypernym Sub. (HS) Macro-Avg
mE5-large-instruct ALEE-F200 0.945 0.882 0.904 0.889 0.905
ALEE-MT61 0.929 0.872 0.899 0.895 0.899
ALEE-BQ275 0.892 0.821 0.835 0.821 0.842
LaBSE ALEE-F200 0.896 0.903 0.884 0.903 0.896
ALEE-MT61 0.920 0.913 0.910 0.925 0.917
ALEE-BQ275 0.807 0.828 0.810 0.847 0.823
Qwen3-Emb-8B ALEE-F200 0.880 0.821 0.836 0.833 0.842
ALEE-MT61 0.872 0.810 0.847 0.826 0.839
ALEE-BQ275 0.778 0.707 0.710 0.697 0.723

Key Findings

  • Semantic Difficulty Hierarchy: Across all models and datasets, polarity negation (PN) is substantially easier than structural and lexical edits, outperforming other categories by 4โ€“7 percentage points. Conversely, thematic role swaps (RS), antonymy (AR), and hypernym substitutions (HS) cluster closely at lower performance levels, highlighting widespread deficiencies in distinguishing deep semantic argument structures and hierarchical taxonomies.
  • Context Length Dilution Effect: Even though only one sentence is edited per foil, paragraph-level evaluation degrades performance significantly. Average accuracy drops by 9.4% from single- to multi-sentence texts and 15.4% from single sentences to full paragraphs on ALEE-MT61 (and 16.0% on ALEE-BQ275), demonstrating that surrounding lexical overlap distracts models from detecting isolated semantic shifts.
  • Decoders Lag Behind Compact Encoders: Despite scaling up to 8B parameters, Qwen3-Embedding-8B (average 0.801) fails to match smaller sub-600M encoders such as LaBSE (0.879) and mE5-large-instruct (0.882), only matching 300M encoder baselines. Parameter scaling within autoregressive decoders does not automatically yield superior fine-grained cross-lingual semantic alignment.
  • Training Prevalence and Tokenization Gaps: Downstream TACC strongly correlates with language token volume in pretraining corpora (CC-100 Spearman \(\rho \approx 0.73\sim0.77\)) and contrastive fine-tuning data, with contrastive fine-tuning demonstrating an even stronger correlation. Furthermore, higher XLM-R subtoken fragmentation correlates negatively with accuracy (\(\rho = -0.516 \sim -0.786\)). In the Romansh dialect case study across six varieties, performance declines in strict accordance with speaker counts and official usage (Rumantsch Grischun > Vallader/Sursilvan > Puter > Surmiran > Sutsilvan), matching an almost perfect negative correlation with subtoken fragmentation (\(\rho = -0.986\)).

Highlights & Insights

  • Scalable English-Centric Decoupling: By placing formal semantic manipulation exclusively on the English side and anchoring to target languages via bitext, ALEE circumvents the bottleneck of absent AMR parsers and generative models in low-resource languages, scaling fine-grained semantic stress tests to 275+ languages at near-zero human annotation cost.
  • Formal Filtering Over Generative Noise: Integrating bidirectional NLI filtering with recursive paragraph rollback guarantees verified semantic divergence, achieving 98%+ grammaticality and 99%+ minimal-pair validity without relying on black-box LLM evaluation prompts.
  • Diagnostic Lens for Low-Resource Model Selection: Aggregate benchmark leaderboards obscure stark per-language disparities; ALEE's per-language diagnostics reveal substantial ranking shifts (e.g., mGTE surging to 5th place on Romansh idioms), providing practical guidance for choosing embeddings for specific low-resource deployment scenarios.

Limitations & Future Work

  • English-Centric Asymmetry: Semantic perturbations originate exclusively on English pivot text, inherently missing typological phenomena that cannot be projected through English syntax (e.g., grammatical gender, rich case systems, honorifics).
  • Selectional Preference Violations in Role Swaps: Exchanging Agent and Patient roles on occasion violates predicate selectional restrictions, generating semantically odd or implausible foils that test formal structural parsing rather than naturalistic usage.
  • Binary Ranking Metric Without Margin Calibration: Triplet Accuracy evaluates relative rank order rather than the absolute distance margin between positive and foil embeddings in representation space.
  • vs Sentence Smith (Li et al., 2025): Sentence Smith pioneered AMR-based dynamic minimal pairs for monolingual English; ALEE extends this paradigm cross-lingually to multi-sentence and paragraph scopes across 275+ languages.
  • vs MultiBLiMP (Jumelet et al., 2026): MultiBLiMP constructs morphosyntactic minimal pairs directly in target languages using specialized rule grammars, which is labor-intensive and covers fewer languages; ALEE scales to 275+ languages fully automatically via bitext projections.
  • vs MTEB (Muennighoff et al., 2023): MTEB evaluates coarse-grained downstream transfer tasks that are susceptible to lexical memorization; ALEE provides orthogonal, controlled minimal-pair diagnostics probing fine-grained semantic representations.

Rating

  • Novelty: โญโญโญโญโญ Decouples cross-lingual semantic perturbation via English AMR and parallel bitext projection with bidirectional NLI filtering.
  • Experimental Thoroughness: โญโญโญโญโญ Spans 275+ languages, 17 distinct embedding models, three parallel corpora, and granular linguistic/dialectal analyses.
  • Writing Quality: โญโญโญโญโญ Rigorous linguistic definitions, clear experimental pipelines, and insightful empirical analyses.
  • Value: โญโญโญโญโญ Establishes an essential diagnostic benchmark and toolkit for fine-grained multilingual embedding evaluation and low-resource model diagnosis.