Skip to content

MIMO: Multilingual Information Retrieval via Monolingual Objectives

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/yjoonjang/MIMO
Area: Information Retrieval & RAG
Keywords: Multilingual Information Retrieval, Knowledge Distillation, Contrastive Learning, Semantic Anchor, Alignment and Uniformity

TL;DR

MIMO is a two-stage training framework for Multilingual Information Retrieval (MLIR) that leverages a high-performing teacher model's English semantic space as an absolute anchor, warming up cross-lingual alignment via distillation and jointly optimizing distillation with contrastive learning to break the alignment-uniformity trade-off and substantially outperform prior baselines.

Background & Motivation

As dense retrieval systems expand into global web applications, real-world search environments are inherently multilingual: a user may formulate a query in English, yet the most pertinent and high-quality passages in a heterogeneous corpus might reside in Korean, Chinese, or Spanish. This paradigm, termed Multilingual Information Retrieval (MLIR), requires a model to identify relevant answers across language boundaries from a mixed-language corpus. However, mainstream multilingual retrieval architectures and evaluation benchmarks are overwhelmingly curated for Multi-Monolingual IR, in which the query and candidate documents are strictly matched in the same language. When evaluated against open mixed-language corpora, existing dense encoders suffer severe performance degradation.

The root cause of this failure lies in the geometry of standard contrastive learning. When models are optimized with conventional InfoNCE objectives, embeddings tend to cluster by superficial linguistic forms rather than shared semantics, fracturing the latent space into isolated language islands. While recent cross-lingual approaches attempt to bridge this divide through cross-lingual contrastive loss (XLCO) or symmetric probability distribution alignment (LaKDA), they remain confined to relative alignment within the student model. Without a stable absolute reference frame, forcing relative cross-lingual alignment inevitably induces representation collapse: optimizing alignment degrades embedding uniformity, eliminating the negative discrimination capability strictly necessary for web-scale passage retrieval.

This paper tackles the challenge by recognizing that high-performing monolingual (specifically English) foundation models already possess a rich, highly uniform, and semantically mature geometric space that can serve as an immutable reference. Core idea: decouple multilingual dense retrieval training into a two-stage pipeline anchored on a frozen teacher's English semantic space, first establishing foundational cross-lingual alignment through sentence-level distillation, and then jointly optimizing monolingual objective distillation with cross-lingual contrastive learning to find an optimal operating point that balances alignment and uniformity.

Method

Overall Architecture

MIMO adopts a teacher-student knowledge distillation architecture, designating the state-of-the-art Qwen3-Embedding-8B model as a frozen teacher encoder \(E_t\) and a compact multilingual encoder (e.g., xlm-roberta-large or mmBERT-base) as the student encoder \(E_s\). The defining principle is that the teacher model processes exclusively English inputs throughout training, acting as an invariant semantic anchor. In contrast, the student model receives semantically parallel inputs in arbitrary languages and maps its output into the teacher's dimensional space via a trainable linear projection layer \(\psi(z) = Wz + b\). Training proceeds across two sequential stages: Stage 1 performs a cross-lingual distillation warmup on large-scale parallel sentence corpora to initialize the multilingual alignment space. Stage 2 executes joint optimization on multilingual passage data, combining monolingual query/passage distillation with cross-lingual contrastive learning to imbue the student with fine-grained retrieval discrimination. At inference time, the projection layer \(\psi\) is discarded, and the bare student encoder directly encodes multilingual queries and passages.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    subgraph S1["Stage 1: Cross-Lingual Distillation Warmup"]
        direction TB
        A1["Multilingual Parallel Pairs<br/>(t_en, t_XX) from OPUS"] --> A2["Teacher E_t encodes t_en<br/>Student E_s encodes t_XX"]
        A2 --> A3["Cosine Distance Distillation Loss<br/>L_Stage1 = 1 - cos(ฯˆ(E_s(t_XX)), E_t(t_en))"]
    end
    subgraph S2["Stage 2: Joint Optimization for MLIR"]
        direction TB
        B1["Cross-Lingual Retrieval Quadruplet<br/>(q_en, p_en, q_XX, p_YY) from mMARCO"] --> B2["Monolingual Distillation Branch<br/>Distill q_XX to q_en and p_YY to p_en"]
        B1 --> B3["Cross-Lingual Contrastive Branch<br/>InfoNCE between q_XX and p_YY"]
        B2 & B3 --> B4["Joint Objective Weighted Sum<br/>L_Stage2 = ฮป L_XLCO + (1-ฮป) L_Distill (ฮป=0.2)"]
    end
    S1 --> S2
    S2 --> C["Inference Deployment<br/>Discard projection ฯˆ; student encodes mixed corpus"]

Key Designs

1. Cross-Lingual Distillation Warmup: Anchoring Multilingual Embeddings to English Semantic Space

Directly initiating contrastive fine-tuning on multilingual corpora without structural pre-alignment forces the model into sub-optimal local minima where language clustering dominates semantic similarity. In Stage 1, MIMO constructs a curated warmup dataset of 5.65 million English-multilingual translation sentence pairs \((t_{\text{en}}, t_{\text{XX}})\) across 14 languages from the OPUS parallel corpus. The teacher model \(E_t\) encodes the English text \(t_{\text{en}}\) into a high-dimensional space (\(m = 3584\)), while the student model \(E_s\) encodes the foreign text \(t_{\text{XX}}\) (\(n = 1024\) for xlm-roberta-large). A linear projection layer \(\psi : \mathbb{R}^n \to \mathbb{R}^m\) bridges the dimension disparity. The warmup objective minimizes the cosine distance between the projected student representation and the frozen teacher representation:

\[L_{\text{Stage1}} = 1 - \cos(\psi(E_s(t_{\text{XX}})), E_t(t_{\text{en}}))\]

This stage operates strictly without complex negative mining or contrastive pairs, rapidly establishing an initial shared geometry where multilingual semantics align with the mature English anchor.

2. Joint Optimization for MLIR: Decoupling Discrimination and Alignment Objectives

While pure distillation transfers cross-lingual alignment, it does not instill the discriminative capacity necessary to rank the single gold document above millions of distracting candidates. Conversely, standard contrastive training degrades embedding uniformity when forced to align languages internally. Stage 2 resolves this tension by jointly optimizing distillation and contrastive objectives using multilingual quadruplets \(\{q_{\text{en}}, p_{\text{en}}, q_{\text{XX}}, p_{\text{YY}}\}\) from mMARCO. The distillation loss forces foreign queries \(q_{\text{XX}}\) and passages \(p_{\text{YY}}\) to independently reach the teacher's English query and passage targets:

\[L_{\text{Distill}} = [1 - \cos(\psi(E_s(q_{\text{XX}})), E_t(q_{\text{en}}))] + [1 - \cos(\psi(E_s(p_{\text{YY}})), E_t(p_{\text{en}}))]\]

Simultaneously, the cross-lingual contrastive loss \(L_{\text{XLCO}}\) acts on arbitrary foreign query-document pairs \((q_{\text{XX}}, p_{\text{YY}})\) with temperature \(\tau\) and large in-batch negatives (\(B = 2048\)):

\[L_{\text{XLCO}} = -\log \frac{\exp(s(q_{\text{XX}}, p_{\text{YY}})/\tau)}{\sum_{j=1}^B \exp(s(q_{\text{XX}}, p_j)/\tau)}\]

The joint loss is balanced via hyperparameter \(\lambda \in [0, 1]\):

\[L_{\text{Stage2}} = \lambda \cdot L_{\text{XLCO}} + (1 - \lambda) \cdot L_{\text{Distill}}\]

Empirical optimization identifies \(\lambda = 0.2\) as the ideal operating point, where distillation maintains cross-lingual alignment and contrastive learning supplies spatial dispersion.

3. Cross-Lingual Alignment-Uniformity Analysis: Geometric Diagnosis of Representation Collapse

To diagnose the underlying mechanics governing multilingual retrieval, the authors adapt the Alignment-Uniformity framework to the MLIR setting using the 14-language parallel Belebele corpus. Cross-lingual alignment \(\ell_{\text{align}}\) measures the expected mean squared distance between semantically identical documents translated into different languages, where lower values denote tighter cross-lingual alignment. Spatial uniformity \(\ell_{\text{uniform}}\) measures the logarithm of the pairwise Gaussian potential across randomly sampled documents from the mixed 14-language collection, where lower values indicate an even spread across the unit hypersphere. The empirical analysis demonstrates that baseline methods operate on flawed extremes: InfoNCE achieves high uniformity but catastrophically poor alignment (\(\ell_{\text{align}} \approx 0.83\)), whereas XLCO and LaKDA improve alignment (\(\approx 0.19\)) at the cost of severe uniformity collapse (\(\approx -3.37\)). MIMO successfully breaks this deadlock, reaching an optimal coordinate (\(\ell_{\text{align}} \approx 0.15, \ell_{\text{uniform}} \approx -3.57\)) that directly translates to superior retrieval accuracy.

Loss & Training

Stage 1 is optimized using AdamW with learning rate \(1 \times 10^{-4}\), batch size 128, sequence length 256, and 1 epoch. Stage 2 uses learning rate \(2 \times 10^{-5}\), sequence length 512, and 1 epoch. To support the global batch size of 2048 required for dense contrastive learning within limited hardware (2 \(\times\) H100 GPUs), Stage 2 implements gradient caching with mini-batches of 32 samples. The InfoNCE scale parameter is set to \(1/\tau = 20.0\). The teacher model Qwen3-Embedding-8B is loaded in bfloat16 with Flash Attention 2 and frozen; an instruction prompt is applied only to queries during teacher encoding in Stage 2. The student encoder uses mean pooling and requires no prompt during training or inference.

Key Experimental Results

Main Results

MLIR performance is evaluated across six diverse benchmarks (Belebele, MLQA, XQuAD, MultiEuP, NeuCLIR22, NeuCLIR23) using nDCG@20 on pooled multilingual corpora. MIMO is compared against representative loss baselines across two student backbones:

Model Method Belebele (14) MLQA (7) XQuAD (8) MultiEuP (6) NeuCLIR22 (2) NeuCLIR23 (2) AVG
xlm-roberta-large InfoNCE 35.86 24.86 42.47 17.23 36.83 31.20 31.41
xlm-roberta-large XLCO 74.02 41.95 77.50 41.02 46.99 37.62 53.18
xlm-roberta-large LaKDA 73.90 41.99 77.54 41.09 46.20 36.96 52.95
xlm-roberta-large MIMO (Ours) 84.18 49.83 86.05 47.06 55.40 41.62 60.69
mmBERT-base InfoNCE 31.45 25.86 39.92 15.61 40.19 32.12 30.86
mmBERT-base XLCO 70.70 40.21 76.00 35.97 45.28 39.34 51.25
mmBERT-base LaKDA 70.79 40.27 75.94 36.08 44.84 39.93 51.31
mmBERT-base MIMO (Ours) 80.27 46.06 83.18 45.11 50.51 43.37 58.08

Ablation Study

An ablation analysis on xlm-roberta-large isolates the contributions of the individual training stages across all six evaluation benchmarks (nDCG@20):

Config Belebele MLQA XQuAD MultiEuP NeuCLIR22 NeuCLIR23 AVG (nDCG@20) Note
Stage1 + Stage2 (Full model) 84.39 49.86 86.41 46.99 55.40 41.62 60.78 Complete pipeline with warmup and joint training
Stage1 Only 83.22 45.26 87.02 40.70 48.50 39.88 57.43 Distillation only; lacks retrieval discrimination
Stage2 Only 73.29 41.10 76.90 40.24 46.83 38.29 52.77 Direct joint training without alignment warmup

Additionally, varying the Stage 2 loss weighting hyperparameter reveals that pure distillation (\(\lambda = 0.0\)) attains 57.43, peaking sharply at \(\lambda = 0.2\) (60.78), and degrading as contrastive learning becomes dominant (\(\lambda = 1.0\) drops to ~54).

Key Findings

  • Monolingual distillation drives cross-lingual alignment: Stage 1 distillation warmup alone achieves an average score of 57.43, directly outperforming fully fine-tuned cross-lingual contrastive baselines XLCO (53.18) and LaKDA (52.95). Anchoring to a mature English semantic space is substantially more potent than training student representations from scratch with relative contrastive objectives.
  • Warmup is non-negotiable for joint optimization: Skipping the Stage 1 warmup and training Stage 2 directly causes average performance to plummet to 52.77โ€”even lower than the Stage 1 warmup alone. Without prior alignment, contrastive fine-tuning struggles to organize the multilingual space.
  • Elimination of language ranking bias: On the Multilingual Rank-based Consistency metric (MRC@10) across 14 languages, InfoNCE exhibits extreme directional bias depending on query language. MIMO delivers consistent, high Spearman rank correlation across all query languages, tracing a regular polygon on radar visualizations.
  • Cross-lingual gains generalize to monolingual search: On the standard MIRACL 10-language Multi-Monolingual benchmark, MIMO achieves an average nDCG@10 of 44.16, significantly surpassing LaKDA (40.63) and InfoNCE (38.32), proving that distilling structured teacher representations improves in-language query-document matching.

Highlights & Insights

  • Solving multilingual retrieval via monolingual anchors: Rather than training on difficult multilingual teachers or relying solely on noisy cross-lingual pairs, MIMO exploits the structural maturity of a frozen English embedding space, drastically simplifying the optimization landscape.
  • Theoretical grounding through Alignment-Uniformity: Extending hyperspherical representation metrics to cross-lingual retrieval provides a rigorous explanation for why prior methods failed (uniformity collapse in relative alignment) and why joint optimization succeeds.
  • High performance with minimal engineering overhead: Operating on public corpora without complex hard-negative mining pipelines, MIMO's 0.6B student model surpasses multilingual-e5-large (38.93) and bge-m3 (59.71), demonstrating that loss formulation can rival massive industrial engineering.

Limitations & Future Work

  • Language scope limited to medium/high-resource families: Training and validation cover the 14 languages of mMARCO; validating whether this anchor-distillation paradigm transfers reliably to hundreds of extremely low-resource languages without rich parallel data remains an open question.
  • Reliance on strict parallel corpora: Both Stage 1 sentence pairs and Stage 2 query-passage sets depend on parallel translation resources, which may act as a bottleneck when scaling to rare dialects.
  • Parallel benchmark bias: Existing MLIR benchmarks constructed from parallel corpora (Belebele, XQuAD) naturally favor exact translation-level alignment; real-world open web retrieval involves more diverse topics and partial relevance.
  • vs InfoNCE (Karpukhin et al., 2020): InfoNCE optimizes solely over monolingual query-document pairs, causing embedding vectors to cluster into language islands and failing completely in mixed-language corpora (31.41 vs. 60.69 nDCG@20).
  • vs XLCO / LaKDA (Chi et al., 2021; Yang et al., 2024): XLCO and LaKDA enforce relative cross-lingual alignment internally within the student, which severely damages spatial uniformity; MIMO introduces an immutable external anchor that maintains both alignment and uniformity.
  • vs Multilingual Sentence-BERT (Reimers and Gurevych, 2020): While m-SBERT explored monolingual teacher distillation, it was limited to MSE matching on STS and monolingual tasks; MIMO establishes a two-stage framework integrating distillation with large-batch contrastive learning for mixed-corpus MLIR.

Rating

  • Novelty: โญโญโญโญโ˜† Anchoring multilingual retrieval to an English monolingual teacher space provides a clean, elegant paradigm, reinforced by insightful cross-lingual Alignment-Uniformity analysis.
  • Experimental Thoroughness: โญโญโญโญโญ Evaluated across six MLIR benchmarks, MIRACL multi-monolingual search, detailed ablations, MRC consistency analysis, and hyperspherical geometric metrics.
  • Writing Quality: โญโญโญโญโญ Lucid problem formulation, clear mathematical exposition, and rigorous diagnostic visualizations.
  • Value: โญโญโญโญโญ Offers a practical, accessible, and high-performing training strategy for global dense retrieval systems.