Skip to content

SemBridge: Language Transfer in Sparse Encoders via Multilingual Semantic Bridge

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/seongtaehong/SemBridge
Area: Information Retrieval & RAG
Keywords: sparse encoders, cross-lingual transfer, tokenizer replacement, multilingual semantic bridge, Entmax sparse weighting

TL;DR

SemBridge proposes a novel embedding initialization framework for transferring English-centric sparse retrieval encoders to diverse target languages by leveraging multilingual dense models as semantic bridges and Entmax-based sparse truncation, reconstructing new target tokens as precise linear combinations of core source synonyms to achieve superior zero-shot retrieval and faster fine-tuning convergence.

Background & Motivation

Sparse encoders (such as SPLADE, SparTerm, and OpenSearch Sparse) have gained significant prominence in neural information retrieval by predicting term-level importance directly within the high-dimensional vocabulary space. This paradigm bridges the gap between semantic understanding and lexical precision: it overcomes lexical mismatch while preserving human-interpretable retrieval scores and full compatibility with standard Inverted Index infrastructures, eliminating the need for expensive approximate nearest neighbor (ANN) search indices. However, state-of-the-art sparse encoders are almost exclusively pre-trained on English corpora (e.g., SPLADE-v3, Granite-30M-Sparse). Unlike dense retrieval models whose latent representations occupy continuous shared vector spaces, a sparse encoder relies on its vocabulary space as the explicit output retrieval space.

This structural constraint poses a fundamental obstacle to cross-lingual language transfer. When examining the vocabulary distributions of existing sparse encoders, non-English tokens represent a vanishingly negligible proportionโ€”for instance, Granite-30M-Sparse contains a mere two Korean tokens, and SPLADE-v3 exhibits severe English bias. Without target-language tokens in the vocabulary, the model suffers from a structural lack of output dimensions through which to allocate term importance in non-English contexts. Standard target-language fine-tuning with naive random or Gaussian initialization fails because the unaligned initial embeddings introduce massive optimization instability and severe gradient shocks, destroying pre-trained retrieval discernment. Meanwhile, pre-training native sparse encoders from scratch for every target language requires immense computational resources, masked language model (MLM) pre-training, and scarce language-specific relevance pairs.

Previous tokenizer replacement and vocabulary initialization approaches fail to bridge this divide cleanly. Statistical baselines (mean, univariate/multivariate Gaussian sampling) fail to capture token-level semantic correspondence. Semantic initialization frameworks like FOCUS rely exclusively on the tiny set of overlapping surface tokens to interpolate unseen tokens, leading to substantial extrapolation errors across different scripts. Decomposition methods like OFA introduce noisy low-rank approximations that assign nonsensical source tokens to target words (e.g., mapping Arabic "home" to "israel" or Chinese "home" to "hush"). The angle of attack in this work is to exploit modern multilingual dense embeddings as a continuous semantic bridge, coupled with exact zero-truncating sparse activation to filter out long-tail vocabulary noise. Core idea: leverage a multilingual dense embedding space as a cross-lingual semantic bridge to compute token-level affinities, and use Entmax-based sparse weighting to reconstruct each unaligned target token as an exact, noise-free linear combination of a few core source-language synonyms.

Method

Overall Architecture

The objective of SemBridge is to replace the source tokenizer \(T_s\) and vocabulary \(V_s\) of a pre-trained sparse encoder \(M\) with a target-language tokenizer \(T_t\) and vocabulary \(V_t\), producing an initialized target embedding matrix \(E_t \in \mathbb{R}^{|V_t| \times d}\) without altering the model's underlying backbone architecture or retrieval dimensions. The workflow operates in two consecutive stages: first, language-agnostic shared tokens are identified and directly copied; second, remaining non-overlapping target tokens are mapped through an auxiliary multilingual dense embedding bridge model, followed by Entmax sparse filtering to perform weighted linear reconstruction over source embeddings.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Input: Source Vocab Vs & Target Vocab Vt"] --> B["Overlapping Token Transfer<br/>Identify numbers, symbols, and shared tokens Vo"]
    A --> C["Remaining Target Tokens<br/>Extract non-overlapping target tokens R"]
    B --> E["Target Embedding Matrix Et"]
    C --> D["Multilingual Semantic Bridge<br/>Dense model encodes cross-lingual representations"]
    D --> F["Sparse Semantic Weighting<br/>Entmax truncates tail noise to retain core synonyms"]
    F --> G["Weighted Linear Reconstruction<br/>Synthesize target embeddings via source linear combination"]
    G --> E
    E --> H["Downstream Retrieval Fine-Tuning / Zero-Shot Retrieval"]

Key Designs

1. Overlapping Token Transfer: Preserving universal semantic anchors without distortion

Although source and target tokenizers are trained on distinct language corpora, they inevitably share language-agnostic tokens, such as numerals, punctuation marks, universal symbols, and common code syntax. Randomly perturbing or re-estimating these shared tokens risks degrading the base model's structural and syntactic priors. SemBridge first identifies the overlapping token set \(V_o = V_t \cap V_s\) using case- and whitespace-normalized string matching. For every token \(x \in V_o\), its embedding vector is transferred via exact direct copying:

\[e_t^x = e_s^x, \quad \forall x \in V_o\]

This zero-cost copying mechanism preserves the expressiveness of universal tokens learned by the source encoder, establishing invariant semantic anchors in the embedding space before adapting language-specific vocabulary.

2. Multilingual Semantic Bridge: Projecting disparate lexical surfaces into a shared semantic continuum

The remaining uninitialized tokens \(R = V_t \setminus V_o\) comprise language-specific morphemes, subwords, and words that share no surface form with English source tokens. To overcome this fundamental lexical mismatch without requiring bilingual parallel lexicons, SemBridge utilizes a pre-trained multilingual dense embedding model \(B\) (defaulting to BGE-M3, with modular support for mGTE or Qwen3-Embedding). Both the full source vocabulary \(V_s\) and the remaining target tokens \(R\) are encoded into dense representations:

\[h_{x_s} = B(x_s), \quad \forall x_s \in V_s; \qquad h_{x_t} = B(x_t), \quad \forall x_t \in R\]

For each uninitialized target token \(x_t \in R\), cosine similarities are computed against all \(N = |V_s|\) source tokens, yielding an exhaustive cross-lingual similarity vector \(s_{x_t} = [\cos(h_{x_t}, h_{x_s^1}), \dots, \cos(h_{x_t}, h_{x_s^N})] \in \mathbb{R}^N\). Stacking these vectors across all \(M = |R|\) remaining tokens forms the complete semantic correspondence matrix \(S \in \mathbb{R}^{M \times N}\), quantifying the semantic affinity between the target vocabulary and the entire source vocabulary.

3. Sparse Semantic Weighting: Filtering long-tail distributional noise via exact Entmax truncation

A dense similarity vector across the entire source vocabulary is inherently plagued by spurious correlations, polysemous drift, and irrelevant subwords. Applying standard Softmax assigns strictly positive probabilities to all \(|V_s|\) tokens, diluting the resulting target embedding into a washed-out vector of averaged noise. Conversely, prior baselines like FOCUS or OFA often map target terms to bizarrely unrelated tokens (e.g., assigning "bray" or "israel" to the concept of "home"). SemBridge eliminates this semantic interference by applying the Entmax transformation with parameter \(\alpha > 1\):

\[p_{x_t} = \text{Entmax}_\alpha(s_{x_t}), \quad \text{s.t.} \sum_{i=1}^N p_{x_t}^i = 1\]

Entmax dynamically projects the probability distribution onto a simplex with exact zero boundaries, discarding the entire long-tail distribution. With \(\alpha=4\), the weight vector becomes extremely sparse, concentrating all mass onto only 2 to 4 direct semantic equivalents (e.g., mapping Korean "์ง‘", Chinese "ๅฎถ", and Arabic "ู…ู†ุฒู„" exclusively to "home", "house", "casa", and "homes"). The new target embedding \(e_t^{x_t}\) is then computed as the weighted linear combination:

\[e_t^{x_t} = \sum_{i=1}^N p_{x_t}^i e_s^{x_s^i}\]

This sparse synthesis guarantees that target tokens inherit the source model's calibrated retrieval capabilities without architectural alterations or semantic pollution.

Loss & Training

Following SemBridge initialization, the model can be directly deployed for zero-shot cross-lingual retrieval. When language-specific retrieval queries and passages are available, the model is fine-tuned using a multi-task objective combining InfoNCE contrastive loss over in-batch negatives and a FLOPs regularization loss enforcing sparse activation:

\[\mathcal{L} = \mathcal{L}_{\text{InfoNCE}} + \lambda_q \mathcal{L}_{\text{FLOPs}}(Q) + \lambda_d \mathcal{L}_{\text{FLOPs}}(D)\]

Here, the document and query FLOPs regularization weights are set to \(\lambda_d = 1 \times 10^{-4}\) and \(\lambda_q = 3 \times 10^{-4}\), respectively. Models are fine-tuned across 4 NVIDIA A100 GPUs for 1 epoch with batch size 64, sequence length 512, bf16 precision, and the AdamW optimizer (learning rate \(2 \times 10^{-5}\) with a 5% linear warmup).

Key Experimental Results

Main Results

The framework was evaluated across four representative sparse encoder architectures (Splade-v3, Splade_PP_en_v1, OpenSearch-Sparse-v1, Granite-30M-Sparse) across five diverse languages (Arabic, Chinese, Hindi, Korean, Russian) on both WebFAQ and MIRACL benchmarks. The table below presents the fine-tuned retrieval performance (nDCG@10) across all languages and architectures (directly extracted from Table 2 of the paper):

Backbone Model Evaluation Benchmark Base (No Transfer) Random Mean Univar. Multivar. FOCUS OFA SemBridge (Ours)
Splade-v3 WebFAQ (Avg.) 0.564 0.395 0.205 0.644 0.644 0.624 0.640 0.697
Splade-v3 MIRACL (Avg.) 0.178 0.080 0.011 0.294 0.307 0.210 0.292 0.319
Splade-PP WebFAQ (Avg.) 0.567 0.383 0.213 0.644 0.641 0.614 0.635 0.684
Splade-PP MIRACL (Avg.) 0.169 0.057 0.009 0.290 0.279 0.219 0.260 0.298
OpenSearch-v1 WebFAQ (Avg.) 0.562 0.143 0.213 0.639 0.639 0.619 0.637 0.683
OpenSearch-v1 MIRACL (Avg.) 0.164 0.009 0.012 0.291 0.284 0.200 0.274 0.296
Granite-30M WebFAQ (Avg.) 0.592 0.297 0.167 0.000 0.521 0.577 0.534 0.666
Granite-30M MIRACL (Avg.) 0.160 0.028 0.008 0.000 0.138 0.179 0.142 0.245

Ablation Study

The authors investigated the impact of bridge model selection, Entmax sparsity hyperparameter \(\alpha\), and computational efficiency tradeoffs. The table below presents the zero-shot retrieval performance (nDCG@10) on Splade-v3 under various bridge models (from Table 5) and sparsity configurations:

Configuration / Bridge Model Bridge Architecture WebFAQ (Avg. nDCG@10) MIRACL (Avg. nDCG@10) Empirical Behavior
FOCUS (Baseline) Static auxiliary embeddings 0.176 0.035 Confined to overlapping tokens; poor generalization
OFA (Baseline) Matrix Factorization + XLM-R 0.351 0.110 Prone to noisy coordinates and semantic drift
SemBridge (Default) BGE-M3 (\(\alpha=4\)) 0.422 0.179 Top zero-shot accuracy; filters out all tail noise
SemBridge w/ mGTE mGTE-multilingual 0.378 0.148 Robust semantic alignment across all scripts
SemBridge w/ Qwen3 Qwen3-Embedding-0.6B 0.306 0.114 Outperforms all external baselines consistently
SemBridge w/ MiniLM paraphrase-multilingual 0.303 0.092 Lightweight bridge remains competitive
SemBridge (\(\alpha=1\), Softmax) BGE-M3 ~0.150 ~0.040 Dense weighting injects overwhelming noise
SemBridge (\(\alpha=2\), Sparsemax) BGE-M3 ~0.350 ~0.130 Substantial improvement, but retains minor distractor words

Key Findings

  • Initialization quality establishes the upper bound for downstream fine-tuning: Even after an identical fine-tuning regimen, models initialized with naive statistical methods or noisy transfer remain severely bottlenecked (e.g., Granite-30M fails completely under univariate Gaussian initialization with 0.000 nDCG@10). SemBridge starts with drastically lower initial training loss and converges rapidly within 0.1 epochs.
  • Entmax truncation is essential for cross-lingual purity: Setting \(\alpha=1\) (Softmax) yields degraded retrieval performance due to the accumulation of thousands of minute, irrelevant token weights. Increasing \(\alpha\) to 4 isolates pure synonyms (e.g., English "house", "home", "casa"), preventing out-of-context gradient interference.
  • Superior Pareto frontier between latency and accuracy: While FOCUS leads to excessive token activation and ballooning FLOPS during inference, SemBridge preserves tight, compact sparsity patterns, maintaining sub-millisecond querying efficiency over standard Inverted Index setups.

Highlights & Insights

  • Bridging continuous dense representations with discrete sparse retrieval: Sparse retrieval adaptation has traditionally struggled due to the absence of a shared continuous latent space. SemBridge elegantly decomposes this problem by using a dense model as an intermediary continuous semantic bridge, and then re-discretizing the mapping through Entmax sparse projection.
  • Zero parameter overhead and pure architectural compatibility: SemBridge operates purely at initialization time. It requires no adapters, no additional projection matrices, and zero architectural modifications, allowing seamless plug-and-play deployment within existing retrieval microservices.
  • Broad utility for language model vocabulary adaptation: The concept of using dense semantic bridges with sparse non-linear weighting can be directly extended to multilingual vocabulary expansion for LLMs, cross-lingual speech encoders, and discrete codebook transfer in multimodal generative models.

Limitations & Future Work

  • Evaluation limited to moderately resourced typologies: While the five evaluated languages span diverse scripts (Arabic, Chinese, Devanagari, Hangul, Cyrillic), they possess relatively mature monolingual tokenizers and pre-trained corpora. Extreme low-resource languages with complex polysynthetic or agglutinative morphologies remain underexplored.
  • Uniform global sparsity threshold: Currently, \(\alpha=4\) is applied statically across all vocabulary tokens. However, closed-class function words, named entities, and highly abstract polysemous nouns possess inherently different semantic breadths. Developing an adaptive, entropy-driven \(\alpha\) schedule represents a promising future research direction.
  • vs FOCUS (Dobler and de Melo, 2023): FOCUS computes embeddings by calculating similarities only against overlapping tokens in an auxiliary static space. When transferring across dissimilar writing scripts where lexical overlap is nearly empty, FOCUS breaks down and generates bloated FLOPS; SemBridge leverages the entire source vocabulary via deep multilingual dense embeddings, achieving vastly superior precision.
  • vs OFA (Liu et al., 2024): OFA factorizes embeddings into primitive bases and coordinates, which often loses fine-grained lexical nuances and introduces phantom associations; SemBridge retains the exact geometrical structure of the source embedding space through direct sparse linear interpolation.

Rating

  • Novelty: โญโญโญโญโ˜† Elegant combination of dense semantic bridging and Entmax sparse truncation specifically tailored to the discrete nature of sparse retrieval encoders.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across 4 sparse encoders, 5 languages, 2 standard benchmarks, with in-depth zero-shot, fine-tuned, loss trajectory, efficiency, and qualitative analyses.
  • Writing Quality: โญโญโญโญโญ Rigorously formulated, logically structured, and clearly motivated with empirical vocabulary distribution statistics.
  • Value: โญโญโญโญโญ Provides an immediate, zero-training-cost engineering solution for deploying high-performance multilingual sparse search engines in industry.