Skip to content

Cross-lingual Representation Learning via Centroid Intervention Fusion

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/VRCMF/CIF.git
Area: LLM Reasoning
Keywords: Cross-Lingual Transfer, Inference-Time Intervention, Centroid Fusion, Representation Alignment, Low-Resource Adaptation

TL;DR

To overcome the parameter explosion and knowledge fragmentation inherent in pairwise cross-lingual activation intervention, this paper proposes Centroid Intervention Fusion (CIF), which reformulates projection matrices as residual perturbations around identity and applies centroid-guided trimmed aggregation to yield a unified, language-shared operator, achieving consistent improvements across multilingual reasoning and generation with frozen LLM parameters.

Background & Motivation

Large language models (LLMs) have achieved remarkable success across diverse natural language processing tasks, yet their multilingual proficiency remains starkly uneven, with low-resource and typologically distant languages lagging substantially behind English. Conventional remedies rely primarily on continued pretraining or multilingual supervised fine-tuning (SFT), both of which demand substantial computational resources and vast parallel corpora. Recently, inference-time activation intervention has emerged as an attractive, parameter-efficient paradigm; representative techniques such as INCLINE learn linear projection matrices from parallel sentence pairs to map hidden representations of non-English inputs into the higher-performing English representation space during the model's forward pass, without modifying any underlying model weights.

However, existing cross-lingual intervention frameworks predominantly follow an independent pairwise design that suffers from severe scalability and knowledge-sharing bottlenecks. When deploying a model across \(N\) target languages across \(L\) transformer layers, pairwise intervention mandates storing and executing \(N \times L\) separate transformation matrices at test time, resulting in memory overhead that scales linearly with the number of supported languages. Crucially, learning isolated transformations for each language pair completely prevents models from sharing geometric alignment knowledge across languages; noisy, scarce parallel data in low-resource pairs frequently causes isolated projection operators to overfit or deviate, leading to brittle cross-lingual transfer.

Empirical studies on multilingual representations reveal that pre-trained language models organize diverse languages within a partially shared geometric subspace, strongly indicating that linear projections aligning various languages toward a common anchor should share an underlying transformation structure. This paper leverages this foundational property by reformulating each language-specific projection matrix as a localized perturbation around the identity transformation, estimating a robust global centroid across languages, and filtering out conflicting outlier directions via trimmed aggregation. Core idea: reformulate language-specific projections as residual perturbations around the identity matrix and fuse them into a single, language-shared intervention operator via centroid-guided trimmed aggregation, achieving robust cross-lingual representation alignment with \(O(L)\) deployment complexity.

Method

Overall Architecture

Centroid Intervention Fusion (CIF) proceeds through three interconnected phases: first, offline ordinary least squares (OLS) is applied on sentence representations to fit linear projection matrices from each source language to the anchor language (English); second, these matrices are converted into residual perturbations relative to the identity transformation, whereupon a geometric centroid is calculated and outlier languages are pruned via trimmed aggregation to construct a unified intervention operator; third, during test-time inference, the LLM backbone parameters remain completely frozen, and the single layer-wise intervention operator is applied to intermediate hidden states during forward computation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Multilingual Parallel Corpus & Frozen LLM"] --> B["Stage 1: Language-Specific Projection Fitting<br/>OLS on last-token hidden states"]
    B --> C["Stage 2: Residual Perturbation Parameterization<br/>Extract intervention offset from identity"]
    C --> D["Stage 3: Centroid-Guided Trimmed Aggregation<br/>Compute geometric centroid & prune outlier languages"]
    D --> E["Stage 4: Unified Shared Operator Construction<br/>Scale by intervention strength & restore identity"]
    E --> F["Inference-Time Activation Transformation<br/>Single shared operator modifies hidden states"]

Key Designs

1. Language-Specific Projection Fitting: Aligning Individual Language Geometries to Anchor Space To bridge the representation discrepancies between source languages and high-resource anchor representations, CIF begins by learning an offline mapping for each source language. Let \(\mathcal{S}\) denote the set of source languages and \(a\) denote the anchor language (e.g., English). For each source language \(s \in \mathcal{S}\), a parallel corpus \(\mathcal{D}_s = \{(x_i^s, x_i^a)\}_{i=1}^{N_s}\) of \(N_s\) sentence pairs is sampled. Sentences are fed through the frozen LLM backbone to extract hidden representations of the last non-padding token at transformer layer \(k\): \(h_i^{s,k}, h_i^{a,k} \in \mathbb{R}^d\). Drawing upon the empirical observation that multilingual representations are approximately linear, an ordinary least-squares objective estimates the optimal linear projection \(W_{s,k}^*\): $\(W_{s,k}^* = \arg\min_{W_{s,k} \in \mathbb{R}^{d \times d}} \frac{1}{N_s} \sum_{i=1}^{N_s} \|W_{s,k} h_i^{s,k} - h_i^{a,k}\|_2^2\)$ This formulation encapsulates the specific coordinate rotation and scaling necessary to steer language \(s\) toward anchor representation space.

2. Residual Perturbation Parameterization: Decoupling Baseline State from Intervention Shifts Directly averaging the full projection matrices \(W_{s,k}^*\) introduces severe interference because language-specific biases corrupt the absolute matrix magnitudes. Because pretrained multilingual models already possess partial intrinsic alignment, cross-lingual intervention fundamentally acts as a localized steering adjustment. CIF therefore reparameterizes each projection matrix as a perturbation offset relative to the identity matrix \(I_d\): $\(\Delta W_{s,k} = W_{s,k}^* - I_d\)$ Here, \(I_d\) corresponds to native, non-intervened forward computation, while \(\Delta W_{s,k}\) strictly captures the directional delta induced by language \(s\). This translation centers all language updates into a common Euclidean tangent space around the origin, permitting well-conditioned linear combination.

3. Centroid-Guided Trimmed Aggregation: Mitigating Negative Interference and Outlier Noise Because parallel data quality and typological divergence vary drastically across languages, certain learned perturbations deviate sharply from the central distribution, causing destructive cross-lingual interference if naively averaged. Inspired by Byzantine-robust distributed estimation, CIF introduces centroid-guided trimming. It first computes an initial centroid across all language perturbations: $\(\Delta \bar{W}_k = \frac{1}{|\mathcal{S}|} \sum_{s \in \mathcal{S}} \Delta W_{s,k}\)$ The Euclidean distance of each language's perturbation from this centroid is computed using the matrix Frobenius norm: \(d_{s,k} = \|\Delta W_{s,k} - \Delta \bar{W}_k\|_F\). Given a trimming parameter \(\text{drop}\), CIF discards the \(\text{drop}\) most distant matrices and computes the trimmed mean over the remaining retained set \(\mathcal{S}_{\text{keep}}^k\): $\(\Delta W_{\text{CIF},k} = \frac{1}{|\mathcal{S}_{\text{keep}}^k|} \sum_{s \in \mathcal{S}_{\text{keep}}^k} \Delta W_{s,k}\)$ Spectral decomposition confirms that the perturbation energy across languages is concentrated within a low-dimensional subspace; trimming effectively discards high-variance, conflicting noise while retaining dominant shared transformation signals.

4. Unified Shared Operator Construction: Single-Matrix Inference-Time Transformation The final shared intervention matrix is assembled by modulating the trimmed centroid perturbation with an intervention strength hyperparameter \(\alpha\): $\(W_{\text{CIF},k}(\alpha, \text{drop}) = I_d + \alpha \Delta W_{\text{CIF},k}\)$ When \(\alpha = 0\), the operator reduces to the identity matrix (no intervention); when \(\text{drop} = 0\) and \(\alpha = 1\), it matches the unpruned arithmetic mean. At test time, for sequence-level scoring tasks (e.g., multi-choice commonsense reasoning), the intervention modifies the hidden state of the final non-padding token \(h_{q,k}\). For autoregressive generation tasks (e.g., machine translation), the intervention is applied at the current decoding step. The modified state is calculated via \(\hat{h}_{q,k} = h_{q,k} + \alpha \Delta W_{\text{CIF},k} h_{q,k}\). Because \(W_{\text{CIF},k}\) is language-agnostic, the pipeline requires no language identification module, and runtime matrix storage drops from \(N \times L\) to strictly \(L\).

Benchmark & Evaluation Protocol

The framework is evaluated across 8 diverse multilingual benchmarks covering discriminative reasoning, natural language inference, factual knowledge editing, and open-ended generation: XCOPA (causal commonsense reasoning, 10 languages), XStoryCloze (story completion, 8 languages), XWinograd (coreference resolution, 6 languages), XCSR (commonsense reasoning, 15 languages), XNLI (natural language inference, 14 languages), MzsRE (zero-shot multilingual knowledge editing, 10 languages), WMT23 (general machine translation, 6 languages), and FLORES-101 (multilingual MT, 10 languages). Four LLM backbones are tested: Qwen2.5-7B-Instruct, BLOOMZ-7B1-MT, LLaMA-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3. All intervention baselines tune \(\alpha \in [-1.0, 1.0]\) with step 0.1 on the validation set, ensuring identical hyperparameter budgets.

Key Experimental Results

Main Results

The main benchmark evaluation compares CIF against zero-shot prompting, an external machine translation pipeline (Translation-NLLB), and four leading inference-time intervention methods (ITI, CAA, SADI, and INCLINE). Results are macro-averaged across the four LLM backbone families.

Method XCOPA (F1) XStoryCloze (F1) XWinograd (F1) XCSR (F1) XNLI (F1) MzsRE (F1) WMT23 (BLEU) FLORES (BLEU)
Zero-shot 53.17 81.39 43.61 39.12 56.06 39.98 39.57 38.45
Translation-NLLB 59.41 76.29 52.45 44.02 49.49 42.81 23.60 32.81
ITI 55.20±0.28 73.05±0.66 35.60±1.03 36.18±0.14 55.83±0.31 39.28±0.22 36.58±0.57 38.23±0.26
CAA 55.24±0.71 71.58±1.57 38.74±0.48 31.10±0.43 53.57±0.82 38.45±0.55 37.44±1.12 37.54±0.69
SADI 55.04±0.39 79.05±0.88 39.58±0.74 31.09±0.19 49.22±0.63 36.02±0.27 39.48±0.59 39.00±0.36
INCLINE (Pairwise SOTA) 53.98±0.17 81.15±0.51 46.85±0.32 38.60±0.09 58.52±0.06 40.62±0.18 42.02±0.33 39.70±0.11
CIF (Ours) 57.36±0.18 82.76±0.81 48.73±0.16 40.19±0.03 60.32±0.26 41.14±0.08 42.70±0.46 40.92±0.13
Gain over INCLINE +3.38 +1.61 +1.88 +1.59 +1.80 +0.52 +0.68 +1.22

Ablation Study

The ablation study validates the cumulative contribution of perturbation centroid fusion and centroid-guided trimming, averaged over all four model backbones:

Config Centroid Fusion Trimmed Drop XCOPA XStoryCloze XWinograd XCSR XNLI MzsRE WMT23 FLORES
CIF Variant (w/o Centroid & Drop) ✗ ✗ 53.98 81.15 46.85 38.60 58.52 40.62 42.02 39.70
CIF Variant (Centroid only) ✓ ✗ 55.81 83.11 48.62 40.00 59.29 41.08 41.86 40.08
CIF (Full Model) ✓ ✓ 57.36 82.76 48.73 40.19 60.32 41.14 42.70 40.92

Key Findings

  • Centroid Fusion Enables Cross-Lingual Knowledge Sharing: Integrating independent projections into a shared perturbation centroid improves XCOPA from 53.98 to 55.81 (+1.83 pp) and XStoryCloze from 81.15 to 83.11 (+1.96 pp), proving that projection operators share universal cross-lingual geometric structures that outperform isolated pairwise fits.
  • Trimming Prunes Divergent Outliers: Incorporating trimmed aggregation provides additional gains across all tasks (+1.55 pp on XCOPA, +1.03 pp on XNLI, +0.84 BLEU on FLORES/WMT). Outlier analysis reveals that languages like Tamil in XCOPA and Hindi in XStoryCloze/FLORES are most frequently pruned due to high syntactic or morphological divergence; pruning them prevents distortion of the central subspace.
  • Robust Out-of-Distribution and Low-Resource Generalization: In granular language breakdowns, CIF achieves massive gains on challenging languages, such as +12.30 pp for Hindi on XCSR, +11.45 pp for Arabic on XCSR, and +5.25 BLEU for Arabic on FLORES. Furthermore, leave-one-language-out experiments confirm that held-out languages still benefit from the fused operator, demonstrating genuine cross-lingual transferability.

Highlights & Insights

  • Perturbation Space Parameterization: Reparameterizing projection operators around identity isolates directional intervention shifts into a zero-centered tangent space, enabling clean geometric distance metrics and linear centroid estimation.
  • Byzantine Trimming Solves Multilingual Interference: Adapting trimmed aggregation from robust statistics into model activation editing successfully purges divergent outlier updates, solving negative interference without human heuristic design.
  • Constant Inference Memory Complexity: By consolidating \(N\) distinct language projection matrices into one shared layer operator, deployment storage is reduced by a factor of \(N\) (from \(N \times L\) to \(L\)), completely eliminating the need for runtime language identification.

Limitations & Future Work

  • Scope Limited to Text Modalities: Current validation is confined to textual reasoning, factual editing, and translation; whether centroid intervention principles transfer to vision-language models or multilingual voice agents remains unexplored.
  • Degradation on Distant Open-Ended Generation: On typologically distant languages in generation tasks (e.g., Chinese in FLORES), CIF exhibits a regression of -3.21 BLEU compared to pairwise INCLINE, indicating that aggressive operator unification can occasionally discard subtle, language-specific syntactic signals necessary for generative decoding.
  • Future Directions: Exploring mixture-of-experts (MoE) intervention where a shared centroid serves as the backbone while lightweight, cluster-level adapters capture family-specific nuances; developing input-dependent dynamic routing for intervention strength.
  • vs INCLINE (Wang et al., 2025a): INCLINE relies on pairwise projection matrices learned and deployed separately for each target-source language pair; CIF unifies these matrices into a single shared operator via centroid fusion, cutting deployment memory by factor \(N\) and surpassing INCLINE on 29 of 32 benchmark configurations.
  • vs ITI / CAA / SADI (Li et al., 2023; Rimsky et al., 2024; Wang et al., 2025b): Standard activation engineering techniques derive steering vectors from positive/negative prompt contrasts in monolingual settings; CIF broadens activation steering to cross-lingual representation space, formalizing operator aggregation and geometric alignment.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Pioneers perturbation-space centroid fusion and trimmed aggregation for inference-time cross-lingual model steering.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 4 LLM backbones, 8 major multilingual benchmarks, dozens of typologically diverse languages, backed by geometric spectral and PCA analyses.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exemplary mathematical clarity, sound theoretical motivations, well-structured prose, and exhaustive appendix details.
  • Value: ⭐⭐⭐⭐⭐ Solves the scalability bottleneck of pairwise cross-lingual intervention, offering a practical, lightweight paradigm for low-resource model adaptation.