Language Chain in Alignment: Cross-lingual Ranking Preference Optimization¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/dltmddbs100/CRPO
Area: LLM Reasoning
Keywords: Cross-lingual Preference Alignment, Learning to Rank, Language Chain, LambdaLoss, Direct Preference Optimization
TL;DR¶
To alleviate language mismatch and degraded response quality in non-English alignment, CRPO reformulates cross-lingual alignment as a hierarchical ranking optimization problem via LambdaLoss, jointly modeling language consistency and generation quality over parallel response quadruplets to substantially outperform conventional binary DPO.
Background & Motivation¶
Reinforcement learning from human feedback and direct preference optimization have become foundational paradigms for steering Large Language Models toward human intent and complex instruction-following behavior. Nevertheless, high-quality preference annotations remain heavily concentrated in English-centric ecosystems. Consequently, alignment benefits do not transfer seamlessly to other languages. When responding to non-English prompts—especially in medium- and low-resource languages—aligned models frequently suffer from severe language confusion (e.g., replying in English to a target-language query or producing awkward code-switching) as well as degraded reasoning and response quality.
Existing multilingual alignment efforts typically depend on expensive human data curation or machine-translated parallel corpora trained via independent, intra-lingual binary preference optimization (e.g., standard DPO). Such approaches suffer from an inherent structural flaw: they isolate the target-language preference optimization from the rich preference knowledge internalized during English pre-training and alignment. Meanwhile, recent attempts at cross-lingual alignment either introduce heavy external translation dependencies or enforce crude binary constraints (e.g., directly penalizing English responses), which often trigger optimization instability, distorted reward dynamics, and catastrophic collapse in low-resource settings.
The key insight of this paper is that cross-lingual adaptation and preference alignment can be harmonized by transplanting Learning-to-Rank principles into the alignment framework, creating an explicit "Language Chain" between English and the target language. By structuring parallel quadruplets across both languages and establishing a hierarchical priority between language consistency and output quality, the model can exploit robust English preference knowledge as an anchor while strictly enforcing target-language generation. Core idea: Formulate cross-lingual preference optimization as a hierarchical ranking task via LambdaLoss (CRPO), establishing the relevance hierarchy \(\psi(y_{tw}) > \psi(y_{ew}) > \psi(y_{tl}) > \psi(y_{el})\) to jointly optimize intra- and inter-lingual preferences on a unified manifold.
Method¶
Overall Architecture¶
CRPO extends the pairwise alignment mechanism of DPO into a global ranking optimization paradigm over multiple candidate responses. Given a prompt \(x_t\) in the target language, CRPO constructs a parallel response quadruplet consisting of: the chosen response in the target language \(y_{tw}\), the rejected response in the target language \(y_{tl}\), the chosen response in English \(y_{ew}\), and the rejected response in English \(y_{el}\).
During training, the policy model \(\pi_\theta\) and frozen reference model \(\pi_{ref}\) compute an implicit reward score \(s_i = \beta \log \frac{\pi_\theta(y_i|x)}{\pi_{ref}(y_i|x)}\) for each candidate \(y_i\). Grounded in the LambdaLoss framework, CRPO assigns hierarchical relevance labels \(\psi\) across these four responses and applies dynamically discounted rank-inversion penalties \(\Delta_{i,j}\) to enforce the intended order. Furthermore, a negative log-likelihood (NLL) objective on \(y_{tw}\) is incorporated to preserve fundamental language modeling and instruction-following fidelity.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Target Language Query<br/>Input prompt x_t"] --> B["Parallel Preference Quadruplet Construction<br/>Bilingual candidates y_tw, y_ew, y_tl, y_el"]
B --> C["Hierarchical Relevance Priority Definition<br/>ψ(y_tw) > ψ(y_ew) > ψ(y_tl) > ψ(y_el)"]
C --> D["Implicit Reward & Lambda-Weight Calculation<br/>Implicit reward s_i & nDCG2 weights Δ_ij"]
D --> E["Dual-Objective Joint Optimization<br/>Ranking alignment loss + NLL regularization"]
E --> F["Aligned Policy Model<br/>Preserving language match & high-fidelity reasoning"]
Key Designs¶
1. Parallel Preference Quadruplet Construction: Beyond Isolated Binary Comparison
Standard preference optimization restricts comparisons to isolated intra-lingual pairs \((y_w, y_l)\), leaving the model's rich English reasoning decoupled from target-language adaptation. CRPO constructs a parallel quadruplet \(\mathcal{Y} = \{y_{tw}, y_{tl}, y_{ew}, y_{el}\}\) for each target prompt \(x_t\) by pairing translated instances with their English source counterparts from UltraFeedback. This formulation enables a single optimization step to supervise both intra-lingual pairs \(((y_{tw}, y_{tl}), (y_{ew}, y_{el}))\) and cross-lingual pairs \(((y_{tw}, y_{el}), (y_{ew}, y_{tl}))\). The preferred English candidate \(y_{ew}\) serves as an informative semantic and logical anchor, providing a smooth preference manifold that prevents low-resource optimization collapse.
2. Hierarchical Relevance Priority Definition: Decoupling Quality and Language Consistency
To balance linguistic fidelity and content quality without allowing either dimension to dominate destructively, CRPO defines a strict four-level relevance hierarchy: $\(\psi(y_{tw}) > \psi(y_{ew}) > \psi(y_{tl}) > \psi(y_{el})\)$ The pivotal design choice is setting \(\psi(y_{ew}) > \psi(y_{tl})\), which ranks a high-quality English response above a low-quality target-language response. In a ranking structure where pairwise gradients are interdependent, demoting \(y_{ew}\) below \(y_{tl}\) would excessively bias gradients toward pure language matching at the expense of output quality. By placing \(y_{tw}\) at the global apex, CRPO guarantees input-output language consistency, while the inter-lingual ranking hierarchy preserves rigorous standards for reasoning and factual correctness.
3. Distance-Aware Gain Function and Weighting Scheme: Tighter Rank Gradients via nDCG2
Standard LambdaLoss in Information Retrieval employs an exponential gain function \(G_i = 2^{\psi_i} - 1\), which yields extreme disparities (e.g., 7, 3, 1, 0) and generates disproportionate local gradients that destabilize intra-lingual tuning. To resolve this, CRPO introduces a balanced linear gain allocation: \(G_{tw} = 9\), \(G_{ew} = 7\), \(G_{tl} = 5\), and \(G_{el} = 4\). Under this setup, the intra-language quality gaps (\(G_{tw} - G_{tl} = 4\), \(G_{ew} - G_{el} = 3\)) strictly exceed the primary cross-lingual gap (\(G_{tw} - G_{ew} = 2\)), ensuring that quality refinement is never eclipsed by language selection. Furthermore, CRPO adopts \(\text{nDCG2}\) as its primary weighting scheme: $\(\Delta_{i,j}^{\text{nDCG2}} = |G_i - G_j| \cdot \left| \frac{1}{D(|\tau(i) - \tau(j)|)} - \frac{1}{D(|\tau(i) - \tau(j)| + 1)} \right|\)$ where \(D(\tau) = \log_2(1 + \tau)\) and \(\tau(i)\) denotes the rank induced by the implicit reward \(s_i\). By focusing on the relative distance \(|\tau(i) - \tau(j)|\) between inverted candidates rather than coarse absolute positions, \(\text{nDCG2}\) supplies localized, well-conditioned gradient signals for robust manifold alignment.
Loss & Training¶
The overall training objective of CRPO is formulated as: $\(\mathcal{L}_{\text{CRPO}} = (1 - \alpha) \cdot \mathcal{L}_{\text{NLL}} + \alpha \cdot \mathcal{L}_{\text{lambda}}(\pi_\theta; \pi_{ref}, \beta)\)$ where the ranking-based LambdaLoss is defined over all candidate pairs satisfying \(\psi_i > \psi_j\): $\(\mathcal{L}_{\text{lambda}}(\pi_\theta; \pi_{ref}, \beta) = \mathbb{E}_{(x, y, \psi) \sim \mathcal{D}} \left[ \sum_{\psi_i > \psi_j} \Delta_{i,j} \log \left( 1 + e^{-(s_i - s_j)} \right) \right]\)$ with implicit rewards \(s_i = \beta \log \frac{\pi_\theta(y_i|x)}{\pi_{ref}(y_i|x)}\) and \(\beta = 0.1\). The regularization term \(\mathcal{L}_{\text{NLL}} = -\log \pi_\theta(y_{tw}|x_t)\) is applied exclusively to the target-language chosen response. Training uses \(\alpha = 0.2\), an effective learning rate of \(8\times 10^{-6}\) with linear decay (10% warmup), a maximum context length of 3072 tokens, and 2 epochs on 4 NVIDIA A100 GPUs.
Key Experimental Results¶
Experiments are evaluated across five diverse languages: Chinese (zh, high-resource), Indonesian (id, mid-resource), Korean (ko, high-resource), Swahili (sw, low-resource), and Bengali (bn, low-resource), using Llama-2-7B, Llama-3-8B, and Mistral-7B-v0.1 backbones.
Main Results¶
The table below reports AlpacaEval win rates against the SFT baseline, comparing length-controlled win rate (LC) and raw win rate (WR):
| Model | Method | Chinese LC / WR (%) | Indonesian LC / WR (%) | Korean LC / WR (%) | Swahili LC / WR (%) | Bengali LC / WR (%) |
|---|---|---|---|---|---|---|
| Llama-2-7B | SFT+DPO | 57.80 / 58.50 | 57.92 / 58.88 | 54.28 / 55.40 | 38.63 / 38.07 | 34.90 / 35.68 |
| CLO | 56.41 / 56.08 | 53.84 / 53.91 | 51.15 / 50.24 | 44.49 / 43.97 | 33.15 / 30.93 | |
| CRPO (Ours) | 64.18 / 64.34 | 63.83 / 64.47 | 63.02 / 63.97 | 44.07 / 43.72 | 37.52 / 36.77 | |
| Llama-3-8B | SFT+DPO | 60.24 / 60.37 | 59.24 / 60.37 | 64.97 / 65.96 | 46.23 / 46.95 | 34.77 / 36.89 |
| CLO | 58.70 / 58.13 | 52.42 / 52.96 | 53.94 / 53.41 | 43.78 / 44.93 | 35.67 / 33.29 | |
| CRPO (Ours) | 62.45 / 62.36 | 67.97 / 68.40 | 68.45 / 68.81 | 61.84 / 62.17 | 53.86 / 55.65 | |
| Mistral-7B | SFT+DPO | 55.65 / 56.14 | 55.98 / 58.50 | 51.49 / 52.85 | 27.95 / 27.33 | 24.63 / 24.59 |
| CLO | 55.18 / 55.03 | 51.29 / 51.98 | 48.14 / 48.32 | 35.15 / 35.40 | 19.98 / 18.01 | |
| CRPO (Ours) | 62.83 / 63.10 | 66.81 / 68.19 | 62.90 / 63.72 | 50.76 / 51.98 | 48.69 / 48.57 |
On multi-lingual knowledge understanding and reasoning benchmarks (zero-shot MMMLU and one-shot Belebele accuracy), CRPO demonstrates consistent gains:
| Model | Benchmark | Indonesian (id) SFT+DPO / CLO / CRPO | Korean (ko) SFT+DPO / CLO / CRPO | Swahili (sw) SFT+DPO / CLO / CRPO |
|---|---|---|---|---|
| Llama-2-7B | MMMLU | 26.33 / 24.69 / 27.85 | 26.20 / 28.57 / 31.07 | 25.08 / 25.62 / 25.12 |
| Belebele | 33.11 / 36.55 / 36.66 | 33.00 / 35.66 / 36.33 | 29.22 / 28.22 / 29.22 | |
| Llama-3-8B | MMMLU | 41.69 / 36.41 / 45.94 | 39.58 / 40.19 / 39.84 | 36.02 / 27.48 / 37.03 |
| Belebele | 64.22 / 35.11 / 65.88 | 57.55 / 59.44 / 68.66 | 47.55 / 28.66 / 49.22 | |
| Mistral-7B | MMMLU | 37.36 / 31.38 / 37.47 | 30.39 / 29.27 / 35.01 | 30.57 / 28.32 / 31.78 |
| Belebele | 48.33 / 30.44 / 47.44 | 49.66 / 35.77 / 51.77 | 33.00 / 33.88 / 34.77 |
Ablation Study¶
Ablation on gain parameter configurations \(G = (G_{tw}, G_{ew}, G_{tl}, G_{el})\) with Llama-3-8B:
| Gain Configuration \(G\) | Property & Rationale | Chinese WR/LC (%) | Korean WR/LC (%) | Indonesian WR/LC (%) | Swahili WR/LC (%) | Bengali WR/LC (%) |
|---|---|---|---|---|---|---|
| (9, 7, 5, 4) | CRPO Default (quality gaps > language gap) | 62.36 / 62.46 | 68.82 / 68.46 | 68.41 / 67.98 | 62.17 / 61.85 | 55.65 / 53.87 |
| (7, 3, 1, 0) | Standard exponential discount (disproportionate gradient) | 60.50 / 60.75 | 66.46 / 66.04 | 67.20 / 66.86 | 66.46 / 66.35 | 54.16 / 52.54 |
| (9, 5, 7, 4) | Inverted hierarchy (\(y_{tl} > y_{ew}\), excessive language bias) | 60.00 / 60.39 | 66.21 / 66.11 | 65.22 / 65.52 | 67.14 / 67.43 | 53.98 / 53.79 |
External quality assessment using Skywork-Reward-V2-Qwen3-8B reveals that CRPO yields markedly higher reward values on generated outputs. For Mistral-7B on Indonesian, while SFT+DPO scores -0.037 and CLO scores -0.665, CRPO elevates the reward score to +0.805. On Chinese, CRPO attains 0.508 compared to 0.255 for SFT+DPO and -0.096 for CLO.
Key Findings¶
- Resistance to Alignment Collapse in Low-Resource Regimes: Conventional baselines frequently suffer severe performance collapse in low-resource languages (e.g., CLO drops to 35.11 on Indonesian Belebele and 33.29% WR on Bengali). In contrast, by anchoring to well-aligned English representations, CRPO maintains high stability, achieving 62.17% WR in Swahili and 55.65% in Bengali with Llama-3-8B.
- Genuine Generative Likelihood Enhancement: Intrinsic analysis of reward and probability distributions shows that while standard DPO expands reward margins primarily by depressing the log-likelihood of disfavored responses, CRPO actively shifts the distribution of chosen responses \(y_{tw}\) to higher log-likelihoods, incentivizing high-quality generation rather than mere negative suppression.
- Mitigation of Alignment Tax on English: Evaluating English AlpacaEval after target-language alignment demonstrates that CRPO consistently preserves or enhances the model's native English capabilities (e.g., Llama-2-7B English LC remains above 60% across all target settings, compared to 53%~57% for baselines).
Highlights & Insights¶
- Repurposing Learning-to-Rank for Cross-lingual Alignment: Instead of viewing multilingual adaptation as fragmented pairwise tasks, applying LambdaLoss transforms cross-lingual transfer into a structured global ranking objective across language boundaries.
- The \(\psi(y_{ew}) > \psi(y_{tl})\) Principle: Demoting the English winner below the target loser severely pollutes the optimization signal with language matching bias; keeping the English winner higher allows the model to absorb English reasoning rigor without sacrificing target-language dominance.
- Linear Gain Calibration over Exponential IR Defaults: Replacing exponential gain functions with calibrated linear offsets prevents localized gradient spikes, establishing an optimal balance where intra-lingual quality gradients supersede inter-lingual language gradients.
Limitations & Future Work¶
- Training Computational Overhead: Because each training step evaluates a quadruplet of four candidate responses rather than a single pair, memory and backpropagation costs are approximately double that of vanilla DPO.
- Benchmark Translation Bias: Evaluation datasets such as multilingual AlpacaEval and MMMLU originate from translated English benchmarks, which may fail to measure nuanced cultural idioms, localized world knowledge, and native pragmatic conventions.
- Static vs. Dynamic Gain Scheduling: The current gain vector \(G=(9, 7, 5, 4)\) is statically assigned; future work should explore dynamic, curriculum-driven gain adaptation based on real-time linguistic distance and training convergence.
Related Work & Insights¶
- vs SFT+DPO: Standard DPO conducts isolated optimization within each language, failing to tap into pre-existing English alignment manifolds and suffering catastrophic drops in low-resource domains; CRPO constructs an explicit cross-lingual gradient highway.
- vs CLO (Cross-Lingual Optimization): CLO relies on a rigid binary objective penalizing English responses, which frequently induces training instability; CRPO provides smooth, distance-weighted ranking penalties across multiple candidates.
- vs PRO (Preference Ranking Optimization): PRO applies Plackett-Luce sequential comparisons that lack top-rank sensitivity; CRPO integrates ranking-aware \(\text{nDCG2}\) weights that heavily penalize rank inversions near the top of the preference list, outperforming PRO on AlpacaEval by 30% to 50%.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulating cross-lingual preference alignment as a LambdaLoss ranking problem with a dual-dimensional hierarchy is conceptually elegant and original.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 3 open-source architectures, 5 diverse language tiers, instruction and reasoning benchmarks, internal likelihood distributions, and external reward models.
- Writing Quality: ⭐⭐⭐⭐⭐ Extremely lucid narrative dissecting the failure modes of binary cross-lingual alignment and validating the theoretical rationale behind ranking metrics.
- Value: ⭐⭐⭐⭐⭐ Provides an effective, general-purpose alignment framework that directly resolves language confusion and low-resource collapse in multilingual LLMs.