From Unverifiable Domain Text to Cross-Lingual RLVR via Code-Switched Task Synthesis¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Reinforcement Learning, Verifiable Rewards (RLVR), Cross-Lingual Transfer, Code-Switching, Task Synthesis (FIM-MCQA)
TL;DR¶
Addressing the severe scarcity of non-English verifiable reward tasks in specialized knowledge-intensive domains (medicine and cybersecurity), this paper proposes code-switched task synthesis to transform unverifiable raw English corpora into multi-view verifiable FIM-MCQA variants with identical answer bindings, significantly boosting target-language reasoning by bridging non-English inputs into dominant English-side reasoning pathways without sacrificing English proficiency.
Background & Motivation¶
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a cornerstone post-training paradigm for scaling test-time reasoning and multi-step problem solving in large language models. However, the viability of RLVR hinges upon task domains whose execution or evaluation can be checked with zero ambiguity through programmatic verification, execution checkers, or ground-truth answer matching. While mathematics and programming naturally yield such unambiguous supervisory signals, specialized knowledge-intensive domains—most notably clinical medicine and cybersecurity—primarily exist as unstructured, unverifiable text corpora. Creating gold-standard verifiable problem sets manually across specialized domains requires prohibitive expert labor and remains bottlenecked at scale.
This challenge is exacerbated in multilingual settings, where specialized non-English text corpora are scarce and verifiable reasoning benchmarks are virtually non-existent. Recent advances attempt to synthesize fill-in-the-middle multiple-choice question answering (FIM-MCQA) tasks from raw documents by masking informative spans to serve as verifiable ground truths. Yet in cross-lingual transfer scenarios, standard pipelines rely almost exclusively on translating English tasks into target languages, creating separate bilingual-monolingual mixtures. Crucially, multilingual reasoning gaps frequently stem from early input-understanding failures and poor cross-lingual semantic alignment, wherein models fail to route non-English domain prompts into their dominant, highly capable English reasoning pathways. Treating languages as isolated instances leaves this representational bridge unoptimized during reinforcement learning.
The central angle of this work is that the form of language presentation during RLVR training should actively construct the cross-lingual bridge rather than leaving it to chance. Core idea: integrate code-switching directly into the verifiable RLVR training signal by synthesizing option-side and passage-level cross-lingual FIM-MCQA variants with identical answer indexes, explicitly forcing the policy model to align non-English context with English-dominant domain reasoning under automated answer rewards.
Method¶
Overall Architecture¶
The framework systematically converts raw, unverifiable English domain texts into multi-view verifiable RLVR training tasks across both source and target languages. The end-to-end pipeline operates in four primary phases: first, raw unstructured documents from cybersecurity (Primus) and clinical medicine (Biomed-Enriched) undergo automated quality and educational filtering to remove non-pedagogical noise; second, a multi-stage synthesizer powered by an LLM with medium reasoning effort extracts self-contained passages, identifies consecutive reasoning-critical spans, applies a four-tier deterministic programmatic verifier to ensure exact substring matching, and generates format- and type-constrained distractors to build English seed FIM-MCQA tasks; third, each seed task is expanded into multi-lingual variants including full target translations, option-side code-switched pairs, and bidirectional passage-level code-switched instances; fourth, the synthesized task pool is balanced with an auxiliary general-reasoning RLVR mixture to train the base model under the Decoupled Actor-Critic Policy Optimization (DAPO) algorithm.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw English Domain Corpora<br/>Primus / Biomed-Enriched"] --> B["Educational Passage & Span Extraction<br/>Quality filtering & 4-tier string verification"]
B --> C["Seed English FIM-MCQA Synthesis<br/>Length- and type-constrained distractor generation"]
C --> D["Option-Side Code-Switching<br/>Cross-pairing passage and answer options"]
C --> E["Passage-Level Bidirectional Code-Switching<br/>Target-in-En and En-in-Target fluent span replacement"]
D --> F["Multi-View Verifiable Task Pool<br/>Invariant ground-truth index and verifier"]
E --> F
F --> G["Co-Training with Auxiliary RLVR Mixture<br/>DAPO optimization with group-filtered rollouts"]
G --> H["Aligned Cross-Lingual Reasoning Policy<br/>Middle-layer representation convergence to English reasoning"]
Key Designs¶
1. Verifiable Seed FIM-MCQA Synthesis: Eliminating Hallucinations to Guarantee Fully Automated Answer Checking
Generating reliable question-answer pairs from free-form domain text in a single LLM call frequently produces hallucinated distractors, paraphrased answer keys, or loose contextual dependencies that break automated evaluation. To establish an unassailable ground truth, the seed synthesis pipeline decomposes generation into educational passage extraction, verbatim answer span extraction, programmatic substring verification, and distractor construction. The synthesizer first extracts a 200–300 word self-contained passage containing causal mechanisms or diagnostic deductions, alongside an embedded 50–100 word answer span representing the key reasoning steps. A deterministic programmatic verifier subjects the candidate span to a four-tier matching protocol (exact match, whitespace strip, NFKC Unicode normalization, and fuzzy string matching). Only upon verified character-for-character identification is the span replaced with a [MASK] token. Finally, the model synthesizes plausible distractors matching the word count, format, and semantic type of the true span, forming a 9-option multiple-choice item with a strictly verified answer index \(y\).
2. Option-Side Code-Switching: Breaking Monolingual Isolation via Cross-Lingual Evidence Matching
Standard bilingual training keeps source and target languages in separate instances, enabling the policy model to solve problems within disconnected linguistic silos. To directly incentivize cross-lingual semantic mapping, option-side code-switching constructs cross-paired task variants where the background evidence and the answer choices reside in opposite languages:
When presented with an English passage \(p_{\text{en}}\) and Korean options \(A_{\text{tgt}}\), or vice versa, the model cannot rely on shallow lexical overlap between the context and the correct option. Instead, it must comprehend the evidence in language \(u\) and project its reasoning into language \(v\) to select the semantically identical masked span, thereby penalizing unidirectional reliance on language-specific superficial patterns.
3. Passage-Level Bidirectional Code-Switching: Contextual Language Mixing to Channel Latent Reasoning States
Even with option-side switching, the passage itself retains a purely monolingual reasoning flow. To encourage non-English inputs to dynamically activate English-proximal domain knowledge during contextual encoding, passage-level code-switching embeds fluent cross-lingual spans directly within the passage:
The transformation operator \(c_{u \to v}\) replaces content-bearing terminology, phrases, and clause-level logical connectors in passage \(p_u\) with their fluent equivalents in language \(v\), while strictly preserving the integrity of the [MASK] token. Both directions are synthesized: \(\text{en} \to \text{tgt}\) (injecting target language vocabulary into English domain passages) and \(\text{tgt} \to \text{en}\) (inserting English technical anchors into target language passages). A strict post-generation filter requires that switched spans constitute at least 15% of the total passage character length. This intra-sentential mixing acts as a semantic conduit, enabling the Transformer's middle layers to map target language inputs onto the model's dominant English reasoning manifold.
4. Co-Training with Auxiliary General-Reasoning Mixture: Format Regularization and Policy Stability
Training large models exclusively on specialized FIM-MCQA instances carries a significant risk of format collapse, where the policy overfits to fill-in-the-middle heuristics and degrades on open-ended multi-step reasoning. To safeguard general reasoning faculties, the training pipeline samples each batch with probability 0.6 from the domain FIM-MCQA pool and probability 0.4 from an auxiliary general RLVR mixture comprising 25,000 scientific reasoning problems, 15,000 math problems, and 10,000 instruction-following prompts. Rollouts are filtered within groups to retain instances exhibiting both correct and incorrect exploratory paths, providing balanced gradient signals that regularize domain adaptation without sacrificing reasoning breadth.
Loss & Training¶
All training variants are optimized using the DAPO algorithm across 3B-scale open-source architectures (Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct, and SmolLM3-3B). For each prompt, 16 rollouts are sampled at temperature \(T = 0.6\) and \(\text{top-}p = 0.95\), with a maximum prompt length of 4,200 tokens and a response limit of 8,192 tokens. Reward calculation is binary exact-match accuracy against ground-truth index \(y\). Models are trained for 2 epochs with a batch size of 256 prompts and a PPO mini-batch size of 64 at a learning rate of \(1 \times 10^{-6}\). The KL penalty and overlong buffers are disabled. Each full training run consumes approximately 2,500 H100 GPU hours.
Key Experimental Results¶
Main Results¶
The method was evaluated across medicine and cybersecurity. In clinical medicine, evaluations were conducted on Korean native and translated benchmarks using Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct. In cybersecurity, evaluations evaluated SmolLM3-3B across four target languages spanning distinct scripts and resource tiers: Korean, Swahili, Hindi, and Japanese.
| Model & Domain | Benchmark | RLVR-Base | +EN | +TGT | +BiMono | +OURS (Full) |
|---|---|---|---|---|---|---|
| Qwen2.5-3B (Med · Korean) | KorMedMCQA-Doctor | 35.06 | 35.34 | 35.29 | 35.34 | 36.67 (+1.33) |
| Qwen2.5-3B (Med · Korean) | MedMCQA* | 34.52 | 33.96 | 34.03 | 33.72 | 34.86 (+0.34) |
| Qwen2.5-3B (Med · Korean) | KMMLU-Pro-Medical | 35.45 | 35.76 | 36.21 | 36.06 | 36.29 (+0.23) |
| Qwen2.5-3B (Med · Korean) | Target Average Avg (Tgt) | 35.01 | 35.02 | 35.18 | 35.04 | 35.94 (+0.90) |
| Qwen2.5-3B (Med · English) | English Average Avg (En) | 51.42 | 52.19 | 52.97 | 52.96 | 52.13 (preserved) |
| Llama-3.2-3B (Med · Korean) | KorMedMCQA-Doctor | 21.10 | 27.93 | 27.47 | 28.79 | 29.31 (+0.52) |
| Llama-3.2-3B (Med · Korean) | MedMCQA* | 24.26 | 27.12 | 27.36 | 29.65 | 29.22 (-0.43) |
| Llama-3.2-3B (Med · Korean) | KMMLU-Pro-Medical | 22.84 | 24.77 | 24.09 | 25.08 | 26.36 (+1.28) |
| Llama-3.2-3B (Med · Korean) | Target Average Avg (Tgt) | 22.73 | 26.61 | 26.31 | 27.84 | 28.30 (+0.46) |
| Llama-3.2-3B (Med · English) | English Average Avg (En) | 51.65 | 56.64 | 56.89 | 57.21 | 57.26 (+0.05) |
| SmolLM3-3B (Cyber · Korean) | Target Average Avg (Tgt) | 50.23 | 54.63 | 58.50 | 55.77 | 61.47 (+5.70) |
| SmolLM3-3B (Cyber · Swahili) | Target Average Avg (Tgt) | 40.57 | 40.93 | 42.00 | 41.27 | 42.77 (+1.50) |
| SmolLM3-3B (Cyber · Hindi) | Target Average Avg (Tgt) | 51.30 | 58.90 | 59.93 | 59.43 | 61.47 (+2.04) |
| SmolLM3-3B (Cyber · Japanese) | Target Average Avg (Tgt) | 59.23 | 59.93 | 60.60 | 60.60 | 61.83 (+1.23) |
Note: Benchmarks marked with an asterisk (*) are translated from English source benchmarks.
Component Ablation Study¶
The component ablation was conducted on Korean cybersecurity benchmarks using SmolLM3-3B, comparing the separate and joint additions of option-side (+Opt) and passage-level (+Pass) code-switching over the bilingual-monolingual (+BiMono) baseline.
| Variant | SecEval* (KO) | CyberMetric* (KO) | CTI-MCQ* (KO) | Target Avg (KO) | English Avg (EN) |
|---|---|---|---|---|---|
| RLVR-Base (Auxiliary only) | 44.3 | 65.8 | 40.6 | 50.23 | 64.87 |
| +En (English domain FIM) | 50.1 | 70.6 | 43.2 | 54.63 | 67.00 |
| +Tgt (Target domain FIM) | 49.8 | 80.2 | 45.5 | 58.50 | 67.83 |
| +BiMono (En + Tgt Monolingual) | 51.1 | 72.4 | 43.8 | 55.77 | 66.80 |
| +Opt (BiMono + Option-side) | 52.0 | 71.0 | 45.9 | 56.30 | 67.20 |
| +Pass (BiMono + Passage-level) | 54.7 | 77.4 | 46.0 | 59.37 | 67.07 |
| +Ours (Opt + Pass Combined) | 57.3 | 80.0 | 47.1 | 61.47 | 68.27 |
Cross-Lingual Decision Consistency¶
Analyzing parallel English and Korean cybersecurity questions reveals how effectively the models maintain language-invariant reasoning choices:
| Configuration | (✓,✓) Both Correct ↑ | (✓,×) En-only Correct ↓ | (×,✓) Ko-only Correct ↓ |
|---|---|---|---|
| Base (No RLVR) | 15.1% | 28.2% | 13.0% |
| En-only | 28.1% | 30.0% | 7.3% |
| Bi-Mono | 29.2% | 29.8% | 7.1% |
| Ours (Code-Switched RLVR) | 34.8% | 27.1% | 6.7% |
Key Findings¶
- Bilingual Monolingual Mixture Fails to Establish Robust Bridges: Merely exposing the policy to parallel monolingual English and target-language instances (+BiMono) does not consistently outperform single-language baselines; in SmolLM3-3B Korean cybersecurity, +BiMono dropped to 55.77% compared to +Tgt (58.50%). In contrast, introducing code-switching boosted performance to 61.47%, indicating that intra-sample language co-occurrence is essential for transfer.
- Passage-Level Switching is the Primary Performance Engine: In the ablation study, passage-level code-switching contributed +3.60 points over BiMono, whereas option-side switching provided +0.53 points. Combining both yielded the optimal 61.47% average, proving that lexical and syntactic mixing inside the passage drives deeper cross-lingual grounding.
- Middle-Layer Latent Representation Alignment: Probing contextual hidden states at the opening
<think>token on CyberMetric showed that +OURS achieves significantly higher Mean-Centered Cosine similarity and Average Neuron-wise Correlation (ANC) across intermediate layers 12–24 compared to all non-code-switched baselines, demonstrating that code-switching narrows the representation gap before reasoning rollout begins. - Enhanced Consistency in High-Level Reasoning Chains: In pairwise evaluations using GPT-5.5 as a judge across parallel reasoning chains, +OURS achieved win rates of 48.3% vs. 40.0% over EN-ONLY on Qwen2.5 and 58.7% vs. 21.7% over TGT-ONLY on SmolLM3, showing substantially closer structural and strategic alignment between English and target-language reasoning trajectories.
Highlights & Insights¶
- Dual-Pronged Task Synthesis as a Data-Centric Bridge: Rather than altering the underlying RL algorithm or modifying reward functions, this approach solves cross-lingual domain adaptation entirely at the data construction level, generating programmatic, verifiable reward tasks from raw, uncurated documents.
- Corroborating the Multilingual Input Bottleneck: The paper shows a near-perfect linear correlation (\(r = 0.998\) on CyberMetric, \(r = 0.967\) on KMMLU-Pro-Medical) between in-domain target-to-English translation quality and retained reasoning performance ratio, proving empirically that non-English reasoning failures stem from inadequate mapping of inputs into English-proximal semantic representations.
- Broad Applicability Across Typologically Diverse Scripts: The method generalizes robustly beyond Korean to Swahili, Hindi, and Japanese, establishing a practical template for extending specialized reasoning capabilities to low- and medium-resource languages without degrading English benchmark performance.
Limitations & Future Work¶
- Significant Computational Footprint Precluding Broad Grid Searches: Each 3B model variant training required approximately 2,500 H100 GPU hours under the DAPO regime, which restricted hyper-parameter sweeps over switching ratios, curriculum schedules, rollout budgets, and larger model scales (e.g., 8B/70B).
- Reliance on Translated Evaluation Sets in Cybersecurity: While Korean medical evaluations utilized native professional licensing examinations, cybersecurity benchmarks across the four non-English languages partially relied on GPT-5.4-mini translations, which may preserve translation artifacts despite rigorous quality vetting with human experts and COMET scores.
- Exploration of Finer-Grained Switching Mechanisms: The current implementation utilizes sentence- and phrase-level syntactic substitution; future directions include investigating dynamically calibrated, grammar-informed code-switching schedules or online curriculum learning during RL exploration.
Related Work & Insights¶
- vs Golden Goose (Lu et al., 2026): While Golden Goose established the paradigm of synthesizing unverifiable web text into FIM-MCQA tasks for monolingual RLVR, it is limited to single-language settings. This paper generalizes the concept to cross-lingual settings, introducing option-side and bidirectional passage-level code-switching to resolve multilingual knowledge transfer.
- vs Code-Switching in Pretraining / SFT (Wang et al., 2025c; Yoo et al., 2025): Previous explorations of code-switching concentrated on token representation learning during pretraining or supervised fine-tuning. This work demonstrates for the first time that code-switched synthesis serves as an effective verifiable reward medium in post-training RLVR without requiring costly human-annotated CoT rationales.
- vs Think-in-English Test-Time Prompting (Tam et al., 2025; Qi et al., 2025): Prompting multilingual models to generate CoTs exclusively in English can cause prompt fragility and output language mismatch. Code-switched RLVR internalizes this capability during training, naturally aligning latent intermediate states with English reasoning pathways while allowing native target-language generation.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant adaptation of FIM-MCQA synthesis into a code-switched formulation for cross-lingual RLVR in data-scarce domains.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Broad evaluations covering two specialized domains, four target languages, extensive ablations, representation probing, and LLM-as-a-judge reasoning trace analyses.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorously structured narrative, clear problem formulation, highly cohesive prose, and meticulous methodological descriptions.
- Value: ⭐⭐⭐⭐⭐ Provides an impactful, data-centric blueprint for bootstrapping non-English reasoning models in technical domains where verifiable rewards are otherwise inaccessible.