Adaptive Knowledge Routing for Complex Reasoning in LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ArmelRandy/llm-reasoning-mt
Area: LLM Reasoning
Keywords: Machine Translation Reasoning, Thinking Tokens, Chain-of-Thought Distillation, Modular Prompting Strategies, Target Refinement & Data Augmentation
TL;DR¶
Through extensive cross-lingual and cross-architecture benchmarking, this paper shows that general-purpose thinking tokens in Large Reasoning Models (LRMs) bring no significant benefit to machine translation, and distilling human-inspired step-by-step CoT traces into student models fails to outperform standard input-output fine-tuning (IOFT); the performance gains from MT-specific modular prompting strategies stem strictly from intermediate translation drafts embedded inside the traces rather than the surrounding reasoning explanations, demonstrating that using teacher models to polish target translations and expand parallel corpora is far more impactful than building "thinking" translation models.
Background & Motivation¶
Large Reasoning Models (LRMs) such as OpenAI o1/o3, DeepSeek-R1, Qwen3, and Claude 4 have established a powerful new paradigm by generating natural language "thinking tokens" before producing a final answer, achieving state-of-the-art results on competition mathematics and complex code synthesis. However, machine translation (MT)—a cornerstone sequence-to-sequence problem in natural language processing—remains largely untouched by rigorous examination under this reasoning paradigm. While prior zero-shot Chain-of-Thought (CoT) prompting experiments on standard LLMs typically led to catastrophic degradation compared to vanilla few-shot prompting, the advent of native thinking models raises a fundamental question: do the internal reasoning chains of modern LRMs encode actionable cross-lingual planning, or do they merely incur costly test-time compute overhead without tangible linguistic benefits?
This ambiguity exposes a sharp tension in current literature. On one hand, knowledge distillation from teacher-generated CoT traces has proven remarkably successful in mathematical problem solving, inspiring recent efforts to build "thinking" translation models by prompting teachers to emulate the step-by-step cognitive practices of human translators (e.g., syntactic tree dissection, lexical disambiguation, back-translation verification). On the other hand, unlike formal mathematics where reasoning steps follow strict deductive logic toward a single verifiable ground truth, translation is inherently open-ended and grounded in surface-level semantic alignment, register appropriateness, and stylistic fluency. Transferring the reasoning distillation playbook to machine translation without probing whether intermediate explanations meaningfully guide the final target risks optimizing for unfaithful or hallucinated rationales.
To resolve this conflict, the authors formulate a comprehensive experimental protocol that scrutinizes thinking tokens across 10 diverse language directions spanning high- and low-resource pairs, varying sampling temperatures, multiple standard benchmarks, and different teacher-student distillation configurations. Core idea: intermediate reasoning explanations provide virtually no benefit for machine translation, and the empirical improvements observed from modular translation traces stem entirely from embedded high-quality translation drafts; hence, leveraging teacher models to curate superior target translations (IOFT-Max / IOFT-BoA) and synthesize diverse parallel training pairs (IOFT-Ext) consistently outperforms distilling thinking tokens while completely eliminating test-time latency.
Method¶
Overall Architecture¶
The paper establishes a decoupled experimental framework to investigate the role of intermediate tokens across three distinct settings: pre-trained LRM benchmarking, CoT supervised fine-tuning (CoTFT), and post-training reinforcement learning (RL). First, off-the-shelf general-purpose LRMs (Qwen3 from 0.6B to 32B and DeepSeek-R1 distillations) are evaluated with thinking enabled versus explicitly suppressed via prefix injection. Second, for training dedicated models, a teacher model (e.g., Llama-4-Scout-17B or Gemma-3-27B-it) generates intermediate traces given source-target pairs under two paradigms: 6 human-inspired CoT templates and 5 modular MT-specific prompting strategies. A student model (Gemma-3-4b-pt) is then fine-tuned via CoTFT to generate <think>traces</think> Final Translation or via standard IOFT directly predicting the target. Crucially, counterfactual ablation variants (IOFT-Max, CoTFT-Max, IOFT-BoA, and IOFT-Ext) isolate the causal impact of translation drafts from pure reasoning explanations.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Parallel MT Corpus (Source, Target)"] --> B["Benchmark LRM Thinking vs. Non-Thinking<br/>Qwen3 & DeepSeek-R1 Family"]
A --> C["Teacher Intermediate Trace Generation<br/>Llama-4-Scout / Gemma-3-27B"]
C --> D["Human CoT Templates vs. Modular MT Strategies<br/>T1-T6 Templates & MAPS / TEaR / CompTra"]
D --> E["Student Fine-Tuning & Causal Disentanglement<br/>CoTFT / IOFT / IOFT-Max / IOFT-Ext"]
E --> F["Reinforcement Learning Verification<br/>GRPO Exploration & Final Translation"]
Key Designs¶
1. Benchmark LRM Thinking vs. Non-Thinking: Testing whether thinking tokens are beneficial or redundant in translation
To establish whether off-the-shelf reasoning models actually translate better when allowed to think, the authors conduct zero-shot and few-shot benchmarking across ten language pairs on FLORES-200, NTREX-128, and TICO-19. Two conditions are strictly contrasted: the standard "thinking mode" where models generate up to 3,500 thinking tokens, and the "non-thinking mode" where reasoning is suppressed by appending <think>\n\n</think> directly to the input prompt. Decoding temperature sweeps from 0.0 to 1.0 further test whether stochastic reasoning helps. Across all model sizes (0.6B to 32B) and families, non-thinking greedy decoding (\(T=0\)) yields equal or superior translation accuracy, demonstrating that the internal thinking tokens of general-purpose LRMs provide no intrinsic advantage for machine translation.
2. Human CoT Templates vs. Modular MT Strategies: Comparing translation-specific cognitive traces against human-like CoT
To investigate whether structured intermediate traces can supervise smaller student models via distillation, the authors synthesize two distinct categories of intermediate tokens: - Human-Inspired CoT Templates (T1–T6): Following He et al. (2025), six prompt templates emulate professional human translation workflows: Hierarchical Translation (T1), Triangulating Translation (T2), Back-Translation (T3), Context-Aware Translation (T4), Translation Explanation (T5), and Structural Transformation (T6). The teacher outputs a first-person reflective analysis dissecting sentence components, syntactic rules, and lexical choices prior to the target translation. - Modular MT-Specific Prompting Strategies: Five structured multi-step translation prompting methods are repurposed to generate execution traces: MAPS (extracting keywords/topics and drafting inspired translations), SBYS (pre-draft research, drafting, refinement, and proofreading), TEaR (drafting, MQM error annotation, and error-based refinement), Self-Refine (iterative multi-round drafting), and CompTra (source sentence proposition decomposition, phrase-level translation, and recombination). Individual sub-step outputs are concatenated to serve as intermediate thinking tokens during student training.
3. Student Fine-Tuning & Causal Disentanglement: Uncoupling reasoning explanations from embedded translation drafts
To uncover the true mechanism driving student improvements under modular prompting strategies, the authors introduce a rigorous disentanglement suite:
- Baseline Pair: Standard IOFT maps source directly to target (\(L_{\max}=512\)); CoTFT trains the student to output concatenated intermediate traces within <think> before emitting the final translation (\(L_{\max}=2048\)).
- Target Selection via Quality Estimation (IOFT-Max & CoTFT-Max): Using the reference-free quality metric BLASER 2.0-QE, the best candidate among the original ground truth and any intermediate drafts embedded inside the teacher's trace is selected. IOFT-Max trains directly on this superior target without generating intermediate tokens, while CoTFT-Max retains the trace.
- Best-of-All (IOFT-BoA) and Corpus Augmentation (IOFT-Ext): IOFT-BoA takes the global best target identified across all five prompting strategies and trains vanilla IOFT. In parallel, IOFT-Ext augments the training corpus with synthetic pairs derived from sentence decomposition (CompTra), Paraphrases (P), Syntactic Paraphrases (SP), and Hard Expressions (H). The results conclusively show that whenever IOFT-Max matches or beats CoTFT-Max, the surrounding non-draft reasoning tokens act purely as dilution or noise.
4. Reinforcement Learning Verification: Evaluating whether reasoning trajectories persist or improve under policy optimization
To test whether thinking tokens unlock benefits during reinforcement learning, checkpoints fine-tuned via IOFT and CoTFT (T3 and all modular strategies) are further optimized using Group Relative Policy Optimization (GRPO) on a distinct parallel corpus for 5,000 steps. The reward function combines lexical overlap (BLEU, chrF++), reference-based neural scores (COMET-22), reference-free quality estimation (BLASER-2.0 QE), and a formatting reward enforcing valid <think> closures. GRPO consistently provides a uniform gain of ~1.3 BLEU and ~1.0 MetricX across all setups, perfectly preserving the relative rank order established during SFT. Because RL without verifiable ground-truth rewards cannot synthesize meaningful reasoning out of thin air, continued SFT on high-quality bitext yields quadruple the gain of GRPO (+6 BLEU vs. +1.3 BLEU).
Loss & Training¶
Student models are trained on a single NVIDIA H100 80GB GPU for 5,000 steps with an effective batch size of 16 (per-device batch size 4 with 4 gradient accumulation steps for IOFT; 16 accumulation steps for CoTFT). Optimization uses AdamW with a peak learning rate of \(1 \times 10^{-5}\), warmed up from \(1 \times 10^{-6}\) over 500 steps followed by a constant schedule. Loss is computed exclusively on the target tokens:
During RL post-training with GRPO, models are hosted on 4 H100 GPUs (1 GPU dedicated to reward model inference), sampling 12 completions per prompt at temperature 1.0 with effective batch size 48, KL coefficient \(\beta = 0.02\), and LoRA rank \(r = 32\), \(\alpha = 64\) across all attention and MLP projection layers.
Key Experimental Results¶
Main Results¶
Zero-shot translation performance on FLORES-200 comparing Qwen3-32B in thinking mode (w/ Thinking) against explicitly suppressed thinking mode (w/o Thinking) using BLEU (higher is better) and MetricX-24-Hybrid-XXL (lower is better, range 0–25). Across 10 diverse language directions, suppressing thinking tokens yields equal or superior performance.
| Language Direction | Metric | Qwen3-32B (w/ Thinking) | Qwen3-32B (w/o Thinking) | Difference (Non-thinking Advantage) |
|---|---|---|---|---|
| English → French | BLEU / MetricX | 51.80 / 2.01 | 50.94 / 2.00 | MetricX -0.01 |
| English → German | BLEU / MetricX | 42.78 / 1.09 | 42.08 / 1.07 | MetricX -0.02 |
| English → Czech | BLEU / MetricX | 34.62 / 4.70 | 33.27 / 4.64 | MetricX -0.06 |
| English → Spanish | BLEU / MetricX | 31.84 / 1.66 | 32.18 / 1.71 | BLEU +0.34 |
| English → Japanese | BLEU / MetricX | 26.44 / 3.88 | 27.33 / 3.62 | BLEU +0.89, MetricX -0.26 |
| English → Finnish | BLEU / MetricX | 24.33 / 7.25 | 24.14 / 7.05 | MetricX -0.20 |
| English → Lithuanian | BLEU / MetricX | 23.70 / 8.41 | 24.10 / 7.58 | BLEU +0.40, MetricX -0.83 |
| English → Kazakh | BLEU / MetricX | 17.33 / 9.54 | 18.13 / 8.39 | BLEU +0.80, MetricX -1.15 |
| English → Turkish | BLEU / MetricX | 31.68 / 5.80 | 30.48 / 5.59 | MetricX -0.21 |
| English → Portuguese | BLEU / MetricX | 51.01 / 1.96 | 51.59 / 1.96 | BLEU +0.58 |
Ablation Study¶
Fine-tuning results on the low-resource English → Xhosa direction using Llama-4-Scout-17B as teacher and Gemma-3-4b-pt as student across 5,000 steps on FLORES-200:
| Fine-Tuning Setup / Strategy | BLEU (↑) | MetricX-24 (↓) | Key Observation vs. IOFT Baseline |
|---|---|---|---|
| Standard IOFT Baseline | 14.1 | 7.9 | Direct mapping from source sentence to target translation |
| CoTFT with T1–T6 (Human Templates Mean) | 13.6 | 8.4 | BLEU drops by 0.5, MetricX worsens by 0.5 (CoT distillation fails) |
| CoTFT with MAPS | 17.6 | 5.9 | +3.5 BLEU, -2.0 MetricX (contains multi-aspect draft translations) |
| CoTFT with CompTra | 16.3 | 6.5 | +2.2 BLEU, -1.4 MetricX (contains clause-level draft translations) |
| IOFT-Max(MAPS) | 18.2 | 5.7 | Polished target alone surpasses CoTFT without generating traces |
| IOFT-BoA (Best-of-All Target Refinement) | 18.8 | 5.5 | +4.7 BLEU, -2.4 MetricX over IOFT with identical training budget |
| IOFT-Ext(Syntactic Paraphrases) | 18.4 | 5.8 | Expanding bitext by \(5\times\) strongly outperforms CoT distillation |
| IOFT-Ext(Paraphrases) | 18.1 | 6.0 | Semantic data diversity yields substantial, trace-free gains |
Key Findings¶
- Human-inspired CoT distillation fails consistently: Across all six human translation templates (T1–T6), training students to produce detailed syntactic breakdowns, semantic reflections, and back-translation checks consistently lags behind standard IOFT by ~0.5 BLEU and ~0.5 MetricX points. Switching the teacher to DeepSeek-R1-Distill-Llama-70B yields the identical negative result.
- Gains in modular CoTFT are driven by embedded drafts, not reasoning: While CoTFT with strategies like MAPS, TEaR, and Self-Refine delivers up to +3.5 BLEU improvements over basic IOFT, simply training standard IOFT on the best translation draft found inside those traces (IOFT-Max) completely outperforms CoTFT. The natural language explanations surrounding the drafts are extraneous and act as noise during optimization.
- Granular phrases vs. full-sentence augmentation: CompTra is the unique case where CoTFT outperforms IOFT-Max. Because CompTra decomposes the sentence into independent sub-sentential propositions, its partial drafts cannot replace the full ground truth; however, presenting them as intermediate context helps the student align complex syntax. Conversely, for full-sentence derivations (Paraphrases and Syntactic Paraphrases), directly appending them as standard parallel data (IOFT-Ext) achieves peak performance (+4.3 BLEU).
Highlights & Insights¶
- Dismantling the "reasoning solves everything" dogma: This paper provides definitive large-scale evidence that natural language thinking tokens are not a silver bullet for language translation, clearly delineating MT (semantic alignment without single verifiable logic steps) from formal mathematical deduction.
- Elegant causal disentanglement methodology: By designing IOFT-Max and IOFT-BoA counterfactuals, the authors elegantly isolate the information contribution of intermediate text, proving that empirical gains from multi-step prompts belong to candidate draft quality rather than the act of "thinking."
- Practical industry recipe for efficient MT deployment: Rather than burdening inference serving with thousands of slow, expensive thinking tokens, practitioners should allocate teacher compute offline to curate superior target bitexts and synthesize syntactic paraphrases, unlocking maximum translation performance at zero extra inference cost.
Limitations & Future Work¶
- Primary focus on sentence-level translation: The main experiments focus on sentence-level MT. While preliminary appendix evaluations on document-level translation (IWSLT TED talks) and stylized translation (Chinese/Portuguese regional variants) echo the same findings, complex discourse phenomena (e.g., long-distance coreference, chapter-level tone maintenance) might still benefit from dedicated structured planning.
- Reference-based evaluation constraints: Automatic metrics like BLEU and MetricX, though standard, may exhibit blindness toward subtle cultural adaptations or figurative idioms where creative reasoning could play a legitimate role.
- Future directions: Investigating whether non-linguistic structured intermediate representations (such as abstract semantic dependency graphs or cross-lingual alignment lattices) can provide inductive biases where free-form natural language explanations fail.
Related Work & Insights¶
- vs. R1-T1 (He et al., 2025) & DRT (Wang et al., 2025a): While recent papers claim that training LLMs to generate human-like translation thought processes via SFT and RL boosts translation capability, this paper proves through strict disentanglement that the reported gains stem from superior teacher rewrites rather than the reasoning process itself; training standard IOFT on those polished targets (IOFT-Max) actually achieves better results.
- vs. Hunyuan-MT (Zheng et al., 2025): While Zheng et al. observed that thinking tokens fail to help MT under RL with final translation rewards, this paper goes deeper by showing that the failure originates at the fundamental SFT and distillation stage across all known CoT formats, providing the complete theoretical and empirical explanation for why CoT fails in MT.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Decisively refutes the assumption that thinking tokens benefit machine translation, establishing an authoritative benchmark for the field.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage of 10+ language pairs, 3 benchmarks, multiple model families (Qwen3, DeepSeek-R1, Llama-4, Gemma-3), and meticulous counterfactual ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear, rigorous, and intellectually disciplined progression from initial empirical surprise to causal mechanism unravelling.
- Value: ⭐⭐⭐⭐⭐ Immense practical and theoretical significance, offering immediate compute savings and clear design principles for production MT systems.