RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment¶

Conference: ACL2026
arXiv: 2604.22520
Code: Not publicly provided in the paper
Area: Machine Translation / LLM Deployment / Sample Routing
Keywords: Hybrid Translation Deployment, Marginal Gain Prediction, in-model router, Budget Allocation, XCOMET

TL;DR¶

RouteLMT formalizes the routing problem in hybrid LLM translation as marginal gain allocation under a fixed large model budget. By using internal representations of the last prompt token from a small translation model to predict "how much improvement a large model can provide relative to the small model," it achieves a better quality-budget Pareto frontier than length-based methods, quality estimation, or external routers across four translation directions.

Background & Motivation¶

Background: Large Language Models (LLMs) exhibit strong performance in machine translation, but production deployment cannot assign all requests to large models due to costs, tail latency, and compute capacity. A common engineering solution is hybrid deployment: the majority of requests are handled by a small model, and only high-value or high-difficulty requests are delegated to a large model.

Limitations of Prior Work: Routing strategies are deceptively simple but prone to budget misallocation. Heuristic routing based on length, rare words, or entropy may waste large model calls on samples where the gain is minimal. Routing based on absolute quality or the difficulty of the small model is not equivalent to identifying where "the large model will provide significant improvement." Furthermore, some post-routing Quality Estimation (QE) methods require small model decoding before scoring, which increases latency and computation.

Key Challenge: The goal of hybrid translation is not to identify the "hardest sentences," but to find sentences where the "gain from the large model relative to the small model is maximized" under a limited budget. Difficult samples might be handled poorly by both models, while simple samples might be significantly improved by the large model due to idioms, abbreviations, or code-switching.

Goal: The authors aim to propose a lightweight, router that requires no external models or prior generation by the small model to directly predict marginal gains, demonstrating that marginal gain is the correct optimization signal for budgeted routing.

Key Insight: The paper utilizes the hidden state of the last prompt token during the prompt prefill stage of the small translation model. Since this representation encodes the source sentence, translation direction, and the model's internal assessment of the input, a simple regression head can be used to predict the gains from a large model upgrade.

Core Idea: Instead of predicting small model quality or input difficulty, the router directly predicts \(g(x;d)=q_{large}(x;d)-q_{small}(x;d)\) and allocates the fixed budget to samples with the highest predicted gains.

Method¶

The core of RouteLMT treats routing as a budget allocation problem. The system consists of a small model \(M_s\) and a large model \(M_l\). For each source sentence and direction, both models have respective translation quality scores. If the large model invocation budget is at most \(p\), the optimal strategy is to select the top-\(p\) samples with the highest marginal gain \(g=q_l-q_s\). Therefore, the training target is marginal gain regression rather than absolute quality.

Overall Architecture¶

During training, the authors first have the small and large models translate training samples, using XCOMET-XXL with human references to calculate quality scores and obtain gain labels. RouteLMT runs a single prefill of the small translation model, extracts the representation from the last prompt token hidden state, and predicts the gain through a lightweight linear head. The model is trained using LoRA to adapt the small model while simultaneously training the regression head.

During inference, the system does not need to generate small model translations first, nor does it require external QE. For offline batch processing, samples are sorted by predicted gain and the top-\(p\) samples are sent to the large model. For streaming deployment, a threshold \(\tau_p\) can be calibrated on held-out traffic to trigger the large model for approximately \(p\) proportion of requests.

Key Designs¶

Marginal Gain as Routing Signal:
- Function: Aligns the routing objective with the budget optimization goal.
- Mechanism: Total quality can be expressed as the small model quality constant plus the expected gain of the routed samples. Thus, when the budget is fixed, the task simplifies to maximizing the sum of gains for selected samples.
- Design Motivation: Difficulty, length, rare words, and absolute quality are only proxies; gain is the direct signal for deployment goals.
in-model hypothesis-free router:
- Function: Predicts whether a sample is worth upgrading to a large model with low overhead.
- Mechanism: A translation prompt is constructed for the source sentence and direction. The small model runs only the prefill stage without decoding. The hidden state of the last prompt token is fed into a linear head to output \(\hat{g}\). Since the translation direction is part of the prompt, the router is naturally direction-aware.
- Design Motivation: External routers ignore the internal representations of the small model, while post-routing QE requires generating a translation first. RouteLMT embeds the router into the small model to balance signal quality and latency.
Quality guard to control negative gain risk:
- Function: Reduces serious regressions where the large model performs significantly worse.
- Mechanism: Candidates are first selected based on gain, followed by a quality filter. "Quality predict" uses in-model quality prediction; "Quality hypo" decodes the small model translation first and filters using a quality scorer.
- Design Motivation: Gain ranking improves average yields but cannot entirely eliminate samples where the large model introduces errors like incorrect disambiguation or over-paraphrasing. A guard can be utilized to trade a portion of gains for lower risk during deployment.

Loss & Training¶

Training labels are derived from \(g(x;d)=\Phi(x,y_l,y^*)-\Phi(x,y_s,y^*)\), where \(\Phi\) is the XCOMET-XXL reference quality score. RouteLMT utilizes MSE loss to regress \(\hat{g}\) against the ground truth gain. In experiments, the small model is LMT-60-0.6B and the large model is LMT-60-8B. LoRA is applied to all linear layers of the small model with rank 8 and alpha 32. Evaluation directions include En-Zh, En-Ru, Zh-En, and Ru-En.

Key Experimental Results¶

Main Results¶

With a fixed large model budget of \(p=0.3\), RouteLMT is the strongest practical router in terms of Spearman correlation, HitRate@p, and MeanDelta@p.

Method	Spearman	HitRate@p Avg.	MeanDelta@p Avg.	Description
Gain Oracle	1.00	100.00	19.48	Ideal upper bound, routing by true gain
Quality Oracle	0.67	75.10	16.73	Quality upper bound is not equivalent to gain upper bound
Random	0.00	30.00	5.83	30% large model budget allocated randomly
Length	0.24	46.39	9.35	One of the strongest heuristics
Entropy	0.09	37.25	7.45	Small model uncertainty is unreliable
sentinel-src-24	0.34	55.00	11.27	Strong baseline for external QE/difficulty estimation
XLM-R-Delta	0.32	53.59	11.02	External model gain prediction
RouteLMT-Q	0.37	56.04	11.77	In-model prediction of small model quality
RouteLMT	0.40	57.33	12.13	In-model marginal gain prediction, best practical method

Ablation Study¶

Configuration	Severe loss	MeanDelta@p	Description
Random	7.10%	5.83	Low routing benefit
Gain	8.19%	12.13	High average gain, but severe negative gains persist
Gain + Quality predict	8.19%	12.24	Predicted quality guard offers limited improvement
Gain + Quality hypo	5.69%	16.73	Post-decoding quality guard significantly reduces losses and boosts gain

Key Findings¶

RouteLMT achieves a MeanDelta@p of 12.13, which is 2.78 higher than the strongest heuristic (Length: 9.35) and more than double the Random baseline, indicating that marginal gain prediction is more aligned with budget objectives.
RouteLMT outperforms RouteLMT-Q, proving that "predicting how much the large model improves" is more effective than "predicting how well the small model performs."
In-model methods outperform external routers like XLM-R, suggesting that internal prompt representations contain useful signals regarding translation directions and difficulty.
Severe negative gains were not entirely eliminated by learned routing, with approximately 8-9% remaining; case analysis shows incorrect entity disambiguation and over-paraphrasing are primary sources of large model degradation.

Highlights & Insights¶

The paper excels at formalizing the deployment problem: budgeted hybrid translation optimizes marginal gain, not difficulty. This mathematical reframing explains why many heuristic routing methods fail.
Using only prefill hidden states for routing is highly practical. It avoids the additional latency of decoding-then-deciding and removes the need for an external QE model, meeting production requirements for simplicity.
The observation that the quality oracle is significantly lower than the gain oracle is persuasive: even if you know the absolute quality of the small model, you don't necessarily know if the large model is worth the invocation.
Guarded routing provides a realistic compromise: using lightweight pre-routing to control costs daily, and post-route verifiers in high-risk scenarios to mitigate severe regressions.

Limitations & Future Work¶

Training supervision relies on XCOMET-XXL reference metrics, which may inherit automatic metric biases and may not perfectly represent user preferences or specific business utility.
The experiments only study the hybrid setup for two models and a fixed route-to-large budget; multi-level cascades, dynamic budgets, latency constraints, and cost fluctuations have not been deeply addressed.
The model combination is fixed to LMT-60-0.6B and LMT-60-8B; whether the same patterns hold for different model families, various capability gaps, or larger-scale models requires further verification.
Language directions only cover En-Zh, En-Ru, Zh-En, and Ru-En. Routing behavior in low-resource languages, morphologically rich languages, and multi-script scenarios may differ.

vs QE-based deferral: Post-routing QE requires generating small model translations before making a decision, leading to higher latency. RouteLMT performs routing before generation, making it more suitable for low-latency deployment.
vs external router: Methods like XLM-R and sentinel only look at external representations of the source sentence. RouteLMT utilizes internal representations of the small translation model, thereby capturing the model's own perception of translation difficulties.
vs difficulty routing: Difficult sentences are not always worth processing by a large model if both models are likely to fail. RouteLMT targets relative gain directly, providing a more accurate objective.
Insight: In LLM system deployment, a router should not merely ask "Is this request difficult?" but instead "How much quality can be gained by upgrading the model?" This gain-aware approach is transferable to other hybrid model services such as summarization, customer service, and code generation.

Rating¶

Novelty: ⭐⭐⭐⭐☆ The formalization of budgeted gain routing is clear, and the in-model representation prediction is practical; the overall setup continues existing routing research.
Experimental Thoroughness: ⭐⭐⭐⭐☆ Analysis across four directions, multiple routers, and risk analysis is comprehensive, though broader language coverage and human preference validation are missing.
Writing Quality: ⭐⭐⭐⭐☆ Motivation and formula derivations are smooth, and experimental tables directly support the conclusions.
Value: ⭐⭐⭐⭐☆ Highly valuable for the production deployment of machine translation, particularly for systems already utilizing small/large model pairs.