RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization¶

Conference: ACL2026
arXiv: 2604.23577
Code: https://github.com/bettyguo/RouteNLP
Area: Model Compression / LLM Routing / Efficient Deployment
Keywords: LLM Routing, Conformal Cascading, Knowledge Distillation, Cost Optimization, Enterprise Deployment

TL;DR¶

RouteNLP is a closed-loop LLM routing and cascading framework. It co-optimizes model ensembles using task-aware routers, conformal calibrated cascading, and failure-cluster-targeted distillation. On a six-task enterprise benchmark, it achieves a cost ratio of 0.159 with a quality ratio of 0.971 and successfully saved 58% of inference costs while maintaining a 91% response acceptance rate in an 8-week customer service pilot.

Background & Motivation¶

Background: Enterprise NLP services typically utilize multiple model tiers: lightweight classifiers, small open-source LLMs, medium MoE models, and expensive frontier APIs. Difficulty varies significantly across requests; while many routine queries do not require the strongest models, critical business requests must satisfy strict quality and latency constraints.

Limitations of Prior Work: Existing LLM routing and cascading methods are mostly evaluated on single benchmarks and rarely consider multi-tasking, SLAs, tail latency, or the evolution of model ensembles in production. crucially, they often treat model ensembles as fixed: the router learns to assign requests to existing models but does not modify cheap models based on routing failures.

Key Challenge: Pure routing can only save costs within existing capability boundaries. If cheap models are systematically deficient in certain high-frequency failure clusters, requests will continuously escalate to expensive models. True cost optimization should be closed-loop: identifying escalation failure patterns, performing targeted distillation on cheaper models, and retraining the router and thresholds.

Goal: The authors propose RouteNLP, integrating a difficulty-aware router, confidence-calibrated cascading, and distillation-routing co-optimization into a production-oriented framework aimed at minimizing costs and SLA violations under quality constraints for each task.

Key Insight: The paper stems from an actual enterprise scenario where partner inference costs exceeded $200,000 per month, yet over 70% of queries were routine tasks. The authors exploit this heavy-tailed difficulty distribution to assign requests to the cheapest model capable of meeting quality thresholds, using failure logs to drive subsequent distillation.

Core Idea: Treat LLM serving as a closed-loop system of "routing + calibrated cascading + ensemble evolution" rather than a one-time router training. Cheaper models gradually absorb high-frequency failure clusters through targeted distillation, allowing more requests to stay within low-cost tiers.

Method¶

RouteNLP assumes a model ensemble $M=\{m1,...,mK\}$ sorted by increasing cost, with each task having a quality threshold $\tau_t$. The system minimizes the cumulative cost required to process requests while ensuring final output quality meets task requirements. It achieves this via three parts: predicting the cheapest viable tier, using uncertainty to decide on escalation, and clustering escalation logs into distillation data to improve low-tier models.

Overall Architecture¶

The ensemble consists of four tiers: T1 is DistilBERT (~$0.01/1K tokens); T2 is Mistral-7B-Instruct (~$0.10/1K tokens); T3 is a quantized Mixtral-8x7B (~$0.80/1K tokens); T4 is the GPT-4-Turbo API (~$8.00/1K tokens). The router first assigns a tier; if the token-level uncertainty after generation exceeds a conformal threshold, the request cascades to the next tier.

Training labels are obtained via offline evaluation of all models on all queries. Quality metrics vary by task: F1 or accuracy for structured tasks, and ROUGE-L or BERTScore for generative tasks. The system is evaluated on a six-task enterprise benchmark covering financial NER/summarization, customer service intent/reply, and legal clause extraction/risk assessment, totaling 40,200 training and 8,800 test samples.

Key Designs¶

Task-Aware Difficulty Router:
- Function: Predicts the minimum acceptable model tier for a given task query.
- Mechanism: Uses DistilBERT-base as a lightweight router, concatenating the [CLS] representation with a 64-dimensional task embedding. A task projection head outputs 4 tier logits. The training loss consists of tier classification, a cost term, and a quality constraint hinge penalty, with $\lambda_c=0.3$ and $\lambda_q=0.5$.
- Design Motivation: Difficulty patterns differ across financial entity extraction, customer service replies, and legal risk assessment. A shared encoder reuses language representations, while task embeddings allow the router to learn task-conditioned difficulty boundaries.
Conformal Confidence-Calibrated Cascading:
- Function: Provides a safety net when the router underestimates difficulty, preventing low-quality outputs from low tiers from being returned.
- Mechanism: 500 calibration samples per task and tier are used to estimate uncertainty thresholds. After generation, token-level uncertainty is calculated as $u=1/L \sum_i (1-p(y_i|y_{<i,x}))$. If $u>\delta_{k,t}$, the request cascades. Thresholds are set via conformal risk control with a target marginal violation rate $\alpha=0.05$.
- Design Motivation: While the router provides a prior judgment, post-generation confidence captures actual output risk. Conformal thresholds provide distribution-free initialization, though the authors note they only guarantee marginal coverage.
Failure Cluster-Driven Distillation-Routing Co-Optimization:
- Function: Enables cheap models to gradually learn to handle high-frequency escalation failures.
- Mechanism: Escalation logs are collected, and router hidden representations are extracted and reduced to 128D via PCA, followed by k-means clustering per task. Clusters are ranked by size multiplied by average quality gap. The top-5 clusters use frontier model outputs as teacher labels for SeqKD on T1 through T3. The router is then retrained and thresholds recalibrated.
- Design Motivation: Random distillation wastes data on samples already handled correctly. Failure clustering identifies systematic weaknesses, providing greater cost reduction with the same amount of data.

Loss & Training¶

The router loss is $L=L_{route}+\lambda_c L_{cost}+\lambda_q L_{quality}$. $L_{route}$ is the cross-entropy for tier classification with labels from full ensemble evaluation; $L_{cost}$ encourages low-cost tiers; $L_{quality}$ applies a hinge penalty when the predicted tier falls below the quality threshold. The distillation loop uses a convergence threshold $\epsilon=0.005$, typically converging in 2-3 rounds. The router has ~67M parameters and trains in ~45 minutes on an A100.

Key Experimental Results¶

Main Results¶

RouteNLP achieves significantly lower costs than Hybrid LLM while maintaining nearly identical quality and drastically reducing SLA violations.

System	Quality Ratio	Cost Ratio	p99 Latency	SLA Violation	Description
Always-T4	1.000	1.000	1847 ms	38.2%	Quality upper bound; highest cost/latency
Always-T2	0.891	0.013	142 ms	0.1%	Low cost but fails quality standards
Random	0.924	0.252	623 ms	12.4%	Unreliable allocation
FrugalGPT	0.967	0.284	986 ms	21.3%	Cascading saves money but worse SLA
Hybrid LLM	0.972	0.312	874 ms	18.7%	Close quality but higher cost
RouteLLM	0.969	0.246	841 ms	17.2%	Preference router baseline
AutoMix	0.958	0.231	1124 ms	24.6%	POMDP-style hybrid model
RouteNLP	0.971	0.159	387 ms	2.3%	Lowest cost with significantly lower SLA violations

Ablation Study¶

Configuration	Quality Ratio	Cost Ratio	T1+T2 Share	T4 Share	Description
Iter 0 (Initial)	0.961	0.203	68%	11%	Initial router and thresholds
Iter 1	0.964	0.178	74%	8%	More requests enter low tiers after distillation
Iter 2	0.969	0.163	79%	6%	Quality continues to recover, cost drops
Iter 3 (Final)	0.971	0.159	81%	5%	Convergence after three rounds

Key Findings¶

Removing cascading results in a 1.9-point quality drop, proving the necessity of the confidence safety net. Removing co-optimization increases costs by 28%, showing that routing alone cannot fully minimize costs.
Compared to random distillation, targeted distillation reduced the cost ratio from 0.203 to 0.159 (21.7% improvement) with the same data volume; random distillation only reached 0.184 (9.4% improvement).
In structured tasks, costs dropped by 78-85% while retaining ~99% quality; in generative tasks, costs dropped by 40-47% with ~96% quality retention.
Human evaluation of 400 generative samples showed 74.5% of routed outputs were equal to or better than T4.
An 8-week customer service pilot (~5K queries/day) showed a real-world cost reduction of 58% with a 4.8% violation rate, close to the 62% prediction from simulations.

Highlights & Insights¶

The major highlight is treating the "model ensemble as a learnable object." While typical routers only learn assignment, RouteNLP uses failure logs to evolve cheap models, which is more representative of long-running enterprise systems.
The use of conformal calibration is presented honestly: the authors specify that guarantees are marginal and depend on exchangeability, noting that domain shifts pushed violations to 8.1%.
Pilot deployment results significantly enhance credibility. Although it was a shadow deployment, 8 weeks of data with 5K queries/day provides stronger evidence than pure simulation.
Failure mode analysis: Multi-step reasoning (42%), domain knowledge (31%), and difficulty ambiguity (27%) were the primary causes. The pilot also identified OCR artifacts and multi-turn references as better suited for escalation than distillation.

Limitations & Future Work¶

The pilot only covered customer service; financial and legal results rely primarily on benchmark simulations.
The co-optimization loop runs on benchmark data rather than production failure logs; new failure modes in the pilot did not perfectly align with benchmark clusters.
As a shadow deployment rather than a randomized A/B test, causal attribution of cost and complaint rate changes is limited.
Conformal coverage worsened from a 5% target to 8.1% under domain shift, suggesting a need for online threshold adaptation.
Evaluation was limited to English; BERTScore consistency for out-of-distribution quality remains unvalidated.
One co-optimization cycle costs ~$2,400, which might not be amortized in low-volume deployments.

vs. FrugalGPT / Hybrid LLM: These focus on call order or binary routing. RouteNLP adds multi-tasking, SLA, conformal calibration, and a distillation loop, making it more practical for complex enterprise ensembles.
vs. RouteLLM: RouteLLM uses preference data for routing but does not evaluate actual business quality of outputs nor modify the ensemble.
vs. Model Compression: Traditional distillation is a one-time compression. RouteNLP performs continuous targeted distillation based on routing failure clusters, aiming to cover high-frequency business gaps rather than creating a universal small model.
Insight: The key to efficient LLM deployment may not be "selecting the optimal small model," but rather building a continuous learning serving system: monitoring failures, clustering them, targeted distillation, recalibrating thresholds, and redeploying.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Routing, cascading, conformal, and distillation are existing modules, but the closed-loop combination and production validation are highly novel.
Experimental Thoroughness: ⭐⭐⭐⭐☆ Solid across six-task benchmarks, ablations, human evaluation, and an 8-week pilot; limited by the non-A/B pilot and English-only scope.
Writing Quality: ⭐⭐⭐⭐☆ Clear engineering details, cost models, and limitations.
Value: ⭐⭐⭐⭐⭐ High reference value for enterprise LLM cost optimization, ensemble governance, and low-cost serving architectures.