From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/NJU-RL/DARS
Area: LLM Evaluation
Keywords: LLM routing, routing supervision, distribution-aware evaluation, risk-aware utility, query rewriting
TL;DR¶
Addressing the fragility and bias of conventional single-shot supervision in LLM routing, this paper proposes DARS (Distribution-Aware Routing Supervision), which models query-level capability distributions via semantics-preserving rewrites and repeated stochastic decoding to construct a risk-aware utility that markedly boosts router effectiveness across architectures.
Background & Motivation¶
Large language models (LLMs) are increasingly deployed as heterogeneous pools rather than isolated monolithic systems. In real-world environments, lightweight models readily resolve routine instructions and factual queries at minimal latency and operational expenditure, while larger or specialized models are reserved for complex mathematical derivations, domain-specific science questions, or long-context reasoning. To optimize the trade-off between inference cost and response quality, dynamic query-dependent LLM routing has become an essential architectural mechanism. Nevertheless, conventional routing algorithms and benchmarks predominantly rely on a point-estimate supervision protocol: for each query-model pair, training targets and evaluation labels are constructed from a single sampled response, recording only its individual task score and token cost.
This single-shot assumption stands in direct conflict with the stochastic nature of free-form LLM generation. Because generation is subject to both decoding randomness and prompt formulation sensitivity, semantically equivalent queries or repeated sampling passes often yield wildly divergent scores. Empirical diagnostic analyses reveal that such stochastic fluctuations cause severe winner-flip anomalies, frequently altering which candidate model appears optimal for a given query. Crucially, this label-level instability does not cancel out as independent noise; instead, it propagates downstream into router training, inducing learned routing policies that overfit to incidental generation artifacts rather than reflecting robust differences in intrinsic model capability.
To resolve this fundamental mismatch, the authors shift the supervision paradigm from isolated sample observations to query-level capability distributions. By systematically accounting for input-side formulation variability and output-side generation stochasticity, the router is supervised not only with expected performance and expected expenditure, but also with an explicit measure of performance volatility. Core idea: shift LLM routing supervision from isolated sampled outcomes to query-level capability distributions, combining semantics-preserving query rewrites and repeated stochastic decoding to construct a risk-aware utility over expected quality, expected cost, and performance variability.
Method¶
Overall Architecture¶
DARS (Distribution-Aware Routing Supervision) decouples the construction of supervision signals from downstream router architectures. The pipeline proceeds through four sequential stages: Repeated Observation Construction, Distributional Capability Estimation, Risk-Aware Supervision Construction, and Routing Policy Learning. During dataset preparation, given an input query, DARS first generates multiple semantics-preserving rewritten queries using a strong LLM to introduce controlled input perturbations. For each candidate model in the pool and each rewrite variant, multiple stochastic decoding runs are performed to populate an observation matrix. These observations are aggregated into expected quality, expected inference cost, and performance variance, which are synthesized into a scalar risk-aware utility. Various router families (regression, classification, ranking, retrieval, and clustering) are then trained using these distributional signals. At inference time, the router accepts only the original raw query and deterministically selects the optimal model, introducing zero extra latency during serving.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input training query x and candidate model pool M"] --> B["Stage 1: Repeated Observation Construction<br/>Prompt rewrites and repeated stochastic decoding"]
B --> C["Stage 2: Distributional Capability Estimation<br/>Aggregating expected quality, cost, and variability"]
C --> D["Stage 3: Risk-Aware Supervision Construction<br/>Formulating scalar utility with cost and risk penalties"]
D --> E["Stage 4: Routing Policy Learning<br/>Adapting to regression, classification, retrieval, or clustering"]
E --> F["Test-time Deployment: Single-query input and greedy model selection"]
Key Designs¶
1. Repeated Observation Construction: Decoupling Input and Output Uncertainty To overcome the instability of single-point evaluation, DARS explicitly captures the two primary sources of query-level generation variance. On the input side, a strong instruction-following model (e.g., GPT-4o) generates \(N\) semantics-preserving rewrites \(\mathcal{X}(x) = \{x^{(1)}, \dots, x^{(N)}\}\) for each training query \(x\), strictly preserving the underlying task semantics, gold answers, and candidate options while altering syntax, phrasing, and formatting. On the output side, for every candidate model \(m \in \mathcal{M}\) and each rewrite variant \(x^{(n)}\), the system executes \(M\) independent stochastic decoding passes at a standard sampling temperature (\(T=0.7\)). This yields an observation matrix \(\mathcal{O}_{x,m} = \{(q_m^{(n,j)}, c_m^{(n,j)})\}_{n=1,j=1}^{N,M}\) consisting of \(N \times M\) entries per query-model pair, recording task-specific quality scores and normalized inference costs.
2. Distributional Capability Estimation: Multi-Statistic Aggregation Rather than extracting isolated scores, DARS aggregates the observation set into summary statistics that capture full query-level capability. Expected quality \(\mu_q(x, m)\) and expected inference cost \(\mu_c(x, m)\) are computed as empirical expectations over all rewrites and decoding runs: $\(\mu_q(x, m) = \frac{1}{NM} \sum_{n=1}^N \sum_{j=1}^M q_m^{(n,j)}, \quad \mu_c(x, m) = \frac{1}{NM} \sum_{n=1}^N \sum_{j=1}^M c_m^{(n,j)}\)$ To quantify sensitivity to phrasing alterations and sampling variance, DARS computes the empirical standard deviation of quality scores as an aggregate performance risk metric: $\(\sigma_q(x, m) = \sqrt{\frac{1}{NM-1} \sum_{n=1}^N \sum_{j=1}^M \left(q_m^{(n,j)} - \mu_q(x, m)\right)^2}\)$ Estimating variance over the joint observation set naturally accounts for the statistical coupling between input reformulations and subsequent decoding dispersion, outperforming naive additive combinations of separate input and output variances.
3. Risk-Aware Supervision Construction: Cost and Robustness Trade-offs To guide routers toward models that balance high accuracy, low cost, and consistent behavior, DARS synthesizes these distributional estimates into a scalar risk-aware utility \(U(x, m)\): $\(U(x, m) = \mu_q(x, m) - \lambda \mu_c(x, m) - \beta \sigma_q(x, m)\)$ where \(\lambda > 0\) (default 0.05) penalizes expensive model invocations, and \(\beta \ge 0\) (default 0.2) penalizes erratic performance variability. The target optimal model is defined as: $\(m^*(x) = \arg\max_{m \in \mathcal{M}} U(x, m)\)$ This formulation penalizes models whose single-shot high scores stem from favorable sampling flukes, steering selection toward candidates that consistently deliver correct answers across varied phrasings.
4. Routing Policy Learning: Paradigm-Agnostic Supervision Adaptation Because DARS produces model-wise capability statistics rather than enforcing a specific neural architecture, it adapts flexibly to diverse router paradigms. In regression-based routers (e.g., MLPRouter), multi-head regressors directly predict \(\mu_q\), \(\mu_c\), and \(\sigma_q\) to compute predicted utility at runtime; in direct utility models, the scalar \(U(x, m)\) serves as the continuous regression target. For deterministic classification, \(m^*(x)\) provides a stable hard class label. For probabilistic classification routers (e.g., RM-Softmax), utilities are mapped to a soft target distribution via temperature-scaled Softmax: \(p_{\text{DARS}}(m|x) = \frac{\exp(U(x, m)/\tau)}{\sum_{m'} \exp(U(x, m')/\tau)}\), preserving fine-grained preferences among competitive models. For non-parametric retrieval (kNNRouter) and clustering routers (AvengersPro), DARS statistics are stored within exemplar query representations or cluster centroids for distance-weighted aggregation.
Loss & Training¶
For parametric neural routers, training objectives align with the chosen learning paradigm. Regression-based models optimize parameters \(\theta\) by minimizing the mean squared error against distributional targets: $\(\mathcal{L}_{\text{reg}}(\theta) = \frac{1}{|\mathcal{D}||\mathcal{M}|} \sum_{x \in \mathcal{D}} \sum_{m \in \mathcal{M}} \left( \hat{U}_\theta(x, m) - U(x, m) \right)^2\)$ Probabilistic classification routers optimize cross-entropy loss against the temperature-softened DARS target distribution: $\(\mathcal{L}_{\text{cls}}(\theta) = -\frac{1}{|\mathcal{D}|} \sum_{x \in \mathcal{D}} \sum_{m \in \mathcal{M}} p_{\text{DARS}}(m|x) \log \hat{p}_\theta(m|x)\)$ To reduce offline sampling expenditures, DARS supports adaptive budget allocation. Starting from a \(2 \times 2\) grid, queries expand to \(3 \times 3\) or \(5 \times 5\) only when estimated risk \(R(x)\) exceeds the first quartile and the utility gap \(\Delta(x)\) between top candidate models falls below the third quartile. On GPQA, this adaptive mechanism terminates 25% of training queries at the \(2 \times 2\) stage, reducing candidate model queries by 13.9% (average 7.75 observations per pair) with negligible impact on final routing utility.
Key Experimental Results¶
Main Results¶
The evaluation uses a heterogeneous candidate pool of six LLMs (Gemma-3-12B-IT, Mistral-Small-3.2-24B-Instruct, Qwen3-32B, Llama-3.3-70B-Instruct, Gemini-2.5-Flash-Lite, DeepSeek-Chat-V3.1) across GPQA, MATH-500, and DROP-800 with cost coefficient \(\lambda = 0.05\). Test queries evaluate rewrite-based utility \(U_{\text{rew}}\) (across 3 semantic rewrites) and decoding-based utility \(U_{\text{dec}}\) (across 3 stochastic decoding runs). Single-shot results report the mean performance over 100 independently sampled training sets.
| Router | Dataset | Single-shot \(U_{\text{rew}}\) | DARS \(U_{\text{rew}}\) | Single-shot \(U_{\text{dec}}\) | DARS \(U_{\text{dec}}\) | Highlighted Improvements |
|---|---|---|---|---|---|---|
| MLP | DROP-800 | 0.661 | 0.675 | 0.709 | 0.726 | Consistent gains across both views |
| GPQA | 0.382 | 0.399 | 0.400 | 0.413 | Marked improvement on difficult QA | |
| MATH-500 | 0.682 | 0.687 | 0.698 | 0.696 | Competitive on constrained math | |
| kNN | DROP-800 | 0.713 | 0.749 | 0.759 | 0.791 | Non-parametric gain (+0.036 on \(U_{\text{rew}}\)) |
| GPQA | 0.441 | 0.461 | 0.469 | 0.489 | Retrieval stability notably enhanced | |
| MATH-500 | 0.720 | 0.730 | 0.733 | 0.742 | Solid improvements across both views | |
| EmbedLLM | DROP-800 | 0.739 | 0.763 | 0.785 | 0.805 | Smoother learned model embeddings |
| GPQA | 0.469 | 0.473 | 0.496 | 0.480 | High robustness under noisy inputs | |
| MATH-500 | 0.706 | 0.719 | 0.728 | 0.747 | Significant decoding gain (+0.019) | |
| AvengersPro | DROP-800 | 0.731 | 0.762 | 0.775 | 0.807 | Cluster-level centroids de-noised |
| GPQA | 0.464 | 0.480 | 0.490 | 0.496 | Robust clustering against label flips | |
| MATH-500 | 0.720 | 0.734 | 0.736 | 0.742 | Systematic utility gains | |
| GraphRouter | DROP-800 | 0.762 | 0.765 | 0.805 | 0.807 | High baseline performance maintained |
| GPQA | 0.462 | 0.496 | 0.487 | 0.516 | Largest absolute jump on GPQA (+0.034 / +0.029) | |
| MATH-500 | 0.746 | 0.757 | 0.755 | 0.758 | Outperforms best single-model alternatives | |
| MIRT | DROP-800 | 0.687 | 0.705 | 0.734 | 0.751 | Item Response Theory parameters stabilized |
| GPQA | 0.409 | 0.444 | 0.433 | 0.468 | Sharp recovery on difficult questions | |
| MATH-500 | 0.697 | 0.759 | 0.714 | 0.760 | Massive mathematical boost (+0.062 on \(U_{\text{rew}}\)) | |
| RM-Softmax | DROP-800 | 0.740 | 0.761 | 0.787 | 0.804 | Regret minimization boundary refined |
| GPQA | 0.453 | 0.477 | 0.475 | 0.512 | Substantial gain on decoding stability | |
| MATH-500 | 0.721 | 0.731 | 0.746 | 0.749 | Cost-aware classification enhanced |
Ablation Study¶
A component ablation on the MIRT routing architecture systematically isolates the impact of distributional inputs, stochastic outputs, and risk modeling.
| Variant | DROP-800 \(U_{\text{rew}}\) | DROP-800 \(U_{\text{dec}}\) | GPQA \(U_{\text{rew}}\) | GPQA \(U_{\text{dec}}\) | MATH-500 \(U_{\text{rew}}\) | MATH-500 \(U_{\text{dec}}\) | Overall Average Utility |
|---|---|---|---|---|---|---|---|
| Single-shot (Baseline) | 0.687 | 0.734 | 0.409 | 0.433 | 0.697 | 0.714 | 0.612 |
| Input-only DA (Rewrites only) | 0.695 | 0.742 | 0.423 | 0.452 | 0.735 | 0.742 | 0.632 |
| Output-only DA (Decoding only) | 0.693 | 0.747 | 0.425 | 0.442 | 0.723 | 0.718 | 0.625 |
| DARS w/o Risk (Mean stats only) | 0.692 | 0.743 | 0.413 | 0.441 | 0.746 | 0.730 | 0.626 |
| DARS w/ Decomp. Risk (Additive variances) | 0.705 | 0.731 | 0.435 | 0.456 | 0.754 | 0.749 | 0.640 |
| Full DARS (Joint distribution & risk) | 0.705 | 0.751 | 0.444 | 0.468 | 0.759 | 0.760 | 0.648 |
Key Findings¶
- Broad gains across 40 out of 42 evaluation settings: DARS universally outperforms single-shot supervision across seven distinct router architectures, confirming that single-shot fragility is an architectural-agnostic issue stemming directly from supervision design.
- Heterogeneous uncertainty profiles across tasks: GPQA is heavily dominated by output-side decoding stochasticity (output variance 0.099 vs. input variance 0.018; winner flip rate 0.970), whereas DROP-800 displays greater sensitivity to input phrasing (input variance 0.0428 vs. output variance 0.0350). Relying on a single perturbation source is insufficient.
- Joint variance modeling outperforms additive decomposition: Full DARS (0.648 average) surpasses the decomposed risk variant (0.640 average). Because prompt rewrites reshape the downstream generation distribution and alter sampling entropy, input and output uncertainties are tightly coupled rather than independent.
- Early saturation in observation budgets: Increasing the grid from \(1 \times 1\) to \(3 \times 3\) captures the vast majority of performance gains, while expanding to \(5 \times 5\) yields diminishing returns. Combining this with adaptive sampling cuts invocation overhead by 13.9% while matching full grid utility.
Highlights & Insights¶
- Rethinking the foundations of routing supervision: Uncovers that conventional single-response labeling introduces severe winner flip rates (up to 0.970 on GPQA), demonstrating that single-point evaluations propagate substantial noise into learned policies.
- Plug-and-play supervision framework: DARS modifies only offline training targets without altering router inference mechanics or runtime overhead, ensuring immediate compatibility with existing regression, classification, and retrieval pipelines.
- Value-at-Risk integration into neural routing: Incorporates performance volatility as an explicit penalty alongside inference cost, inherently steering routers away from erratic models that succeed only by sampling chance.
Limitations & Future Work¶
- Offline observation collection overhead: Populating an \(N \times M\) observation matrix incurs multiple LLM forward passes during training data collection, which remains computationally heavier than single-shot labeling despite adaptive pruning.
- Single-turn scope: The current framework is validated strictly on single-turn query routing; extending distribution-aware capability estimation to multi-turn conversational agents and interactive tool execution remains an open challenge.
- Dynamic model pools: When models are frequently added, deprecated, or hot-swapped in production pools, maintaining up-to-date capability grids may require lightweight surrogate estimators or distillation techniques.
Related Work & Insights¶
- vs RouteLLM / FrugalGPT: Early routing strategies pioneered cost-quality trade-offs and cascade mechanisms but relied on single-shot response scores; DARS operates orthogonally by replacing fragile point targets with robust capability distributions.
- vs GraphRouter / IRT-Router: Advanced routing models enhance structural representation via graph edges or latent ability parameters; DARS directly improves the supervision quality feeding these models, unlocking substantial compounding gains.
- vs Semantic Entropy / SelfCheckGPT: Uncertainty estimation techniques leverage multiple generations to detect hallucinations; DARS adapts this uncertainty insight to model selection, translating response dispersion into an actionable risk penalty.
Rating¶
- Novelty: โญโญโญโญโ Pinpoints the single-shot supervision bottleneck in LLM routing and introduces a principled distribution-aware utility framework.
- Experimental Thoroughness: โญโญโญโญโญ Rigorous validation across six diverse LLMs, three reasoning benchmarks, and seven router families with in-depth diagnostic ablations.
- Writing Quality: โญโญโญโญโญ Exceptionally clear narrative, clean mathematical formulation, and compelling empirical evidence.
- Value: โญโญโญโญโญ Sets a new reliability standard for LLM routing supervision, facilitating dependable deployment of heterogeneous model fleets.