Rubric-Guided Process Reward for Stepwise Model Routing¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Stepwise Model Routing, Process Reward, Rubric-based Evaluation, Large Reasoning Models, Reinforcement Learning
TL;DR¶
Addressing the deceptive alignment trap of outcome-only rewards in stepwise model routing, RoRo trains a Rubricor and a Judge via alternating optimization and statistical validation gates to deliver dense process rewards, significantly boosting reasoning accuracy under tight compute budgets via GRPO.
Background & Motivation¶
Large Reasoning Models (LRMs) have demonstrated exceptional capabilities across challenging mathematical reasoning, formal logic, and complex multi-hop question answering. However, their superior performance largely stems from generating extended, explicit chain-of-thought trajectories that recursively decompose problems into intermediate stages. This computational paradigm incurs prohibitive inference latency and execution costs, presenting severe deployment bottlenecks in real-time or resource-constrained production environments. To reconcile reasoning quality with inference efficiency, researchers have turned to stepwise model routing, operating on the insight that individual steps within a single reasoning trajectory differ substantially in intrinsic difficulty. In typical reasoning paths, only a handful of critical pivot steps demand the full power of an expensive LRM, whereas routine computational or paraphrasing steps can be seamlessly delegated to a lightweight Small Reasoning Model (SRM).
Early routing paradigms primarily estimated single-step uncertainty or confidence in isolation, triggering LRM escalation whenever local heuristics exceeded fixed thresholds. Recent advances, notably TRIM, formalize routing as a sequential decision process and train an adaptive routing policy via reinforcement learning, allowing the router to condition its choices on both prefix trajectory history and cumulative resource consumption. Nevertheless, a glaring conceptual mismatch persists: while these methods model routing as a sequential process, they continue to optimize the policy almost exclusively using outcome-level supervision, such as terminal answer correctness and global token expenditure. Extensive empirical evidence in reward modeling indicates that outcome-only feedback is vulnerable to deceptive alignment: policies can adopt erratic or brittle model-switching behaviors that accidentally arrive at the correct final answer on training queries, while failing to acquire genuine, generalizable routing competence.
The fundamental hurdle in establishing process-level supervision for model routing is the total absence of gold standard routing labels, as there is no deterministic ground truth specifying whether a model escalation at step \(i\) was optimal. Inspired by recent breakthroughs in rubric-based reinforcement learning for chain-of-thought generation, this work explores using natural language criteria to evaluate non-verifiable routing dynamics. Core idea: collect diverse routing trajectories to construct multi-faceted preference pairs, train a Rubricor and a Judge via alternating optimization to generate and score query-specific routing rubrics under strict statistical validation gates, and leverage the resulting dense process rewards alongside outcome feedback to optimize a lightweight routing policy with GRPO.
Method¶
Overall Architecture¶
The RoRo framework comprises three sequential phases: routing preference data construction, rubric-guided process reward modeling, and process-aware policy optimization via GRPO. During inference, both the heavy Rubricor and Judge are frozen and completely omitted from the active serving path; a compact 2-layer MLP routing policy inspects a 5-dimensional feature summary from the SRM candidate draft and dynamically decides whether to accept the step or escalate to the LRM for regeneration.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
Q["Input Query & Diverse Trajectory Pool<br/>SRM/LRM/Random/Entropy/Confidence/Outcome-only"] --> D1["1. Route Preference Data Construction<br/>Outcome / Cost / Seed Rubric Tri-Rule Pairing"]
D1 --> D2["2. Rubricor & Judge Alternating Optimization<br/>Rubricor Generates Candidate Rubrics ↔ BT Judge Scoring"]
D2 --> D3["3. Statistical Validation Gate<br/>Partial Correlation / Score Variance / Outcome Leakage"]
D3 --> D4["4. Joint Process-Outcome GRPO Policy Optimization<br/>Frozen G & J, Group-Normalized Advantages on 2-Layer MLP"]
D4 --> OUT["Inference Deployment: Ultra-Lightweight Stepwise Router"]
Key Designs¶
1. Route Preference Data Construction: Multi-policy Trajectory Pooling and Cascaded Tri-rule Pairing To overcome the absence of gold routing labels, RoRo pools routing trajectories for each query across six heterogeneous policies: SRM-only, LRM-only, random routing, entropy-threshold routing, confidence-threshold routing, and an outcome-only RL router. From this diverse trajectory pool \(\mathcal{T}_q\), preference pairs \((\tau^+, \tau^-)\) are established via a cascaded three-tier rule: (1) outcome preference prioritizes trajectories yielding correct final answers; (2) cost preference selects the trajectory with lower LRM token consumption when both answers are correct; and (3) process preference resolves ties in outcome and cost using an expert-designed seed rubric (\(r_{\text{seed}}\)), preferring trajectories that demonstrate timely escalation at initial error points and effective trajectory recovery.
2. Rubricor & Judge Alternating Optimization: Query-Adaptive Natural Language Criteria and Bradley-Terry Evaluation Because a fixed, static rubric fails to capture the idiosyncratic reasoning demands of diverse problem domains, RoRo co-trains a Rubricor \(G_\psi\) and a Judge \(J_\phi\) (both parameterized by Qwen3-8B). After warming up the Judge on seed rubrics via a Bradley-Terry objective, the models undergo alternating optimization: $\(L_J = -\log \sigma \left( J_\phi(q, \tau^+, r_{\text{seed}}) - J_\phi(q, \tau^-, r_{\text{seed}}) \right)\)$ In iteration \(t\), Phase 1 freezes the Judge and prompts the Rubricor to generate \(M\) candidate rubrics \(r_{q,m}\); each rubric's scalar reward \(\rho_{q,m} = \log \sigma(\bar{\Delta} J_{q,m}^{(t)})\) is defined by the Judge's margin in separating preferred from dispreferred trajectories, updating \(G_\psi\) via policy gradient. In Phase 2, the Rubricor is frozen to sample validated query-specific rubrics \(\tilde{r}_q^{(t)}\), and the Judge is refined using preference pairs scored under \(\tilde{r}_q^{(t)}\). This co-evolution prevents rubric stagnation and enables the Judge to assign calibrated continuous process scores.
3. Statistical Validation Gate: Eliminating Outcome Leakage and Spurious Intermediate Signals Candidate natural-language criteria generated by LLMs are susceptible to hallucination, trivial tautologies, or outcome leakage (evaluating final answer correctness rather than routing quality). RoRo routes all sampled candidate rubrics through a rigorous three-stage statistical filter evaluated on 1,200 held-out rollouts across 200 validation queries: - Partial Correlation Significance: Controls for answer correctness and normalized LRM cost using partial Pearson correlation, retaining only criteria with two-sided \(p < 0.05\) after Holm-Bonferroni correction across all criteria in the candidate rubric. - Score Variance Floor: Enforces a minimum standard deviation \(\sigma_{\min} = 0.05\) across rollouts to prune non-discriminative, flat criteria. - Outcome Leakage Ceiling: Estimates mutual information \(MI(v_i; y)\) between criterion scores and binary outcome labels using the KSG estimator (\(k=5\)), strictly discarding any criterion where \(MI > 0.1\text{ nats}\). If fewer than two criteria pass all three tests, the entire candidate rubric is discarded, dropping the discard rate from 12% in early rounds to below 5% at convergence.
4. Joint Process-Outcome GRPO Policy Optimization: Value-Free Group Advantage Normalization During router policy training, \(G_\psi\) and \(J_\phi\) are frozen. For each training query, the router samples a rollout group of \(K=8\) trajectories \(\{\tau^{(k)}\}_{k=1}^K\). The Rubricor provides a shared validated rubric \(\tilde{r}_q\), under which the Judge scores each rollout to generate process rewards \(R_{\text{process}}^{(k)}\). The total trajectory reward balances correctness, computational cost, and process quality: $\(R_{\text{total}}^{(k)} = R_{\text{outcome}}^{(k)} - \lambda C^{(k)} + \beta R_{\text{process}}^{(k)}\)$ where \(C^{(k)}\) is the LRM token count, \(\lambda = 0.001\), and \(\beta = 0.5\). By applying Group Relative Policy Optimization (GRPO), trajectory-level advantages \(A^{(k)} = (R_{\text{total}}^{(k)} - \mu_K) / \sigma_K\) are calculated directly from intra-group statistics without training a separate Value/Critic network. The lightweight 2-layer MLP router (128 hidden units) is updated via PPO-style clipped objectives.
Loss & Training¶
The routing policy takes a 5-dimensional feature vector: the current step's candidate uncertainty (average token entropy from the drafting model), the prefix minimum uncertainty, the prefix average uncertainty, the step token count normalized by a constant, and the normalized step index. Training is conducted on the MATH training split with temperature 0.7, top-p 0.95, and maximum rollout horizon of 16,384 tokens. Crucially, inference requires zero additional LM token generation for routing decisions, maintaining minimal memory and latency footprints.
Key Experimental Results¶
Main Results¶
RoRo is evaluated under both same-family (Qwen3-1.7B SRM with Qwen3-14B LRM) and cross-family (Qwen3-1.7B SRM with DeepSeek-R1-Distill-Qwen-14B LRM) configurations across five benchmarks: MATH-500, AIME 2025, OmniMath (in-domain math reasoning), and GSM8K, GPQA (out-of-domain generalization). Budgeted Accuracy (BA) is reported at 20%, 40%, and 60% of LRM-only FLOPs.
| Model Setup / Method | MATH-500 (@20/@40/@60) | AIME 2025 (@20/@40/@60) | OmniMath (@20/@40/@60) | GSM8K (@20/@40/@60) | GPQA (@20/@40/@60) | Average (@20/@40/@60) |
|---|---|---|---|---|---|---|
| Same-family (Qwen3-1.7B + Qwen3-14B) | ||||||
| SRM-only (1.7B) | 68.6 / - / - | 32.4 / - / - | 51.5 / - / - | 87.3 / - / - | 32.8 / - / - | - / - / - |
| LRM-only (14B) | 86.2 / - / - | 66.6 / - / - | 63.4 / - / - | 97.0 / - / - | 52.3 / - / - | - / - / - |
| Random | 70.4 / 74.2 / 76.1 | 35.4 / 43.1 / 50.8 | 52.7 / 55.0 / 56.3 | 88.3 / 90.3 / 92.3 | 33.4 / 36.5 / 39.7 | 56.0 / 59.8 / 63.4 |
| RSD | 72.8 / 75.8 / 79.2 | 36.9 / 45.8 / 53.7 | 54.7 / 57.6 / 59.8 | 89.5 / 93.5 / 95.3 | 35.7 / 41.6 / 46.2 | 57.9 / 62.9 / 66.8 |
| SpecCoT | 73.4 / 77.8 / 78.8 | 37.3 / 44.9 / 59.4 | 52.2 / 56.4 / 58.3 | 89.8 / 91.5 / 93.7 | 34.2 / 38.1 / 42.5 | 57.4 / 61.7 / 66.5 |
| SpecReason | 73.9 / 76.9 / 79.4 | 37.6 / 45.5 / 56.2 | 57.0 / 58.2 / 60.1 | 90.3 / 92.4 / 94.2 | 34.6 / 39.0 / 43.3 | 58.7 / 62.4 / 66.6 |
| STEER | 74.3 / 77.4 / 79.9 | 37.1 / 45.9 / 57.5 | 54.1 / 57.3 / 62.5 | 91.2 / 93.6 / 95.0 | 35.1 / 40.5 / 44.8 | 58.4 / 62.9 / 67.3 |
| GlimpRouter | 76.0 / 78.0 / 80.1 | 38.2 / 47.6 / 58.8 | 55.2 / 57.5 / 60.3 | 92.3 / 94.2 / 95.4 | 35.8 / 42.0 / 46.5 | 59.5 / 63.9 / 68.2 |
| TRIM (RL baseline) | 75.1 / 77.5 / 79.8 | 34.8 / 45.3 / 60.3 | 55.9 / 57.2 / 60.0 | 94.1 / 95.5 / 95.8 | 36.4 / 43.6 / 47.0 | 59.3 / 63.8 / 68.6 |
| RoRo (Ours) | 75.5 / 79.8 / 81.9 | 40.6 / 47.0 / 60.5 | 56.5 / 59.9 / 62.8 | 95.2 / 96.1 / 96.5 | 37.2 / 44.6 / 48.6 | 61.0 / 65.5 / 70.0 |
| Cross-family (Qwen3-1.7B + DeepSeek-R1-14B) | ||||||
| SRM-only (1.7B) | 68.6 / - / - | 32.4 / - / - | 51.5 / - / - | 87.3 / - / - | 32.8 / - / - | - / - / - |
| LRM-only (DeepSeek-14B) | 82.6 / - / - | 65.1 / - / - | 62.7 / - / - | 94.8 / - / - | 48.6 / - / - | - / - / - |
| Random | 69.8 / 73.5 / 75.4 | 34.6 / 42.0 / 49.5 | 52.0 / 54.3 / 56.6 | 87.9 / 89.8 / 91.6 | 30.2 / 33.8 / 37.1 | 54.9 / 58.7 / 62.2 |
| RSD | 71.5 / 74.3 / 77.8 | 36.0 / 44.3 / 52.2 | 53.4 / 56.1 / 58.5 | 88.7 / 91.8 / 93.9 | 32.4 / 37.6 / 42.3 | 56.4 / 60.8 / 64.9 |
| SpecCoT | 72.1 / 75.8 / 78.0 | 36.4 / 43.5 / 56.8 | 51.6 / 55.4 / 57.2 | 89.2 / 91.2 / 93.5 | 31.5 / 36.0 / 41.2 | 56.2 / 60.4 / 65.3 |
| SpecReason | 72.6 / 75.2 / 78.4 | 37.0 / 44.6 / 53.8 | 52.8 / 55.7 / 58.0 | 89.5 / 91.9 / 93.5 | 35.3 / 37.0 / 41.6 | 57.4 / 60.9 / 65.1 |
| STEER | 73.1 / 75.8 / 78.8 | 36.5 / 44.2 / 55.9 | 53.2 / 56.2 / 58.6 | 90.1 / 92.3 / 93.8 | 32.6 / 38.0 / 42.5 | 57.1 / 61.3 / 65.9 |
| GlimpRouter | 73.4 / 78.8 / 79.2 | 39.6 / 45.0 / 57.0 | 54.0 / 56.6 / 59.0 | 90.9 / 92.7 / 93.8 | 33.0 / 39.2 / 43.2 | 58.2 / 62.5 / 66.4 |
| TRIM (RL baseline) | 73.8 / 76.2 / 78.5 | 34.0 / 44.0 / 58.2 | 54.6 / 56.3 / 58.8 | 91.6 / 93.1 / 94.0 | 33.4 / 40.3 / 44.0 | 57.5 / 62.0 / 66.7 |
| RoRo (Ours) | 74.2 / 78.2 / 80.6 | 38.9 / 45.7 / 59.0 | 55.3 / 58.5 / 61.1 | 92.4 / 93.6 / 94.2 | 34.7 / 41.8 / 46.1 | 59.1 / 63.6 / 68.2 |
Ablation Study¶
The ablation study isolates the contributions of process rewards, the learned Rubricor, the learned Judge, and statistical validation on MATH-500 and OmniMath (averaged across BA@20, @40, and @60).
| Experimental Configuration | MATH-500 (Avg. BA) | Relative Drop (\(\Delta\)) | OmniMath (Avg. BA) | Relative Drop (\(\Delta\)) | Description |
|---|---|---|---|---|---|
| RoRo (Qwen3-14B) | 79.3 | — | 59.9 | — | Full model configuration |
| w/o process reward | 77.1 | -2.2 | 56.5 | -3.4 | Complete removal of process reward (outcome-only RL) |
| w/o Learned Rubricor | 77.8 | -1.5 | 58.4 | -1.5 | Replaces generated rubrics with fixed seed rubric |
| w/o Learned Judge | 78.2 | -1.1 | 58.7 | -1.2 | Replaces learned Judge with direct uncalibrated scoring |
| w/o validation | 78.5 | -0.8 | 58.1 | -1.8 | Skips statistical validation gate |
| RoRo (DeepSeek-R1-14B) | 78.0 | — | 58.6 | — | Cross-family full model |
| w/o process reward | 75.5 | -2.5 | 55.2 | -3.4 | Complete removal of process reward |
| w/o Learned Rubricor | 76.4 | -1.6 | 57.0 | -1.6 | Fixed seed rubric |
| w/o Learned Judge | 76.8 | -1.2 | 57.3 | -1.3 | Direct scoring without Judge training |
| w/o validation | 77.2 | -0.8 | 56.8 | -1.8 | Skips statistical validation gate |
Key Findings¶
- Process Reward is Paramount: Stripping process rewards causes the steepest degradation (-3.4 points on OmniMath, -2.2 to -2.5 on MATH-500), effectively reducing RoRo to TRIM-like performance and validating that terminal outcome supervision fails to prevent deceptive alignment in intermediate routing.
- Learned Components Transcend Human Priors: Replacing the dynamically generated rubric with the fixed hand-crafted seed rubric incurs a 1.5–1.6 point drop, indicating that the Rubricor discovers crucial problem-specific evaluation angles uncaptured by generic human rubrics.
- Timely Early Escalation: Detailed trace audits on OmniMath (Cases 01261 and 01924) reveal that baseline TRIM uniformly distributes LRM calls across the trajectory or clusters them in late stages, failing to repair early mathematical setup blunders (using 24/30 and 21/30 LRM calls yet still failing). RoRo strategically deploys LRM calls in the initial 3–9 steps to lock in the proper problem formulation and returns to SRM for routine execution, reaching correct answers with 13 and 10 LRM calls and cutting FLOPs by 36%–50%.
- Substantial Wall-Clock Acceleration: When paired with Qwen3-14B at BA@20, RoRo reduces latency from 104.095s to 41.375s (2.516× speedup). Scaling to Qwen3-32B expands the speedup to 2.765× (156.143s down to 56.473s), confirming that algorithmic FLOP reductions translate directly to tangible deployment speedups that scale with model disparity.
Highlights & Insights¶
- Extends rubric-based RL beyond natural language synthesis into the meta-decision domain of dynamic multi-model routing, solving the verification gap in the absence of gold routing labels.
- Integrates a rigorous tripartite statistical validation gate (Holm-Bonferroni corrected partial correlation, score variance thresholding, and KSG mutual information leakage bound) ensuring process supervision remains uncorrupted by answer shortcuts.
- Features an asymmetric compute design: heavy 8B models handle offline rubric generation and trajectory scoring, while active serving relies purely on a nanosecond-latency 2-layer MLP consuming a 5D feature state, adding zero auxiliary tokens during inference.
- Empirical error analysis proves that mathematical reasoning exhibits a steep "intervention criticality curve" where early errors snowball irreversibly; process-guided routing autonomously discovers this phenomenon without explicit step weighting.
Limitations & Future Work¶
- Reliance on Internal Token Logits: Feature extraction assumes access to token-level probabilities (e.g., average entropy) from the small drafting model, restricting applicability when only black-box text APIs are accessible.
- Offline Training Overhead: The alternating optimization and rubric validation phases require executing multiple inference passes and statistical checks across thousands of rollouts using 8B models, incurring non-trivial upfront offline training costs.
- Evaluation Domain Scope: Empirical validation is predominantly centered on mathematical and scientific reasoning benchmarks; extending RoRo to code synthesis, open-ended tool orchestration, and long-horizon multimodal agents remains an open frontier.
Related Work & Insights¶
- vs TRIM (Kapoor et al., 2026): TRIM pioneered sequential decision routing under RL but relies on outcome accuracy and token cost penalties; RoRo identifies the resulting deceptive alignment and demonstrates that rubric-guided process rewards deliver consistent 1.5–3.4 point gains, especially under restricted budgets.
- vs RSD / SpecCoT / SpecReason (Liao et al., 2025; Shi et al., 2025; Pan et al., 2025): Speculative reasoning frameworks rely on auxiliary verification passes or external process reward models during active inference; RoRo distills process supervision into a lightweight MLP router, avoiding all inference-time verification token overhead.
- vs RLCER / OpenRubrics (Sheng et al., 2026; Liu et al., 2025a): Prior rubric RL works focused on guiding text generation within single models; RoRo re-engineers the rubric generation and validation pipeline to evaluate inter-model collaboration trajectories.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates process reward modeling for stepwise model routing and introduces alternating rubric optimization with strict leakage-preventing statistical gates.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Thoroughly evaluated across 5 benchmarks, 2 model pairings, 3 compute budgets, rigorous ablations, latency profiling, and difficulty-stratified usage analyses.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, disciplined mathematical formulation of the validation tests and GRPO advantages, and intuitive case studies.
- Value: ⭐⭐⭐⭐⭐ Offers a highly practical, deployment-ready paradigm for serving high-parameter reasoning models within realistic latency and compute envelopes.