From Rollouts to Recipes: Self-Contained Post-Training for LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Alignment & RLHF
Keywords: LLM post-training, reinforcement learning, Self-Routing, on-policy self-distillation, dynamic recipe routing
TL;DR¶
Addressing the limitation that standard verifier-based post-training applies a single uniform optimization recipe to all samples regardless of learning state, this paper proposes Self-Routingโa self-contained framework that leverages rollout correctness and predictive entropy to dynamically assign samples to GRPO, on-policy self-distillation, regularization, or skipping, achieving consistent gains across mathematical reasoning benchmarks without external teachers or extra sampling overhead.
Background & Motivation¶
Post-training is central to improving the reasoning capability of large language models on verifiable domains such as mathematical problem solving, program synthesis, and code repair. In these domains, verifier-based reinforcement learning (RLVR)โexemplified by PPO, GRPO, and DAPOโleverages outcome rewards such as unit tests or execution pass rates without requiring labor-intensive process-level supervision or external chain-of-thought (CoT) annotations. However, prevailing post-training recipes operate almost exclusively at a global scope: they enforce a single, static optimization mechanism or a fixed mixture of objective functions across the entire training dataset.
This global paradigm overlooks the reality that current-policy rollouts intrinsically disclose the local, sample-level learning state of the model. For instance, problems where rollouts fail uniformly yield zero relative advantage gradient under standard outcome verifiers, rendering RL updates uninformative; conversely, instances exhibiting mixed success and failure lie in the most sensitive, high-variance regime where policy gradient exploration provides maximum return. Furthermore, problems that the model already solves consistently and with high certainty derive marginal benefit from aggressive updates and risk policy drift, whereas problems where the model confidently fails repeatedly often represent high-entropy noise that can destabilize training. Existing heuristics, such as dropping all-correct or all-wrong groups and test-time majority voting, operate as binary data selection or answer reranking filters rather than answering how the training dynamics should adaptively engage with heterogeneous sample states.
The fundamental departure of this work is the transition from rollouts to recipes: rollout behaviors should directly guide the optimization pathway of each sample rather than functioning merely as acceptance gates. Critically, this mechanism must remain completely self-containedโrequiring no external larger teacher models, fixed offline reference corpora, or redundant inference calls. Core idea: construct an empirical behavior state from on-policy rollout correctness decomposition and token-level predictive entropy confidence, and use it to adaptively route each sample into one of four disjoint training queuesโGRPO exploration, on-policy self-distillation (OPSD), policy regularization (REG), or skipping (SKIP)โin a unified post-training loop.
Method¶
Overall Architecture¶
Self-Routing operates within an iterative verifier-guided post-training pipeline. At each iteration, the current policy generates a group of on-policy rollouts for each prompt in the sampled mini-batch, and an outcome verifier computes binary task correctness. Lightweight behavioral metrics are extracted to form a continuous state representation, which is mapped into probability scores over four candidate training recipes. Samples are partitioned into disjoint queues through categorical sampling, their respective objective losses are computed, and only the active queues contribute to the aggregated gradient update.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Batch of Prompts ๐<br/>Sample G on-policy rollouts per prompt"] --> B["Accuracy Decomposition and Confidence Calibration<br/>Extract empirical accuracy ax and token entropy cx"]
B --> C["Behavior State to Recipe Probability Mapping<br/>Compute 4-branch routing scores and categorical sampling"]
C --> D{"Disjoint Queue Partition and Loss Aggregation"}
D -->|Bg queue: Mixed correctness| E["GRPO Exploration Update<br/>Group-relative advantage policy gradient"]
D -->|Bo queue: Low accuracy and low confidence| F["On-Policy Self-Distillation OPSD<br/>Imitate target trajectory from answer-conditioned base model"]
D -->|Br queue: High accuracy and high confidence| G["Policy Regularization REG<br/>Constrain KL divergence against reference policy"]
D -->|Bs queue: High confidence persistent failure| H["Dynamic Skipping SKIP<br/>Exclude zero-gradient low-signal samples"]
E --> I["Aggregate Active Queue Losses and Update ๐
๐ฝ"]
F --> I
G --> I
Key Designs¶
1. Accuracy Decomposition and Confidence Calibration: From Discrete Outcomes to Continuous Behavior States Directly partitioning samples with hard thresholding on empirical accuracy induces severe routing instability when group size \(G\) is small (e.g., \(G=8\)), because a single rollout outcome can arbitrarily flip the assignment while ignoring model certainty. To build a robust representation, the authors decompose the scalar rollout accuracy \(a_x = \frac{1}{G}\sum_{i=1}^G r_i\) (where \(r_i \in \{0, 1\}\)) into three smooth Gaussian membership scores centered at 0, 0.5, and 1: low-accuracy \(l_x\), uncertain \(m_x\), and high-accuracy \(h_x\), with fixed bandwidths \(\sigma_l = \sigma_h = 0.18\) and \(\sigma_m = 0.16\), normalized into \(\omega(a_x) = [l, m, h]\). Concurrently, internal model certainty is measured via sequence-level mean predictive entropy \(\bar{H}(o_i, x)\) across generated tokens, normalized within the batch to yield sample confidence \(c_x\). Subtracting batch mean confidence \(\bar{c}_B\) produces calibrated relative deviation \(\Delta c_x\) and normalized components \(\varphi(c_x) = (\tilde{c}, \tilde{c}^+, \tilde{c}^-, \Delta c)\). The combined tuple \(b_x = (\omega(a_x), \varphi(c_x))\) establishes a complete internal-external behavioral signature.
2. Behavior State to Recipe Probability Mapping: Interpretable Four-Branch Routing To align behavioral states with optimal learning dynamics, preliminary diagnostics revealed that samples with intermediate accuracy possess maximal reward variance suitable for RL, low-accuracy low-confidence samples benefit most from dense corrective imitation, stable solved samples require stabilization rather than perturbation, and overconfident failures degrade both active updates. Translating these insights, the system defines four transparent scoring functions: $\(s_{\text{GRPO}} = m + h(1 - \tilde{c}) + l(1 - \tilde{c}^-) \cdot \tilde{c}\)$ $\(s_{\text{OPSD}} = l(1 - \tilde{c}^-)\)$ $\(s_{\text{REG}} = h \cdot \tilde{\Delta c} \cdot \tilde{c}^+\)$ $\(s_{\text{SKIP}} = l \cdot \tilde{c}^- \cdot \tilde{c}\)$ The raw scores are normalized into a multinomial probability vector \(p_x = [p_{\text{GRPO}}, p_{\text{OPSD}}, p_{\text{REG}}, p_{\text{SKIP}}]\). Rather than using an argmax assignment, each sample draws its training recipe via categorical sampling \(t_x \sim \text{Categorical}(p_x)\), which preserves stochastic exploration and prevents premature batch collapse onto a single branch.
3. On-Policy Self-Distillation OPSD: Dense Guidance Without External Teachers When a sample falls into the \(B_o\) queue, verifier rewards provide no contrasting signal. Instead of calling a disproportionately large external teacherโwhich introduces stylistic divergence, verbose formatting, and trajectory distribution mismatchโthe framework adopts On-Policy Self-Distillation. Target trajectories \(y_x^T = (y_{x,1}^T, \dots, y_{x,T_x}^T)\) are generated offline by prompting the identical base model with the problem input concatenated with the ground-truth answer. The student model then performs token-level imitation on these target sequences under standard cross-entropy loss: $\(\mathcal{L}_{\text{OPSD}} = -\frac{1}{|B_o|}\sum_{x \in B_o}\sum_{t=1}^{T_x} \log \pi_\theta(y_{x,t}^T \mid x, y_{x,<t}^T)\)$ This injects dense, state-appropriate correction without requiring external annotation or additional inference services during training.
4. Policy Regularization and Dynamic Skipping: Stability Preservation and Noise Elimination For samples in \(B_r\) where solutions are already reliable, continuing aggressive policy gradient updates yields diminishing returns and threatens catastrophic forgetting. Self-Routing routes them to a regularization objective penalizing KL divergence against a reference policy \(\pi_{\text{ref}}\): $\(\mathcal{L}_{\text{REG}} = \frac{1}{|B_r|}\sum_{x \in B_r} D_{\text{KL}}(\pi_\theta(\cdot \mid x) \parallel \pi_{\text{ref}}(\cdot \mid x))\)$ This preserves existing problem-solving representations and prevents policy drift. Simultaneously, unrecoverable, overconfident failures in \(B_s\) are assigned \(\mathcal{L}_{\text{SKIP}} = 0\) and completely ignored, expelling uninformative gradient noise and conserving backward-pass computation.
Loss & Training¶
During each step, the mini-batch is partitioned into disjoint queues \(B_g, B_o, B_r, B_s\) with respective cardinalities \(n_g, n_o, n_r, n_s\). The training loss aggregates contributions strictly across active queues: $\(\mathcal{L} = \frac{n_g \mathcal{L}_{\text{GRPO}} + n_o \mathcal{L}_{\text{OPSD}} + n_r \mathcal{L}_{\text{REG}}}{n_g + n_o + n_r}\)$ Here \(\mathcal{L}_{\text{GRPO}}\) optimizes the clipped surrogate objective using normalized group-relative advantages \(A_i = \frac{r_i - \mu_x}{\sigma_x + \epsilon}\); \(\mathcal{L}_{\text{OPSD}}\) minimizes token cross-entropy against self-generated targets; and \(\mathcal{L}_{\text{REG}}\) minimizes KL divergence. Skipped samples contribute zero gradient and are omitted from the normalization denominator. Training is conducted via ms-swift on 8 NVIDIA A100 GPUs using DAPO-Math-17K, with rollout count fixed at \(G=8\).
Key Experimental Results¶
Main Results¶
The authors evaluate Self-Routing across six model configurations spanning two architecture generations (Qwen3 and Qwen3.5) from 0.6B to 4B parameters on in-domain (ID) mathematical reasoning (GSM8K, MATH-500, AIME24, AIME25) and out-of-domain (OOD) general reasoning (MMLU-Pro, GPQA-diamond).
| Model | Method | GSM8K (ID) | MATH-500 (ID) | AIME24 (ID) | AIME25 (ID) | MMLU-Pro (OOD) | GPQA-dia. (OOD) | Macro Avg. |
|---|---|---|---|---|---|---|---|---|
| Qwen3-0.6B | Base | 65.8 | 35.7 | 11.6 | 8.7 | 23.4 | 20.6 | 27.6 |
| Naive-GRPO | 70.4 | 42.8 | 13.9 | 10.5 | 20.9 | 22.1 | 30.1 | |
| Naive-OPSD | 71.6 | 44.1 | 13.5 | 11.4 | 21.5 | 21.7 | 30.6 | |
| Self-Routing | 73.2 | 45.8 | 15.6 | 13.0 | 22.6 | 23.8 | 32.3 | |
| Qwen3-1.7B | Base | 77.1 | 44.3 | 33.7 | 24.2 | 51.4 | 34.2 | 44.1 |
| Naive-GRPO | 81.6 | 50.4 | 39.6 | 29.8 | 47.9 | 37.8 | 47.9 | |
| Naive-OPSD | 83.4 | 52.1 | 42.7 | 32.4 | 49.3 | 39.1 | 49.8 | |
| Self-Routing | 86.8 | 55.9 | 46.3 | 36.7 | 50.6 | 42.5 | 53.1 | |
| Qwen3-4B | Base | 82.3 | 54.4 | 66.9 | 54.1 | 59.2 | 49.3 | 61.0 |
| Naive-GRPO | 89.6 | 62.8 | 74.3 | 65.4 | 54.6 | 53.8 | 66.8 | |
| Naive-OPSD | 93.5 | 66.1 | 79.4 | 70.2 | 56.4 | 56.7 | 70.4 | |
| Self-Routing | 95.2 | 69.8 | 83.6 | 74.9 | 58.1 | 60.4 | 73.7 | |
| Qwen3.5-0.8B | Base | 32.9 | 18.3 | 4.9 | 3.1 | 37.1 | 8.8 | 17.5 |
| Naive-GRPO | 38.6 | 22.7 | 6.8 | 4.7 | 33.8 | 11.3 | 19.6 | |
| Naive-OPSD | 39.8 | 24.1 | 6.4 | 5.5 | 34.7 | 10.9 | 20.2 | |
| Self-Routing | 43.1 | 26.8 | 8.6 | 7.3 | 36.0 | 13.6 | 22.6 | |
| Qwen3.5-2B | Base | 69.1 | 44.2 | 24.1 | 17.3 | 57.4 | 31.5 | 40.6 |
| Naive-GRPO | 77.9 | 50.7 | 36.8 | 28.1 | 53.8 | 36.9 | 47.4 | |
| Naive-OPSD | 80.8 | 54.3 | 42.5 | 32.6 | 55.1 | 39.7 | 50.8 | |
| Self-Routing | 84.6 | 58.2 | 48.4 | 38.5 | 56.7 | 44.3 | 55.1 | |
| Qwen3.5-4B | Base | 87.4 | 77.2 | 78.9 | 76.8 | 71.6 | 69.1 | 76.8 |
| Naive-GRPO | 91.3 | 81.7 | 84.2 | 82.5 | 66.8 | 72.3 | 79.8 | |
| Naive-OPSD | 93.4 | 84.8 | 88.6 | 86.7 | 68.7 | 75.9 | 83.0 | |
| Self-Routing | 96.1 | 88.3 | 93.2 | 91.4 | 70.9 | 79.6 | 86.6 |
Ablation Study¶
Ablations on Qwen3-4B isolate the relative importance of individual routing signals, calibration operations, and conservative branches across ID Math, OOD verifiable tasks (SATBench, AutoLogi, LiveCodeBench-v5), and general reasoning benchmarks.
| Configuration / Variant | ID Math Avg. | OOD-Verifiable (OOD-V) | General Benchmark Avg. | Design Implication |
|---|---|---|---|---|
| Self-Routing (Full) | 80.9 | 71.1 | 59.3 | Complete 4-branch routing with calibrated confidence and smooth accuracy |
| Accuracy-only | 79.6 | 70.4 | 58.1 | Omits confidence signals; routes solely on empirical rollout accuracy |
| w/o conf. calibration | 79.0 | 70.0 | 57.2 | Uses unnormalized predictive entropy without batch-level debiasing |
| Confidence-only | 74.2 | 67.6 | 54.8 | Relies strictly on token entropy, lacking explicit correctness feedback |
| w/o behavior assignment | 77.8 | 69.0 | 56.4 | Replaces state-conditioned scores with naive or uniform assignment |
| w/o REG / SKIP | 79.1 | 70.3 | 56.9 | Enforces active updates (GRPO or OPSD) across all samples |
| Naive-OPSD | 77.3 | 69.5 | 56.6 | Applies uniform on-policy self-distillation to all training instances |
| Naive-GRPO | 73.0 | 68.5 | 54.2 | Applies uniform reinforcement learning to all training instances |
Key Findings¶
- Adaptive Allocation Dominates Uniform Baselines: Across all tested backbones, Self-Routing consistently outscores single-recipe methods. On Qwen3-4B, its macro average reaches 73.7, exceeding Naive-GRPO (66.8) and Naive-OPSD (70.4) by 6.9 and 3.3 points, respectively. On Qwen3.5-4B, it reaches 86.6.
- Natural Shift in Training Dynamics: Integrating routing ratios across training reveals an average branch allocation of 30.8% GRPO, 30.4% OPSD, 25.5% REG, and 13.3% SKIP. In early stages, OPSD exceeds 50% to correct fundamental reasoning errors; midway, GRPO peaks as the model generates high-variance exploration trajectories; in late stages, REG dominates to stabilize acquired capabilities, while SKIP steadily declines.
- Structural FLOPs Efficiency: Under standard Transformer cost models normalized by \(NTBL\), setting \(G=8\) rollouts gives normalized FLOPs of 64.0 for Naive-GRPO, 24.0 for Naive-OPSD, and 34.7 for Self-Routing. While not intended as an absolute wall-clock reduction over cheap SFT, Self-Routing uses only ~54% of the computation required by uniform GRPO while substantially outperforming it.
- Mitigating General Knowledge Forgetting: While math RLVR typically incurs noticeable degradation on MMLU-Pro (e.g., Qwen3-4B Base drops from 59.2 to 54.6 under GRPO), Self-Routing preserves performance at 58.1, demonstrating that behavior-conditioned REG and SKIP branches shield non-mathematical competencies from aggressive distribution shifts.
Highlights & Insights¶
- Promoting Rollout Behavior from Filtering to Routing: Traditional RL pipelines utilize rollout statistics solely for binary acceptance (discarding uninformative batches) or inference-time voting. Self-Routing redefines rollout behavior as a rich controller that dictates how a model should update on each sample.
- Fully Self-Contained Supervision: By conditioning the base model itself on ground-truth answers to generate teacher rollouts offline, the system avoids relying on external larger models or human annotations, preserving stylistic and architectural alignment throughout distillation.
- Pluggable Architecture: The four-branch dispatch framework cleanly decouples state estimation, probability scoring, and objective optimization, making it readily adaptable to broader verifiable domains such as theorem proving and code optimization.
Limitations & Future Work¶
- Heuristic Rather Than Theoretical Recipe Mapping: The alignment between rollout patterns (e.g., mixed correctness to GRPO, low accuracy to OPSD) is rooted in empirical diagnostics rather than a closed-form theoretical proof in optimization dynamics.
- Scope Constrained to Verifiable Tasks: The current framework relies on binary verifier feedback to estimate rollout accuracy, leaving its application to open-ended dialogue, creative writing, or ambiguous instruction-following tasks an open challenge.
- Dependence on Ground-Truth Target Answers: Constructing OPSD teacher traces requires ground-truth labels during offline preprocessing, limiting applicability in entirely unsupervised or pure self-play discovery scenarios.
Related Work & Insights¶
- vs Naive-GRPO / DAPO / PODS: Conventional RLVR applies uniform policy gradients, wasting computation on all-correct or all-wrong groups. Self-Routing focuses GRPO strictly on high-variance samples while using OPSD and REG to handle edge cases.
- vs GKD / On-Policy Distillation: Standard on-policy distillation relies on external teachers, incurring cross-model distribution and format shifts. Self-Routing uses answer-conditioned self-distillation (OPSD), keeping training completely self-contained.
- vs Curriculum Learning & Data Selection: Curriculum learning adjusts sample scheduling over time, and data selection decides whether to train on a sample. Self-Routing solves the orthogonal question: once a sample is selected, what loss function best serves its immediate learning state?
Rating¶
- Novelty: โญโญโญโญโญ Elegant shift from binary sample filtering to behavior-conditioned optimization recipe routing in a fully self-contained loop.
- Experimental Thoroughness: โญโญโญโญโญ Rigorous evaluation across 6 model checkpoints, multiple ID/OOD benchmarks, FLOPs profiling, and training dynamic tracking.
- Writing Quality: โญโญโญโญโญ Clear exposition, well-grounded motivations, transparent mathematical definitions, and detailed ablation analyses.
- Value: โญโญโญโญโญ Delivers high practical value for RLVR and post-training pipelines by improving both gradient signal-to-noise ratio and compute allocation.