Skip to content

From Rollouts to Recipes: Self-Contained Post-Training for LLMs

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Alignment & RLHF
Keywords: LLM post-training, reinforcement learning, Self-Routing, on-policy self-distillation, dynamic recipe routing

TL;DR

Addressing the limitation that standard verifier-based post-training applies a single uniform optimization recipe to all samples regardless of learning state, this paper proposes Self-Routingโ€”a self-contained framework that leverages rollout correctness and predictive entropy to dynamically assign samples to GRPO, on-policy self-distillation, regularization, or skipping, achieving consistent gains across mathematical reasoning benchmarks without external teachers or extra sampling overhead.

Background & Motivation

Post-training is central to improving the reasoning capability of large language models on verifiable domains such as mathematical problem solving, program synthesis, and code repair. In these domains, verifier-based reinforcement learning (RLVR)โ€”exemplified by PPO, GRPO, and DAPOโ€”leverages outcome rewards such as unit tests or execution pass rates without requiring labor-intensive process-level supervision or external chain-of-thought (CoT) annotations. However, prevailing post-training recipes operate almost exclusively at a global scope: they enforce a single, static optimization mechanism or a fixed mixture of objective functions across the entire training dataset.

This global paradigm overlooks the reality that current-policy rollouts intrinsically disclose the local, sample-level learning state of the model. For instance, problems where rollouts fail uniformly yield zero relative advantage gradient under standard outcome verifiers, rendering RL updates uninformative; conversely, instances exhibiting mixed success and failure lie in the most sensitive, high-variance regime where policy gradient exploration provides maximum return. Furthermore, problems that the model already solves consistently and with high certainty derive marginal benefit from aggressive updates and risk policy drift, whereas problems where the model confidently fails repeatedly often represent high-entropy noise that can destabilize training. Existing heuristics, such as dropping all-correct or all-wrong groups and test-time majority voting, operate as binary data selection or answer reranking filters rather than answering how the training dynamics should adaptively engage with heterogeneous sample states.

The fundamental departure of this work is the transition from rollouts to recipes: rollout behaviors should directly guide the optimization pathway of each sample rather than functioning merely as acceptance gates. Critically, this mechanism must remain completely self-containedโ€”requiring no external larger teacher models, fixed offline reference corpora, or redundant inference calls. Core idea: construct an empirical behavior state from on-policy rollout correctness decomposition and token-level predictive entropy confidence, and use it to adaptively route each sample into one of four disjoint training queuesโ€”GRPO exploration, on-policy self-distillation (OPSD), policy regularization (REG), or skipping (SKIP)โ€”in a unified post-training loop.

Method

Overall Architecture

Self-Routing operates within an iterative verifier-guided post-training pipeline. At each iteration, the current policy generates a group of on-policy rollouts for each prompt in the sampled mini-batch, and an outcome verifier computes binary task correctness. Lightweight behavioral metrics are extracted to form a continuous state representation, which is mapped into probability scores over four candidate training recipes. Samples are partitioned into disjoint queues through categorical sampling, their respective objective losses are computed, and only the active queues contribute to the aggregated gradient update.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Batch of Prompts ๐““<br/>Sample G on-policy rollouts per prompt"] --> B["Accuracy Decomposition and Confidence Calibration<br/>Extract empirical accuracy ax and token entropy cx"]
    B --> C["Behavior State to Recipe Probability Mapping<br/>Compute 4-branch routing scores and categorical sampling"]
    C --> D{"Disjoint Queue Partition and Loss Aggregation"}
    D -->|Bg queue: Mixed correctness| E["GRPO Exploration Update<br/>Group-relative advantage policy gradient"]
    D -->|Bo queue: Low accuracy and low confidence| F["On-Policy Self-Distillation OPSD<br/>Imitate target trajectory from answer-conditioned base model"]
    D -->|Br queue: High accuracy and high confidence| G["Policy Regularization REG<br/>Constrain KL divergence against reference policy"]
    D -->|Bs queue: High confidence persistent failure| H["Dynamic Skipping SKIP<br/>Exclude zero-gradient low-signal samples"]
    E --> I["Aggregate Active Queue Losses and Update ๐…๐œฝ"]
    F --> I
    G --> I

Key Designs

1. Accuracy Decomposition and Confidence Calibration: From Discrete Outcomes to Continuous Behavior States Directly partitioning samples with hard thresholding on empirical accuracy induces severe routing instability when group size \(G\) is small (e.g., \(G=8\)), because a single rollout outcome can arbitrarily flip the assignment while ignoring model certainty. To build a robust representation, the authors decompose the scalar rollout accuracy \(a_x = \frac{1}{G}\sum_{i=1}^G r_i\) (where \(r_i \in \{0, 1\}\)) into three smooth Gaussian membership scores centered at 0, 0.5, and 1: low-accuracy \(l_x\), uncertain \(m_x\), and high-accuracy \(h_x\), with fixed bandwidths \(\sigma_l = \sigma_h = 0.18\) and \(\sigma_m = 0.16\), normalized into \(\omega(a_x) = [l, m, h]\). Concurrently, internal model certainty is measured via sequence-level mean predictive entropy \(\bar{H}(o_i, x)\) across generated tokens, normalized within the batch to yield sample confidence \(c_x\). Subtracting batch mean confidence \(\bar{c}_B\) produces calibrated relative deviation \(\Delta c_x\) and normalized components \(\varphi(c_x) = (\tilde{c}, \tilde{c}^+, \tilde{c}^-, \Delta c)\). The combined tuple \(b_x = (\omega(a_x), \varphi(c_x))\) establishes a complete internal-external behavioral signature.

2. Behavior State to Recipe Probability Mapping: Interpretable Four-Branch Routing To align behavioral states with optimal learning dynamics, preliminary diagnostics revealed that samples with intermediate accuracy possess maximal reward variance suitable for RL, low-accuracy low-confidence samples benefit most from dense corrective imitation, stable solved samples require stabilization rather than perturbation, and overconfident failures degrade both active updates. Translating these insights, the system defines four transparent scoring functions: $\(s_{\text{GRPO}} = m + h(1 - \tilde{c}) + l(1 - \tilde{c}^-) \cdot \tilde{c}\)$ $\(s_{\text{OPSD}} = l(1 - \tilde{c}^-)\)$ $\(s_{\text{REG}} = h \cdot \tilde{\Delta c} \cdot \tilde{c}^+\)$ $\(s_{\text{SKIP}} = l \cdot \tilde{c}^- \cdot \tilde{c}\)$ The raw scores are normalized into a multinomial probability vector \(p_x = [p_{\text{GRPO}}, p_{\text{OPSD}}, p_{\text{REG}}, p_{\text{SKIP}}]\). Rather than using an argmax assignment, each sample draws its training recipe via categorical sampling \(t_x \sim \text{Categorical}(p_x)\), which preserves stochastic exploration and prevents premature batch collapse onto a single branch.

3. On-Policy Self-Distillation OPSD: Dense Guidance Without External Teachers When a sample falls into the \(B_o\) queue, verifier rewards provide no contrasting signal. Instead of calling a disproportionately large external teacherโ€”which introduces stylistic divergence, verbose formatting, and trajectory distribution mismatchโ€”the framework adopts On-Policy Self-Distillation. Target trajectories \(y_x^T = (y_{x,1}^T, \dots, y_{x,T_x}^T)\) are generated offline by prompting the identical base model with the problem input concatenated with the ground-truth answer. The student model then performs token-level imitation on these target sequences under standard cross-entropy loss: $\(\mathcal{L}_{\text{OPSD}} = -\frac{1}{|B_o|}\sum_{x \in B_o}\sum_{t=1}^{T_x} \log \pi_\theta(y_{x,t}^T \mid x, y_{x,<t}^T)\)$ This injects dense, state-appropriate correction without requiring external annotation or additional inference services during training.

4. Policy Regularization and Dynamic Skipping: Stability Preservation and Noise Elimination For samples in \(B_r\) where solutions are already reliable, continuing aggressive policy gradient updates yields diminishing returns and threatens catastrophic forgetting. Self-Routing routes them to a regularization objective penalizing KL divergence against a reference policy \(\pi_{\text{ref}}\): $\(\mathcal{L}_{\text{REG}} = \frac{1}{|B_r|}\sum_{x \in B_r} D_{\text{KL}}(\pi_\theta(\cdot \mid x) \parallel \pi_{\text{ref}}(\cdot \mid x))\)$ This preserves existing problem-solving representations and prevents policy drift. Simultaneously, unrecoverable, overconfident failures in \(B_s\) are assigned \(\mathcal{L}_{\text{SKIP}} = 0\) and completely ignored, expelling uninformative gradient noise and conserving backward-pass computation.

Loss & Training

During each step, the mini-batch is partitioned into disjoint queues \(B_g, B_o, B_r, B_s\) with respective cardinalities \(n_g, n_o, n_r, n_s\). The training loss aggregates contributions strictly across active queues: $\(\mathcal{L} = \frac{n_g \mathcal{L}_{\text{GRPO}} + n_o \mathcal{L}_{\text{OPSD}} + n_r \mathcal{L}_{\text{REG}}}{n_g + n_o + n_r}\)$ Here \(\mathcal{L}_{\text{GRPO}}\) optimizes the clipped surrogate objective using normalized group-relative advantages \(A_i = \frac{r_i - \mu_x}{\sigma_x + \epsilon}\); \(\mathcal{L}_{\text{OPSD}}\) minimizes token cross-entropy against self-generated targets; and \(\mathcal{L}_{\text{REG}}\) minimizes KL divergence. Skipped samples contribute zero gradient and are omitted from the normalization denominator. Training is conducted via ms-swift on 8 NVIDIA A100 GPUs using DAPO-Math-17K, with rollout count fixed at \(G=8\).

Key Experimental Results

Main Results

The authors evaluate Self-Routing across six model configurations spanning two architecture generations (Qwen3 and Qwen3.5) from 0.6B to 4B parameters on in-domain (ID) mathematical reasoning (GSM8K, MATH-500, AIME24, AIME25) and out-of-domain (OOD) general reasoning (MMLU-Pro, GPQA-diamond).

Model Method GSM8K (ID) MATH-500 (ID) AIME24 (ID) AIME25 (ID) MMLU-Pro (OOD) GPQA-dia. (OOD) Macro Avg.
Qwen3-0.6B Base 65.8 35.7 11.6 8.7 23.4 20.6 27.6
Naive-GRPO 70.4 42.8 13.9 10.5 20.9 22.1 30.1
Naive-OPSD 71.6 44.1 13.5 11.4 21.5 21.7 30.6
Self-Routing 73.2 45.8 15.6 13.0 22.6 23.8 32.3
Qwen3-1.7B Base 77.1 44.3 33.7 24.2 51.4 34.2 44.1
Naive-GRPO 81.6 50.4 39.6 29.8 47.9 37.8 47.9
Naive-OPSD 83.4 52.1 42.7 32.4 49.3 39.1 49.8
Self-Routing 86.8 55.9 46.3 36.7 50.6 42.5 53.1
Qwen3-4B Base 82.3 54.4 66.9 54.1 59.2 49.3 61.0
Naive-GRPO 89.6 62.8 74.3 65.4 54.6 53.8 66.8
Naive-OPSD 93.5 66.1 79.4 70.2 56.4 56.7 70.4
Self-Routing 95.2 69.8 83.6 74.9 58.1 60.4 73.7
Qwen3.5-0.8B Base 32.9 18.3 4.9 3.1 37.1 8.8 17.5
Naive-GRPO 38.6 22.7 6.8 4.7 33.8 11.3 19.6
Naive-OPSD 39.8 24.1 6.4 5.5 34.7 10.9 20.2
Self-Routing 43.1 26.8 8.6 7.3 36.0 13.6 22.6
Qwen3.5-2B Base 69.1 44.2 24.1 17.3 57.4 31.5 40.6
Naive-GRPO 77.9 50.7 36.8 28.1 53.8 36.9 47.4
Naive-OPSD 80.8 54.3 42.5 32.6 55.1 39.7 50.8
Self-Routing 84.6 58.2 48.4 38.5 56.7 44.3 55.1
Qwen3.5-4B Base 87.4 77.2 78.9 76.8 71.6 69.1 76.8
Naive-GRPO 91.3 81.7 84.2 82.5 66.8 72.3 79.8
Naive-OPSD 93.4 84.8 88.6 86.7 68.7 75.9 83.0
Self-Routing 96.1 88.3 93.2 91.4 70.9 79.6 86.6

Ablation Study

Ablations on Qwen3-4B isolate the relative importance of individual routing signals, calibration operations, and conservative branches across ID Math, OOD verifiable tasks (SATBench, AutoLogi, LiveCodeBench-v5), and general reasoning benchmarks.

Configuration / Variant ID Math Avg. OOD-Verifiable (OOD-V) General Benchmark Avg. Design Implication
Self-Routing (Full) 80.9 71.1 59.3 Complete 4-branch routing with calibrated confidence and smooth accuracy
Accuracy-only 79.6 70.4 58.1 Omits confidence signals; routes solely on empirical rollout accuracy
w/o conf. calibration 79.0 70.0 57.2 Uses unnormalized predictive entropy without batch-level debiasing
Confidence-only 74.2 67.6 54.8 Relies strictly on token entropy, lacking explicit correctness feedback
w/o behavior assignment 77.8 69.0 56.4 Replaces state-conditioned scores with naive or uniform assignment
w/o REG / SKIP 79.1 70.3 56.9 Enforces active updates (GRPO or OPSD) across all samples
Naive-OPSD 77.3 69.5 56.6 Applies uniform on-policy self-distillation to all training instances
Naive-GRPO 73.0 68.5 54.2 Applies uniform reinforcement learning to all training instances

Key Findings

  • Adaptive Allocation Dominates Uniform Baselines: Across all tested backbones, Self-Routing consistently outscores single-recipe methods. On Qwen3-4B, its macro average reaches 73.7, exceeding Naive-GRPO (66.8) and Naive-OPSD (70.4) by 6.9 and 3.3 points, respectively. On Qwen3.5-4B, it reaches 86.6.
  • Natural Shift in Training Dynamics: Integrating routing ratios across training reveals an average branch allocation of 30.8% GRPO, 30.4% OPSD, 25.5% REG, and 13.3% SKIP. In early stages, OPSD exceeds 50% to correct fundamental reasoning errors; midway, GRPO peaks as the model generates high-variance exploration trajectories; in late stages, REG dominates to stabilize acquired capabilities, while SKIP steadily declines.
  • Structural FLOPs Efficiency: Under standard Transformer cost models normalized by \(NTBL\), setting \(G=8\) rollouts gives normalized FLOPs of 64.0 for Naive-GRPO, 24.0 for Naive-OPSD, and 34.7 for Self-Routing. While not intended as an absolute wall-clock reduction over cheap SFT, Self-Routing uses only ~54% of the computation required by uniform GRPO while substantially outperforming it.
  • Mitigating General Knowledge Forgetting: While math RLVR typically incurs noticeable degradation on MMLU-Pro (e.g., Qwen3-4B Base drops from 59.2 to 54.6 under GRPO), Self-Routing preserves performance at 58.1, demonstrating that behavior-conditioned REG and SKIP branches shield non-mathematical competencies from aggressive distribution shifts.

Highlights & Insights

  • Promoting Rollout Behavior from Filtering to Routing: Traditional RL pipelines utilize rollout statistics solely for binary acceptance (discarding uninformative batches) or inference-time voting. Self-Routing redefines rollout behavior as a rich controller that dictates how a model should update on each sample.
  • Fully Self-Contained Supervision: By conditioning the base model itself on ground-truth answers to generate teacher rollouts offline, the system avoids relying on external larger models or human annotations, preserving stylistic and architectural alignment throughout distillation.
  • Pluggable Architecture: The four-branch dispatch framework cleanly decouples state estimation, probability scoring, and objective optimization, making it readily adaptable to broader verifiable domains such as theorem proving and code optimization.

Limitations & Future Work

  • Heuristic Rather Than Theoretical Recipe Mapping: The alignment between rollout patterns (e.g., mixed correctness to GRPO, low accuracy to OPSD) is rooted in empirical diagnostics rather than a closed-form theoretical proof in optimization dynamics.
  • Scope Constrained to Verifiable Tasks: The current framework relies on binary verifier feedback to estimate rollout accuracy, leaving its application to open-ended dialogue, creative writing, or ambiguous instruction-following tasks an open challenge.
  • Dependence on Ground-Truth Target Answers: Constructing OPSD teacher traces requires ground-truth labels during offline preprocessing, limiting applicability in entirely unsupervised or pure self-play discovery scenarios.
  • vs Naive-GRPO / DAPO / PODS: Conventional RLVR applies uniform policy gradients, wasting computation on all-correct or all-wrong groups. Self-Routing focuses GRPO strictly on high-variance samples while using OPSD and REG to handle edge cases.
  • vs GKD / On-Policy Distillation: Standard on-policy distillation relies on external teachers, incurring cross-model distribution and format shifts. Self-Routing uses answer-conditioned self-distillation (OPSD), keeping training completely self-contained.
  • vs Curriculum Learning & Data Selection: Curriculum learning adjusts sample scheduling over time, and data selection decides whether to train on a sample. Self-Routing solves the orthogonal question: once a sample is selected, what loss function best serves its immediate learning state?

Rating

  • Novelty: โญโญโญโญโญ Elegant shift from binary sample filtering to behavior-conditioned optimization recipe routing in a fully self-contained loop.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous evaluation across 6 model checkpoints, multiple ID/OOD benchmarks, FLOPs profiling, and training dynamic tracking.
  • Writing Quality: โญโญโญโญโญ Clear exposition, well-grounded motivations, transparent mathematical definitions, and detailed ablation analyses.
  • Value: โญโญโญโญโญ Delivers high practical value for RLVR and post-training pipelines by improving both gradient signal-to-noise ratio and compute allocation.