SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/machine981/SCOPE
Area: LLM Reasoning
Keywords: On-Policy Distillation, Policy Optimization, Reasoning Alignment, Perplexity Calibration, Dual-Path Adaptive Weighting
TL;DR¶
SCOPE addresses the uniform signal-quality assumption in On-Policy Distillation by partitioning student rollouts by outcome correctness into two complementary supervision paths: applying inverse teacher-perplexity-weighted reverse KL distillation to filter out misleading noise from flawed prefixes on incorrect trajectories, and student-perplexity-weighted MLE to amplify low-confidence capability-boundary exploration on correct trajectories, coupled with group-level normalization to preserve reasoning diversity and boost pass rates.
Background & Motivation¶
In the reasoning alignment of large language models (LLMs), on-policy reinforcement learning with verifiable rewards (such as RLVR and GRPO) has emerged as the standard paradigm. However, because rewards are awarded solely as sparse, scalar outcomes at the final token, credit assignment across extended multi-step reasoning chains becomes notoriously challenging, requiring enormous rollout budgets to converge. On-Policy Distillation (OPD) tackles this limitation by superimposing dense, token-level KL divergence feedback from an authoritative teacher model onto the student's self-generated rollouts, providing fine-grained step-level supervisory signals that substantially accelerate optimization and mitigate credit assignment bottlenecks.
Nevertheless, conventional OPD operates under an unquestioned foundational premise: it assumes that the teacher's dense supervision is uniformly dependable across all generated trajectories. In practical reasoning domains, this disregard for signal-quality heterogeneity induces a severe double-sided failure mode. For incorrect rollouts, the teacher is forced to evaluate tokens conditioned on a flawed, student-generated reasoning prefix; empirical measurements reveal that when the teacher encounters high perplexity on these damaged contexts, its next-token predictive distributions collapse into high-entropy noise, compelling the student to internalize hallucinations. Conversely, for successful rollouts, enforcing rigid teacher KL constraints stifles the student's own valid yet unconventional reasoning paths; even when falling back to standard Maximum Likelihood Estimation (MLE), equal-weight reinforcement disproportionately rewards already-mastered dominant solutions, marginalizing boundary-case successes and precipitating the "Pass@k paradox" where Pass@1 increases while Pass@32 severely degrades.
The fundamental tension resides in the dual requirement of reasoning optimization: failed attempts require the teacher's genuine corrective capability rather than its corrupted prefix noise, while successful attempts require expanding the student's exploration boundary rather than reinforcing over-represented majority modes. Resolving this challenge demands a framework that disentangles rollouts by verification outcomes and adaptively calibrates signal reliability through dual complementary perspectives. Core idea: propose Signal-Calibrated On-Policy Distillation Enhancement (SCOPE), which routes on-policy rollouts by correctness into two distinct paths, employing inverse teacher-perplexity-weighted KL distillation to prune noisy guidance on flawed trajectories, and positive student-perplexity-weighted MLE to amplify capability-boundary exploration on correct ones, all stabilized via within-group softmax normalization.
Method¶
Overall Architecture¶
The architecture of SCOPE is organized around outcome-driven group branching and Dual-Path Adaptive Weighting (DPAW). For any given input prompt \(x\), the student policy \(\pi_\theta\) first samples a group of \(N\) candidate trajectories \(Y_x = \{y_1, y_2, \dots, y_N\}\) in an on-policy rollout. An automated verifier evaluates each candidate with a binary reward \(R_i \in \{0, 1\}\), explicitly partitioning the group into two disjoint subsets: the correct set \(\Omega_x^c = \{y_i \in Y_x \mid R_i = 1\}\) and the incorrect set \(\Omega_x^w = \{y_i \in Y_x \mid R_i = 0\}\). If either subset is empty, its corresponding branch is gracefully skipped, allowing all-correct, all-incorrect, and mixed groups to generate consistent gradient updates.
Following subset partitioning, SCOPE applies asymmetric, signal-aware adaptive weighting tailored to the distinct supervisory demands of each path. For failed trajectories in \(\Omega_x^w\), SCOPE computes the sequence perplexity under the teacher model \(\pi_T\) and assigns an inverse-perplexity weight to filter out corrupted contexts where the teacher is confused, conducting reverse KL distillation. For successful trajectories in \(\Omega_x^c\), SCOPE computes the student's own rollout perplexity and assigns a positive-perplexity weight to prioritize low-confidence solutions near the capability frontier, performing self-reinforcement via weighted MLE. Both sets of weights undergo local softmax normalization within each group before joint backpropagation under a unified loss.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Prompt x"] --> B["Student On-Policy Rollout<br/>Sample N Candidate Trajectories"]
B --> C["Outcome-Driven Group Branching<br/>Binary Verifier Routing"]
C -->|Correct Trajectories Ri=1| D["Valid Trajectory Exploitation<br/>Student-Perplexity Up-Weighting"]
C -->|Incorrect Trajectories Ri=0| E["Flawed Trajectory Rectification<br/>Teacher-Perplexity Down-Weighting"]
D --> F["Group-Level Normalization & Unified Loss<br/>Weighted MLE + Weighted OPD"]
E --> F
F --> G["Updated Student Policy Model"]
Key Designs¶
1. Outcome-Driven Group Branching: Decoupling Autonomous Exploitation and External Guidance Standard OPD applies reverse KL divergence indiscriminately across all sampled rollouts, suppressing reasoning diversity on correct trajectories while suffering from contaminated prefixes on incorrect ones. SCOPE establishes an explicit routing boundary driven by verifiable rewards \(R_i\), decoupling training into two fundamentally distinct objectives. For correct rollouts in \(\Omega_x^c\), external teacher guidance is bypassed in favor of self-reinforcing surrogate likelihood optimization (\(L_{\text{MLE}}\)) with importance sampling correction, cultivating the model's native reasoning strengths. For failed rollouts in \(\Omega_x^w\), where self-supervision lacks valid targets, SCOPE engages a frozen teacher policy \(\pi_T\) to deliver dense token-level corrective gradients through an on-policy surrogate distillation loss (\(L_{\text{OPD}}\)).
2. Flawed Trajectory Rectification: Inverse Teacher-Perplexity Weighting to Prune Prefix Noise When a student's reasoning sequence derails due to arithmetic flaws, hallucinated premises, or repetitive loops, external teacher completion faces the "Flawed Prefix Trap." Conditioning the teacher on severely degraded student prefixes disrupts its attention and spreads probability mass, yielding uninformative, high-entropy distributions. Empirical error-recovery audits confirm that teacher recovery accuracy collapses when prefixes exhibit high perplexity. To prevent the student from internalizing this toxic supervisory noise, SCOPE evaluates the teacher's sequence perplexity \(\text{PPL}_T(y_i \mid x) = \exp(-\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \log \pi_T(a_{i,t} \mid x, a_{i,<t}))\) and formulates an inverse teacher-guided weight \(w_i^{\text{tea}}\): $\(w_i^{\text{tea}} = \frac{\exp\left(\frac{1}{\tau |y_i|} \log \pi_T(y_i \mid x)\right)}{\sum_{j \in \Omega_x^w} \exp\left(\frac{1}{\tau |y_j|} \log \pi_T(y_j \mid x)\right)} = \frac{\text{PPL}_T(y_i \mid x)^{-1/\tau}}{\sum_{j \in \Omega_x^w} \text{PPL}_T(y_j \mid x)^{-1/\tau}}\)$ Controlled by temperature \(\tau\), this soft-gating mechanism exponentially down-weights failed trajectories where the teacher exhibits high perplexity, assigning near-zero supervisory contribution to degraded contexts and ensuring distillation gradients originate strictly from coherent prefixes where the teacher possesses high corrective confidence.
3. Valid Trajectory Exploitation: Student-Perplexity Weighting to Amplify Unconventional Paths Uniformly optimizing all self-generated correct solutions inevitably induces mode collapse by overwhelmingly rewarding dominant, high-frequency reasoning trajectories while suffocating rare, unconventional, yet structurally sound alternatives. To overcome the Pass@k paradox, SCOPE introduces student-guided adaptive weights \(w_i^{\text{stu}}\) calibrated by the student's own rollout perplexity \(\text{PPL}_S(y_i \mid x)\): $\(w_i^{\text{stu}} = \frac{\exp\left(-\frac{1}{\tau |y_i|} \log \pi_S(y_i \mid x)\right)}{\sum_{j \in \Omega_x^c} \exp\left(-\frac{1}{\tau |y_j|} \log \pi_S(y_j \mid x)\right)} = \frac{\text{PPL}_S(y_i \mid x)^{1/\tau}}{\sum_{j \in \Omega_x^c} \text{PPL}_S(y_j \mid x)^{1/\tau}}\)$ This formulation operates as an inverse-probability sampling reweighting: correct rollouts characterized by higher perplexity (i.e., lower generation probability) reside directly at the student's current capability frontier. By dynamically assigning amplified weights to these low-confidence successful trajectories, SCOPE expands the effective reasoning horizon and safeguards exploration diversity across multi-sample generation.
4. Group-Level Normalization and Stability: Cross-Prompt Scale Calibration and Gradient Bounding Because individual prompts inherently vary in conceptual difficulty, linguistic length, and reasoning complexity, raw perplexity magnitudes differ widely across instances. Directly applying unnormalized raw PPL values as loss scales would skew optimization toward long or high-entropy prompts. SCOPE resolves this by enforcing local within-group softmax normalization, guaranteeing \(\sum_{i \in \Omega_x^c} w_i^{\text{stu}} = 1\) and \(\sum_{i \in \Omega_x^w} w_i^{\text{tea}} = 1\). Analytically, this constrains the update along each active branch to a convex combination of individual trajectory gradients, bounding the overall gradient norm by the maximum single-sample gradient and providing rigorous numerical stability throughout distributed training.
Loss & Training¶
Given an on-policy rollout buffer collected under behavior policy \(\pi_{\text{old}}\), the token-level importance sampling ratio is defined as \(\rho_{i,t}(\theta) = \frac{\pi_\theta(a_{i,t} \mid x, a_{i,<t})}{\pi_{\text{old}}(a_{i,t} \mid x, a_{i,<t})}\). The surrogate objectives for the valid and flawed branches are formulated as: $\(L_{\text{MLE}}(x, y_i; \theta) = - \sum_{t=1}^{|y_i|} \rho_{i,t}(\theta)\)$ $\(L_{\text{OPD}}(x, y_i; \theta) = \sum_{t=1}^{|y_i|} \rho_{i,t}(\theta) \left[ \log \bar{\pi}_\theta(a_{i,t} \mid x, a_{i,<t}) - \log \pi_T(a_{i,t} \mid x, a_{i,<t}) \right]\)$ where \(\bar{\pi}_\theta\) denotes detached policy parameters. Integrating both branches with their respective DPAW weights over dataset \(\mathcal{D}\) produces the unified SCOPE objective: $\(J_{\text{SCOPE}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}} \left[ \sum_{i \in \Omega_x^c} w_i^{\text{stu}} \cdot L_{\text{MLE}}(x, y_i) + \lambda \sum_{i \in \Omega_x^w} w_i^{\text{tea}} \cdot L_{\text{OPD}}(x, y_i) \right]\)$ where \(\lambda\) governs the branch trade-off. Optimization is executed with a global prompt batch size of 256, sampling \(G=8\) rollouts per prompt, a maximum prompt length of 4,096 tokens, a maximum completion length of 12,288 tokens, a generation temperature of 0.6, a weight temperature \(\tau=1.0\), a constant learning rate of \(5 \times 10^{-5}\), and an AdamW optimizer with weight decay 0.01.
Key Experimental Results¶
Main Results¶
SCOPE was comprehensively evaluated across six challenging mathematical reasoning benchmarks spanning competition problems and STEM domains: AIME 2024, AIME 2025, AMC 2023, MATH-500, Minerva, and OlympiadBench. Models were evaluated using Avg@32 (expected single-sample stability) and Pass@32 (coverage and multi-sample exploration capacity). Baseline methods include GRPO, offline Knowledge Distillation (KD), and standard On-Policy Distillation (OPD).
| Setup & Method | AIME24 (A/P@32) | AIME25 (A/P@32) | AMC23 (A/P@32) | MATH500 (A/P@32) | Minerva (A/P@32) | Olympiad (A/P@32) | Overall Avg@32 | Overall Pass@32 |
|---|---|---|---|---|---|---|---|---|
| Student: R1-Distill-1.5B (Base) | 29.4 / 76.5 | 23.9 / 46.9 | 72.7 / 94.7 | 84.6 / 97.3 | 32.3 / 55.9 | 44.2 / 67.5 | 47.9 | 73.1 |
| + GRPO | 35.5 / 68.3 | 24.5 / 45.1 | 75.1 / 95.0 | 87.0 / 96.7 | 35.1 / 53.5 | 40.5 / 67.7 | 49.6 | 71.1 |
| + KD | 26.6 / 71.4 | 22.2 / 45.6 | 69.1 / 96.3 | 84.1 / 97.4 | 30.7 / 54.1 | 39.7 / 66.9 | 45.4 | 72.0 |
| + OPD | 40.2 / 75.0 | 28.9 / 48.5 | 75.9 / 95.0 | 89.0 / 97.7 | 34.9 / 53.0 | 44.9 / 69.3 | 52.3 | 73.1 |
| + SCOPE (Ours) | 42.7 / 77.9 | 30.4 / 50.9 | 80.9 / 97.2 | 89.8 / 97.9 | 37.8 / 55.1 | 49.7 / 70.9 | 55.2 | 75.0 |
| Relative gain vs. OPD (\(\Delta\%\)) | +6.22% / +3.87% | +5.19% / +4.95% | +6.59% / +2.32% | +0.90% / +0.20% | +8.31% / +3.96% | +10.69% / +2.31% | +5.54% | +2.60% |
| Student: Qwen3-1.7B-Base (Base) | 3.7 / 26.7 | 3.2 / 20.7 | 24.6 / 69.4 | 54.8 / 88.6 | 11.2 / 41.6 | 16.3 / 48.0 | 19.0 | 49.2 |
| + GRPO | 7.5 / 27.2 | 4.2 / 24.5 | 33.1 / 60.0 | 62.9 / 89.8 | 25.3 / 49.8 | 25.9 / 50.9 | 26.5 | 50.4 |
| + KD | 4.6 / 18.9 | 5.8 / 24.8 | 29.8 / 66.1 | 50.2 / 86.2 | 16.6 / 43.2 | 13.3 / 41.4 | 20.1 | 46.8 |
| + OPD | 12.2 / 31.5 | 10.6 / 29.7 | 43.1 / 80.1 | 67.9 / 90.7 | 24.2 / 49.4 | 25.3 / 53.7 | 30.6 | 55.9 |
| + SCOPE (Ours) | 13.3 / 31.5 | 12.1 / 35.6 | 46.3 / 83.0 | 70.9 / 93.0 | 27.9 / 54.0 | 24.6 / 54.6 | 32.5 | 58.6 |
| Relative gain vs. OPD (\(\Delta\%\)) | +9.02% / +0.00% | +14.15% / +19.87% | +7.42% / +3.62% | +4.42% / +2.54% | +15.29% / +9.31% | -2.77% / +1.68% | +6.21% | +4.83% |
In comparative evaluations against recent OPD variants on R1-Distill-1.5B (Teacher: SkyWork-OR1-7B), SCOPE achieves an average of 51.3% in Avg@32 and 75.3% in Pass@32 across AIME24/25 and AMC23, consistently surpassing ExOPD (46.5% / 72.0%) and RLAD (48.5% / 72.4%). In code generation tasks across HumanEval, Codeforces, and LiveCodeBench, SCOPE improves Avg@32 by +4.69% over OPD (reaching 44.6% vs 42.6%). Furthermore, scaling up to a Qwen3-32B teacher distilling into a Qwen3-8B student demonstrates sustained gains, with SCOPE achieving 71.3% Avg@32 and 90.3% Pass@32 compared to 68.8% / 87.4% for OPD.
Ablation Study¶
Ablation experiments conducted on AIME24 and AIME25 dissect the individual contributions and directional alignment of the DPAW components:
| Configuration Variant | AIME24 Avg@32 (%) | AIME24 Pass@32 (%) | AIME25 Avg@32 (%) | AIME25 Pass@32 (%) | Mechanistic Impact |
|---|---|---|---|---|---|
| Full SCOPE Model | 42.7 | 77.9 | 30.4 | 50.9 | Full dual-path perplexity calibration active |
| w/o DPAW (Uniform Weighting) | 40.2 | 75.0 | 28.9 | 45.7 | Reverts to uniform OPD, causing AIME25 Pass@32 to plunge by 5.2% |
| w/o Student-guided Weight (\(w^{\text{stu}}\)) | 41.5 | 74.1 | 29.5 | 47.2 | Equal weighting on correct rollouts dampens boundary exploration |
| w/o Teacher-guided Weight (\(w^{\text{tea}}\)) | 40.8 | 76.5 | 29.1 | 48.6 | Fails to filter noisy teacher gradients on flawed prefixes |
| w/ Opposite Student-guided Weight | 40.0 | 73.0 | 28.4 | 46.0 | Reinforces dominant paths and induces severe diversity collapse |
| w/ Opposite Teacher-guided Weight | 38.6 | 74.8 | 27.6 | 46.5 | Up-weights corrupted prefixes, reducing AIME24 Avg@32 by 4.1% |
Key Findings¶
- Resolution of the Pass@k Paradox: While GRPO and baseline OPD suffer from diversity collapse (GRPO degrades Pass@32 on AIME24 from 76.5% to 68.3%), SCOPE achieves gains in both single-sample accuracy and multi-sample exploration. Tracking policy entropy during training indicates that SCOPE sustains healthy exploration dynamics without premature exploitation plateaus.
- Toxicity of Flawed Prefix Distillation: Reversing the teacher-guided weight (assigning higher weights to high-PPL error rollouts) causes AIME24 Avg@32 to drop precipitously from 42.7% to 38.6%, significantly worse than baseline OPD. This provides direct empirical validation that high teacher perplexity acts as a strong indicator of corrupted prefix noise.
- Sensitivity to Temperature \(\tau\): Testing \(\tau \in \{0.5, 1.0, 2.0\}\) indicates that \(\tau=1.0\) yields optimal performance. A smaller value (\(\tau=0.5\)) overly sharpens the weight distribution, concentrating updates on outlier trajectories, whereas a larger value (\(\tau=2.0\)) flattens weights toward uniform OPD, sacrificing signal quality discrimination.
Highlights & Insights¶
- Perplexity as an Empirical Proxy for Corrective Feasibility: The preliminary prefix truncation study reveals that the teacher's recovery rate is up to +19.4% higher on low-perplexity student errors (Q1: 64.9% vs Q4: 45.4% at 0.2 truncation), providing rigorous justification for using perplexity as a dynamic signal-quality filter.
- Boundary-Targeted Self-Reinforcement: Rather than treating all correct samples equally, up-weighting correct rollouts with higher student perplexity concentrates gradient updates on low-confidence instances near the model's capability threshold, naturally diversifying reasoning paths without external entropy regularization.
- Zero-Inference-Overhead Architecture: DPAW computes trajectory weights entirely from the log-probabilities generated during standard forward rollout and distillation passes, requiring no secondary reward models, process annotators, or complex hyperparameter scheduling.
Limitations & Future Work¶
- Reliance on Verifiable Outcome Rewards: SCOPE currently requires deterministic outcome verification (e.g., exact-match answers in math or unit test suites in code) to route rollouts into correct and incorrect branches. Extending SCOPE to open-ended dialogue, creative writing, or general alignment tasks requires integration with reliable learned reward models.
- Synchronous Teacher Query Overhead: In the current synchronous implementation, teacher model scoring adds noticeable per-step wall-clock latency (641.9s per step for SCOPE vs. 459.0s for GRPO). Implementing asynchronous pipelines where teacher scoring overlaps with student rollouts is an important engineering optimization.
- Scale and Architecture Exploration: Empirical evaluations were bounded at 32B teacher and 8B student configurations. Scaling the framework to frontier-scale foundation models (>70B) and Mixture-of-Experts (MoE) architectures remains an open direction.
Related Work & Insights¶
- vs. Standard On-Policy Distillation (OPD): Conventional OPD (Lu and Lab, 2025; Agarwal et al., 2024) enforces uniform token-level KL supervision across all student rollouts, risking diversity suppression on correct solutions and gradient contamination on failed attempts. SCOPE solves this via correctness routing and dual-path adaptive weighting.
- vs. Reinforcement Learning with Verifiable Rewards (RLVR / GRPO): GRPO (Shao et al., 2024) relies purely on terminal scalar rewards, facing severe credit assignment challenges and steep entropy decay. SCOPE leverages dense teacher guidance for error correction while using student-perplexity weighting to preserve reasoning diversity and prevent multi-sample performance saturation.
- vs. Process Reward Models (PRMs): While PRMs (Lightman et al., 2024) provide step-level supervision, they demand expensive data collection and transfer poorly across domains. SCOPE achieves equivalent process-level noise gating purely through sequence-level perplexity scores from pre-trained foundation models.
Rating¶
- Novelty: โญโญโญโญ [Clever dual-path formulation turning perplexity into a directional signal-quality filter for on-policy reasoning alignment]
- Experimental Thoroughness: โญโญโญโญโญ [Extensive evaluations across six math benchmarks, code generation, model scaling, ablation suites, and error-recovery case studies]
- Writing Quality: โญโญโญโญโญ [Clear motivation, rigorous mathematical derivations, and transparent discussion of limitations and trade-offs]
- Value: โญโญโญโญโญ [A plug-and-play, highly effective framework that bridges distillation efficiency and reinforcement learning diversity for reasoning LLMs]