SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/machine981/SCOPE
Area: LLM Reasoning
Keywords: On-Policy Distillation, Signal Calibration, Dual-Path Adaptive Weighting, Reasoning Diversity, Pass@k Paradox
TL;DR¶
SCOPE addresses the structural flaw of uniform dense supervision in On-Policy Distillation (OPD) by routing rollouts by correctness into a dual-path adaptive weighting framework: filtering flawed prefix noise on incorrect rollouts via inverse teacher perplexity, and amplifying under-explored boundary reasoning on correct rollouts via student perplexity, decisively breaking the Pass@k paradox.
Background & Motivation¶
In the reasoning alignment of large language models (LLMs), reinforcement learning with verifiable rewards (RLVR, e.g., GRPO) has emerged as the dominant paradigm. However, sparse, outcome-level scalar rewards dispensed only at terminal tokens render token-level credit assignment notoriously difficult, demanding massive exploration iterations to converge. To alleviate this sample inefficiency, On-Policy Distillation (OPD) has gained substantial traction: it allows a weaker student model to self-sample exploration rollouts while receiving dense token-level reverse Kullback-Leibler (KL) divergence supervision from a capable teacher model, striking an effective balance between distribution consistency and learning efficiency.
Despite this progress, prevailing OPD algorithms operate under a strong implicit assumption—that the teacher's dense supervisory feedback is uniformly reliable across all student-sampled trajectories. In complex multi-step reasoning, this assumption breaks down in two critical aspects. On successful trajectories, uniform teacher supervision or uniform positive sample reinforcement drives the student's policy toward dominant reasoning modes, causing severe mode collapse and the "Pass@k paradox" (where Pass@1 increases while multi-sample exploration Pass@32 severely degrades). Conversely, on failed trajectories, weaker students frequently generate logically corrupted, repetitive, or numerically exploded reasoning prefixes; conditioning a strong teacher on these flawed prefixes traps it in an out-of-distribution regime, causing its outputs to degenerate into high-entropy predictive noise rather than informative corrections.
This reveals a profound duality in signal calibration: incorrect rollouts require teacher guidance but must be filtered according to the teacher's actual grasp of the context (indicated by teacher perplexity), whereas correct rollouts should bypass rigid teacher constraints and be adaptively weighted by the student's own uncertainty to consolidate fragile capability boundaries. Core idea: Formulate a Signal-Calibrated On-Policy Distillation Enhancement (SCOPE) framework that routes on-policy rollouts by verified correctness, applying teacher-perplexity-weighted KL distillation to filter noise on failed trajectories and student-perplexity-weighted MLE to amplify unconventional valid paths on successful ones under group-level normalization.
Method¶
Overall Architecture¶
SCOPE discards the conventional approach of applying uniform reverse KL divergence to all sampled rollouts. Its pipeline operates across three cohesive stages: outcome-driven group branching, dual-path adaptive weighting (DPAW), and unified group-normalized policy optimization. Given an input prompt \(x\), the student model samples a cohort of \(N\) candidate responses \(Y_x = \{y_1, y_2, \dots, y_N\}\) on-policy. A deterministic verifier evaluates each rollout to produce binary outcomes \(R_i \in \{0, 1\}\), partitioning the group into a correct subset \(\Omega_x^c\) and an incorrect subset \(\Omega_x^w\). Correct rollouts bypass the teacher model to preserve exploration diversity and are optimized via student-perplexity-weighted Maximum Likelihood Estimation (MLE). Incorrect rollouts are assigned to the teacher model for reverse KL distillation, where weights are inversely proportional to teacher perplexity to discard prefix-induced hallucinations. Local Softmax normalization within each non-empty subset eliminates cross-prompt difficulty discrepancies before the unified objective updates the student policy.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["Input Prompt x"] --> B["Student On-Policy Sampling<br/>Rollout Candidate Group Y_x"]
B --> C{"Outcome-Driven Group Branching<br/>Verifier Binary Reward R_i"}
subgraph DPAW["Dual-Path Adaptive Weighting (DPAW)"]
direction TB
C -->|"Correct Rollouts R_i=1 (Ω_x^c)"| D["Student-Perplexity-Weighted MLE<br/>Amplify Boundary Explorations"]
C -->|"Incorrect Rollouts R_i=0 (Ω_x^w)"| E["Teacher-Perplexity-Weighted OPD<br/>Filter Prefix-Induced Noise"]
D --> F["Group-Level Softmax Normalization<br/>Calibrate Prompt Difficulty Spread"]
E --> F
end
F --> G["Unified Optimization Loss J_SCOPE<br/>Weighted Gradient Backpropagation"]
Key Designs¶
1. Outcome-Driven Group Branching: Decoupling verification subsets to prevent mode collapse and misleading imitation
Standard OPD forces all trajectories—whether successful or failed—to match the teacher's next-token distributions, which needlessly distorts valid unconventional reasoning paths discovered by the student while risking the imitation of corrupted teacher completions on failed prefixes. SCOPE routes responses strictly based on deterministic reward verification into \(\Omega_x^c = \{y_i \in Y_x \mid R_i = 1\}\) and \(\Omega_x^w = \{y_i \in Y_x \mid R_i = 0\}\). This formulation dynamically accommodates all-correct, all-incorrect, or mixed cohorts without requiring fixed generation proportions, seamlessly passing through whichever branches are active for a given prompt.
2. Student-Perplexity-Weighted MLE: Amplifying unconventional valid paths to resolve the Pass@k paradox
For correct rollouts (\(\Omega_x^c\)), uniform reinforcement over-allocates gradient updates to easily mastered, high-probability reasoning chains, causing rapid entropy collapse and stifling alternative reasoning routes. SCOPE leverages the student's own sequence perplexity \(\text{PPL}_S(y_i \mid x) = \exp(-\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \log \pi_S(a_{i,t} \mid x, a_{i,<t}))\) as a proxy for trajectory-level uncertainty. The adaptive weight scales positively with student perplexity: $\(w_i^{\text{stu}} = \frac{\exp\left(-\frac{1}{\tau |y_i|} \log \pi_S(y_i \mid x)\right)}{\sum_{j \in \Omega_x^c} \exp\left(-\frac{1}{\tau |y_j|} \log \pi_S(y_j \mid x)\right)} = \frac{\text{PPL}_S(y_i \mid x)^{1/\tau}}{\sum_{j \in \Omega_x^c} \text{PPL}_S(y_j \mid x)^{1/\tau}}\)$ This weighting acts as an inverse-probability correction: it reduces the dominant sampling frequency exponent in the expected gradient from \(\pi_S\) to \(\pi_S^{1 - 1/(\tau|y|)}\), thereby actively amplifying low-probability yet valid paths at the student's capability boundary and counteracting the Pass@k paradox.
3. Teacher-Perplexity-Weighted OPD: Filtering out prefix-induced noise on flawed trajectories
On failed rollouts (\(\Omega_x^w\)), the student's reasoning prefix is frequently derailed by faulty premises, logic contradictions, or repetitive loops. Empirical truncation tests reveal that when the teacher exhibits high perplexity on an erroneous prefix (Q4 bucket, mean PPL=2.383), its recovery rate drops severely to 28.6%~45.4%, indicating that its token distributions have collapsed into uninformative noise. SCOPE assigns weights inversely proportional to teacher perplexity: $\(w_i^{\text{tea}} = \frac{\exp\left(\frac{1}{\tau |y_i|} \log \pi_T(y_i \mid x)\right)}{\sum_{j \in \Omega_x^w} \exp\left(\frac{1}{\tau |y_j|} \log \pi_T(y_j \mid x)\right)} = \frac{\text{PPL}_T(y_i \mid x)^{-1/\tau}}{\sum_{j \in \Omega_x^w} \text{PPL}_T(y_j \mid x)^{-1/\tau}}\)$ By penalizing trajectories where the teacher is confused and focusing on low-PPL instances (Q1 bucket, mean PPL=1.361, achieving up to 64.9% recovery rate), SCOPE extracts pure, reliable corrective gradients while ignoring corrupt prefixes.
4. Group-Level Softmax Normalization: Stabilizing updates across heterogeneous prompt difficulties
Mathematical and algorithmic prompts inherently vary in structural complexity, token length, and baseline perplexity scale. Global or unnormalized weighting would allow outlier prompts with extreme raw perplexity values to dominate gradient magnitudes. SCOPE confines the Softmax normalization strictly within the candidate rollout group for each prompt, ensuring \(\sum_{i \in \Omega} w_i = 1\) with \(0 < w_i < 1\). Under triangle inequality bounds, the gradient of each active branch forms a convex combination of individual trajectory gradients (\(\|\sum w_i g_i\| \le \max \|g_i\|\)), preserving numerical stability while effectively injecting signal-calibrated selectivity.
Loss & Training¶
Using the token-level on-policy importance sampling ratio \(\rho_{i,t}(\theta) = \frac{\pi_\theta(a_{i,t} \mid x, a_{i,<t})}{\pi_{\text{old}}(a_{i,t} \mid x, a_{i,<t})}\), the unified SCOPE loss objective \(J_{\text{SCOPE}}\) over dataset \(\mathcal{D}\) is formulated as: $\(J_{\text{SCOPE}} = \mathbb{E}_{x \sim \mathcal{D}} \left[ \sum_{i \in \Omega_x^c} w_i^{\text{stu}} \cdot \mathcal{L}_{\text{MLE}}(x, y_i; \theta) + \lambda \sum_{i \in \Omega_x^w} w_i^{\text{tea}} \cdot \mathcal{L}_{\text{OPD}}(x, y_i; \theta) \right]\)$ where: $\(\mathcal{L}_{\text{MLE}}(x, y_i; \theta) = - \sum_{t=1}^{|y_i|} \rho_{i,t}(\theta)\)$ $\(\mathcal{L}_{\text{OPD}}(x, y_i; \theta) = \sum_{t=1}^{|y_i|} \rho_{i,t}(\theta) \left[ \log \bar{\pi}_\theta(a_{i,t} \mid x, a_{i,<t}) - \log \pi_T(a_{i,t} \mid x, a_{i,<t}) \right]\)$ Here \(\bar{\pi}_\theta\) denotes detached model parameters and \(\lambda\) is a branch balancing coefficient. Experiments were conducted on a 20-GPU distributed cluster (16 GPUs for the student, 4 GPUs for the teacher). Optimization employs a global batch size of 256 prompts, \(G=8\) rollouts per prompt, rollout temperature \(\tau=0.6\), and weighting temperature \(\tau=1.0\).
Benchmark & Evaluation Protocol¶
Main Mathematical Reasoning Results¶
Evaluated on six challenging mathematical reasoning benchmarks across two diverse model architectures: DeepSeek-R1-Distill-Qwen-1.5B (student) paired with SkyWork-OR1-7B (teacher), and Qwen3-1.7B-Base (student) paired with Qwen3-8B (teacher). The metrics report expected accuracy stability Avg@32 (A@32) and multi-sample exploration capability Pass@32 (P@32):
| Student Model | Training Method | AIME24 A@32 / P@32 | AIME25 A@32 / P@32 | AMC23 A@32 / P@32 | MATH500 A@32 / P@32 | Olympiad A@32 / P@32 | Average A@32 / P@32 |
|---|---|---|---|---|---|---|---|
| R1-Distill-1.5B | Base Model | 29.4 / 76.5 | 23.9 / 46.9 | 72.7 / 94.7 | 84.6 / 97.3 | 44.2 / 67.5 | 47.9 / 73.1 |
| (Teacher: | + GRPO | 35.5 / 68.3 | 24.5 / 45.1 | 75.1 / 95.0 | 87.0 / 96.7 | 40.5 / 67.7 | 49.6 / 71.1 |
| SkyWork-OR1-7B) | + KD | 26.6 / 71.4 | 22.2 / 45.6 | 69.1 / 96.3 | 84.1 / 97.4 | 39.7 / 66.9 | 45.4 / 72.0 |
| + OPD | 40.2 / 75.0 | 28.9 / 48.5 | 75.9 / 95.0 | 89.0 / 97.7 | 44.9 / 69.3 | 52.3 / 73.1 | |
| + SCOPE (Ours) | 42.7 / 77.9 | 30.4 / 50.9 | 80.9 / 97.2 | 89.8 / 97.9 | 49.7 / 70.9 | 55.2 / 75.0 | |
| Relative vs. OPD | +6.22% / +3.87% | +5.19% / +4.95% | +6.59% / +2.32% | +0.90% / +0.20% | +10.69% / +2.31% | +5.54% / +2.60% | |
| Qwen3-1.7B-Base | Base Model | 3.7 / 26.7 | 3.2 / 20.7 | 24.6 / 69.4 | 54.8 / 88.6 | 16.3 / 48.0 | 19.0 / 49.2 |
| (Teacher: | + GRPO | 7.5 / 27.2 | 4.2 / 24.5 | 33.1 / 60.0 | 62.9 / 89.8 | 25.9 / 50.9 | 26.5 / 50.4 |
| Qwen3-8B) | + KD | 4.6 / 18.9 | 5.8 / 24.8 | 29.8 / 66.1 | 50.2 / 86.2 | 13.3 / 41.4 | 20.1 / 46.8 |
| + OPD | 12.2 / 31.5 | 10.6 / 29.7 | 43.1 / 80.1 | 67.9 / 90.7 | 25.3 / 53.7 | 30.6 / 55.9 | |
| + SCOPE (Ours) | 13.3 / 31.5 | 12.1 / 35.6 | 46.3 / 83.0 | 70.9 / 93.0 | 24.6 / 54.6 | 32.5 / 58.6 | |
| Relative vs. OPD | +9.02% / 0.00% | +14.15% / +19.87% | +7.42% / +3.62% | +4.42% / +2.54% | -2.77% / +1.68% | +6.21% / +4.83% |
Against recent state-of-the-art OPD variants on the 1.5B student across AIME24, AIME25, and AMC23, SCOPE achieves 51.3% Avg@32 and 75.3% Pass@32, outperforming ExOPD (46.5% / 72.0%) and RLAD (48.5% / 72.4%). On code generation tasks (HumanEval, Codeforces, LiveCodeBench), SCOPE delivers an average Avg@32 of 44.6% (+4.69% relative to OPD) and Pass@32 of 60.3%. In model scaling experiments (Qwen3-32B teacher \(\rightarrow\) Qwen3-8B student), SCOPE achieves 71.3% Avg@32 and 90.3% Pass@32, consistently maintaining its superiority over OPD (68.8% / 87.4%).
Ablation Studies¶
Conducted on the challenging AIME24 and AIME25 benchmarks to dissect the individual components of DPAW and verify the directional correctness of the perplexity weighting:
| Configuration Variant | AIME24 Avg@32 (%) | AIME24 Pass@32 (%) | AIME25 Avg@32 (%) | AIME25 Pass@32 (%) | Empirical Finding & Mechanism |
|---|---|---|---|---|---|
| Full SCOPE (DPAW) | 42.7 | 77.9 | 30.4 | 50.9 | Optimal trade-off between accuracy and diversity |
| w/o Teacher-guided Weight | 40.5 | 76.5 | 29.1 | 48.0 | Fails to suppress prefix noise; degraded rectification |
| w/o Student-guided Weight | 41.2 | 74.1 | 29.5 | 47.2 | Suppresses unconventional solutions; severe diversity drop |
| w/o DPAW (Uniform Weights) | 39.8 | 74.0 | 28.5 | 45.7 | Treats all signals equally; AIME25 Pass@32 drops 5.2% |
| Opposite Teacher Weight | 38.6 | 73.8 | 27.6 | 46.1 | Heavily weights corrupted prefixes; Avg@32 drops 4.1% |
| Opposite Student Weight | 40.8 | 72.9 | 29.0 | 46.8 | Over-reinforces mastered modes; accelerates collapse |
Key Findings¶
- Teacher error recovery is bounded by prefix perplexity: Conditioning the teacher model on truncated error trajectories reveals that low-PPL prefixes (Q1 bucket) consistently achieve substantially higher recovery rates than high-PPL ones (Q4 bucket), displaying an advantage margin of up to +19.4% (64.9% vs. 45.4% at 20% truncation). High teacher perplexity directly diagnoses semantic incoherence and noise generation.
- Diversity preservation breaks the Pass@k paradox: Unlike GRPO which exhibits catastrophic entropy decay from the initial training steps, SCOPE sustains healthy policy entropy throughout training. By actively boosting boundary valid trajectories, SCOPE continues scaling performance up to \(k=32\), avoiding the premature saturation observed in OPD and GRPO.
- Unimodal temperature response: Setting \(\tau = 1.0\) achieves the optimal balance. An overly sharp temperature (\(\tau=0.5\)) concentrates gradients excessively on extreme outliers, whereas a wide temperature (\(\tau=2.0\)) flattens the distribution back toward flawed uniform weighting.
Highlights & Insights¶
- Sequence perplexity as an elegant zero-overhead proxy for signal reliability: Rather than introducing compute-heavy and error-prone Process Reward Models (PRMs), SCOPE cleverly utilizes sequence-level length-normalized log-probabilities natively calculated during on-policy rollout passes to quantify signal quality.
- De-mythologizing teacher infallibility: SCOPE systematically uncovers the "flawed prefix trap" in on-policy distillation, proving that powerful teachers can be led astray by logically derailed student prefixes. Down-weighting noisy teacher evaluations is critical to stable alignment.
- Seamless generalization to verifiable tasks: The framework relies solely on verifiable binary outcomes, enabling immediate plug-and-play adoption across code generation, automated theorem proving, and multi-turn agent tool-use domains.
Limitations & Future Work¶
- Reliance on deterministic outcome verifiers: SCOPE currently requires automated, verifiable binary signals to branch trajectories. Extending the approach to open-ended generation, creative writing, or subjective instruction-following will require integration with learned reward models or multi-agent verification juries.
- Wall-clock latency in synchronous training: In the experimental setup, student sampling and teacher log-probability scoring are executed synchronously, resulting in a per-step wall-clock time of 641.9s compared to 459.0s for GRPO (with teacher scoring accounting for 200.0s). Adopting an asynchronous execution pipeline would eliminate this idle time.
- Scaling to ultra-large and MoE architectures: The largest tested student-teacher pair was 8B-32B. Investigating how dual-path signal calibration behaves with 70B+ dense models and sparsely gated Mixture-of-Experts (MoE) remains an exciting open avenue.
Related Work & Insights¶
- vs. Standard On-Policy Distillation (OPD): Prior OPD works (Lu & Thinking Machines, 2025; Agarwal et al., 2024) blindly enforce reverse KL distillation across all student trajectories. SCOPE identifies the severe signal heterogeneity between correct and incorrect paths, using dual-path weighting to filter prefix noise and stimulate diversity simultaneously.
- vs. Reinforcement Learning with Verifiable Rewards (RLVR / GRPO): Algorithms like GRPO (Shao et al., 2024) rely solely on terminal scalar rewards, triggering rapid entropy collapse and the Pass@k paradox. SCOPE preserves exploration diversity on correct rollouts via perplexity weighting while providing dense, noise-filtered guidance on failed rollouts.
- vs. Recent RL-KD Hybrid Frameworks (ExOPD / RLAD): Recent methods either extrapolate rewards beyond the teacher's distribution (ExOPD) or inject teacher signals into reward shaping (RLAD). However, both presume uniform teacher reliability. SCOPE introduces a necessary confidence gate that consistently yields superior accuracy across mathematical and coding benchmarks.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates an insightful critique of uniform supervision in on-policy distillation, elegantly resolving the flawed prefix trap and Pass@k paradox via dual-path perplexity calibration.
- Empirical Rigor: ⭐⭐⭐⭐⭐ Comprehensive evaluation across six math benchmarks, three coding tasks, model scaling, truncation recovery studies, and thorough ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Cohesive narrative progression from empirical motivation to rigorous theoretical derivations and qualitative case analyses.
- Significance: ⭐⭐⭐⭐⭐ Provides a vital, highly generalizable optimization blueprint for post-training reasoning alignment that bridges RLVR and knowledge distillation.