SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training

Conference: NeurIPS 2026
arXiv: 2510.08141
Code: https://github.com/597358816/SCOPE-RL
Area: LLM Alignment
Keywords: Reinforcement Learning Alignment, Policy Entropy Collapse, Exploration Control, GRPO, LLM Reasoning

TL;DR

SCOPE-RL tackles the pervasive entropy collapse in GRPO-based post-training of reasoning LLMs by uncovering a temperature-sign asymmetry in entropy dynamics and introducing a lightweight (<2% compute overhead) temperature-adaptive positive-sample regularizer that reliably stabilizes exploration and boosts both Pass@1 and Pass@k.

Background & Motivation

Reinforcement learning with verifiable rewards (RLVR) has become the predominant paradigm for unlocking complex mathematical and logical reasoning capabilities in large language models. In particular, Group Relative Policy Optimization (GRPO) serves as the primary training backbone due to its critic-free design, high training throughput, and robust scalability across distributed clusters. However, policies trained via GRPO consistently suffer from an insidious failure mode known as entropy collapse: policy entropy decreases monotonically throughout training, model outputs become increasingly homogeneous, and stochastic exploration vanishes prematurely. As a result, the model ceases to explore alternative reasoning branches and merely sharpens behavior patterns that were already latent within the base pre-trained model.

Previous attempts to mitigate entropy collapse generally follow two avenues: explicitly adding maximum-entropy bonuses to rewards or advantages (e.g., Entropy-Reg, Entropy-Adv), or adopting asymmetric clip-higher ranges (e.g., DAPO, CISPO) to encourage larger policy steps. Unfortunately, both paradigms introduce severe optimization instability in practice. Policy entropy tends to oscillate erratically between collapse and explosion, often accompanied by irreversible reward drops. Consequently, practitioners have lacked a stable, predictable mechanism to steer policy entropy into a healthy exploration regime and quantitatively investigate how exploration intensity interacts with downstream task performance.

Revisiting entropy dynamics at the sample trajectory level reveals a previously overlooked temperature-sign asymmetry: under high-temperature sampling, positive feedback trajectories counter entropy decay while negative ones exacerbate collapse; under low-temperature sampling, this dynamic completely reverses. Core idea: construct an auxiliary regularization branch using less than 2% of positive samples sampled at a temperature dynamically adapted to the policy's entropy deficit, achieving stable, quantitative entropy regulation without disrupting the primary reward maximization.

Method

Overall Architecture

SCOPE-RL builds upon the standard RLVR and GRPO framework. Rather than injecting coarse global entropy penalties or aggressive asymmetric clipping bounds into the main objective, SCOPE-RL decouples exploration stabilization into a lightweight auxiliary regularization branch. During each training iteration, the main actor samples standard rollouts at temperature \(T=1.0\) and updates parameters using standard group-normalized advantages. Concurrently, the auxiliary branch tracks the policy's previous entropy \(\mathcal{H}(\pi_{\theta_{\text{old}}})\) relative to a chosen target threshold \(\mathcal{H}_0\), sets an adaptive sampling temperature \(T\), and filters for verified correct solutions (\(R=1\)) to compute an auxiliary clipped policy gradient that offsets the negative entropy drift.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Query Input q"] --> B["GRPO Main Rollout<br/>T=1.0, group size G=8"]
    A --> C["Monitor Policy Entropy<br/>Calculate Entropy Deficit"]
    C --> D["Adaptive Temperature Sampling<br/>T=clip(1+H0-H_old, 0.8, 1.2)"]
    D --> E["Positive Sample Filtering<br/>Keep only R(q,o)=1 rollouts"]
    B --> F["GRPO Group-Normalized Update"]
    E --> G["Auxiliary Regularization Branch<br/>Scaled by weight α=1/64"]
    F --> H["Joint Policy Gradient Optimization<br/>Maintains Target Entropy H0"]
    G --> H

Key Designs

1. Temperature-Sign Entropy Dynamics Asymmetry: Decoupling Sign and Temperature Interactions

Conventional intuition suggests that increasing global sampling temperature during training should naturally preserve diversity. However, empirical analysis shows that merely altering global temperature in vanilla GRPO does not stop entropy collapse. Investigating sample-level gradient updates reveals that the sign of the reward signal crucially governs entropy behavior: at high sampling temperatures, positive samples exert a positive derivative on policy entropy, acting as a natural restorative force against collapse, while negative samples accelerate collapse. At low temperatures, this dynamic inverts, but negative samples cause immediate reward collapse. In a contextual bandit abstraction with softmax policy parameterization, when one-step entropy drift \(\Delta\mathcal{H}_{\theta_k}(1) < 0\) (the condition of impending entropy collapse), the derivative of the entropy change with respect to temperature at \(T=1\) is strictly positive under positive-sample updates: $\(\left.\frac{\mathrm{d}\Delta\mathcal{H}_{\theta_k}(T)}{\mathrm{d}T}\right|_{T=1} > 0\)$ This theoretical insight provides the foundation for using high-temperature positive trajectories to stabilize policy entropy.

2. Adaptive Temperature Regulation: Closed-Loop Negative Feedback Control

Leveraging the above theoretical result, SCOPE-RL implements an adaptive negative-feedback loop. Given a user-specified target entropy level \(\mathcal{H}_0\) and the evaluated empirical entropy of the previous checkpoint \(\mathcal{H}(\pi_{\theta_{\text{old}}})\), the auxiliary sampling temperature adjusts dynamically. When \(\mathcal{H}(\pi_{\theta_{\text{old}}}) < \mathcal{H}_0\), indicating that the policy is collapsing toward determinism, the system increases temperature (\(T > 1\)) to recruit entropy-restoring positive samples. Conversely, when entropy drifts above \(\mathcal{H}_0\), temperature drops (\(T < 1\)) to prevent uncontrolled divergence. The temperature is bounded to maintain numerical stability: $\(T = \mathrm{clip}\left(1 + \mathcal{H}_0 - \mathcal{H}(\pi_{\theta_{\text{old}}}),\, 0.8,\, 1.2\right)\)$ This feedback loop steadily locks policy entropy within a tight band around \(\mathcal{H}_0\), providing the first predictable and quantitative exploration controller for RL post-training.

3. Minimal-Overhead Positive-Sample Filtering: Purifying Auxiliary Exploration Signals

Within the auxiliary branch, SCOPE-RL strictly discards all incorrect trajectories, retaining exclusively verifiable correct solutions (\(R(q, o_i) = 1\)). Incorporating negative samples introduces high gradient variance and risks precipitous reward degradation. Furthermore, the auxiliary batch requires only \(G'=64\) positive samples compared to the 4096 samples in the standard rollout batch (less than 2% computational overhead). By dynamically estimating the required auxiliary query count from the prior step's empirical pass rate, SCOPE-RL injects a pristine exploration regularizer at virtually zero additional compute cost.

Loss & Training

The overall SCOPE-RL training objective blends the vanilla GRPO objective with the auxiliary positive-sample regularizer: $\(\mathcal{J}_{\text{SCOPE}}(\theta) = \mathcal{J}_{\text{GRPO}}(\theta) + \alpha\,\mathbb{E}_{q \sim P(Q),\, \{o_i\}_{i=1}^{G'} \sim \pi_{\theta_{\text{old}}}^T}\left[\frac{1}{G'}\sum_{i=1}^{G'} \mathbf{1}[R(q,o_i)=1]\frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\left(r_{i,t}(\theta),\, \mathrm{clip}(r_{i,t}(\theta), 1-\epsilon, 1+\epsilon)\right)\right]\)$ where the likelihood ratio is computed relative to the unscaled reference policy: \(r_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t}\mid q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}\mid q, o_{i,<t})}\). Crucially, the intentional distribution mismatch between sampling from \(\pi_{\theta_{\text{old}}}^T\) and evaluating against \(\pi_{\theta_{\text{old}}}\) generates the necessary entropy-steering bias; ablations demonstrate that reweighting this ratio with importance sampling eliminates entropy control. Default parameters include clipping bound \(\epsilon=0.2\), auxiliary weight \(\alpha=64/4096=1/64\), and learning rate \(1\times 10^{-6}\).

Key Experimental Results

Main Results

On the DAPO-17K benchmark across three base model families, SCOPE-RL consistently outperforms GRPO and existing anti-collapse baselines on in-domain mathematical reasoning benchmarks (Pass@1).

Base Model & Method	AIME24	AIME25	AMC23	GSM8K	MATH	Minerva	Olympiad	Average
Qwen2.5-Math-7B	15.5	7.81	42.1	65.4	59.4	11.0	26.7	32.56
+ GRPO	32.1	11.0	72.4	88.7	80.6	34.6	41.8	51.60
+ Entropy-Reg (\(\lambda=0.01\))	34.2	10.1	73.1	88.7	79.8	35.3	40.8	51.71
+ DAPO	33.3	13.0	73.9	87.9	79.6	34.6	40.8	51.87
+ CISPO	35.0	13.8	75.1	89.7	80.2	37.1	41.6	53.21
+ KL-cov	32.1	13.5	73.9	89.9	79.8	35.6	41.7	52.36
+ SCOPE-RL	36.4	18.9	75.3	89.5	82.6	38.2	43.0	54.84 (+3.23)
Qwen2.5-7B + GRPO	17.1	7.60	65.8	92.3	75.6	36.8	38.8	47.71
Qwen2.5-7B + SCOPE-RL	17.5	12.9	69.3	92.9	78.0	37.8	40.3	49.81 (+2.10)
Qwen3-4B + GRPO	52.9	41.5	86.1	95.2	92.0	46.7	60.0	67.77
Qwen3-4B + SCOPE-RL	56.5	43.7	89.7	95.0	92.8	48.9	61.5	69.73 (+1.96)

In multi-sample reasoning evaluations (Pass@k) on AIME24 and AIME25, GRPO experiences marked degradation as training progresses (falling behind the original base model), whereas SCOPE-RL preserves vital exploratory coverage:

Method (Qwen2.5-7B)	AIME24 Pass@512	AIME24 Pass@1024	AIME25 Pass@512	AIME25 Pass@1024
Base Model	71.7	80.0	61.7	73.3
+ GRPO	63.3	73.3	60.0	63.3
+ SCOPE-RL	70.0	86.7	66.7	76.7

Ablation Study

Component ablations and target entropy sensitivity analyses on Qwen2.5-Math-7B confirm the necessity of each architectural component:

Configuration / Setting	AIME24	AIME25	AMC	MATH	Average	Entropy Dynamics
Full SCOPE-RL (\(\mathcal{H}_0=0.50\))	36.4	18.9	75.3	82.6	54.84	Stable at \(\approx 0.50\)
w/o temperature adaptation	31.4	11.8	74.5	79.0	51.31 (-3.53)	Entropy collapses to 0.018
positive + negative samples	32.7	11.0	73.2	79.0	51.56 (-3.28)	Entropy collapses to 0.020
only negative samples	28.9	12.7	65.1	77.6	49.71 (-5.12)	Severe instability / explodes to 7.23
Target Entropy \(\mathcal{H}_0=0.25\)	37.9	11.3	74.4	79.8	53.00 (-1.84)	Insufficient exploration
Target Entropy \(\mathcal{H}_0=0.75\)	33.6	15.1	77.0	79.2	53.45 (-1.39)	Excessive entropy, slight drift
Target Entropy \(\mathcal{H}_0=1.00\)	33.2	15.6	74.8	79.6	53.13 (-1.71)	Over-exploration degrades accuracy

Key Findings

• Non-monotonic Entropy-Performance Curve: Varying target entropy demonstrates that reasoning performance follows an inverted U-shaped relationship with exploration, peaking at moderate entropy (\(\mathcal{H}_0 = 0.50\)).
• Crucial Role of Positive-Sample Exclusivity: Including negative samples in the auxiliary branch causes rapid entropy collapse or catastrophic variance spikes, demonstrating that the entropy-stabilizing property is strictly unique to positive rollouts.
• Genuine Trajectory Discovery: Tracking prompt-level correctness categories across training epochs reveals that SCOPE-RL continuously converts "all-wrong" problems into solved problems while maintaining a higher fraction of "mixed" prompts, confirming that retained entropy translates into genuine reasoning discovery rather than random token noise.

Highlights & Insights

• Formulates a rigorous theoretical explanation for the temperature-sign asymmetry in policy entropy updates, moving beyond empirical heuristics.
• Delivers an exceptionally lightweight plug-and-play solution requiring less than 2% auxiliary rollouts, avoiding the complexity of critic networks or invasive modifications to the primary GRPO loss.
• Eliminates the steep diversity tax typically imposed by RL post-training, directly benefiting test-time compute scaling and best-of-N sampling pipelines.

Limitations & Future Work

• Theoretical Scope: The formal derivation is conducted in a contextual bandit framework with tabular softmax assumptions, leaving token-level autoregressive MDP guarantees for future theory.
• Global Scalar Granularity: The method regulates a single scalar global entropy, treating routine formatting tokens and pivotal reasoning fork points uniformly. Future work could incorporate token-level or step-level uncertainty awareness.
• Dependency on Base Model Competence: The auxiliary mechanism assumes the base model can generate at least some correct solutions (\(\approx 30\%\) pass rate); extreme cold-start tasks will still require initial demonstration bootstrapping.

Related Work & Insights

• vs GRPO [DeepSeekMath / DeepSeek-R1]: GRPO provides a cost-effective policy gradient framework but suffers from inevitable entropy collapse. SCOPE-RL acts as a non-invasive regularizer that sustains long-horizon exploration.
• vs DAPO [Yu et al., 2025]: DAPO utilizes asymmetric clipping to stimulate entropy expansion, which often precipitates reward collapse or optimization instability. SCOPE-RL replaces heuristic clipping with closed-loop temperature control.
• vs Entropy-Reg [Hou et al., 2025]: Global maximum-entropy penalties penalize deterministic behavior across all tokens and degrade solution quality. SCOPE-RL restricts entropy management to verified positive trajectories, avoiding adverse optimization bias.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Identifies and formalizes the temperature-sign asymmetry in sample-level entropy dynamics.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously tested across 4 model families, multiple random seeds, Pass@k metrics, and out-of-domain benchmarks.
• Writing Quality: ⭐⭐⭐⭐⭐ Crystal-clear presentation, intuitive motivations, and consistent mathematical notation.
• Value: ⭐⭐⭐⭐⭐ Provides an immediately deployable, low-overhead solution to one of the biggest bottlenecks in open reasoning RL.

Related Papers

• [ICLR 2026] Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead
• [NeurIPS 2026] Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation
• [NeurIPS 2026] Diffusion LLMs are Natural Adversaries for any LLM
• [NeurIPS 2026] When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy
• [NeurIPS 2026] Consolidating Rewarded Perturbations for LLM Post-Training