ERR+: Sequential Entropy Resolution for Efficient and Decisive LLM Reasoning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/XrkArul/err_response
Area: LLM Reasoning
Keywords: Large Language Models, Reinforcement Learning, Process Reward, Sequential Entropy Resolution, Reasoning Efficiency
TL;DR¶
To address within-group advantage collapse under binary rewards and the accuracy degradation caused by naive length penalties in RLVR, ERR+ introduces a two-phase framework that first rewards token-level entropy drops via Entropy Relief Reward (ERR) and subsequently refines conciseness using within-group tanh-standardized length penalties (RRER), simultaneously improving reasoning accuracy and reducing generation length.
Background & Motivation¶
In the prevailing paradigm of post-training large reasoning models via reinforcement learning with verifiable rewards (RLVR, such as GRPO and DAPO), models are encouraged to generate extended chain-of-thought (CoT) traces enclosed in <think>...</think> tokens to tackle complex mathematical and algorithmic challenges. However, when models achieve high pass rates within a sampled response group, binary correctness verification leads to a severe structural limitation: because all correct rollouts receive identical positive rewards, the within-group advantage variance collapses to zero. Consequently, the policy receives virtually no gradient guidance regarding internal reasoning quality, remaining indifferent between crisp, logically decisive deductions and rambling, overly verbose justifications that happen to reach the same final answer.
To incentivize conciseness and process efficiency, recent approaches have attempted to incorporate token-level entropy penalties or absolute length constraints into the reward function. Yet fundamental empirical analyses reveal that during reasoning generation, high-entropy tokens—which account for only a small minority of the trace (e.g., the top 20%)—represent indispensable exploratory junctions and branching forking points where the policy explores alternative hypotheses and critical problem-solving paths. Methods like PEAR directly penalize thinking-phase entropy as a length proxy, while step-function or linear length penalties exert indiscriminate pressure across all problems. By suffocating high-entropy exploration, these approaches degrade problem-solving accuracy on difficult tasks, cementing an undesirable trade-off between reasoning accuracy and token efficiency.
This paper tackles the challenge from a distinct angle: what distinguishes high-quality reasoning is not the suppression of exploratory high entropy, but whether the model decisively resolves uncertainty into low-entropy commitment once an exploratory phase concludes. Empirical tracking confirms that correct rollouts exhibit significantly more frequent and deeper token-level entropy drops compared to incorrect ones. Core idea: decouple reasoning structure optimization from length compression into a sequential two-phase framework, where Phase 1 rewards cumulative entropy relief (ERR) to foster decisive commitment without penalizing exploration, and Phase 2 introduces difficulty-aware tanh within-group relative efficiency rewards (RRER) to prune safe redundant spans after early gradient conflicts dissolve.
Method¶
Overall Architecture¶
The ERR+ framework executes across two sequential training stages. In Phase 1, the model is trained with the Entropy Relief Reward (ERR), which identifies local token-level entropy drops within the thinking sequence, normalizes their sum sub-linearly using a logarithmic length factor, and gates the bonus on final answer correctness to build a decisive solution backbone. Once Phase 1 converges and the proportion of unresolved high-entropy states drops to near zero, Phase 2 engages the Robust Relative Efficiency Reward (RRER). RRER standardizes response length against co-generated group peers using a tanh-transformed z-score, safely pruning verbose justifications between commitment points while preventing early gradient interference between exploration and compression.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Query and Group Rollout<br/>G sampled candidate reasoning paths"] --> B["Entropy Relief Reward (ERR)<br/>Log-normalized cumulative entropy drops"]
B --> C["Sequential Decoupled Schedule<br/>Phase switch after Theorem 1 gradient conflict resolves"]
C --> D["Robust Relative Efficiency Reward (RRER)<br/>Tanh-saturated within-group length z-score"]
D --> E["Decisive and Concise Reasoning Policy"]
Key Designs¶
1. Entropy Relief Reward (ERR): Rewarding uncertainty resolution rather than penalizing exploration
To prevent the detrimental suppression of exploratory thinking states while avoiding the perverse incentive to generate longer traces purely to collect more drop bonuses, ERR operates strictly on the positive drops of adjacent token entropies. For positions \(t \in [2, T_k]\) within the thinking span (where \(T_k\) denotes the </think> index), token-level relief is defined as \(r_t = \max(H_{t-1} - H_t - \epsilon, 0)\), filtering out ambient entropy fluctuations via noise margin \(\epsilon\). Instead of linear normalization, which unfairly penalizes hard problems requiring genuine multi-step chains, the cumulative drop magnitude is normalized sub-linearly by \(\log(T_k + 1)\):
Crucially, ERR is gated by correctness in the composite reward \(R_1(y, y^*) = \min(R_b + \lambda \text{ERR}(y), R_{\max})\): only responses reaching the verified correct answer receive the bonus. This design guarantees that spurious entropy drops within invalid reasoning paths are never reinforced, converting entropy drops into reliable indicators of fruitful deductive commitments.
2. Robust Relative Efficiency Reward (RRER): Difficulty-aware outlier-robust length shaping To overcome the limitations of fixed-budget length penalties that penalize difficult prompts disproportionately, RRER uses the empirical mean length of co-generated responses within the same group as a dynamic proxy for problem difficulty. For each response length \(L_i\), a within-group standard score \(z_i = (L_i - \mu_L) / (\sigma_L + \varepsilon)\) is calculated and transformed via a hyperbolic tangent mapping governed by scale hyperparameter \(\gamma\):
The tanh saturation naturally bounds \(\lambda_i \in (-1, 1)\), insulating the learning signal against extreme outlier trajectories (such as anomalous rollouts that reach the maximum generation limit). Moreover, RRER is allocated asymmetrically: correct solutions receive positive or negative shaping \(\alpha \lambda_i\), whereas incorrect or unparseable solutions are restricted to non-positive penalties \(\alpha \min(0, \lambda_i)\). This prevents the policy from hacking length rewards by outputting trivially brief wrong answers.
3. Sequential Decoupled Schedule: Resolving early gradient conflicts prior to structural pruning To theoretically explain why joint multi-objective optimization fails during early training, the authors formalize the interaction between the policy gradients of expected negative reward \(L_1(\theta)\) and \(L_2(\theta)\), denoted \(g_1(\theta)\) and \(g_2(\theta)\). During early training, a substantial fraction \(\rho(\theta)\) of the thinking tokens occupy the mid-exploration set \(T_\rho(y) = \{t \in T(y) : H_t > \bar{H}, r_t = 0\}\)—high-entropy states not yet followed by a committed drop. A joint update applying length pressure via \(g_2\) preferentially suppresses these non-rewarded high-entropy tokens, which directly removes the prerequisite context needed for subsequent entropy relief drops, establishing an explicit negative inner product:
where \(C = \lambda \Delta \bar{H} / \log(T_k + 1) > 0\). Theorem 1 proves that joint training strictly impairs Phase 1 learning by \(\eta C \rho(\theta) + O(\eta^2)\). However, once Phase 1 converges to \(\theta_1^*\), virtually all high-entropy states transition into committed drops, driving \(\rho(\theta_1^*) \approx 0\) and shifting the gradient cosine from \(-0.45\) to \(-0.05\) (and \(+0.02\) post Phase 2). Pruning can then safely target low-relief redundant spans without degrading the underlying reasoning structure.
Loss & Training¶
Both phases utilize standard Group Relative Policy Optimization (GRPO) advantage normalization without a dedicated value network, implemented within the open-source verl framework. The model is trained on 7,473 GSM8K training instances with a batch size of 128, learning rate \(1 \times 10^{-6}\), temperature 0.6, top-\(p\) 0.95, and 8 rollouts per prompt. Phase 1 runs for \(N_1\) steps until validation accuracy peaks, after which Phase 2 optimizes \(R_2\) for \(N_2\) steps. Default hyperparameters are \(\epsilon = 0.01\), \(\gamma = 0.5\), \(\lambda = 0.3\), \(\alpha = 0.3\), \(R_b = 1.0\), \(R_f = 0.1\), and \(R_{\max} = 1.5\).
Key Experimental Results¶
Main Results¶
Evaluated on DeepSeek-R1-Distill-Qwen-1.5B across five challenging reasoning benchmarks, ERR+ outperforms standard RLVR and competitive length-control baselines in both Pass@1 accuracy and average token length.
| Method | GSM8K (Acc / Tok) | AIME24 (Acc / Tok) | AMC23 (Acc / Tok) | MATH-500 (Acc / Tok) | MMLU-STEM (Acc / Tok) | Average (Acc / Tok) |
|---|---|---|---|---|---|---|
| Base (Zero-shot) | 84.6% / 2076 | 26.7% / 12958 | 70.0% / 8063 | 82.1% / 4729 | 51.7% / 6266 | 63.0% / 6818 |
| DEER | 85.7% / 1553 | 26.7% / 10692 | 75.0% / 5556 | 83.8% / 3534 | 50.4% / 6391 | 64.3% / 5545 |
| Dynasor | 85.1% / 1916 | 26.7% / 11687 | 75.0% / 7287 | 82.0% / 4351 | 49.6% / 6670 | 63.7% / 6382 |
| PEAR | 85.3% / 1949 | 33.3% / 12260 | 67.5% / 8015 | 81.0% / 4848 | 50.7% / 6033 | 63.6% / 6621 |
| GRPO (Binary reward) | 86.6% / 2281 | 30.0% / 12939 | 70.0% / 8336 | 82.6% / 5206 | 51.6% / 7440 | 64.2% / 7240 |
| Laser-L2048 | 82.8% / 1850 | 26.7% / 10515 | 75.0% / 7503 | 80.3% / 4529 | 50.4% / 5632 | 63.0% / 6006 |
| DAPO | 85.4% / 2123 | 26.7% / 12956 | 70.0% / 8065 | 83.4% / 4432 | 49.9% / 6966 | 63.1% / 6908 |
| GRPO+ERR+ (Ours) | 88.6% / 1324 | 33.3% / 10670 | 77.5% / 6288 | 84.8% / 3857 | 58.0% / 5109 | 68.4% / 5450 |
| DAPO+ERR+ (Ours) | 87.1% / 1465 | 30.0% / 10542 | 75.0% / 5603 | 85.2% / 3296 | 54.2% / 4742 | 66.3% / 5130 |
On larger backbones (Qwen3-4B, Qwen3-8B, and DeepSeek-R1-Distill-Qwen-7B), ERR+ consistently achieves superior performance. On the 7B model, GRPO+ERR+ reaches 82.5% average accuracy with 4495 tokens (compared to GRPO's 79.4% with 5509 tokens), and improves AIME24 accuracy from 50.0% to 60.0%.
Ablation Study¶
The ablation isolates the contribution of each training stage and normalizer choice on DeepSeek-R1-Distill-Qwen-1.5B.
| Variant | GSM8K (Acc / Tok) | AIME24 (Acc / Tok) | AMC23 (Acc / Tok) | MATH-500 (Acc / Tok) | MMLU-STEM (Acc / Tok) | Average (Acc / Tok) | Note |
|---|---|---|---|---|---|---|---|
| ERR+ (Full sequential) | 88.6% / 1324 | 33.3% / 10670 | 77.5% / 6288 | 84.8% / 3857 | 58.0% / 5109 | 68.4% / 5450 | Best accuracy and balanced conciseness |
| w/o Phase 1 (Phase 2 only) | 85.0% / 1244 | 26.7% / 10729 | 67.5% / 6933 | 84.0% / 4253 | 51.4% / 4168 | 62.9% / 5465 | Length compressed prematurely; -5.5% avg acc |
| w/o Phase 2 (Phase 1 only) | 88.1% / 1612 | 33.3% / 12807 | 72.5% / 8061 | 84.6% / 4752 | 55.6% / 5895 | 66.8% / 6625 | Strong accuracy, but responses remain 21.5% longer |
| Joint training (\(R_1 + R_2\)) | 84.2% / 2005 | 23.3% / 12105 | 72.5% / 7245 | 83.0% / 4470 | 49.9% / 6470 | 62.6% / 6459 | Severe early gradient conflict; worst accuracy |
Comparing Phase 1 normalizers, \(\log(T_k + 1)\) yields 68.4% accuracy with 5450 tokens, whereas linear \(T_k\) over-penalizes extended reasoning, collapsing accuracy to 48.6% (2063 tokens), and square-root \(\sqrt{T_k}\) achieves only 53.3% (3314 tokens).
Key Findings¶
- Structure-then-compress mechanics: Fine-grained trace inspection reveals that Phase 1 doubles the frequency of large entropy drops (>0.5 nats) from 6.5 to 12.8 per 100 tokens and tightens the average gap between drops from 15.4 to 7.8 tokens. Consequently, safe-prune accuracy (continuation accuracy after replacing all low-relief spans with ellipsis) surges from 72.4% to 84.3%. Phase 2 subsequently reduces the proportion of low-relief spans from 35.8% to 24.5% while holding safe-prune accuracy firm at 84.8%, empirically confirming that Phase 2 safely compresses the exact low-value material that Phase 1 structured.
- Resistance to reward hacking: Monitoring confirms that the model does not exploit ERR via artificial entropy oscillation. The ratio of cumulative upward entropy movement to rewarded drops (Up/Drop) remains invariant at 1.0007–1.0011 throughout training, and the frequency of entropy-direction switches per trace decreases from 1012.98 (GRPO) to 609.77 (ERR+), demonstrating authentic convergence toward decisive reasoning.
- Cross-domain out-of-distribution generalization: When tested on the graduate-level scientific benchmark GPQA Diamond without any in-domain training, GSM8K-trained ERR+ achieves 31.6% accuracy with 10,961 tokens, exceeding both baseline GRPO (29.3% / 14,528 tokens) and Base (27.8% / 13,875 tokens), proving that decisive reasoning habits transfer across domains.
Highlights & Insights¶
- From entropy suppression to entropy resolution: Instead of treating high entropy as a pathology to be penalized, ERR+ recognizes high-entropy states as necessary exploratory precursors and rewards the decisive resolution of uncertainty.
- Theoretically founded schedule: Theorem 1 provides mathematical rigor for understanding gradient interference between exploration and efficiency rewards, demonstrating that sequential training is an analytical necessity rather than an empirical trick.
- Outlier-robust relative shaping: Combining within-group peer normalization with tanh saturation provides scale-invariant, difficulty-aware length shaping that resists gradient pollution from runaway outlier rollouts.
Limitations & Future Work¶
- Lack of human-centric pedagogical objectives: Optimizing strictly for accuracy and token efficiency can produce dense, austere reasoning steps that may be less pedagogically accessible to human readers seeking gradual step-by-step exposition.
- Supervised phase-switching threshold: Transitioning from Phase 1 to Phase 2 currently relies on validation accuracy checkpoint monitoring; future work could formulate adaptive gradient-alignment thresholds for autonomous transitions.
Related Work & Insights¶
- vs PEAR: PEAR penalizes average thinking entropy as a length proxy, directly dampening the exploratory positions vital for solving hard problems and causing severe accuracy drops on AIME24; ERR+ leaves high entropy unpenalized and rewards only downstream drops.
- vs LASER / Kimi k1.5: LASER employs rigid step-function length penalties and Kimi k1.5 relies on extreme-value normalization; ERR+ leverages group-relative z-scores saturated via tanh, offering superior resilience to outliers and adapting dynamically to varying problem difficulties.
- vs DAPO / TokenSqueeze: While DAPO stabilizes RLVR through dynamic sampling and TokenSqueeze uses post-hoc token pruning, ERR+ instills intrinsic, decisive reasoning habits during policy optimization, naturally complementing algorithmic sampling improvements.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates an innovative perspective that rewards entropy drops rather than penalizing entropy presence, supported by a formal gradient conflict proof.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 4 model backbones, 5 diverse benchmarks, bootstrap running-mean traces, anti-hacking metrics, and safe-prune analyses.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative progression connecting empirical observations, mathematical formulations, and microscopic mechanistic verification.
- Value: ⭐⭐⭐⭐⭐ Provides an effective, non-invasive post-training recipe for long CoT reasoning models without requiring expensive step-level process reward annotations.