Skip to content

InfoDensity: Rewarding Information-Dense Traces for Efficient Reasoning

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/amao0o0/InfoDensity
Area: LLM Reasoning
Keywords: Reinforcement Learning, Efficient Reasoning, Entropy Trajectory, Suffix-Max Envelope, Accuracy-Efficiency Trade-off

TL;DR

InfoDensity addresses reasoning verbosity and length-penalty reward hacking by identifying that high-quality reasoning traces exhibit low uncertainty convergence and fast descent along their entropy trajectories, and introduces a single suffix-max envelope reward coupled with group-relative length scaling in reinforcement learning to simultaneously compress response lengths and enhance accuracy.

Background & Motivation

Large reasoning models (LRMs) such as DeepSeek-R1 and the Qwen series achieve state-of-the-art problem-solving capabilities by producing extensive multi-step Chain-of-Thought (CoT) traces. However, these models inherently generate verbose, repetitive, and redundant intermediate steps, imposing severe token overhead, high inference latency, and exorbitant serving costs. To curtail excessive deliberation, conventional reinforcement learning (RL) techniques have incorporated explicit length penalties or token-budget schedules into policy optimization objectives. Nevertheless, optimizing exclusively for final sequence length and terminal answer correctness leaves intermediate reasoning steps entirely unsupervised. Under aggressive length minimization, models frequently fall prey to reward hacking—learning to emit deceptively concise answers that satisfy strict length budgets while omitting essential deductive logic, which fundamentally degrades reasoning fidelity.

Supervising the intrinsic quality of intermediate reasoning steps without costly human annotations poses a significant challenge. While Process Reward Models (PRMs) require extensive step-level labels or computationally prohibitive tree searches, information-theoretic formulations offer a promising path by utilizing the model's self-predictive token distributions. However, whether information-theoretic metrics can reliably distinguish step validity has lacked thorough empirical validation. A fine-grained step-level analysis on ProcessBench reveals that single-step uncertainty reduction (\(\Delta H_t\)) is overwhelmingly noisy across models and datasets, exhibiting negligible effect sizes (Cohen's d between \(-0.02\) and \(+0.12\)) and ROC-AUC scores hovering around \(0.50\). Consequently, treating localized step-wise entropy changes as an error detection signal proves largely ineffective.

Crucially, shifting the analytical lens from isolated steps to the macro-level predictive entropy trajectory \(\{H_1, H_2, \dots, H_T\}\) across the entire reasoning trace reveals two pronounced, consistent characteristics: correct traces systematically achieve lower predictive entropy at the conclusion of thinking (low uncertainty convergence) and exhibit a significantly steeper rate of entropy reduction across intermediate steps (fast uncertainty descent). Conversely, erroneous traces maintain elevated entropy throughout or progressively accumulate confusion. These findings indicate that high-quality reasoning traces are fundamentally information-dense—each generated token meaningfully contributes toward resolving the model's uncertainty regarding the correct answer. Core idea: supervise reasoning quality directly by constructing a single suffix-max envelope over the entropy trajectory that jointly captures low uncertainty convergence and rapid uncertainty descent, combined with group-relative length scaling, to incentivize information-dense and concise reasoning during RL.

Method

Overall Architecture

The InfoDensity training pipeline integrates an entropy trajectory-based quality reward with group-relative length scaling within a Group Relative Policy Optimization (GRPO) framework. During rollout, the policy autoregressively generates reasoning traces for input queries. Token-level predictive entropy is extracted from top-\(K\) log probabilities via vLLM and partitioned into fixed chunks (\(C=50\) tokens) to construct stable chunk-level entropy sequences \(\{H_1, H_2, \dots, H_T\}\). The framework computes a right-to-left suffix-max envelope to penalize residual uncertainty and reward rapid descent, yielding an intrinsic quality score \(R_{\text{quality}}(\tau)\). Concurrently, a group-relative length scaling factor \(R_L(\tau)\) adjusts rewards based on candidate length relative to its rollout group. Finally, the multiplicative composite reward \(R_{\text{InfoDensity}}(\tau)\) is gated by final-answer correctness to update policy weights.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Query X<br/>GRPO sample rollout group"] --> B["Chunk-level Entropy Computation<br/>top-K truncated entropy with C=50 chunking"]
    B --> C["Suffix-Max Envelope Aggregation<br/>capture low convergence and fast descent"]
    C --> D["Group-Relative Length Scaling<br/>unbiased conciseness incentive without fixed targets"]
    D --> E["Answer-Gated Joint Reward<br/>assign joint reward to correct traces, zero otherwise"]
    E --> F["Policy Optimization<br/>GRPO update for information-dense reasoning"]

Key Designs

1. Chunk-level Entropy Computation: Mitigating Step Delimitation Drift and Micro-noise

Standard step-level evaluations rely on newline markers (\n\n) to parse reasoning boundaries, yet exploratory RL policies frequently produce non-standard formatting that breaks heuristic delimiters. Furthermore, raw per-token entropy suffers from lexical and syntactic noise unrelated to logical progression. InfoDensity addresses this by extracting per-token Shannon entropy using the top-\(K\) (\(K=20\)) truncated next-token probability distribution returned directly by vLLM: $\(H(y_{t,k} \mid c_{t,k}) = -\sum_{v \in V_K} \tilde{p}_\theta(v \mid c_{t,k}) \log \tilde{p}_\theta(v \mid c_{t,k})\)$ Because reasoning distributions concentrate overwhelming probability mass on top-20 candidates, this truncated estimate closely approximates the full softmax distribution while maintaining an absolute theoretical upper bound of \(\log K\). Rather than relying on fragile string splitting, the reasoning trace is sliced into uniform chunks of \(C=50\) tokens, averaging token entropies within each chunk into a discrete sequence \(H_t\). This fixed chunking eliminates syntactic parsing failures and smooths localized high-frequency token fluctuations.

2. Suffix-Max Envelope Aggregation: Jointly Constraining Trajectory Convergence and Descent

A naive trajectory mean \(\frac{1}{T}\sum H_t\) fails to catch instances where a model appears decisive early on but deteriorates into late-stage confusion or loops; conversely, penalizing only the terminal entropy step leaves intermediate rambling unconstrained. InfoDensity constructs a right-to-left running maximum—termed the suffix-max envelope: $\(\text{suffix}_t = \max(H_t, H_{t+1}, \dots, H_T), \quad t = 1, 2, \dots, T\)$ Geometrically, the envelope forms an upper bound over the trajectory. If high uncertainty manifests near the tail, the \(\max\) operator propagates this penalty backward across all preceding positions, severely degrading the score. Conversely, if the trajectory converges decisively to a low-entropy conclusion, \(\text{suffix}_t \approx H_t\) throughout, and a steeper descent minimizes the overall envelope area. Normalizing against the maximum possible entropy anchor \(\log K\), the quality reward is formulated as: $\(R_{\text{quality}}(\tau) = 1 - \frac{1}{T \log K} \sum_{t=1}^T \text{suffix}_t\)$ This parameter-free operator yields a bounded \([0, 1]\) metric that simultaneously enforces low uncertainty convergence and fast uncertainty descent in a single formulation.

3. Group-Relative Length Scaling: Dynamic Conciseness Incentive Without Static Budgets

While the quality reward naturally suppresses non-informative deliberation, equally sound and fully converged traces require an additional gradient signal to prefer more concise formulations. To prevent the rigid failure modes associated with static token limits across diverse problem complexities, InfoDensity employs group-relative length scaling: $\(R_L(\tau) = \exp\left(-\lambda \cdot \frac{L(\tau) - \mu_L}{\sigma_L}\right)\)$ Here, \(L(\tau)\) denotes generated sequence length, \(\mu_L\) and \(\sigma_L\) represent the mean and standard deviation of response lengths within the current rollout group (\(n=5\)), and \(\lambda > 0\) controls compression pressure (set to \(\lambda=0.05\)). Generated outputs shorter than the group average achieve \(R_L > 1\), while longer-than-average trajectories receive exponential attenuation, providing an adaptive pressure toward brevity without requiring external difficulty estimators.

4. Answer-Gated Joint Reward: Preventing Shortcut Trajectories via Correctness Conditioning

To ensure length incentives do not induce aggressive skipping of vital derivation steps, InfoDensity pairs the composite reward strictly with ground-truth verification: $\(R(\tau) = \begin{cases} R_{\text{quality}}(\tau) \cdot R_L(\tau), & \text{if } \tau \text{ is correct} \\ 0, & \text{otherwise} \end{cases}\)$ Unsound or erroneous derivations receive zero reward regardless of how brief or superficially confident their token sequences may be. Consequently, policy optimization is bounded by accuracy, compelling the model to maximize information density per token rather than pursuing vacuous brevity.

Loss & Training

InfoDensity is trained for one epoch (207 optimization steps) on a curated 7,000-problem subset of DeepMath-103K (difficulty levels 5–10) using LoRA (rank \(r=32\), scaling factor \(\alpha=64\)). Training uses GRPO with rollout group size \(n=5\), micro-batch size 32, learning rate \(1\times 10^{-5}\), KL divergence coefficient \(\beta=0.005\), and maximum sequence cutoff of 16,384 tokens. The default length-scaling hyperparameter is \(\lambda=0.05\).

Key Experimental Results

Main Results

Evaluations are conducted on competition-level mathematics (AMC23, AIME24, MATH500) and expert-level multidisciplinary QA (GPQA-Diamond) under greedy decoding. Performance is measured across pass@1 Accuracy (Acc, %), average generation Length (Len, in thousands of tokens), and Accuracy–Efficiency Score (AES, \(\alpha=1, \beta=3, \gamma=5\)):

Base Model Method AMC23 (Acc/Len/AES) AIME24 (Acc/Len/AES) MATH500 (Acc/Len/AES) GPQA-D (Acc/Len/AES) Overall (Acc/Len/AES)
DSR-1.5B Original 50.0 / 9.6k / 0.00 20.0 / 14.3k / 0.00 69.6 / 6.2k / 0.00 6.6 / 14.4k / 0.00 36.5 / 11.1k / 0.00
DEER 57.5 / 7.0k / 0.72 23.3 / 11.8k / 0.67 65.6 / 3.1k / 0.21 5.6 / 12.5k / -0.63 38.0 / 8.6k / 0.35
NoThink 62.5 / 2.4k / 1.49 6.7 / 7.6k / -2.86 71.8 / 1.3k / 0.89 9.1 / 1.5k / 2.03 37.5 / 3.2k / 0.79
LCPO 67.5 / 7.6k / 1.25 30.0 / 12.3k / 1.64 74.2 / 5.2k / 0.36 8.6 / 13.5k / 0.97 45.1 / 9.7k / 0.83
O1-Pruner 60.0 / 8.4k / 0.72 16.7 / 13.9k / -0.81 68.2 / 6.3k / -0.12 8.1 / 14.2k / 0.71 38.2 / 10.7k / 0.17
PEAR 60.0 / 6.5k / 0.92 36.7 / 11.7k / 2.68 77.0 / 4.4k / 0.61 11.1 / 12.8k / 2.16 46.2 / 8.8k / 1.00
ETR 77.5 / 5.0k / 2.13 23.3 / 11.6k / 0.69 78.0 / 4.3k / 0.66 11.1 / 12.4k / 2.19 47.5 / 8.3k / 1.15
InfoDensity 80.0 / 5.0k / 2.28 33.3 / 9.8k / 2.31 80.2 / 3.4k / 0.91 9.6 / 11.8k / 1.54 50.8 / 7.5k / 1.49
DSR-7B Original 80.0 / 6.6k / 0.00 43.3 / 11.8k / 0.00 85.0 / 4.2k / 0.00 24.2 / 11.3k / 0.00 58.1 / 8.5k / 0.00
PEAR 92.5 / 4.4k / 0.80 60.0 / 8.5k / 1.44 90.2 / 2.5k / 0.59 36.9 / 5.2k / 2.11 69.8 / 5.1k / 1.00
ETR 87.5 / 2.4k / 0.92 56.7 / 4.6k / 1.54 90.6 / 1.5k / 0.84 37.3 / 2.5k / 2.40 68.0 / 2.8k / 1.18
InfoDensity 95.0 / 3.6k / 1.02 60.0 / 6.5k / 1.60 91.0 / 2.1k / 0.70 42.9 / 5.6k / 2.82 72.2 / 4.5k / 1.20
Qwen3-8B Original 90.0 / 8.0k / 0.00 63.3 / 12.2k / 0.00 90.6 / 5.4k / 0.00 52.0 / 9.9k / 0.00 74.0 / 8.9k / 0.00
ETR 92.5 / 4.2k / 0.56 73.3 / 8.6k / 0.77 93.6 / 2.4k / 0.65 57.1 / 5.2k / 0.77 79.1 / 5.1k / 0.63
InfoDensity 90.0 / 3.7k / 0.54 73.3 / 6.9k / 0.91 93.0 / 2.1k / 0.69 56.1 / 4.2k / 0.81 78.1 / 4.2k / 0.69

Ablation Study

Problem-level outcome partition on DSR-1.5B across 768 pooled test items (387 base-correct, 381 base-wrong) and length-scaling coefficient (\(\lambda\)) stability:

Baseline Partition Subgroup Share Outcome Shift Length Evolution Analysis & Characterization
Base Correct (\(n=387\)) 92.2% (\(n=357\)) Preserved (✓) 2,471 \(\to\) 1,622 (-34%) Cleanly prunes repetitive double-checking while retaining validity
Base Correct (\(n=387\)) 7.8% (\(n=30\)) Regression (✗) 4,684 \(\to\) 10,323 (+120%) Bimodal: 21 cases run to budget limit, 9 over-compress and slip
Base Wrong (\(n=381\)) 27.6% (\(n=105\)) Rescued (✓) 15,228 \(\to\) 2,700 (-82%) Breaks infinite self-doubt loops, committing upon solving
Base Wrong (\(n=381\)) 72.4% (\(n=276\)) Still Wrong (✗) 14,954 \(\to\) 12,159 (-19%) 66.7% exhaust 16k tokens; 22.8% terminate early around 3.8k

Length-Scaling Coefficient \(\lambda\) Sensitivity: - For \(\lambda \in \{0.01, 0.05, 0.1\}\): Training exhibits stable convergence, reaching 57%–60% validation accuracy while condensing average generation length to 1.4k–2.5k tokens. - For extreme \(\lambda = 0.5\): Severe reward hacking emerges; the model collapses into degenerate brevity, with validation accuracy plunging to \(\sim 37\%\).

Key Findings

  • Rescuing Overthinking Trajectories: InfoDensity resolves 27.6% of problems previously failed by the base model due to budget exhaustion. Qualitative analysis confirms that baseline models frequently reach the correct intermediate derivation within the first 2,000 tokens but spiral into repetitive verification loops until reaching the 16,384-token cutoff; InfoDensity eliminates this unproductive deliberation.
  • Superior Accuracy-Efficiency Frontier: On DSR-1.5B, InfoDensity elevates accuracy from 36.5% to 50.8% (+14.3 points) while reducing mean response length from 11.1k to 7.5k tokens (-32%), yielding an overall AES of 1.49 (surpassing ETR by 30%).
  • Nature of Regressions: Among the 7.8% regression cases, the majority stem from elementary calculation slips (e.g., arithmetic signs or counting bounds) caused by bypassed redundant checking, rather than structural breakdowns in deductive planning.

Highlights & Insights

  • Macro Trajectory Envelope over Micro Step Signals: Demonstrating that localized step entropy reduction (\(\Delta H_t\)) yields near-chance classification accuracy (\(d \le 0.12\)), the paper shifts focus to macro trajectory geometry, using the suffix-max operator to capture both tail decisiveness and descent rate.
  • Unifying Two Axiomatic Properties into One Operator: The suffix-max envelope elegantly couples low tail convergence and rapid descent without requiring multi-objective balancing or heuristic weighting.
  • Group-Relative Multiplicative Scaling: Multiplying the quality metric by a group-relative exponential length term bypasses arbitrary token constraints and cleanly discourages length hacking by gating rewards with answer correctness.

Limitations & Future Work

  • External Proxy Traces in Formative Analysis: Empirical trajectory distributions were derived from ProcessBench using external generation traces rather than on-policy rollouts, which might partially diverge from intrinsic generator dynamics.
  • Architecture-Specific Trajectory Discrepancies: On Qwen3 base models, entropy descent on MATH and Omni-MATH appears markedly flatter than on LLaMA or DeepSeek models, likely reflecting differing pre-training data overlaps.
  • Absence of Dedicated Verification Safeguards: Suppressing redundant verification steps occasionally increases susceptibility to minor arithmetic or sign slips; integrating targeted lightweight verifiers could address this failure mode.
  • vs NoThink / DEER: NoThink cuts thinking prompts entirely and DEER relies on decoding-time confidence triggers; both collapse on challenging problems (e.g., AIME24 drops to 6.7% for NoThink). InfoDensity trains the internal policy to remain expressive yet concise, achieving 33.3% on AIME24.
  • vs LCPO / O1-Pruner: These methods enforce strict token budgets or baseline length matching, which limits adaptability across easy versus hard questions. InfoDensity's group-relative scaling adjusts flexibly across problem difficulty.
  • vs PEAR / ETR: PEAR averages entropy across rigid thinking/answer phases and ETR relies on momentum-smoothed stepwise descent. InfoDensity's suffix-max envelope forms a tighter mathematical bound over trajectory convergence and descent, exhibiting stronger cross-architecture stability.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant introduction of the suffix-max envelope to quantify information density in reasoning trajectories.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 4 model scales, 4 competitive benchmarks, greedy and sampling protocols, and qualitative error dissections.
  • Writing Quality: ⭐⭐⭐⭐⭐ Cohesive narrative, rigorous empirical motivations, and clear mathematical formulations.
  • Value: ⭐⭐⭐⭐⭐ Establishes a principled information-theoretic framework to tackle the accuracy-efficiency trade-off and curb reward hacking in large reasoning models.