Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Trae1ounG
Area: LLM Reasoning
Keywords: reinforcement learning, internal policy decomposition, residual stream, bottom-up optimization, mathematical reasoning
TL;DR¶
By decomposing the LLM policy across the Transformer residual stream into layer and modular policies, this paper identifies a universal exploration-to-convergence transition and introduces Bottom-up Policy Optimization (BuPO), which aligns critical internal boundary layers during early RL training to reconstruct foundational reasoning representations and substantially boost complex reasoning performance.
Background & Motivation¶
Reinforcement learning with verifiable rewards (RLVR)—exemplified by recent reasoning breakthroughs such as DeepSeek-R1, GRPO, and PPO—has become the standard post-training paradigm for scaling long Chain-of-Thought (CoT) reasoning in large language models. However, existing policy optimization methods universally treat the LLM as a monolithic, unified policy. All policy gradients, advantage assignments, and clipping penalties are applied exclusively to the final token probability distribution generated at the very top of the Transformer stack. This coarse-grained, top-down supervision completely overlooks the rich, incremental evolution of information unfolding across layers and submodules within the internal residual stream.
Mechanistic interpretability tools such as the logit lens have established that intermediate hidden states can be decoded into token vocabulary distributions, revealing that reasoning representations evolve throughout model depth. Yet, prior interpretability literature treats intermediate representations predominantly as passive probes for inspecting discrete output tokens rather than active decision-making policies in reinforcement learning. This creates a fundamental tension: Transformer architectures naturally build reasoning hierarchically from lower layers to higher layers—where lower layers maintain high uncertainty to explore candidate semantic hypotheses and upper layers gradually resolve ambiguities to execute final predictions. Optimizing solely the final layer forces credit signals to backpropagate through dozens of non-linear transformations, which frequently dilutes gradient signals at the base, leaving lower layers without robust, early guidance for foundational reasoning abstractions.
To resolve this limitation, this paper introduces a policy-centric perspective grounded in Transformer mechanics: any internal hidden state combined with the unembedding matrix intrinsically constitutes a valid, trainable internal policy. Because internal reasoning representations emerge bottom-up, this insight raises a pivotal algorithmic hypothesis: can aligning internal layer policies during early training stages force lower layers to preemptively acquire high-level reasoning representations, thereby providing a more resilient foundation for the entire model? Core idea: decompose the language model policy into internal layer and modular policies via the residual stream, identify the boundary layer where FFN entropy transitions from exploration to integration, and perform Bottom-up Policy Optimization (BuPO) by aligning internal policies first before transitioning to full-policy GRPO.
Method¶
Overall Architecture¶
Bottom-up Policy Optimization (BuPO) establishes a principled reinforcement learning pipeline that bridges mechanistic interpretability with policy optimization. The input is a reasoning prompt \(q\), and the output is a fully optimized reasoning language model policy.
The pipeline comprises four sequential phases. First, hidden states from the residual stream are projected through the unembedding matrix without layer normalization to formulate explicit layer-level (\(\pi_l^{\text{Layer}}\)) and modular (\(\pi_l^{\text{ATTN}}, \pi_l^{\text{FFN}}\)) policies. Second, a lightweight one-time forward diagnostic calculates token-level entropy change (\(\Delta H_l^{\text{FFN}}\)) across layers to identify the architectural "Boundary Layer" where FFN dynamics shift from positive exploration to stable integration. Third, in the initial training stage (\(s_{\text{cur}} \le s_{\text{inter}}\)), Internal Policy Alignment (IPA) restricts gradient updates strictly to the boundary layer and all preceding lower layers, inducing internal feature refinement. Fourth, once foundational representations are established (\(s_{\text{cur}} > s_{\text{inter}}\)), the training regime switches seamlessly to standard whole-model GRPO to achieve global policy convergence.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Prompt & Rollout Trajectories"] --> D1["Internal Policy Decomposition<br/>Residual stream mapped to layer/modular policies"]
D1 --> D2["Boundary Layer Identification<br/>FFN entropy change marks exploration transition"]
D2 --> D3["Internal Policy Alignment IPA<br/>First sinter steps optimize boundary and lower layers"]
D3 --> D4["Global Policy Optimization GRPO<br/>Full model converges on refined foundation"]
D4 --> Out["High-Performance Reasoning Policy"]
Key Designs¶
1. Internal Policy Decomposition: treating residual hidden states as continuous policy distributions without layer normalization
Conventional logit-lens methods project intermediate states into discrete vocabulary space with LayerNorm (LN) to inspect top token labels. However, the authors demonstrate that incorporating LN introduces unstable variance scaling that degrades interpretability and hampers policy optimization. BuPO formulates the forward pass of an \(L\)-layer Transformer along the residual stream: \(H_l = H^{(2l)} = H^{(0)} + \sum_{i=1}^l A_i + \sum_{j=1}^l F_j\), where \(A_i\) and \(F_j\) denote self-attention and FFN outputs, respectively. By combining hidden states with the unembedding matrix \(E_u \in \mathbb{R}^{N \times d_{\text{model}}}\), internal policies are defined directly:
Omitting LayerNorm guarantees smooth and stable entropy dynamics across layers and modules. Ablation experiments (Table 5) confirm that BuPO without LN consistently outperforms BuPO with LN across all reasoning benchmarks (e.g., 54.06 vs. 47.81 on AIME24 for Qwen3-8B).
2. Boundary Layer Identification: detecting the critical transition from exploration to integration via FFN entropy change
To determine which internal layer policy provides the optimal training leverage, the paper tracks internal policy entropy \(H_l^{\text{Layer}} = -\sum_{j=1}^{|V|} P_{l,j} \log P_{l,j}\) and defines modular entropy change as \(\Delta H_l = H_l^{\text{Output}} - H_l^{\text{Input}}\). While all inspected models exhibit a global progression from high-entropy exploration in early layers to near-zero deterministic predictions in the final layer, their fine-grained module dynamics reveal stark architectural contrasts: - Llama models maintain weak positive entropy changes throughout almost all layers, converging abruptly only within the last 1–3 layers. - Qwen models exhibit a pronounced three-stage progression in FFN entropy: an initial exploration phase where \(\Delta H_l^{\text{FFN}} > 0\) (layers 1–6 in Qwen3-4B) writing new orthogonal features into the residual stream; a broad middle integration phase where \(\Delta H_l^{\text{FFN}} \approx 0\) retrieving and consolidating parametric knowledge; and an upper convergence phase where \(\Delta H_l^{\text{FFN}} < 0\) driving final token collapse.
The Boundary Layer is precisely defined as the last layer with positive FFN entropy change (\(\Delta H_l^{\text{FFN}} > 0\)), demarcating the end of representation expansion. For Qwen3-4B/8B, this transition occurs at layer 6; for OctoThinker-3B, at layer 26; and for OctoThinker-8B, at layer 30. Computing this diagnostic requires only a single baseline forward pass taking ~209 seconds (0.44% of total RL training time, with a FLOPs ratio of 1:698.2), making it a negligible one-time investment.
3. Internal Policy Alignment (IPA): restricting advantage gradients to lower layers for targeted feature refinement
Directly injecting raw reward feedback into lower layers without safeguards risks catastrophic representation drift. BuPO introduces Internal Policy Alignment (IPA), which optimizes a selected internal layer policy \(\pi_l^{\text{Layer}}\) via an advantage-weighted clipped objective:
where \(\rho_{i,t}^l = \frac{\pi_l^{\text{Layer}}(o_{i,t}|q, o_{i,<t})}{\pi_{l,\text{old}}^{\text{Layer}}(o_{i,t}|q, o_{i,<t})}\) represents the internal policy probability ratio on the sampled tokens, and \(\hat{A}_{i,t}\) is the verifiable reward advantage computed from group outcomes. Due to the directed acyclic graph of the residual stream:
Parameters above layer \(l\) (\(k > l\)) receive zero gradients and remain frozen, ensuring that optimization acts as a dedicated supervisory signal for layers \(0 \dots l\) and \(E_u\). Mechanistic probing demonstrates that IPA induces dramatic feature refinement: cosine similarity between the intermediate hidden state \(H_6\) and the topmost layers surges from near-zero to above 0.8 within dozens of steps, proving that early layers preemptively encode high-level reasoning abstractions.
4. Staged Bottom-up Transition: scheduling step-budgeted alignment to prevent representation collapse
While IPA successfully refines foundational features, prolonged internal optimization proves detrimental: empirical analysis reveals that beyond ~50 steps, the KL divergence between \(\pi_6^{\text{Layer}}\) and subsequent layers escalates rapidly, language perplexity (PPL) degrades, and models succumb to repetitive generation. BuPO resolves this through a step-wise two-stage optimization schedule:
During the first \(s_{\text{inter}}\) steps (\(s_{\text{inter}} = 30\) for Qwen3-4B, \(s_{\text{inter}} = 20\) for other backbones out of 300 total steps), the model reconstructs lower-layer reasoning representations. Thereafter, training transfers to standard global GRPO (\(\mathcal{J}_{\text{GRPO}}\)), enabling the upper layers to calibrate on top of an already robust, feature-refined foundational substrate.
Loss & Training¶
- Phase 1 (\(s_{\text{cur}} \le s_{\text{inter}}\)): Optimize \(\mathcal{J}_{\text{IPA}}\) on Boundary Layer policy \(\pi_l^{\text{Layer}}\), updating weights \(\theta_{0 \dots l}\) and \(E_u\).
- Phase 2 (\(s_{\text{cur}} > s_{\text{inter}}\)): Optimize standard \(\mathcal{J}_{\text{GRPO}}\) on the final language model policy \(\pi_\theta\), updating all model parameters.
- Hyperparameters: Implemented within the veRL framework using AdamW with learning rate \(1\times 10^{-6}\) for policy and \(1\times 10^{-5}\) for PPO critic. Prompt batch size is 128 (8 rollouts per prompt), mini-batch size is 32, rollout temperature is 1.0, clipping parameter \(\epsilon=0.2\), with zero KL penalty (\(\beta=0\)) and no entropy regularization. Max response length is 7168 tokens for Qwen and 3072 tokens for Llama backbones. Training samples are drawn from DeepMath-103k (5k random subset).
Key Experimental Results¶
Main Results¶
Evaluations across four competitive mathematical reasoning benchmarks (AMC23, MATH500, AIME24, and AIME25) compare BuPO against standard RL baselines. Results report Avg@K (K=16 for AMC23/MATH500, K=32 for AIME24/AIME25).
| Model Backbone | Method | AMC23 (Avg@16) | MATH500 (Avg@16) | AIME24 (Avg@32) | AIME25 (Avg@32) | Overall Average |
|---|---|---|---|---|---|---|
| Qwen3-4B | Vanilla (Zero-shot) | 67.66 | 80.29 | 23.20 | 18.60 | 47.44 |
| PPO | 77.03 | 83.64 | 32.60 | 27.60 | 55.22 | |
| Reinforce++ | 63.44 | 80.63 | 17.40 | 18.65 | 45.03 | |
| RLOO | 77.66 | 82.73 | 30.83 | 24.79 | 54.00 | |
| GRPO | 76.88 | 82.41 | 32.19 | 28.85 | 55.08 | |
| BuPO (Ours) | 81.09 (+4.21) | 84.90 (+2.49) | 36.88 (+4.69) | 31.15 (+2.30) | 58.51 (+3.43) | |
| Qwen3-8B | Vanilla (Zero-shot) | 67.34 | 80.46 | 26.98 | 19.17 | 48.49 |
| PPO | 87.03 | 86.20 | 37.81 | 22.60 | 58.41 | |
| Reinforce++ | 82.66 | 86.05 | 41.77 | 31.15 | 60.41 | |
| RLOO | 86.41 | 87.32 | 46.67 | 33.02 | 63.36 | |
| GRPO | 85.94 | 88.05 | 49.48 | 33.54 | 64.23 | |
| BuPO (Ours) | 89.22 (+3.28) | 87.76 | 54.06 (+4.58) | 34.38 (+0.76) | 66.36 (+2.13) | |
| OctoThinker-3B | Vanilla (Zero-shot) | 1.24 | 5.26 | 0.21 | 0.00 | 1.68 |
| GRPO | 27.50 | 46.07 | 0.63 | 0.10 | 18.58 | |
| BuPO (Ours) | 27.50 (+0.00) | 49.79 (+3.72) | 0.63 (+0.00) | 0.42 (+0.32) | 19.59 (+1.01) | |
| OctoThinker-8B | Vanilla (Zero-shot) | 4.53 | 9.84 | 0.52 | 0.10 | 3.75 |
| GRPO | 34.84 | 56.89 | 2.50 | 2.19 | 24.11 | |
| BuPO (Ours) | 37.66 (+2.82) | 62.05 (+5.16) | 4.69 (+2.19) | 6.77 (+4.58) | 27.79 (+3.68) |
Ablation Study¶
1. Impact of Internal Optimization Steps \(s_{\text{inter}}\) (Qwen3-4B with \(\pi_6^{\text{Layer}}\))¶
| Step Budget (\(s_{\text{inter}}\)) | AMC23 | MATH500 | AIME24 | AIME25 | Average |
|---|---|---|---|---|---|
| GRPO (\(s_{\text{inter}} = 0\)) | 76.88 | 82.41 | 32.19 | 28.85 | 55.08 |
| BuPO (\(s_{\text{inter}} = 30\), Default) | 81.09 | 84.90 | 36.88 | 31.15 | 58.51 |
| BuPO (\(s_{\text{inter}} = 50\)) | 62.66 | 79.00 | 14.17 | 12.60 | 42.11 |
| BuPO (\(s_{\text{inter}} = 70\)) | 14.14 | 25.20 | 0.21 | 0.00 | 9.89 |
2. Target Layer Selection Across Backbones (Validating Boundary Layer Hypothesis)¶
| Backbone | Target Layer Strategy | AMC23 | MATH500 | AIME24 | AIME25 | Average |
|---|---|---|---|---|---|---|
| Qwen3-4B | Boundary Layer (\(\pi_6^{\text{Layer}}\)) | 81.09 | 84.90 | 36.88 | 31.15 | 58.51 |
| Middle Integration (\(\pi_{26}^{\text{Layer}}\)) | 81.56 | 84.40 | 35.42 | 30.52 | 57.98 | |
| Penultimate Layer (\(\pi_{35}^{\text{Layer}}\)) | 82.66 | 85.14 | 35.42 | 30.12 | 58.34 | |
| Qwen3-8B | Boundary Layer (\(\pi_6^{\text{Layer}}\)) | 89.22 | 87.76 | 54.06 | 34.38 | 66.36 |
| Middle Integration (\(\pi_{21}^{\text{Layer}}\)) | 75.55 | 84.40 | 32.60 | 21.67 | 53.56 | |
| Penultimate Layer (\(\pi_{35}^{\text{Layer}}\)) | 83.75 | 86.51 | 42.81 | 29.58 | 60.66 | |
| OctoThinker-3B | Boundary Layer (\(\pi_{26}^{\text{Layer}}\)) | 27.50 | 49.79 | 0.63 | 0.42 | 19.59 |
| Early Intermediate (\(\pi_{16}^{\text{Layer}}\)) | 16.17 | 11.40 | 0.00 | 0.00 | 6.89 | |
| Lower Layer (\(\pi_6^{\text{Layer}}\)) | 7.80 | 9.60 | 0.21 | 0.10 | 4.43 | |
| OctoThinker-8B | Boundary Layer (\(\pi_{30}^{\text{Layer}}\)) | 37.66 | 62.05 | 4.69 | 6.77 | 27.79 |
| Early Intermediate (\(\pi_{16}^{\text{Layer}}\)) | 3.91 | 12.20 | 0.00 | 0.00 | 4.03 | |
| Lower Layer (\(\pi_6^{\text{Layer}}\)) | 7.58 | 9.40 | 0.00 | 0.10 | 4.27 |
3. Comparison with Conventional Layer-Aware GRPO Baselines¶
On Qwen3-4B, BuPO (58.51) outperforms freezing lower layers (55.65) and layerwise learning rate decay (\(0.1\times\) LR on upper layers: 58.12). On OctoThinker-3B, BuPO (19.59) similarly surpasses frozen layers (17.70) and layerwise LR (17.95), proving that gains stem from explicit probability distribution alignment rather than coarse parameter weighting.
Key Findings¶
- Exceptional Gains on Elite Math Olympiads: On AIME24, BuPO achieves a +4.69 point increase on Qwen3-4B and +4.58 points on Qwen3-8B over GRPO. On AIME25, OctoThinker-8B surges from 2.19% to 6.77% (+4.58 absolute, >200% relative gain).
- Unbiased Pass@K Dominance: As evaluated across \(K \in [1, 256]\) with 300 samples (Figure 6), BuPO establishes consistent Pareto improvements over GRPO, achieving gains of +7.48 on OctoThinker-3B and +7.93 on OctoThinker-8B at Pass@256.
- Robust Out-of-Domain Generalization: Evaluated on seven diverse benchmarks (Table 9), BuPO outperforms GRPO by an average of +1.83 points on Qwen3-4B and +0.62 on Qwen3-8B, with standout gains on GPQA-Diamond (+4.10) and HumanEval (+3.60), confirming that bottom-up feature alignment instills transferable reasoning capabilities.
- Sensitivity of Alignment Budget: Excessive IPA steps (\(s_{\text{inter}} \ge 50\)) trigger severe PPL degradation and policy degeneracy (dropping to 9.89 at 70 steps), underscoring that lower-layer alignment is strictly an inductive initialization phase rather than an end-state objective.
Highlights & Insights¶
- Mathematically Formulating Residual Streams as Action Policies: Reconceptualizing intermediate activations from passive probes into genuine probability distributions unlocks direct policy gradient intervention within the model's internal hierarchy.
- Entropy Dynamics Expose Architectural Discrepancies: The \(\Delta H_l^{\text{FFN}}\) metric successfully uncovers why Qwen models excel at post-training knowledge acquisition (exhibiting structured three-phase progression) while Llama models suffer from abrupt terminal-layer collapse.
- Ultra-Low Cost Diagnostic with High Engineering ROI: Boundary layer identification requires only ~200 seconds of compute without backward passes, providing an accessible, highly effective plug-and-play enhancement for modern RL training pipelines.
Limitations & Future Work¶
- High Rollout Compute Requirements: Like all RLVR approaches, generating long reasoning trajectories (up to 7k tokens) across large sample counts remains computationally intensive, requiring 8× A100 GPU nodes.
- Static Time-Step Scheduling: Switching from IPA to GRPO relies on a fixed, empirically tuned step threshold (\(s_{\text{inter}}\)); developing adaptive scheduling based on real-time internal representation similarity or KL divergence is a promising direction.
- Exploration Beyond Verifiable Reasoning: Current evaluations focus on mathematical reasoning and code generation; validating BuPO on creative open-ended dialogue, tool calling, and multimodal planning remains open for exploration.
Related Work & Insights¶
- vs GRPO / PPO / RLVR (Shao et al., 2024; Sutton et al., 1998): Standard RLVR optimizes the network holistically from the final output; BuPO decomposes the internal residual stream and selectively optimizes the lower-layer boundary policy first to prime representation learning.
- vs Logit-Lens / Tuned-Lens (nostalgebraist, 2020; Belrose et al., 2023): Logit-lens inspects intermediate states via discrete token decoding with layer normalization; BuPO adopts an LN-free policy-centric perspective, directly calculating policy entropy and computing policy gradients.
- vs Attention-Guided RL (Li et al., 2025; Liu et al., 2025): While prior work utilizes attention patterns to guide token exploration, BuPO leverages the residual stream to coordinate both attention exploration and FFN parametric knowledge consolidation.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering framing of residual states as explicit, trainable internal policies]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated across 4 model scales, 4 main math benchmarks, 7 OOD datasets, and multi-seed trials]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear progression from mechanistic discoveries to theoretical framing and practical RL formulation]
- Value: ⭐⭐⭐⭐⭐ [Highly practical, low-overhead optimization paradigm for large-scale reasoning post-training]