Skip to content

Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Trae1ounG
Area: LLM Reasoning
Keywords: Reinforcement Learning, Internal Policy, Residual Stream Decomposition, Policy Optimization, Mechanistic Interpretability

TL;DR

By projecting Transformer residual streams with the unembedding matrix, this paper decomposes an LLM into internal layer and modular policies, uncovers divergent exploration-to-convergence entropy dynamics across model families, and proposes Bottom-up Policy Optimization (BuPO) to pre-align foundational boundary layers before global RL training, substantially boosting mathematical reasoning at negligible diagnostic cost.

Background & Motivation

In the post-training paradigm of large language models, reinforcement learning with verifiable rewards (RLVR, exemplified by DeepSeek-R1 and GRPO) has become the dominant driving force for eliciting advanced multi-step reasoning and reflective chains of thought. Despite these successes, existing policy gradient algorithms treat an LLM strictly as a monolithic, unified policy. Under this black-box formulation, the reward signal is computed solely from the final output token distribution and backpropagated homogeneously across dozens of stacked Transformer layers, completely ignoring the hierarchical computational structure and progressive feature specialization unfolding inside the network.

Internal representations along the Transformer's residual stream harbor intricate reasoning dynamics. While prior mechanistic interpretability techniques like the Logit-Lens have inspected intermediate hidden states by projecting them onto discrete vocabulary tokens, they remain passive, diagnostic tools that fail to connect internal distributions with formal Markov Decision Process (MDP) policy formulations. Furthermore, optimizing only the holistic policy often leaves lower and intermediate layers drifting in an unconstrained search space during the fragile early training phase, missing the chance to actively cultivate foundational representations before full policy convergence.

To resolve the dissonance between holistic policy optimization and fine-grained internal information flow, this work bridges mechanistic residual stream decomposition with reinforcement learning policy theory. By treating the combination of intermediate hidden states and the unembedding matrix as legitimate token-level action distributions, the authors systematically probe how internal reasoning policies evolve from bottom to top through an entropy-centric lens. Core idea: decompose the language model policy into internal layer policies across the residual stream, identify the cognitive transition boundary via FFN entropy dynamics, and apply Bottom-up Policy Optimization (BuPO) to pre-align foundational representations before holistic RL, prompting lower layers to capture high-level reasoning features early.

Method

Overall Architecture

BuPO breaks away from the conventional paradigm of optimizing an LLM as a single end-to-end policy by establishing a structured workflow: "Internal Policy Decomposition & Probing → Boundary Layer Identification → Sequential Bottom-up Alignment". During a standard forward pass, the residual hidden state \(H_l\) at any layer \(l\) is mapped to a valid probability distribution over the vocabulary via the shared unembedding matrix \(E_u\). By running a one-time forward diagnostic pass on base model rollouts, the framework tracks the token-level policy entropy change rate \(\Delta H_l\) across layers and submodules, pinpointing the exact "Boundary Layer" where feed-forward networks (FFN) transition from positive exploratory entropy expansion to zero or negative integration. In the training phase, BuPO operates sequentially: during the initial \(s_{\text{inter}}\) steps, it applies Internal Policy Alignment (IPA) exclusively on the designated boundary layer, directing gradient updates strictly to layers \(0 \le k \le l\) and inducing pre-emptive feature refinement; once step count exceeds \(s_{\text{inter}}\), the algorithm seamlessly switches to holistic GRPO on the final layer policy \(\pi_\theta\) to consolidate overall task performance.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Reasoning Prompt and Context q, o_{<t}"] --> B["Residual Stream Decomposition and Policy Formulation<br/>Project H_l with unembedding E_u onto vocabulary"]
    B --> C["Internal Policy Entropy Dynamics and Boundary Probing<br/>Identify transition layer where ΔH_l^FFN shifts sign"]
    C --> D{"Training Stage Check<br/>Current step s_{cur} ≤ s_{inter} ?"}
    D -->|Yes: Early Training Stage| E["Internal Policy Alignment (IPA)<br/>Targeted gradient update to layers 0~l inducing feature refinement"]
    D -->|No: Later Holistic Fine-tuning| F["Global Policy Optimization (GRPO)<br/>Full-model parameter update for macro task convergence"]
    E --> G["Converged High-Capacity Policy π_θ"]
    F --> G

Key Designs

1. Residual Stream Decomposition and Policy Formulation: Clean Probability Distributions without LayerNorm Distortions

Unlike standard logit-lens formulations that incorporate top Layer Normalization (LN) to view discrete top-1 token predictions, this design treats internal states as formal policy distributions within an RL framework. Leveraging the additive nature of the Transformer residual stream, the hidden state at layer \(l\) is decomposed as \(H_l = H^{(0)} + \sum_{i=1}^l A_i + \sum_{j=1}^l F_j\), where \(A_i\) and \(F_j\) denote attention and FFN incremental outputs. The internal layer policy is formulated by directly projecting \(H_l\) through the unembedding matrix \(E_u \in \mathbb{R}^{N \times d_{\text{model}}}\):

\[\pi_l^{\text{Layer}} \equiv P_l^{\text{Layer}} = \text{softmax}(H_l E_u^\top)\]

Similarly, modular policies isolating specific attention and feed-forward updates are defined as \(\pi_l^{\text{ATTN}} = \text{softmax}(A_l E_u^\top)\) and \(\pi_l^{\text{FFN}} = \text{softmax}(F_l E_u^\top)\). Rigorous ablation demonstrates that intentionally omitting Layer Normalization (BuPO-wo-LN) prevents artificial scaling distortions across depths, producing more monotonic, interpretable entropy trajectories and superior downstream downstream benchmark accuracy.

2. Internal Policy Entropy Dynamics and Boundary Probing: Pinpointing the Transition from Exploration to Integration

To diagnose internal reasoning mechanisms across model architectures, the authors introduce token-level Internal Policy Entropy \(H_l^{\text{Layer}} = - \sum_{j=1}^{|V|} P_{l,j}^{\text{Layer}} \log P_{l,j}^{\text{Layer}}\) and its incremental change rate across a layer or module \(\Delta H_l = H_l^{\text{Output}} - H_l^{\text{Input}}\). Probing popular open-source models uncovers striking architectural divergences: while both Qwen and Llama universally evolve from high-entropy exploration in early layers to near-zero deterministic predictions at the top, their pacing differs fundamentally. Llama maintains broad uncertainty across over 85% of its depth and experiences an abrupt, sudden collapse within the final 3 layers; conversely, Qwen displays a progressive, gradual reduction of uncertainty layer by layer.

Analyzing submodule dynamics reveals that the FFN module serves as the primary regulator of parametric knowledge retrieval and convergence. In Qwen models, lower FFN layers exhibit positive entropy growth (\(\Delta H_l^{\text{FFN}} > 0\)) to widen the search space, followed by a wide middle region (\(\Delta H_l^{\text{FFN}} \approx 0\)) for stable factual integration, and upper layers with negative entropy (\(\Delta H_l^{\text{FFN}} < 0\)) for decisive convergence. The authors designate the final layer with positive FFN entropy expansion as the "Boundary Layer" (layer 6 for Qwen3-4B/8B, layer 26 for Llama-OctoThinker-3B, and layer 30 for OctoThinker-8B). This entropy-indicated inflection point provides a mathematically grounded, computationally cheap signal for choosing the ideal internal alignment target.

3. Internal Policy Alignment (IPA): Targeted Gradient Flow Driving Pre-emptive Feature Refinement

Having identified the boundary layer \(l\), the authors propose Internal Policy Alignment (IPA) to inject outcome reward signals directly into that intermediate distribution. IPA defines an internal policy update ratio \(\rho_{i,t}^l = \frac{\pi_l^{\text{Layer}}(o_{i,t} \mid q, o_{i,<t})}{\pi_{l,\text{old}}^{\text{Layer}}(o_{i,t} \mid q, o_{i,<t})}\) and applies a clipped surrogate objective with group-normalized advantage values \(\hat{A}_{i,t}\):

\[\mathcal{J}_{\text{IPA}}(\pi_\theta, \pi_l^{\text{Layer}}) = \mathbb{E}_{q \sim \mathcal{Q}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot \mid q)} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \min \left( \rho_{i,t}^l \hat{A}_{i,t}, \, \text{clip}(\rho_{i,t}^l, 1-\epsilon, 1+\epsilon)\hat{A}_{i,t} \right) \right]\]

By the chain rule, the gradient with respect to parameter \(\theta_k\) at layer \(k\) satisfies \(\frac{\partial \mathcal{J}_{\text{IPA}}}{\partial \theta_k} = \frac{\partial \mathcal{J}_{\text{IPA}}}{\partial \pi_l^{\text{Layer}}} \cdot \frac{\partial \pi_l^{\text{Layer}}}{\partial H_l} \cdot \frac{\partial H_l}{\partial \theta_k}\). Because \(\frac{\partial H_l}{\partial \theta_k} = 0\) for all \(k > l\), the gradient is strictly zero for all layers above \(l\). This guarantees that IPA supervises only layers \(0\) through \(l\) and the unembedding matrix \(E_u\), shielding lower representations from volatile top-layer gradients. Empirical analysis confirms that intermediate state \(H_l\) undergoes pronounced "feature refinement" under IPA, exhibiting rapidly climbing cosine similarity with top-most representations and establishing a solid, mature representational foundation for subsequent holistic reasoning.

Loss & Training

The overall training objective of BuPO follows a two-stage bottom-up schedule governed by current training step \(s_{\text{cur}}\):

\[\mathcal{J}_{\text{BuPO}} = \begin{cases} \mathcal{J}_{\text{IPA}}(\pi_\theta, \pi_l^{\text{Layer}}), & s_{\text{cur}} \le s_{\text{inter}} \\ \mathcal{J}_{\text{GRPO}}(\pi_\theta), & s_{\text{cur}} > s_{\text{inter}} \end{cases}\]

Here, group-relative advantages \(\hat{A}_{i,t} = \frac{R_i - \text{mean}(R)}{\text{std}(R)}\) are calculated over \(G=8\) rollouts per prompt with binary verifiable rewards \(R_i \in \{0, 1\}\). Training hyperparameters include AdamW optimizer, policy learning rate \(1\times 10^{-6}\), clipping threshold \(\epsilon = 0.2\), mini-batch size of 32 prompts, and maximum context length of 9,216 tokens. The number of internal alignment steps \(s_{\text{inter}}\) is set to 30 for Qwen3-4B and 20 for other backbones. Over-aligning the bottom layers (e.g., \(s_{\text{inter}} \ge 50\)) causes representational misalignment with frozen upper layers, triggering sharp perplexity degradation. The diagnostic entropy profiling runs at step 0 in just 209.25 seconds on a single node of 8 NVIDIA A100 GPUs (amounting to 0.44% of total RL runtime and a FLOPs ratio of 1:698.2), making BuPO extremely lightweight and practical.

Key Experimental Results

Main Results

BuPO is comprehensively evaluated across four rigorous mathematical reasoning benchmarks: MATH500, AMC23, AIME24, and AIME25. Performance is measured via Avg@K (AMC/MATH with \(K=16\), AIME with \(K=32\)):

Backbone Model Optimization Method AMC (Avg@16) MATH500 (Avg@16) AIME24 (Avg@32) AIME25 (Avg@32) Benchmark Average
Qwen3-4B Vanilla (Base) 67.66 80.29 23.20 18.60 47.44
Qwen3-4B PPO 77.03 83.64 32.60 27.60 55.22
Qwen3-4B Reinforce++ 63.44 80.63 17.40 18.65 45.03
Qwen3-4B RLOO 77.66 82.73 30.83 24.79 54.00
Qwen3-4B GRPO 76.88 82.41 32.19 28.85 55.08
Qwen3-4B BuPO (Ours) 81.09 (+4.21) 84.90 (+2.49) 36.88 (+4.69) 31.15 (+2.30) 58.51 (+3.43)
Qwen3-8B Vanilla (Base) 67.34 80.46 26.98 19.17 48.49
Qwen3-8B PPO 87.03 86.20 37.81 22.60 58.41
Qwen3-8B Reinforce++ 82.66 86.05 41.77 31.15 60.41
Qwen3-8B RLOO 86.41 87.32 46.67 33.02 63.36
Qwen3-8B GRPO 85.94 88.05 49.48 33.54 64.23
Qwen3-8B BuPO (Ours) 89.22 (+3.28) 87.76 54.06 (+4.58) 34.38 (+0.76) 66.36 (+2.13)
OctoThinker-3B Vanilla (Base) 1.24 5.26 0.21 0.00 1.68
OctoThinker-3B GRPO 27.50 46.07 0.63 0.10 18.58
OctoThinker-3B BuPO (Ours) 27.50 (+0.00) 49.79 (+3.72) 0.63 (+0.00) 0.42 (+0.32) 19.59 (+1.01)
OctoThinker-8B Vanilla (Base) 4.53 9.84 0.52 0.10 3.75
OctoThinker-8B GRPO 34.84 56.89 2.50 2.19 24.11
OctoThinker-8B BuPO (Ours) 37.66 (+2.82) 62.05 (+5.16) 4.69 (+2.19) 6.77 (+4.58) 27.79 (+3.68)

Ablation Study

The ablation investigates the target internal policy selection (Boundary Layer vs. other layers) and internal optimization step count \(s_{\text{inter}}\):

Model Backbone Variant / Target Layer Policy AMC MATH500 AIME24 AIME25 Benchmark Average Note / Behavior
Qwen3-4B Boundary Layer (\(\pi_6^{\text{Layer}}\)) 81.09 84.90 36.88 31.15 58.51 Optimal FFN exploration-to-integration inflection
Qwen3-4B Mid Integration Layer (\(\pi_{26}^{\text{Layer}}\)) 81.56 84.40 35.42 30.52 57.98 Plateaued knowledge integration, limited exploration
Qwen3-4B Penultimate Layer (\(\pi_{35}^{\text{Layer}}\)) 82.66 85.14 35.42 30.12 58.34 Suffers from token repetition and response length explosion
Qwen3-4B Bottom steps \(s_{\text{inter}} = 0\) (GRPO) 76.88 82.41 32.19 28.85 55.08 Vanilla baseline without internal alignment
Qwen3-4B Bottom steps \(s_{\text{inter}} = 50\) 62.66 79.00 14.17 12.60 42.11 Excessive alignment harms cross-layer coordination
Qwen3-4B Bottom steps \(s_{\text{inter}} = 70\) 14.14 25.20 0.21 0.00 9.89 Severe representational gap and PPL collapse
OctoThinker-8B Boundary Layer (\(\pi_{30}^{\text{Layer}}\)) 37.66 62.05 4.69 6.77 27.79 Llama exploration boundary resides in deeper layers
OctoThinker-8B Mid Layer (\(\pi_{16}^{\text{Layer}}\)) 3.91 12.20 0.00 0.00 4.03 Misaligned with entropy transition, poor optimization
OctoThinker-8B Shallow Layer (\(\pi_6^{\text{Layer}}\)) 7.58 9.40 0.00 0.10 4.27 Insufficient capacity in lower layers for Llama

Key Findings

  • Boundary layer optimization is decisive across architectures: Across both Qwen3 and Llama-OctoThinker families, aligning the boundary layer indicated by positive-to-negative FFN entropy transition consistently yields the highest average scores. Targeting too shallow a layer in Llama (e.g. layer 6) causes optimization failure (collapsing to 4.27), while optimizing the penultimate layer (\(\pi_{35}^{\text{Layer}}\)) triggers severe repetition loops and response lengths skyrocketing to 8,000 tokens.
  • Strict, non-linear sensitivity to internal alignment duration: A clear inverted-U trajectory governs \(s_{\text{inter}}\). On Qwen3-4B, 30 steps of IPA provide a +3.43 average gain over GRPO. However, expanding \(s_{\text{inter}}\) to 50 or 70 steps triggers catastrophic representational dissonance between the aligned bottom layers and the untouched top layers, causing KL divergence spikes, runaway perplexity, and severe benchmark degradation.
  • Superiority over conventional layer-aware baselines: Contrasting BuPO against layer-freezing baselines (freezing layers up to \(l\)) and layer-wise learning rates (applying a \(0.1\times\) rate to layers above \(l\)) confirms that BuPO's advantages do not stem from merely weighting lower parameters. On Qwen3-4B, BuPO outperforms frozen layers and layerwise learning rate schemes by +2.86 and +0.39 points respectively, and by +1.89 and +1.64 points on OctoThinker-3B, validating that the mathematical formulation of IPA genuinely reconstructs intermediate policy distributions.

Highlights & Insights

  • Formulating internal hidden states as genuine RL policies: The paper bridges the gap between passive mechanistic interpretability and active policy optimization by defining internal layer and modular policies directly over vocabulary space via \(E_u\), enabling clipped policy gradient algorithms to supervise internal representations.
  • Entropy change as an architectural cognitive signature: The discovery of Qwen's progressive, continuous reasoning contraction versus Llama's sudden top-layer convergence provides concrete empirical evidence explaining why different LLM architectures exhibit distinct post-training sensitivities and knowledge absorption profiles.
  • Pre-emptive feature refinement at negligible compute cost: Aligning the boundary layer forces lower layers to capture high-level reasoning representations before full-policy RL kicks in; furthermore, identifying this boundary requires only a single forward pass taking 209 seconds (~0.44% of total RL runtime), rendering it immediately adoptable in production training pipelines.

Limitations & Future Work

  • High infrastructure footprint for large-scale RL rollouts: Despite the negligible overhead of entropy diagnostics, the overall training and evaluation pipeline still necessitates 8\(\times\) NVIDIA A100 GPUs and expensive multi-sample Pass@K generation (e.g., Pass@256 with 300 sampled rollouts) due to the high variance of mathematical reasoning trajectories.
  • Single-layer hard switching schedule: BuPO currently relies on a hard threshold \(s_{\text{inter}}\) to transition abruptly from the boundary layer to the holistic policy. Designing smooth multi-layer curriculum schedules or dynamically annealed loss combinations could further stabilize multi-stage optimization.
  • Extension to open-ended agentic domains: The current experiments center primarily on rule-verifiable mathematical problem solving and competitive coding. Validating BuPO on multi-turn interactive tool use, web navigation, and multimodal reasoning tasks remains a valuable future direction.
  • vs Holistic RLVR (GRPO / PPO / DeepSeek-R1): Holistic methods treat the network as an end-to-end black box, propagating reward gradients blindly across all layers and causing bottom layers to wander during initial exploration; BuPO pre-trains foundational reasoning capabilities at the boundary layer, guiding the overall policy along a more structured optimization path.
  • vs Logit-Lens and Tuned-Lens: Prior lens methods serve strictly as passive visualization probes for discrete token extraction, often requiring auxiliary learned probes or preserving LayerNorm; BuPO formalizes un-normalized internal states into unconstrained continuous policy distributions and uses them as active optimization targets.
  • vs Heuristic Layer-wise Optimization: Conventional techniques simply scale learning rates or freeze bottom layers based on intuition; BuPO derives its target layer rigorously from FFN entropy transition dynamics and uses advantage-weighted clipped updates to achieve genuine representational refinement.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulating Transformer residual states as optimizable internal RL policies and linking them to entropy dynamics is exceptionally creative and theoretically rigorous.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates multiple model families across different scales, complete with fine-grained entropy flow visualizations, Pass@K scaling curves, cosine similarity tracking, and extensive out-of-domain validations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Cohesive narrative, crisp mathematical derivations, and beautifully articulated structural insights with highly informative figures.
  • Value: ⭐⭐⭐⭐⭐ Successfully marries mechanistic interpretability with post-training policy optimization, offering an effective, production-ready recipe for RLVR reasoning models.