Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/QwenLM/Confident-Decoding
Area: LLM Reasoning
Keywords: Dynamic Decoding, Alignment Tax, Entropy-Guided Backward Search, Optimal Stopping, LLM Reasoning
TL;DR¶
Addressing the "alignment tax" where final layers disrupt fragile logic chains, this paper reveals a three-phase Guess–Refine–Perturb dynamic in LLM forward passes and proposes Confident Decoding—a training-free, drop-in decoding strategy that dynamically rolls back to near-final local entropy valleys, restoring deep reasoning fidelity with zero extra KV-cache memory and under 2% latency overhead.
Background & Motivation¶
Standard autoregressive generation in modern large language models (LLMs) conventionally decodes each token from the final transformer layer. This universal practice implicitly relies on a monotonic-depth assumption: shallow layers perform low-level feature extraction, intermediate layers progressively synthesize multi-hop contextual dependencies, and the deepest layer naturally provides the most calibrated and discriminative next-token predictive distribution. However, mechanistic probing and layer-wise representational analyses demonstrate that this monotonic-depth assumption frequently breaks down. Intermediate layers often crystallize task-critical, highly confident semantic representations, whereas the terminal layers can compress, redirect, or perturb these refined internal states before projecting onto the vocabulary.
This terminal degradation is intrinsically tied to post-training procedures designed for human preference alignment, including Supervised Fine-Tuning (SFT), RLHF, RLAIF, and Direct Preference Optimization (DPO). Representation engineering demonstrates that alignment objectives steer representations via low-rank steering vectors that concentrate heavily in the uppermost layers. While this late-stage regularization functions as a vital guardrail for conversational tone, instruction formatting, and safety compliance, it introduces a severe "planning–pragmatics tradeoff" in complex reasoning scenarios. When synthesizing delicate mathematical proofs, algorithmic code, or scientific deductions, the specialized logic distribution (\(P_{\text{logic}}\)) diverges sharply from the generic, safe distribution (\(P_{\text{align}}\)). Consequently, the final layer imposes an "alignment tax" by actively shifting high-information, domain-specific tokens toward common, uninformative functional words and punctuation.
Existing inference-time interventions, such as contrastive decoding (DoLa, SLED) or early-exit architectures, fail to resolve this fundamental conflict. Contrastive techniques compute logit differentials across early and late layers under strict assumptions of representational subspace homogeneity that crumble in modern hybrid or Mixture-of-Experts architectures. Meanwhile, traditional early-exit methods focus on latency reduction by prematurely aborting the forward pass, which sacrifices essential global attention context. This paper breaks away from the assumption that the final layer is the optimal decoding interface, asking instead where inside the model true predictive confidence peaks. Core idea: without altering the forward pass or KV cache, use an entropy-guided conservative backward search over a near-final candidate window to dynamically select the first local "entropy valley," filtering terminal alignment perturbations and unlocking the latent reasoning ceiling in aligned LLMs.
Method¶
Overall Architecture¶
Confident Decoding operates as an entirely training-free, drop-in decoding strategy. For each active token generation step, the model executes the complete \(L\)-layer forward pass, ensuring that KV-cache representations and full-context attention operations remain untouched. Prior to sampling, the algorithm extracts hidden states across a candidate window of near-final layers \(\mathcal{C} = \{L-M+1, \dots, L\}\). Applying the model's exact final RMSNorm (including the learned scaling factor \(\gamma\)) and unmodified output unembedding matrix \(W_U\), candidate distributions and predictive Shannon entropies \(H(p_t^{(\ell)})\) are computed in parallel. The algorithm then performs a conservative backward search starting from layer \(L\) toward layer \(L-K\), identifying the first local entropy trough (the "entropy valley" \(\ell^\star_t\)). The logits from this valley layer are routed directly to the sampler, dynamically bypassing terminal perturbations.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Full L-Layer Forward Pass<br/>x_t^(1) → ... → x_t^(L)"] --> B["Forward Dynamics Diagnosis<br/>Guess (I) → Refine (II) → Perturb (III)"]
B --> C["Near-Final Candidate Extraction<br/>Shared RMSNorm & Unmodified W_U Projection"]
C --> D["Entropy Evaluation & Conservative Backward Search<br/>Scan for First Local Entropy Valley ℓ⋆"]
D --> E["Optimal Stopping & Stochastic Fallback<br/>Select Valley with Prob. p / Fallback to L with 1-p"]
E --> F["Deliver Logits to Sampler z_t^(ℓ⋆)"]
Key Designs¶
1. Layer-wise Dynamics Diagnosis: Formulating the Guess–Refine–Perturb Progression
To demystify terminal layer degradation, the authors profile residual stream vectors \(h_t^{(l)} = h_t^{(l-1)} + m_t^{(l)}\) by tracking Relative Contribution Norm \(\text{Norm Ratio}(l) = \|m_t^{(l)}\|_2 / \|h_t^{(l-1)}\|_2\) and Residual I/O Cosine Similarity \(\text{IO-CosSim}(l) = \langle h_t^{(l)}, h_t^{(l-1)} \rangle / (\|h_t^{(l)}\|_2 \|h_t^{(l-1)}\|_2)\). Empirical measurements on Qwen3.5-35B-A3B reveal three distinct computational phases: - Phase I (Guess, \(l \lesssim 0.15L\)): Massive write intensity and volatile directional deflections as the model establishes crude statistical approximations under high entropy; - Phase II (Refine, \(0.15L \lesssim l \lesssim 0.95L\)): Stabilized contribution norms (0.23–0.57) and exceptionally high directional fidelity (0.91–0.97), where attention mechanisms steadily integrate contextual dependencies and drive monotonic entropy reduction; - Phase III (Perturbation, \(l \gtrsim 0.95L\)): Write intensity surges 2–3\(\times\) higher than Phase II while cosine similarity plummets to \(\approx 0.69\), representing an off-axis perturbation that overwrites the fragile reasoning trajectory formed during Phase II.
2. Predictive Entropy as Confidence Metric: Uncovering the "Entropy Valley"
Within the late refinement corridor, \(\text{IO-CosSim}\) remains high (0.91–0.97), providing strong geometric justification for evaluating intermediate states with the shared final unembedding matrix \(W_U\). Candidate logits, probability distributions, and entropies are computed as: $\(z_t^{(\ell)} = W_U \text{RMSNorm}_L(x_t^{(\ell)}), \quad p_t^{(\ell)} = \text{softmax}(z_t^{(\ell)}), \quad H_t^{(\ell)} = -\sum_{v \in \mathcal{V}} p_t^{(\ell)}(v) \log p_t^{(\ell)}(v)\)$ When late-stage alignment tax pulls predictions toward generic distributions, predictive uncertainty rebounds. The algorithm formalizes the "Entropy Valley" as the local entropy minimum encountered during backward traversal: $\(\ell^\star_t = \min \left\{ \ell \in [L-K, L] : H_t^{(\ell)} < H_t^{(\ell+1)} < \dots < H_t^{(L)} \right\}\)$ This local valley identifies the exact representational zenith where the model reaches peak task-specific certainty before terminal corruption emerges.
3. Optimal Stopping Theory: Conservative Backward Search and Local Optimality
Dynamic layer selection is formulated as an Optimal Stopping Problem over observable entropy decomposed as \(\hat{H}(l) = H^*(l) + \epsilon(l) + \eta(l)\), where \(H^*(l)\) is strictly decreasing along the refinement corridor \([V_{\text{onset}}, V^*]\) (\(\Delta H^*(l) \le 0\)), \(\epsilon(l)\) denotes bounded projection noise (\(|\epsilon(l)| \le \epsilon_{\text{max}}\)), and \(\eta(l)\) represents per-token terminal alignment perturbation. Theorem 1 proves that under bounded probe noise and dominant terminal perturbation (\(\Delta \eta(l) > 2\epsilon_{\text{max}}\)), the conservative backward rule \(\hat{V} = \max\{l < L \mid \hat{H}(l-1) \ge \hat{H}(l)\}\) never halts within Phase III and strictly terminates at \(\hat{V} \in [V_{\text{onset}}, V^*]\). This provides a rigorous guarantee: the search sacrifices at most a bounded sliver of unfinished refinement while entirely eliminating catastrophic terminal drift.
4. Systems Implementation: Graph-Safe Execution with Zero Memory Overhead
Integrating dynamic layer decoding into high-throughput production engines like vLLM faces challenges with continuous batching, tensor parallelism, and CUDA graph replay. Confident Decoding encapsulates stateful logic within eager wrappers while keeping the compiled graph pure: the compiled forward pass returns raw hidden state tensors without in-graph mutation. Candidate extraction and entropy evaluation are vectorized into fused tensor operations with boolean freezing masks. Crucially, in empirical lazy execution, 88.5% of generated tokens require no auxiliary unembedding projection, resulting in an average of only 0.116 extra projections per token. This confines the empirical FLOPs increase to +0.87%, adds exactly 0 MB to the KV cache, and keeps end-to-end wall-clock latency overhead under 2%.
Benchmark & Evaluation Protocol¶
The method is evaluated across 6 diverse model architectures spanning dense networks (Qwen3.5-27B, Gemma-4-31B) and sparse MoE topologies (Qwen3.5-35B-A3B, Qwen3.5-122B-A10B, gpt-oss-20b, gpt-oss-120B). Evaluations cover seven challenging benchmarks: GPQA-Diamond (graduate science), HLE (frontier academic reasoning), Omni-MATH (Olympiad-level math), LongBench v2 (long-context understanding), LiveCodeBench v6 (code generation), Air-Bench 2024 (safety guardrails), and WritingBench (open-ended writing). Experiments use default lookback window \(K=10\), deterministic selection \(p=1.0\), and greedy temperature \(T=0.0\).
Key Experimental Results¶
Main Results¶
Benchmark performance across diverse model families (three-run means ± unbiased sample standard deviations):
| Model Backbone | Decoding Strategy | GPQA-D | HLE | LCB-v6 | LongBench-v2 | Omni-MATH | Air-Bench | WritingBench |
|---|---|---|---|---|---|---|---|---|
| Qwen3.5-27B | Last Layer | 78.2±1.5 | 16.0±0.2 | 63.9±0.4 | 62.8±0.8 | 76.0±0.1 | 65.7±1.1 | 66.7±0.1 |
| Confident (Ours) | 79.4±1.2 | 16.8±0.4 | 73.3±0.9 | 64.9±0.8 | 76.2±0.1 | 64.7±0.6 | 66.4±0.9 | |
| Absolute \(\Delta\) | +1.2 | +0.8 | +9.4 | +2.1 | +0.2 | -1.0 | -0.3 | |
| Qwen3.5-35B-A3B | Last Layer | 76.3±0.9 | 9.2±2.1 | 70.1±0.1 | 63.2±0.6 | 72.3±0.3 | 51.7±1.1 | 65.2±0.1 |
| Confident (Ours) | 82.8±1.0 | 11.2±1.5 | 74.4±1.2 | 63.9±0.2 | 73.0±0.2 | 55.4±1.2 | 65.3±0.1 | |
| Absolute \(\Delta\) | +6.5 | +2.0 | +4.3 | +0.7 | +0.7 | +3.7 | +0.1 | |
| Qwen3.5-122B-A10B | Last Layer | 83.3±0.8 | 14.7±0.5 | 76.8±0.3 | 66.4±0.4 | 78.3±0.2 | 66.0±0.8 | 72.6±0.2 |
| Confident (Ours) | 85.4±0.7 | 16.5±0.3 | 79.2±0.5 | 66.7±0.3 | 78.5±0.1 | 67.0±0.6 | 72.7±0.3 | |
| Absolute \(\Delta\) | +2.1 | +1.8 | +2.4 | +0.3 | +0.2 | +1.0 | +0.1 | |
| Gemma-4-31B | Last Layer | 76.8±1.2 | 9.2±0.8 | 75.1±0.3 | 58.6±0.7 | 68.3±0.3 | 74.0±0.9 | 64.2±0.2 |
| Confident (Ours) | 80.8±0.9 | 11.0±0.6 | 78.3±0.5 | 59.1±0.5 | 69.0±0.2 | 76.0±0.7 | 64.4±0.3 | |
| Absolute \(\Delta\) | +4.0 | +1.8 | +3.2 | +0.5 | +0.7 | +2.0 | +0.2 | |
| gpt-oss-120B | Last Layer | 76.3±1.0 | 10.8±0.4 | 83.1±0.3 | 54.7±0.6 | 70.2±0.3 | 88.0±0.5 | 67.5±0.2 |
| Confident (Ours) | 80.8±0.8 | 12.6±0.3 | 84.5±0.4 | 55.6±0.5 | 71.0±0.2 | 90.0±0.4 | 67.8±0.2 | |
| Absolute \(\Delta\) | +4.5 | +1.8 | +1.4 | +0.9 | +0.8 | +2.0 | +0.3 |
Ablation Study & Baseline Comparison¶
Comparison with contrastive decoding baselines on Qwen3.5-35B-A3B (Table 7):
| Method | GPQA-D | HLE | LCB-v6 | LongBench-v2 | Omni-MATH | Air-Bench | WritingBench |
|---|---|---|---|---|---|---|---|
| Last Layer Decoding | 76.3 | 9.2 | 70.1 | 63.2 | 72.3 | 51.7 | 65.2 |
| DoLa (Chuang et al., 2023) | 77.3 | 7.8 | 70.9 | 63.4 | 72.4 | 52.0 | 65.2 |
| SLED (Zhang et al., 2024) | 78.8 | 7.4 | 71.7 | 63.2 | 72.1 | 53.0 | 65.3 |
| Confident Decoding (Ours) | 82.8 | 11.2 | 74.4 | 63.9 | 73.0 | 55.4 | 65.3 |
Base vs. Instruct comparison demonstrating alignment tax amplification (Table 3):
| Model Variant | Strategy | GPQA-D | HLE | LCB-v6 | LongBench-v2 | Omni-MATH | Air-Bench | WritingBench | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-35B-A3B-Base | Last Layer | 70.3 | 8.0 | 55.0 | 59.0 | 61.3 | 67.3 | 64.0 | 55.0 |
| Confident | 72.2 | 9.0 | 57.4 | 58.9 | 61.1 | 70.2 | 63.8 | 56.1 | |
| Absolute \(\Delta\) | +1.9 | +1.0 | +2.4 | -0.1 | -0.2 | +2.9 | -0.2 | +1.1 | |
| Qwen3.5-35B-A3B (Instruct) | Last Layer | 76.3 | 9.2 | 70.1 | 63.2 | 72.3 | 51.7 | 65.2 | 58.3 |
| Confident | 82.8 | 11.2 | 74.4 | 63.9 | 73.0 | 55.4 | 65.3 | 60.9 | |
| Absolute \(\Delta\) | +6.5 | +2.0 | +4.3 | +0.7 | +0.7 | +3.7 | +0.1 | +2.6 |
Task-difficulty stratification on MATH and Omni-MATH (gpt-oss-20b, Table 12): - Level 1 (Easy): Saturated baseline, minor variation: MATH 98.2% vs 97.8% (-0.4%), Omni-MATH 97.0% vs 92.7% (-4.3%); - Level 4 (Extreme): Near baseline collapse, massive recovery: MATH improves from 2.1% to 24.6% (+22.5%), Omni-MATH improves from 1.1% to 23.5% (+22.4%).
Key Findings¶
- Gains Concentrate on Complex Reasoning: Performance on open-ended composition and long-context retrieval remains stable (WritingBench varies within \(\pm 0.3\%\)), while gains erupt on syntax-sensitive coding and frontier reasoning (+9.4% on LiveCodeBench for Qwen3.5-27B; +6.5% on GPQA-D for Qwen3.5-35B-A3B).
- Direct Validation of Alignment Tax: Instruct models yield substantially higher intervention gains than Base models (+2.6 vs. +1.1 points on average). Token-level non-final selection rates rise from 10.4% in Base to 12.8% in Instruct, confirming that preference tuning intensifies terminal generic-token bias.
- Substitution Analysis: At substitution positions, Standard Decoding disproportionately favors high-frequency function words and punctuation (e.g., "the", "is", "so", "."), whereas Confident Decoding preserves domain-specific, content-bearing semantic entities (e.g., "mass", "radius", "approximately", "Cartesian").
Highlights & Insights¶
- Challenging the Monotonic-Depth Dogma: Formalizes the Guess–Refine–Perturb dynamic, providing concrete mechanistic evidence that the final layer is not inherently optimal for complex reasoning due to post-training distortions.
- Non-Destructive Vertical TTC: Preserves the complete forward pass and full KV cache, distinguishing itself from conventional early-exit methods by serving as an efficacy-driven vertical test-time computation paradigm rather than a latency-cutting heuristic.
- Robustness Over Contrastive Baselines: By independently evaluating predictive entropy per candidate layer rather than taking cross-layer logit differences, Confident Decoding avoids amplifying architectural noise caused by expert switching in MoEs and layer alternation in hybrid architectures.
Limitations & Future Work¶
- Unembedding Probe Mismatch: The framework relies on projecting intermediate hidden states through the shared final unembedding matrix \(W_U\). On compact hybrid backbones with frequent layer-type switching (e.g., Qwen3.5-9B interleaving DeltaNet and full attention), subspace geometry shifts can induce structured probe noise \(\epsilon_{\text{type}}\), leading to localized regressions on benchmarks with shallow refinement gradients.
- Inference-Time Patching vs. Training-Time Fix: The method mitigates symptoms of late-stage distributional drift during inference but does not rectify the underlying structural deficiencies of RLHF/DPO objectives.
- Fixed Lookback Window: The current fixed window \(K=10\) could benefit from dynamic, model-depth-aware lookback boundaries.
Related Work & Insights¶
- vs. Contrastive Decoding (DoLa / SLED): DoLa and SLED contrast representations across layers under the assumption of subspace comparability, which breaks down under MoE routing and hybrid attention. Confident Decoding evaluates entropy independently per layer, offering architectural robustness across dense, hybrid, and MoE networks.
- vs. Early-Exit Decoding (CALM / LayerSkip): Early-exit methods truncate the forward pass to save compute, severely degrading multi-step reasoning capabilities. Confident Decoding retains the entire \(L\)-layer forward pass and KV cache, optimizing purely for readout fidelity.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates the Guess-Refine-Perturb dynamic, models layer selection via optimal stopping, and provides an elegant training-free solution.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across 6 model architectures, 9 scale tiers, 7 diverse benchmarks, token-level substitutions, and difficulty stratification.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical framing, clear narrative arc, and transparent exposition of both successes and failure modes.
- Value: ⭐⭐⭐⭐⭐ Drop-in compatibility with zero extra KV-cache footprint and negligible overhead (<2% latency), offering immediate utility for inference engines like vLLM and SGLang.