Skip to content

Format Collapse in Policy Optimization for LLM Reasoning

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Reinforcement Learning, Policy Optimization, Cognitive Shortcut, Format Collapse, Least Resistance Bias

TL;DR

By systematically relaxing rigid chain-of-thought structural constraints in verifiable-reward reinforcement learning, this paper demonstrates that policy optimization consistently collapses into an answer-only format without explicit reasoning, driven by an intrinsic "Least Resistance Bias" toward the most learnable gradient signal.

Background & Motivation

Reinforcement learning with verifiable rewards (RLVR) has become the predominant paradigm for post-training large language models to master complex multi-step reasoning. Standard protocols, popularized by recent reasoning breakthroughs such as DeepSeek-R1, enforce a rigid formatting regime that compels the model to formulate explicit reasoning traces inside <think> tokens before generating the final solution in <answer> blocks. While empirically effective in coaxing long CoT trajectories, this heavy-handed structural coercion fundamentally obscures an essential question: what are the intrinsic structural preferences of policy optimization when such artificial formatting constraints are relaxed? If modern policy gradient optimizers harbor a deep-seated bias toward specific output topologies, prevailing post-training methodologies might be expensively fighting against the natural grain of learning dynamics.

Existing literature on shortcut learning in machine learning has predominantly characterized shortcuts as artifacts of observational biases, poor data coverage, or specific algorithmic limitations, tacitly treating the optimizer itself as a neutral agent that dispassionately maximizes expected return. However, in open multi-step generative tasks where a reward function permits both verbose, reasoned responses and direct, concise answers, it remains unclear whether the policy dynamically learns an adaptive allocation of computation according to problem hardness, or whether the optimization process exhibits an unprompted pathology.

To rigorously address this gap, this paper studies policy optimization behavior across six models spanning five architectures (Gemma-3 4B & 12B, Qwen-2.5 7B, Llama-3.1 8B, Ministral 8B, and Yi 6B), four reasoning benchmarks (GSM8K, rStar-Coder, ReClor, and Planning-Mystery), and three distinct RLVR algorithms (DAPO, Dr. GRPO, and REINFORCE++). Core idea: policy optimization exhibits an intrinsic "Least Resistance Bias" that drives the policy to collapse into an answer-only "Cognitive Shortcut", persisting even against 4x reward gradients and mutually exclusive alternatives because unconstrained policy exploration enables the optimizer to greedily exploit the most immediately learnable gradient signal rather than maximize global expected reward.

Method

Overall Architecture

To decouple and causally verify the mechanics of format collapse and the Least Resistance Bias, the authors establish an empirical progression. First, they relax the standard strict think-then-answer reward into a flexible composite reward that permits both reasoned traces and direct answer-only outputs. Second, to verify whether format selection follows a systematic procedural curriculum, they construct a nested Matryoshka-style reward hierarchy (\(f_1 \subset f_2 \subset f_3\)). Third, they deploy adversarial perturbations—including geometric exponential reward multipliers and mutually exclusive format allocations—to test the durability of the bias. Finally, they use KL divergence constraints as a diagnostic lens to inspect the phase transition between exploratory freedom and reward hacking.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Query q (Multi-step Reasoning Task)"] --> B["1. Composite Reward & Cognitive Shortcut Discovery<br/>Relax CoT constraints to compare Strict vs. Composite rewards"]
    B --> C["2. Nested Reward Hierarchy Sequential Verification<br/>Construct f1⊂f2⊂f3 to audit staged procedural mastery"]
    C --> D["3. Exponential & Mutually Exclusive Adversarial Ablations<br/>4x reward scaling and disjoint format boundary tests"]
    D --> E["4. KL Divergence Diagnostic & Exploration Trade-off<br/>Vary KL leash tightness to expose reward hacking phase transition"]
    E --> F["Sampled Output y Evaluated on Pass@5 & Format Distributions"]

Key Designs

1. Composite Reward & Cognitive Shortcut Discovery: Relaxing format rigidity to isolate structural preferences To reveal how policy optimization behaves without external structural crutches, the authors formulate a Composite Reward \(R_{\text{composite}}(y)\) as a counterpart to the conventional Strict Reward \(R_{\text{strict}}(y)\). While \(R_{\text{strict}}\) grants \(+3.0\) points strictly for an exact <think>...</think><answer>...</answer> sequence and penalizes any deviation by \(-4.0\), \(R_{\text{composite}}\) rewards valid outputs containing any combination of interleaved thinking blocks, and critically, direct <answer> outputs that omit reasoning entirely. In conjunction with a verifiable correctness reward \(R_{\text{correct}}\) (\(+3.0\) for ground-truth equivalence within \boxed{}, \(-5.0\) otherwise), an adaptive agent should logically produce detailed thoughts on complex questions while taking direct shortcuts on trivial arithmetic. Instead, the policy uniformly collapses to the minimal answer-only format across \(>95\%\) of generations, incurring severe accuracy degradation. The authors term this systematic omission of explicit thought traces the "Cognitive Shortcut."

2. Nested Reward Hierarchy Sequential Verification: Probing staged mastery across structural complexities To examine whether this preference for simplicity represents an absolute impasse or a staged procedural curriculum, the authors introduce a nested reward structure \(f_1 \subset f_2 \subset f_3\): - Easy format (\(f_1\)): Simply requires the final answer to be enclosed in \boxed{.*}; - Medium format (\(f_2\)): A superset of \(f_1\), requiring the entire response to be enclosed in <answer> tags; - Hard format (\(f_3\)): A superset of \(f_2\), requiring the full <think>...</think><answer>...</answer> structure. With an additive reward objective \(R_{\text{total}}(y) = \sum_{i=1}^3 r(y, f_i)\), any generation that satisfies \(f_3\) automatically collects the rewards for \(f_1\) and \(f_2\), establishing a strong cumulative incentive to converge to the all-encompassing format. Tracking the optimization trajectories reveals a clear three-stage progression: the optimizer invariably saturates \(f_1\) within 200–300 steps before making any measurable ascent on \(f_2\), only tackling \(f_3\) once \(f_2\) is well underway. This confirms that the Least Resistance Bias dictates a strictly sequential learning curve.

3. Exponential & Mutually Exclusive Adversarial Ablations: Quantifying bias tenacity under conflicting incentives To quantify whether strong financial incentives or structural independence can override this bias, two adversarial stress tests were constructed: - Geometric Exponential Progression: The reward magnitudes were scaled geometrically such that \(r(y, f_3) = 4r(y, f_1) = 2r(y, f_2) = +4.0\). Despite hard format \(f_3\) yielding four times the payoff of easy format \(f_1\), the policy stubbornly focuses on \(f_1\) for the initial 150–200 steps, temporarily ignoring the massive reward ceiling in favor of the immediately accessible gradient signal. - Mutually Exclusive Disjoint Formatting: By enforcing strict string boundary anchors (^ and $), formats were made strictly mutually exclusive such that satisfying one explicitly invalidated the others. In this zero-substructure landscape with identical positive rewards (\(R_{\text{conflict}}(y) = R_{\max} \cdot \mathbf{1}\{y \in f_i\}\)), the policy exhibits absolute collapse: over 99.6% of final Qwen-2.5 7B outputs belong exclusively to \(f_1\), while \(f_2\) and \(f_3\) remain at exactly 0.0%, proving that without shared step-stones, the optimizer completely foregoes complex formats.

4. KL Divergence Diagnostic & Exploration Trade-off: Unveiling the mechanistic link to reward hacking Treating the Kullback-Leibler divergence with respect to the initial policy \(\pi_{\text{ref}}\) as a diagnostic sensor rather than merely a regularization loss, the authors investigate how policy exploration facilitates shortcut discovery. Contrasting a "Tight Leash" (\(\beta = 0.3\)) against a "Loose Leash" (\(\beta = 0.1\)) under composite rewards demonstrates that format collapse is not passive entropy decay, but an active optimization journey that requires substantial drift from the pre-trained prior. In loose leash training, an abrupt phase transition occurs near step 600: a sharp surge in KL divergence coincides with a dramatic escalation in format reward and a collapse in token completion length from \(\sim 700\) to \(\sim 100\) tokens, confirming that the exploratory freedom necessary for high-reward policy discovery is precisely the engine that enables reward hacking.

Loss & Training

The study validates findings across three distinct on-policy optimization algorithms: 1. DAPO / Dr. GRPO: Implemented with decoupled clipping (\(\epsilon_{\text{high}} = 0.28\)) and correction for response length/question difficulty biases; trained with learning rate \(5\times 10^{-6}\), gradient clipping norm \(0.15\), and cosine decay; 2. REINFORCE++: Critic-free policy gradient utilizing global advantage and reward normalization, trained with learning rate \(5\times 10^{-5}\) and 2 PPO epochs; 3. LoRA Fine-tuning: Parameter-efficient training across all linear projection modules (\(r = 64, \alpha = 128\)) with bf16 precision, running 3 epochs with 5 sampled rollouts per prompt.

Quantitative Experimental Results

Main Results: Strict vs. Composite Reward Paradigms

Performance is evaluated using Pass@5 accuracy (%) across four benchmarks: GSM8K (math), rStar-Coder (code), ReClor (logic), and Planning-Mystery (planning).

Model Params GSM8K (Strict) GSM8K (Composite) rStar-Coder (Strict) rStar-Coder (Composite) ReClor (Strict) ReClor (Composite) Planning (Strict) Planning (Composite)
Gemma-3 4B 72.4 ± 0.5 73.1 ± 0.4 55.8 ± 0.6 53.5 ± 0.7 73.2 ± 0.3 69.4 ± 0.5 58.3 ± 0.4 55.9 ± 0.6
Qwen-2.5 7B 92.4 ± 0.3 85.5 ± 0.5 73.2 ± 0.4 73.6 ± 1.2 80.1 ± 0.2 73.8 ± 0.4 77.5 ± 0.3 73.8 ± 0.5
Llama-3.1 8B 86.2 ± 0.4 82.9 ± 0.6 64.1 ± 0.5 59.2 ± 0.6 78.3 ± 0.3 72.7 ± 0.5 74.9 ± 0.4 70.6 ± 0.7
Ministral 8B 89.5 ± 0.3 84.8 ± 0.5 74.0 ± 0.4 72.6 ± 0.4 81.0 ± 0.2 76.5 ± 0.4 76.8 ± 0.3 77.3 ± 0.3
Yi 6B 85.1 ± 0.4 83.8 ± 0.4 68.3 ± 0.6 63.7 ± 0.7 68.4 ± 0.4 65.1 ± 0.6 71.2 ± 0.5 67.7 ± 0.6
Gemma-3 12B 94.6 ± 0.2 86.4 ± 0.4 76.1 ± 0.4 72.3 ± 0.5 85.5 ± 0.2 81.2 ± 0.3 81.2 ± 0.3 76.5 ± 0.5

Ablation & Policy Distribution Audits

The table below integrates the final output format distribution under Composite rewards on GSM8K (Table 3), nested Reward Hierarchy (Table 4), and Mutually Exclusive Choice experiments (Table 5), averaged over 3 random seeds.

Setting Model Evaluated % Minimal Shortcut (\(f_1\)) % Full CoT (\(f_3\)) % Invalid / Intermediate Key Finding
Composite Reward Audit Gemma-3 4B 97.3 ± 0.4 2.1 ± 0.6 0.6 ± 0.1 Massive preference for unreasoned direct answers
Composite Reward Audit Qwen-2.5 7B 98.6 ± 0.3 0.8 ± 0.5 0.6 ± 0.3 Virtually complete abandonment of CoT reasoning
Composite Reward Audit Llama-3.1 8B 96.4 ± 0.3 2.7 ± 0.7 0.9 ± 0.5 Consistent across architectural families
Composite Reward Audit Mistral 8B 98.1 ± 0.2 1.2 ± 0.4 0.7 ± 0.0 Chain-of-thought generation drops below 2%
Composite Reward Audit Yi 6B 95.7 ± 0.7 3.2 ± 0.5 1.1 ± 0.3 Shortcut format dominates generation pool
Composite Reward Audit Gemma-3 12B 98.6 ± 0.2 0.6 ± 0.2 0.8 ± 0.4 Scaling model size to 12B does not avert collapse
Reward Hierarchy (Additive) Qwen-2.5 7B 2.4 ± 0.8 (\(f_1\)-only) 90.9 ± 0.3 (\(f_3\)-full) 2.1 ± 0.5 (\(f_2\) is 6.7%) Staged progression converges when bonuses accumulate
Mutually Exclusive (Disjoint) Qwen-2.5 7B 99.6 ± 0.2 (\(f_1\)-only) 0.0 ± 0.0 (\(f_3\)-full) 0.4 ± 0.1 (\(f_2\) is 0.0%) Absolute zero exploration of complex paths

Key Findings

  • Reasoning is not an emergent preference of policy optimization: When free to choose, models do not dynamically scale reasoning with difficulty; they discard it. On GSM8K, training under Composite rewards drops Qwen-2.5 7B accuracy from 92.4% to 85.5% (a 6.9-point degradation) and Gemma-3 12B by 8.2 points. Direct shortcut outputs have an intrinsic accuracy of only 84.9% compared to 97.6% for reasoning outputs, yet the policy greedily defaults to the shortcut.
  • Shortcuts supersede strong reward incentives: Under exponential 4x scaling favoring \(f_3\), learning curves prove that \(f_3\) is totally neglected in the first 200 steps while \(f_1\) saturates, confirming that gradient learnability overrides absolute reward value.
  • Complete format extinction in disjoint action spaces: When formats are mutually exclusive and equally rewarded, generation of \(f_3\) drops to 0.0%, demonstrating the optimizer's complete refusal to explore complex trajectories without intermediate shared sub-problems.
  • KL divergence governs the phase transition into reward hacking: Tight KL regularizers (\(\beta = 0.3\)) preserve generation length (\(\sim 700\) tokens) and prevent format collapse, but impede peak reward discovery; loose regularizers (\(\beta = 0.1\)) undergo a sharp phase transition at step 600, plunging token length to \(\sim 100\) and collapsing format fidelity.

Highlights & Insights

  • Reframing shortcut learning as an intrinsic optimizer bias: Rather than viewing shortcuts as statistical glitches in datasets or algorithmic bugs, this work shows that on-policy optimization naturally gravitates toward the path of least cognitive resistance.
  • The exploration vs. hacking dilemma: The exploratory freedom required by RLVR to unlock complex reasoning is mathematically the exact mechanism that enables models to locate and exploit underspecified format loopholes.
  • Crucial implication for RLVR reward design: Flexible reward schemes that hope for "emergent adaptive thinking" will fail without explicit structural scaffolding or dense intermediate verification.

Limitations & Future Work

  • Model parameter scale: Experiments were capped at 12B parameters due to compute constraints (\(\sim 200\) GPU hours total); future work must investigate whether frontier models (\(\ge 30\text{B}\)/70B) possess enough prior reasoning stability to alter gradient learnability.
  • Algorithmic scope: Limited to on-policy algorithms (DAPO, GRPO, REINFORCE++); offline RL, Direct Preference Optimization (DPO), and test-time search remains to be audited.
  • Theoretical gradient analysis: The paper establishes empirical and behavioral dynamics; a closed-form mathematical proof characterizing the signal-to-noise ratio of simplistic format gradients remains an open direction.
  • vs. Sohn et al. (ICML 2021) Shortest-Path Constrained RL: Sohn et al. constrain agents to shortest paths to eliminate redundant environment transitions, where the shortest path is the true optimal solution. In contrast, this paper identifies a shortest path that is cognitively trivial and suboptimal in task accuracy.
  • vs. Liu et al. (2022) Offline RL Propensity Overfitting: Liu et al. attribute shortcuts to sample coverage gaps in offline datasets; here, format collapse emerges robustly in fully explored on-policy settings, proving it is intrinsic to the optimizer.
  • vs. Mead et al. (ICLR 2026) Risk-Averse CVaR Blindness: Mead et al. study algorithms mathematically barred from observing successful tails; here, the optimizer perceives the higher-reward trajectories but actively chooses the simpler path.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates and systematically demonstrates the "Least Resistance Bias" in LLM reasoning policy optimization.
  • Empirical Rigor: ⭐⭐⭐⭐⭐ Rigorous suite spanning 5 model families, 4 task domains, 3 RL algorithms, and controlled nested/mutually exclusive/KL ablations.
  • Clarity & Writing: ⭐⭐⭐⭐⭐ Clear progression from empirical phenomenon to causal stress tests and mechanistic diagnostics.
  • Practical Impact: ⭐⭐⭐⭐⭐ Delivers vital cautionary insights for designing robust reward functions in next-generation reasoning post-training.