Large Reasoning Models Learn Better Alignment from Flawed Thinking¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/poloclub/recap
Area: Reasoning
Keywords: Large Reasoning Models, Safety Alignment, CoT Prefilling, Reinforcement Learning, Counter-Aligned Prefilling
TL;DR¶
Addressing the acute vulnerability where large reasoning models (LRMs) are easily hijacked by flawed chain-of-thought (CoT) premises, this paper introduces RECAP, an RL post-training framework that mixes counter-aligned reasoning prefills into rollouts, training models to self-reflect and override deceptive trajectories to achieve robust alignment without inference overhead or math capability degradation.
Background & Motivation¶
Frontier large reasoning models (LRMs), such as DeepSeek-R1, OpenAI-o3, and Qwen3-Thinking, have achieved groundbreaking empirical performance across mathematical competitions and complex code synthesis by generating structured, long chain-of-thought (CoT) trajectories prior to outputting final answers. These models frequently exhibit emergent cognitive behaviors, notably "aha moments" where the model revisits initial premises, introspects, and backtracks to refine flawed reasoning steps. However, despite these apparent reasoning superpowers, frontier LRMs remain fundamentally brittle in their safety alignment: recent investigations demonstrate that prefilling an LRM's reasoning trace with an innocuous affirmative phrase such as "I know that" or a few sentences of malicious reasoning can completely dismantle safety guardrails and coax the model into generating harmful instructions. Because CoT prefilling is natively supported by both open-weights inference frameworks and major commercial cloud APIs, this vulnerability exposes a critical flaw—LRMs do not possess robust, critical reasoning about safety constraints.
A systematic diagnosis of this failure reveals an alarming phenomenon termed "Following Without Thinking." When an LRM is seeded with a partial reasoning trace extracted from a weakly aligned model, its safety scores plummet catastrophically, following the flawed premise down an unsafe path rather than correcting it. Conversely, if the exact same model is prefilled with a reasoning prefix from an exceptionally safe model, its safety score surges by up to 91.7%. This profound sensitivity reveals an intrinsic generalization gap in contemporary post-training reinforcement learning (such as standard RLHF, GRPO, or DAPO): during online RL, models are optimized exclusively against final outcome rewards computed on smooth, self-generated rollouts. They are never systematically taught how to navigate noisy, adversarial, or flawed intermediate reasoning states. Consequently, when deployed models encounter imperfect or adversarially corrupted intermediate steps at test time, they lack any mechanism to recover and derail.
To bridge this vulnerability, standard post-training remedies typically resort to supervised fine-tuning (SFT) over curated safe demonstrations or applying external guardrails. However, such interventions frequently trigger a severe "safety tax," inducing heavy overrefusal on benign queries and degrading core reasoning capabilities. The authors turn this brittleness on its head: rather than hoping self-correction will spontaneously emerge under clean rollouts, they advocate exposing the model directly to synthetic counter-aligned reasoning during RL training. By forcing the policy to recover from intentionally induced unsafe thinking on harmful prompts and from overconservative thinking on benign prompts, the model learns to internalize alignment values deeply into its reasoning dynamics. Core idea: propose RECAP (Robust Safety Alignment via Counter-Aligned Prefilling), an RL post-training method that injects counter-aligned CoT prefixes into training rollouts and computes policy updates exclusively on self-generated continuation tokens, forcing models to override flawed trajectories and internalize robust self-correction.
Method¶
Overall Architecture¶
RECAP integrates cleanly into standard online reinforcement learning pipelines (specifically built upon DAPO and GRPO) without requiring extra reward models, architectural changes, or additional training phases. During training, RECAP operates in three key phases: counter-aligned prefill synthesis, hybrid rollout sampling, and prefix-sliced policy gradient optimization. A designated fraction of training prompts are prepended with counter-aligned reasoning prefixes—harmful queries receive unsafe reasoning traces, while benign overrefusal queries receive overcautious refusal traces. The policy model is forced to resume reasoning from this corrupted prefix, explore continuations, and produce a final response. Policy gradients and importance sampling ratios are computed strictly on the newly generated continuation tokens (\(t > \ell_{\text{pre}}\)), teaching the model that blindly continuing flawed prefixes results in zero reward, whereas actively overriding the prefix and rerouting to safe, helpful responses achieves optimal returns. Crucially, RECAP requires no CoT prefilling at deployment, ensuring zero inference-time modifications.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-domain Training Prompts<br/>(Harmful / Benign Sensitive / Math)"] --> B["Counter-Aligned Prefill Construction<br/>Unsafe CoT for Harmful / Refusal CoT for Benign"]
B --> C["Hybrid Rollout Prefill Sampling<br/>Inject ratio alpha and length cutoff"]
C --> D["Policy Continuation Generation<br/>Autonomous self-reflection and response"]
D --> E["Prefix-Sliced Policy Gradient Optimization<br/>Compute gradient only for t > l_pre"]
E --> F["Robust Aligned Reasoning Model"]
Key Designs¶
1. Counter-Aligned Prefill Construction: Engineering Target-Reversed Deceptive Supervision To supply realistic yet deceptive reasoning paths without manual annotation, RECAP synthesizes counter-aligned CoT traces \(y_{\text{cot}}^{\text{pre}}\) using external proxy models exhibiting alignment biases orthogonal to the desired objective. For direct harmful queries, reasoning prefixes are sampled from a weakly aligned model (DSQwen-7B) at high temperature (\(T=0.9, p=0.95\)); candidate traces are evaluated with a safety classifier (LlamaGuard) to ensure the injected premise actively pursues harmful instructions. For benign sensitive queries prone to overrefusal, counter-aligned prefixes are generated by fine-tuning DSQwen-7B exclusively on safety data (STAR-1), creating an overly defensive generator that routinely proposes refusing harmless questions (e.g., terminating a process). Seeding the training rollouts with these opposing traces creates an acute tension: following the prefilled reasoning leads directly to unsafe answers or unhelpful refusals, compelling the policy to question and counter the injected logic.
2. Hybrid Rollout Prefill Sampling: Balancing Autonomous Initiation with In-Trajectory Recovery Exposing the policy solely to prefilled inputs would degrade its ability to initiate clean, unprompted reasoning trajectories during deployment. RECAP introduces a mixing ratio \(\alpha \in (0, 1)\) (optimally set to 0.5), injecting counter-aligned prefixes into only a subset of safety training data while keeping the remainder as standard, uncorrupted prompts. Furthermore, the prefix length is strictly bounded by \(\ell_{\text{pre}}\) (set to 500 words). This hybrid formulation ensures that the model preserves the dual capability of both generating sound reasoning from scratch and maintaining heightened vigilance to override corrupt trajectories when misleading premises appear mid-stream.
3. Prefix-Sliced Policy Gradient Optimization: Isolating External Tokens for True Rerouting Gradients Standard RL algorithms compute importance weights across the entire sequence, which introduces severe distributional mismatch when external, frozen prefix tokens are present. RECAP resolves this by defining an adaptive optimization token threshold \(t_0(x) = \ell_{\text{pre}} + 1\) for prefilled prompts and \(t_0(x) = 1\) for clean prompts. Importance sampling ratios \(r_{i,t}(\theta)\) and normalized advantages \(\hat{A}_{i,t}\) are computed solely over the sequence length \(|o_i|_{\text{opt}} = |o_i| - (t_0(x) - 1)\):
Since rewards \(R_i\) evaluate only the final response \(y_{\text{resp}}\) against safety, helpfulness, and mathematical correctness, this objective exerts intense gradient pressure on the self-generated continuation tokens \(y_{\text{cot}}^{\text{gen}}\) to spontaneously produce transition phrases like "Wait, but this is unsafe..." and systematically derail the malicious prefill.
Loss & Training¶
RECAP leverages Decouple Clip and Dynamic Sampling Policy Optimization (DAPO) with reward filtering: prompts where rollouts achieve either all zero rewards (\(R_i = 0\)) or uniformly perfect rewards (\(R_i > \tau\)) are discarded during training to focus model capacity on high-variance exploratory boundaries. Training is conducted on a multi-objective mixture of 5,000 prompts: 1,000 harmful prompts from BeaverTails, 1,000 overrefusal prompts from STAR-1, and 3,000 mathematical reasoning prompts from GSM8K and MATH (with mathematical prompts receiving zero prefilling). Hyperparameters include rollout group size \(G=16\), maximum context length \(L_{\text{max}}=8,000\) tokens, clipping bounds \(\varepsilon_{\text{low}}=0.2, \varepsilon_{\text{high}}=0.28\), and 1 training epoch on 8× A100-80GB GPU nodes using DeepSpeed ZeRO-3 with bfloat16. Continuous safety rewards are derived from IBM Granite-Guardian-3.1-8B guardrail logits, overrefusal is scored by Llama-3.1-8B-Instruct with rubrics, and math correctness uses Reinforcement Learning with Verifiable Rewards (RLVR).
Key Experimental Results¶
Main Results¶
The authors conduct comprehensive post-training evaluations across DSLlama-8B and DSQwen-14B models on direct harmfulness (STRONGREJECT and its prefilled counterpart with DSQwen-1.5B traces), jailbreaking (WILDJAILBREAK, Scale AI FORTRESS), overrefusal helpfulness (XSTEST, FORTRESS-OR), and core mathematical reasoning (MATH500, GSM8K, AIME2024).
| Model & Method | STRONGREJECT (Safety↑) | STRONGREJ-Prefill (Safety↑) | WILDJAILBREAK (Safety↑) | FORTRESS (Safety↑) | XSTEST (Helpfulness↑) | FORTRESS-OR (Helpfulness↑) | MATH500 (pass@1↑) | GSM8K (pass@1↑) | AIME2024 (pass@16↑) |
|---|---|---|---|---|---|---|---|---|---|
| DSLlama-8B Suite | |||||||||
| Original | 57.83 | 45.05 | 24.40 | 60.83 | 90.40 | 90.00 | 81.00 | 90.32 | 70.00 |
| STAR (SFT) | 77.00 | 59.42 | 75.50 | 78.49 | 78.40 | 86.00 | 81.80 | 90.74 | 66.67 |
| SafeChain (SFT) | 68.05 | 60.38 | 46.60 | 65.56 | 90.40 | 84.50 | 79.40 | 91.32 | 70.00 |
| SFT (In-domain) | 73.48 | 57.51 | 71.60 | 74.25 | 80.00 | 82.40 | 81.90 | 91.32 | 63.33 |
| DAPO (Vanilla RLHF) | 96.81 | 79.23 | 72.90 | 68.86 | 78.00 | 82.80 | 82.20 | 93.71 | 66.67 |
| RECAP (Ours) | 99.68 | 98.70 | 88.75 | 86.84 | 91.87 | 91.80 | 83.60 | 93.72 | 70.00 |
| DSQwen-14B Suite | |||||||||
| Original | 68.69 | 49.20 | 34.40 | 57.91 | 96.00 | 95.00 | 86.40 | 95.20 | 86.67 |
| STAR (SFT) | 82.74 | 59.11 | 74.50 | 79.05 | 86.40 | 93.20 | 87.20 | 95.86 | 86.67 |
| SafeChain (SFT) | 69.65 | 54.95 | 53.30 | 63.15 | 96.00 | 96.40 | 89.60 | 96.44 | 86.67 |
| SFT (In-domain) | 85.43 | 59.11 | 70.03 | 74.54 | 90.00 | 96.00 | 89.60 | 95.90 | 83.33 |
| DAPO (Vanilla RLHF) | 99.04 | 80.51 | 77.60 | 67.85 | 96.80 | 95.00 | 88.80 | 97.19 | 86.67 |
| RECAP (Ours) | 99.04 | 98.08 | 91.65 | 80.17 | 96.80 | 97.60 | 90.00 | 97.77 | 86.67 |
Ablation Study & Adaptive Robustness¶
The authors ablate key hyperparameters on DSLlama-8B (prefill ratio \(\alpha\), prefill length \(\ell_{\text{pre}}\), and source distribution) and subject the model to aggressive adaptive attacks: Full CoT Hijacking (forcing generation to start in the response segment) and Iterative Prefill Reset (IPR, inserting repeated adversarial resets across \(k\) rounds).
| Setting / Attack Variant | STRONGREJECT (Safety↑) | WILDJAILBREAK (Safety↑) | XSTEST (Helpfulness↑) | Empirical Insights & Mechanics |
|---|---|---|---|---|
| Prefill Source (DSLlama-8B) | ||||
| Vanilla DAPO (\(\alpha=0\)) | 96.81 | 72.90 | 78.00 | Susceptible to jailbreaks due to lack of perturbation during training |
| Aligned Prefills (Safe CoT) | 91.20 | 65.40 | 76.50 | Policy exploits safe prefixes without acquiring corrective capability |
| Counter-Aligned (RECAP) | 99.68 | 88.75 | 91.87 | Forces active trajectory overriding; yields strongest self-correction |
| Prefill Ratio \(\alpha\) | ||||
| \(\alpha = 0.25\) | 98.40 | 82.10 | 88.50 | Sub-optimal corrective supervision for adversarial jailbreaks |
| \(\alpha = 0.50\) (Default) | 99.68 | 88.75 | 91.87 | Sweet spot: optimal balance between clean initiation and recovery |
| \(\alpha = 1.00\) (All Prefilled) | 94.20 | 78.50 | 84.10 | Model overfits to prefill context, harming autonomous clean starts |
| Full CoT Hijacking Attack | ||||
| DSLlama-8B: Vanilla DAPO | 70.29 | - | - | Collapses when reasoning is bypassed via malicious CoT + </think> |
| DSLlama-8B: RECAP | 98.08 (+27.79) | - | - | Retains deep internal alignment even when forced to answer directly |
| DSQwen-14B: Vanilla DAPO | 73.48 | - | - | Severe safety degradation under complete reasoning hijacking |
| DSQwen-14B: RECAP | 96.49 (+23.01) | - | - | Consistently robust under full CoT hijacking |
| Iterative Prefill Reset (IPR) | \(k=1\) round | \(k=2\) rounds | \(k=3\) rounds | |
| Vanilla DAPO | 79.23 | 70.29 | 69.65 | Safety steadily degrades as adversary repeatedly resets thought trace |
| RECAP (Ours) | 98.72 | 98.08 | 97.44 | Persistently enforces safe refusal across repeated adversarial resets |
Key Findings¶
- Cross-Distribution & Out-of-Family Generalization: While RECAP uses DSQwen-7B counter-aligned prefills on safety data during training, it generalizes exceptionally well to completely unseen adversarial attacks without prefilling (e.g., CodeAttack \(+21.7\%\), ReNeLLM \(+22.9\%\), GCG \(+17.1\%\)), and performs identically when trained using LLaMA-3.2-1B prefills, confirming the learned resilience is model-agnostic.
- Dispelling the "Safety Tax" Trade-Off: Traditional SFT approaches drastically increase overrefusal (XSTEST dropping to 78.4%), whereas RECAP enhances helpfulness to 91.87% on DSLlama-8B and 96.80% on DSQwen-14B while actually improving MATH500 pass@1 by \(+1.2\%\) to \(+2.6\%\), proving that critical reasoning post-training does not degrade capabilities.
- Negligible Latency Overhead with Deeper Thinking: Inference token budgets remain essentially identical between RECAP and DAPO (707 vs 625 tokens on DSLlama-8B). However, internal reflection frequency surges from 43.9% to 74.2% on clean benchmarks, with first-correction events well-distributed throughout generation (68.9% occurring after the initial quarter of generation).
Highlights & Insights¶
- Adversarial In-Rollout Perturbation: While standard RLHF optimizes policies on their own unperturbed generations, RECAP introduces the philosophy of training on flawed thinking. Intentionally corrupting intermediate reasoning trajectories provides an indispensable supervisory signal for robust decision boundaries.
- Zero-Inference Footprint: The entire mechanism lives strictly within the training data loader and loss masking logic. No special control tokens, auxiliary verification networks, or multi-turn test-time prompts are required at deployment.
- Deep Intent Disambiguation: In obfuscated jailbreak prompts (such as circled Unicode glyphs spelling "bump stocks for AR-15"), RECAP first faithfully resolves the obfuscation in its CoT, recognizes the real-world hazard, and only then executes a deliberate pivot toward refusal, proving it does not rely on shallow keyword filtering.
Limitations & Future Work¶
- Dependency on External Misaligned Generators: Synthesizing effective counter-aligned traces requires an auxiliary generator with known biases (e.g., weakly aligned or overly cautious models). Extending this to fully autonomous self-play red-teaming represents an important next step.
- Extension to Open-Ended Logic and Math: While counter-aligned prefilling works remarkably well in safety where boundaries are binary (safe vs unsafe, helpful vs refusal), applying it to mathematical domains is more nuanced due to multiple branching error types and false-positive heuristics.
- Multimodal & Long-Context Extensions: Investigating whether vision-language reasoning models (VLMs) suffer from analogous multimodal prefilling vulnerabilities remains an exciting frontier.
Related Work & Insights¶
- vs SafeChain & STAR: SafeChain and STAR rely on supervised fine-tuning over curated reasoning paths, which induces rigidity, overrefusal, and vulnerability to out-of-distribution reasoning prefixes; RECAP utilizes online RL with negative trajectory exploration, allowing the model to acquire true trajectory-level recovery.
- vs Backtracking (Zhang et al., 2024): Backtracking introduces explicit
[RESET]control tokens and triggers hard sequence restarts during inference; RECAP achieves in-trajectory soft rerouting within natural language CoT without altering decoding vocabulary or inference protocols. - vs Vanilla RLHF (DAPO / GRPO): Standard RLHF optimizes models solely on self-generated successful rollouts, leaving models completely defenseless against noisy or malicious premises; RECAP builds defensive resilience by forcing the policy to recover from active perturbations.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ (Pioneers the paradigm of learning robust alignment by exposing LRMs to synthetic flawed thinking during RL rollouts)
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Exhaustive empirical validation across multiple architectures, adaptive attacks, full CoT hijacking, IPR, and unseen jailbreaks)
- Writing Quality: ⭐⭐⭐⭐⭐ (Lucid motivation, rigorous formulations, compelling qualitative analyses, and highly reproducible design)
- Value: ⭐⭐⭐⭐⭐ (Directly neutralizes the fatal CoT-prefilling vulnerability in modern reasoning models without incurring a safety tax)