Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/mz-kim/RwR
Area: LLM Reasoning
Keywords: Safety Alignment, False Refusal, Response Decomposition, Refusal Statement, Rationale-Only
TL;DR¶
By structurally decomposing safety-tuning responses into boilerplate refusal statements and explanatory rationales, this work uncovers that initial refusal statements act as drivers of superficial cue reliance and false refusals; training models exclusively on request-specific rationales (Rationale-Only) substantially decreases over-refusal on benign queries while maintaining robust safety defense.
Background & Motivation¶
Balancing helpfulness and harmlessness remains one of the most foundational tensions in aligning large language models (LLMs). Mainstream post-training pipelines rely heavily on supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) on curated safety data to teach models to reject dangerous, illegal, or unethical requests. However, this aggressive safety conditioning frequently leads models to an overly cautious failure mode known as false refusal (or exaggerated safety / over-refusal): models mistakenly reject benign user queries simply because they superficially contain risky vocabulary, ambiguous sensitive phrases, or taboo themes (e.g., "How do I blow up a balloon?" or "Where can I shoot a good photo?"). Such unwarranted refusals degrade general utility and undermine user trust.
Prior attempts to mitigate false refusals have largely operated at inference time through prompt optimization, contrastive decoding, internal representation steering, and decision-boundary shifting. While these post-hoc techniques achieve empirical improvements, they inevitably introduce non-trivial inference latency, require fragile manual hyperparameter tuning, or risk eroding general knowledge. From a representation learning perspective, an input prompt \(x\) carries both underlying semantic intent \(\phi_{\text{sem}}(x)\) and superficial lexical patterns \(\phi_{\text{surf}}(x)\). When alignment datasets repeatedly pair templated refusal formulations with risky surface tokens, language models succumb to shortcut learning, latching onto \(\phi_{\text{surf}}(x)\) rather than resolving \(\phi_{\text{sem}}(x)\).
Rather than treating the symptoms at inference time, this work shifts attention to the training data itself. Standard safety datasets (e.g., Safety-Tuned LLaMAs) consistently present defense responses in a rigid, two-part structure: a boilerplate refusal statement (e.g., "I'm sorry, but I cannot fulfill this request") immediately followed by a rationale explaining why the action is hazardous or unlawful. Core idea: by structurally decomposing safety responses into boilerplate refusal statements and refusal rationales, the paper identifies the leading refusal statement as the primary catalyst inducing surface-level keyword anchoring and false refusals, demonstrating that training exclusively on request-specific rationales (Rationale-Only) eliminates over-refusals while preserving full defensive robustness.
Method¶
Overall Architecture¶
The methodology centers on structural decomposition, controlled data synthesis, and internal attribution probing across diverse model families (Llama-3.1, Mistral, Gemma-2, and Qwen2.5). First, safety responses from standard benchmarks are normalized and partitioned into two clean constituent elements: the boilerplate refusal statement and the substantive refusal rationale. Second, researchers systematically manipulate the structural position of the statement (beginning, middle, end), isolate component combinations (Statement-Only, Rationale-Only, Statement and Rationale), and modulate the semantic granularity of the rationale (Request-Specific vs. Generic). Finally, the fine-tuned base models are evaluated under a rigorous evaluation protocol designed to detect pseudo-compliance, accompanied by first-token entropy computation and causal token-level embedding ablations to characterize internal shortcut mechanisms.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw Safety-Tuning Dataset<br/>Harmful Prompts + Responses"] --> B["Safety Response Decomposition<br/>Statement vs. Rationale Partitioning"]
B --> C["Positional & Component Interventions<br/>Beginning / Middle / End / Isolated Subsets"]
C --> D["Rationale Semantic Specificity<br/>Request-Specific vs. Generic Formulation"]
D --> E["Cross-Family QLoRA Fine-Tuning<br/>Identical Optimization & Base Configurations"]
E --> F["Entropy & Token Attribution Probing<br/>Causal Embedding Ablation on Risky Cues"]
F --> G["Dual-Track Strict Evaluation<br/>Harmful Defense vs. Pseudo-Harmful Compliance"]
Key Designs¶
1. Safety Response Decomposition and Normalization: Separating Formulas from Reasons
To dismantle the confounding elements embedded in conventional safety responses, this design establishes a rigorous normalization and decomposition protocol. A standard refusal response is explicitly separated into (i) a formulaic refusal statement lacking case-specific logical content (e.g., "I apologize, but I am unable to assist with that request"), and (ii) a refusal rationale that substantiates the refusal by articulating legal, ethical, or physical risks (e.g., "Counterfeiting currency violates federal law and destabilizes financial systems..."). Utilizing Llama-3.3-70B-Instruct, 256 core safety instances sampled from the Safety-Tuned LLaMAs dataset are standardized into a Base Format featuring a single concise initial statement followed by a coherent rationale of at least two sentences. From this anchor, three cleanly isolated training variants are generated—Statement-Only, Rationale-Only, and Statement and Rationale—preserving underlying factual justifications and lexical substance across conditions without introducing arbitrary content drift.
2. Positional and Structural Statement Manipulation: Probing Leading Attention Biases
To determine whether the negative impact of refusal statements stems from their existence or their privileged position at the sequence onset, this intervention systematically relocates the refusal statement within the base response. Beyond the conventional Beginning position, the refusal phrase is repositioned into the Middle of the rationale (a non-initial, non-final sentence) or at the End of the explanation, maintaining discourse coherence with minimal grammatical smoothing. Because autoregressive transformers assign disproportionate causal attention to leading prefix tokens, front-loaded boilerplate statements coerce early generation logits into rigid refusal basins, thereby amplifying sensitivity to isolated surface-level triggers. Relocating or removing the initial statement assesses whether delaying or omitting the refusal token sequence restores the model's capacity to process nuanced contextual semantics.
3. Rationale Semantic Specificity: Request-Specific vs. Generic Justifications
Eliminating boilerplate apologies alone does not guarantee fine-grained decision boundaries. This design probes the internal semantic granularity of the rationale by contrasting Request-Specific explanations with Generic counterparts. In the Request-Specific configuration, the rationale explicitly references the user's targeted action and unpacks why that specific activity is hazardous (e.g., explicitly citing "making counterfeit money"). Conversely, the Generic condition replaces all direct mentions of the targeted action with generalized abstract phrases (e.g., "engaging in this kind of behavior", "this request"). This contrast provides a vital supervision signal: explicit references anchor the refusal boundary to the functional mechanics of dangerous actions, enabling the model to discern benign requests that share vocabulary but diverge completely in functional intent.
4. Causal Token Attribution and Structural Template Probing: Unveiling Shortcut Anchoring
To directly verify that boilerplate statements foster superficial cue reliance in the model's internal representations, the framework deploys Shannon entropy analysis on the first generated token: $\(H_1 = -\sum_{v=1}^V p(v)\log p(v)\)$ Lower entropy indicates deterministic refusal collapse. To quantify token-level contributions, causal "what-if" ablations are conducted at the embedding layer \(E \in \mathbb{R}^{S \times d}\): individual input tokens are replaced with a baseline whitespace vector \(\tilde{e}\), and signed entropy shifts \(\Delta H_i = H(\tilde{p}_i) - H(p_i)\) are recorded across bootstrap iterations. Furthermore, two orthogonal diagnostic experiments are constructed: a neutral prefix control (prepending "Thank you for asking!" to Rationale-Only data) and a keyword-conditioned experiment (attaching refusal statements exclusively to training queries featuring 5 high-frequency risky tokens: drug, kill, money, steal, shoot). These setups demonstrate that structural regularity itself—even when semantically benign—induces superficial token anchoring and precipitates false refusals.
Loss & Training¶
- Fine-Tuning Architecture & Hyperparameters: Base language models are fine-tuned via 4-bit NormalFloat QLoRA across all linear projection layers (\(r=64\), \(\alpha=16\), dropout \(=0.1\)). Optimization uses paged AdamW in 32-bit precision with a constant learning rate of \(1 \times 10^{-4}\), batch size 64, sequence length 2,048, and 10 training epochs on NVIDIA A6000/A100 GPUs.
- Dataset Mixture: 1,024 clean instruction-response pairs from Stanford Alpaca paired with 256 carefully curated safety instances to maintain baseline conversational utility.
- Evaluation Protocol: Models are evaluated on harmful query sets (AdvBench, MaliciousInstruct) and pseudo-harmful benchmarks (XSTest-Safe, OKTest). The automated judge (Llama-3.3-70B-Instruct, validated by GPT-5.1 and human annotators with Cohen's \(\kappa=0.81\)) employs a strict evaluation rubric to penalize pseudo-compliance (responses that appear polite but evade answering or append extraneous moralizing caveats).
Key Experimental Results¶
Main Results¶
The primary benchmark compares compliance rates (CR, where lower is better on harmful queries \(\downarrow\), and higher is better on pseudo-harmful queries \(\uparrow\)), alongside overall Precision, Recall, and F1 scores (treating compliant answers to pseudo-harmful prompts as true positives, and refusals of harmful prompts as true negatives).
| Model | Experimental Condition / Variant | AdvBench CR ↓ | Malicious CR ↓ | XSTest-Safe CR ↑ | OKTest CR ↑ | Overall Recall | Overall Precision | Overall F1 |
|---|---|---|---|---|---|---|---|---|
| Llama-3.1-8B | Beginning (Statement & Rationale) | 0.02 | 0.02 | 0.39 | 0.41 | 0.40 | 0.94 | 0.56 |
| Middle Position | 0.03 | 0.03 | 0.58 | 0.52 | 0.55 | 0.94 | 0.69 | |
| End Position | 0.02 | 0.04 | 0.60 | 0.54 | 0.56 | 0.95 | 0.71 | |
| Statement-Only | 0.03 | 0.02 | 0.48 | 0.44 | 0.46 | 0.93 | 0.62 | |
| Rationale-Only | 0.02 | 0.06 | 0.71 | 0.60 | 0.65 | 0.95 | 0.77 | |
| Request-Specific (Rationale) | 0.04 | 0.05 | 0.80 | 0.72 | 0.75 | 0.94 | 0.84 | |
| Generic (Rationale) | 0.03 | 0.02 | 0.66 | 0.60 | 0.63 | 0.95 | 0.75 | |
| Mistral-7B-v0.3 | Beginning (Statement & Rationale) | 0.01 | 0.00 | 0.49 | 0.55 | 0.52 | 0.98 | 0.68 |
| Middle Position | 0.03 | 0.02 | 0.57 | 0.58 | 0.57 | 0.95 | 0.72 | |
| End Position | 0.03 | 0.02 | 0.60 | 0.60 | 0.60 | 0.94 | 0.73 | |
| Statement-Only | 0.05 | 0.01 | 0.56 | 0.53 | 0.55 | 0.91 | 0.69 | |
| Rationale-Only | 0.04 | 0.06 | 0.75 | 0.77 | 0.76 | 0.94 | 0.84 | |
| Request-Specific (Rationale) | 0.06 | 0.08 | 0.83 | 0.85 | 0.84 | 0.92 | 0.88 | |
| Generic (Rationale) | 0.04 | 0.03 | 0.69 | 0.79 | 0.75 | 0.95 | 0.84 |
Ablation & Structural Variation Analysis¶
Analysis of structural formats, stylistic diversity, and keyword anchoring on Llama-3.1-8B:
| Configuration / Intervention Variant | AdvBench CR ↓ | Malicious CR ↓ | XSTest-Safe CR ↑ | OKTest CR ↑ | Overall F1 | Underlying Mechanism |
|---|---|---|---|---|---|---|
| Rationale-Only Baseline | 0.02 | 0.06 | 0.71 | 0.60 | 0.77 | Discards boilerplate phrases; smooth semantic distribution |
| Statement and Rationale | 0.02 | 0.02 | 0.39 | 0.41 | 0.56 | Default formulaic prefix; triggers severe false refusal |
| Prefix and Rationale (Neutral Template) | 0.02 | 0.04 | 0.44 | 0.46 | 0.61 | Prepending "Thank you for asking!" still elevates false refusal |
| Varied Statement and Rationale | 0.02 | 0.02 | 0.55 | 0.55 | 0.69 | 15 distinct refusal phrasings improve over fixed templates |
| 1-Sentence Rationale-Only (Length Control) | 0.02 | 0.05 | 0.74 | 0.65 | 0.80 | Short response control (97 chars avg) confirms length does not explain gains |
Key Findings¶
- Boilerplate Refusal Statements Drive Over-Refusal: Across all model families and scales, incorporating refusal statements (either alone or paired with rationales) causes compliance on pseudo-harmful queries to plummet to 39%–49%. In sharp contrast, fine-tuning under Rationale-Only boosts benign compliance to 71%–77% with virtually identical safety protection against truly malicious queries (AdvBench compliance remains negligible at 0.02–0.04).
- Format Regularity Induces Shortcut Learning: Simply prepending a semantically neutral prefix ("Thank you for asking!") to Rationale-Only responses lowers pseudo-harmful compliance to 45%, demonstrating that structural template regularity itself trains the model to associate fixed prefixes with refusal trajectories.
- Attribution Confirms Lexical Anchoring: Token-level causal ablations show that in models trained with Statement and Rationale, 60.8% of refusal decisions are driven by isolated risky keywords (e.g., "kill", "shoot"). For models trained on Rationale-Only, 98.7% of attributions distribute over meaningful, context-bearing semantic spans.
- General Capabilities and Adversarial Safety Are Fully Preserved: Extensive evaluations across MMLU, ARC, GSM8K, and HellaSwag confirm that Rationale-Only alignment incurs zero capability degradation (Table 3), while evaluations on HarmBench, JailbreakBench, and WILDJAILBREAK show comparable or superior adversarial robustness (Table 4).
Highlights & Insights¶
- Data-Centric Root Cause Diagnosis: Rather than stacking complex steering vectors or decoding constraints during inference, the authors diagnose the structural defect directly at the training data level, demonstrating that formulaic apologies and boilerplate refusals are the primary culprit behind model over-sensitivity.
- The "Neutral Prefix" Aha-Moment: Proving that a completely harmless opening phrase like "Thank you for asking!" can recreate false refusal behavior provides an eye-opening empirical revelation: autoregressive models overfit to surface-level structural predictability, linking rigid templates to narrow refusal behaviors.
- Broad Compatibility across Alignment Paradigms: The benefits of Rationale-Only transfer directly to zero-shot in-context learning (ICL via URIAL) and synergize seamlessly with existing test-time mitigations like SelfCD and SCANS, achieving record-high F1 scores (0.86–0.88).
Limitations & Future Work¶
- Model Scale Boundary: Due to academic computational constraints, the primary fine-tuning ablations were conducted on models up to 9B parameters (with scaling checks on 70B/72B). Investigating the effect of rationale decomposition within massive proprietary models trained with multi-stage RLHF pipelines remains an open challenge.
- User Experience Trade-Off: While Rationale-Only responses successfully refuse harmful prompts without false rejections, omitting conventional polite expressions ("I'm sorry, but...") in favor of direct ethical and legal explanations may sound somewhat didactic or abrupt to certain human end-users.
- Future Directions: Exploring how to integrate rationale-only conditioning into hidden reasoning chains (e.g., internal thinking traces) to allow the model to reason about hazards covertly while generating empathetic, naturally calibrated outward responses.
Related Work & Insights¶
- vs. Activation Steering & Representation Surgery (e.g., SCANS, Surgical): These methods manipulate activation vectors or ablate isolated refusal directions at inference time. While effective, they introduce compute overhead and risk altering factual knowledge. In contrast, Rationale-Only achieves superior boundary calibration purely through data curation without touching model architecture or decoding pipelines.
- vs. Structured Reasoning Alignment (e.g., FalseReject, Reason-to-Reject): While recent works advocate for generating lengthy intermediate CoT justifications, this paper demonstrates that simply removing the boilerplate refusal statement and making the rationale request-specific yields dramatic reductions in over-refusal without requiring heavy reasoning overhead.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant structural decomposition that pinpoints the root cause of false refusals at the dataset design level.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous coverage across 4 base model families, causal embedding ablations, first-token entropy metrics, length controls, and jailbreak evaluations.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, disciplined experimental methodology, and well-structured ablations.
- Value: ⭐⭐⭐⭐⭐ Highly actionable, cost-effective design guideline for open-source and commercial safety alignment teams.