Skip to content

gspr aligning llm safeguards as generalizable safety policy reasoners

content_hash: 053c1826896257ad ---# GSPR: Aligning LLM Safeguards as Generalizable Safety Policy Reasoners

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/HKUST-KnowComp/GSPR
Area: LLM Safety
Keywords: LLM Guardrails, Policy Reasoning, Reinforcement Learning, Fine-grained Risk Categorization, Generalization Ability

TL;DR

Addressing the rigid dependency of existing safeguards on fixed policies and their inability to generalize to unseen risk taxonomies, GSPR treats any safety taxonomy as dynamic variables in the prompt, incorporates cold-start distillation with multi-dimensional rule-based rewards under GRPO, and achieves generalizable fine-grained reasoning with ultra-low token costs.

Background & Motivation

As large language models (LLMs) become deeply embedded across critical real-world applications, mitigating safety vulnerabilities in user-model interactions has emerged as a top operational priority. Malicious queries from users, such as jailbreak prompts, adversarial red-teaming, and indirect prompt injections, can hijack model behaviors to generate restricted or hazardous materials. Furthermore, in complex scenarios like retrieval-augmented generation (RAG), even when user queries are entirely benign, retrieved external documents can corrupt model generations with toxic or harmful outputs. In response, alongside internal mechanisms like pre-training corpus filtering and post-training alignment, deploying external safeguard models (Guardrails, such as Llama Guard, ShieldGemma, and WildGuard) to moderate input prompts and output responses has become an industry standard for protecting safety guardrails without compromising foundational model utility.

However, existing guardrails exhibit fundamental structural limitations when facing open and evolving real-world safety policies. Distinct safety benchmarks (e.g., Aegis, SafeRLHF, BeaverTails) each introduce unique taxonomies and specialized guidelines. In contrast, existing open-source safeguards and even recent RL-aligned models like RSafe hardcode fixed, narrow safety policies (e.g., 14 or 18 fixed categories) into their instruction prompts. Consequently, current guardrails either collapse into coarse-grained binary classifiers distinguishing only between safe and unsafe, or fail catastrophically when encountering new or domain-specific guidelines (such as academic integrity policies in education or custom HR regulations in corporate enterprises). When evaluated on unseen taxonomies, they can neither comprehend the novel rules nor attribute violations to specific policies with faithful explanations.

To resolve these bottlenecks, this paper reframes safety moderation from a static pattern classification task into dynamic, policy-constrained logical reasoning. Core Idea: Treat any arbitrary safety taxonomy as dynamic prompt variables, leverage lightweight cold-start SFT alongside multi-faceted rule-based rewards (format compliance, binary safety, and fine-grained category matching), and align the guardrail via GRPO reinforcement learning to incentivize generalizable, compact policy reasoning over arbitrary unseen rules.

Method

Overall Architecture

Given moderation content \(x\) (an input prompt \(p\) or a prompt-response pair \((p, T(p))\)) and any arbitrary safety taxonomy \(S = \{c_i\}_{i=1}^{|S|}\), GSPR aims to determine its binary safety state \(y \in \{\text{safe}, \text{unsafe}\}\), predict the violated fine-grained policy category \(c \in S \cup \{\text{not applicable}\}\), and produce a concise Chain-of-Thought reasoning trajectory explaining the judgment.

The overall training pipeline encompasses three key stages: dynamic policy prompt formatting across multi-source safety benchmarks, high-quality cold-start supervised fine-tuning (Cold-start SFT), and multi-reward reinforcement learning alignment via Group Relative Policy Optimization (GRPO). The end-to-end pipeline is illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Sample & Taxonomy<br/>Moderation content x and policy set S"] --> B["Dynamic Policy Prompt Formatting<br/>Construct prompt q with policy variables & output format"]
    B --> C["Cold-start Supervised Fine-tuning<br/>Distill high-quality per-policy reasoning traces"]
    C --> D["Group Sampling & Long CoT Generation<br/>Sample G reasoning trajectories & structured labels"]
    D --> E["Multi-dimensional Rule-based Rewards<br/>Compute format penalty, safety reward & category reward"]
    E --> F["GRPO Group Relative Policy Optimization<br/>Group advantage normalization & KL regularization update"]
    F --> G["Final Output<br/>Reasoning trace + safety verdict + fine-grained category"]

Key Designs

1. Dynamic Policy Prompt Formatting: Eliminating Hardcoded Dependency on Fixed Taxonomies
To overcome the catastrophic generalization failure of prior safeguards on unseen safety policies, GSPR introduces a flexible prompt formatting function \(q = f(x, S)\). The template comprises four key components: an explicit moderation instruction, a dynamic list of safety policies extracted from the target taxonomy \(S\), the moderation content \(x\) (demarcating user prompt vs. agent response), and a strict box-based output specification (requiring the thinking process within <think> tags, safety verdict inside \safety{}, and categorical attribution inside \category{}). During training, instead of memorizing a fixed set of 14-18 rules, GSPR incorporates 19 distinct safety taxonomies spanning 167 fine-grained policies from diverse datasets (including Aegis, SafeRLHF, BeaverTails, WildGuard, OR-Bench, and GUARDSET-X's HR and education domains). By sub-sampling policy subsets with a sampling ratio \(s\) while retaining the ground-truth category and occasionally inserting an others category, GSPR forces the guardrail to perform genuine logical reasoning over dynamically provided policy definitions.

2. Cold-start Supervised Fine-tuning: Establishing Structured Policy Reasoner Priors
Directly applying reinforcement learning exploration on open-weight base models (such as Qwen2.5-7B or Qwen3-8B) over long policy lists and strict dual-label format constraints easily induces formatting collapse or chaotic reasoning. To address this, GSPR implements a cold-start strategy: leveraging Gemini-2.5-Flash across all 19 training taxonomies (sampling 80 instances per taxonomy) to generate detailed, per-policy verification traces conditioned on ground-truth safety and category labels. Following strict regex-based parsing and format filtering, 1,383 high-quality cold-start instances are retained for lightweight SFT (2 epochs). This step equips the base model with structured policy reasoning priors—checking criteria, weighing contextual evidence, and formulating precise verdicts—establishing a solid starting point for subsequent RL exploration.

3. Multi-dimensional Rule-based Rewards and GRPO Alignment: Balancing Reasoning Depth, Verdict Accuracy, and Token Efficiency
During RL alignment, GSPR adopts Group Relative Policy Optimization (GRPO), eliminating the need for an expensive and potentially biased critic network. For each prompt \(q\), the old policy \(\mathcal{G}_{\theta_{\text{old}}}\) samples a group of \(G=5\) candidate rollouts \(\{o_1, \dots, o_G\}\). Utilizing ground-truth annotations from the safety benchmarks, GSPR computes deterministic rule-based composite rewards: $\(R_i = R_{fi} \times (\alpha_1 R_{si} + \alpha_2 R_{ci})\)$ where the safety reward is \(R_{si} = \mathbb{I}(\hat{y}_i = y)\) and the category reward is \(R_{ci} = \mathbb{I}(\hat{c}_i = c)\), with weighting hyperparameters \(\alpha_1 = 0.55, \alpha_2 = 0.45\). Crucially, the format reward \(R_{fi}\) enforces structural tag ordering, single-language consistency, and 5-gram repetition penalties. Furthermore, recognizing that outcome-driven RL often induces verbose, low-quality babbling after the </think> tag, GSPR introduces a strict length-ratio penalty: $\(R_{fi} = \begin{cases} 1.0, & \text{correct format and } |o_i| < 2.5 |\hat{r}_i| \\ 0.5, & \text{correct format but post-tag bloat } |o_i| \ge 2.5 |\hat{r}_i| \\ 0.0, & \text{incorrect format} \end{cases}\)$ This penalty discourages superficial token padding around final answers and compels the model to concentrate its reasoning budget strictly inside the <think> reasoning trajectory, substantially cutting inference costs.

Loss & Training

GSPR optimizes the policy model \(\mathcal{G}_\theta\) using the standard GRPO clipped surrogate objective: $\(\mathcal{J}_{\text{GRPO}}(\theta) = \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \mathbb{E} \left[ \min\left(r_{i,t} \hat{A}_{i,t}, \text{clip}(r_{i,t}, 1-\epsilon, 1+\epsilon) \hat{A}_{i,t}\right) \right] - \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})\)$ where the probability ratio is \(r_{i,t} = \frac{\pi_\theta(o_{i,t} | q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} | q, o_{i,<t})}\), and the advantage is computed solely from normalized group-level relative rewards: $\(\hat{A}_{i,t} = R_i - \frac{1}{G}\sum_{j=1}^G R_j\)$ Training is executed on a single compute node featuring 8 NVIDIA H800 (80GB) GPUs using the VERL distributed framework. The global batch size is 128 (mini_batch_size=64, per-GPU batch_size=8), optimized with Adam at a constant learning rate of \(1\times 10^{-7}\) for 1 epoch. Rollout sampling uses temperature 0.7, top-p 0.8, and repetition_penalty 1.2.

Key Experimental Results

Main Results

GSPR is comprehensively benchmarked across in-domain test splits (WildGuard, Aegis, SafeRLHF, BeaverTails) and completely unseen out-of-domain taxonomies (OpenAI Moderation, HEx-PHI, T2T Synthetic Safety Bench, Do-Not-Answer). Metrics include Safety Accuracy (S-Acc), Safety F1 (S-F1), and fine-grained Category Accuracy (C-Acc).

The table below summarizes performance on the in-domain test suites (quoted directly from Table 2 of the paper):

Model Family Model Name Prompt Safety (WildGuard) S-Acc / C-Acc Prompt Safety (Aegis) S-Acc / C-Acc Response Safety (SafeRLHF) S-Acc / C-Acc Response Safety (BeaverTails) S-Acc / C-Acc Overall S-Acc Overall C-Acc
Frontier APIs o3-mini 92.54 / 80.34 58.53 / 37.88 86.87 / 78.37 61.31 / 54.05 74.81 62.66
Frontier APIs Gemini-2.5-Flash 85.61 / 72.54 63.27 / 47.28 85.05 / 74.20 58.17 / 53.19 73.02 61.80
Frontier APIs GPT-5 90.34 / 72.50 79.43 / 71.15 89.36 / 83.98 77.92 / 68.79 84.83 75.37
Guardrail Baselines ShieldGemma-9B 86.87 / 68.22 75.02 / 64.62 88.18 / 81.32 76.17 / 65.77 81.56 69.98
Guardrail Baselines LlamaGuard3-8B 83.69 / 1.00 80.71 / 1.67 80.59 / 4.23 69.55 / 0.19 78.64 1.77
Guardrail Baselines GuardReasoner-8B 83.34 / 0.00 74.09 / 0.00 88.84 / 0.00 82.65 / 0.00 82.23 0.00
Qwen2.5-7B Series RSafe (RL-aligned) 86.80 / 39.14 77.93 / 40.18 88.60 / 26.09 83.56 / 15.27 84.22 30.17
Qwen2.5-7B Series GSPR w/ Cold-start (Ours) 87.01 / 72.57 82.99 / 79.04 90.32 / 86.14 82.39 / 75.51 85.68 78.32
Qwen3-8B Series RSafe (RL-aligned) 86.87 / 33.61 76.90 / 32.10 90.59 / 9.60 81.61 / 11.54 84.00 21.71
Qwen3-8B Series GSPR w/ Cold-start (Ours) 89.10 / 72.04 82.18 / 76.74 91.46 / 86.94 82.71 / 75.84 86.36 77.89

The table below presents generalization results on out-of-domain benchmarks with completely unseen taxonomies (quoted directly from Table 3 of the paper):

Model Family Model Name OpenAI Moderation S-Acc / C-Acc HEx-PHI S-Acc / C-Acc T2T Safety Bench S-Acc / C-Acc Do-Not-Answer S-Acc / C-Acc Overall OOD S-Acc Overall OOD C-Acc
Guardrail Baselines ShieldGemma-9B 77.46 / 74.57 97.59 / 51.38 99.19 / 95.05 90.84 / 88.60 91.27 77.40
Guardrail Baselines LlamaGuard3-8B 89.79 / 0.06 95.86 / 7.24 99.09 / 16.42 82.32 / 0.11 91.77 5.96
Guardrail Baselines GuardReasoner-8B 73.97 / 0.00 78.62 / 0.00 99.49 / 0.00 92.01 / 0.00 86.02 0.00
Qwen2.5-7B Series RSafe 78.83 / 53.63 97.33 / 1.72 98.30 / 22.89 91.77 / 22.68 91.56 25.23
Qwen2.5-7B Series GSPR w/ Cold-start 83.42 / 80.80 97.24 / 58.28 99.44 / 90.05 91.26 / 89.67 92.84 79.70
Qwen3-8B Series RSafe 78.55 / 25.37 97.93 / 13.45 98.68 / 26.23 91.82 / 8.73 91.75 18.45
Qwen3-8B Series GSPR w/ Cold-start 83.33 / 79.38 97.58 / 53.45 99.24 / 96.26 92.28 / 90.31 93.11 79.85

Ablation Study

Ablations and quantitative response statistics evaluate reasoning efficiency, language stability, and output redundancy (quoted directly from Table 4 of the paper):

Configuration Base Model Avg Word # Mix % (Language Switching) Repeat % (5-gram Repetitions) Note
o3-mini Closed API 172.89 0.02 0.10 Moderate token cost
GuardReasoner-8B Llama-3.2 211.04 0.00 0.00 Verbose traces, fails category prediction
RSafe Qwen2.5-7B 187.98 0.68 1.68 High word count, low category accuracy
Cold-start SFT Qwen2.5-7B 291.41 0.36 0.15 Highly verbose per-policy enumeration
GSPR (safety only) Qwen2.5-7B 128.44 0.11 0.03 Lacks category reward, looser traces
GSPR w/o Cold-start Qwen2.5-7B 93.18 3.56 0.60 Elevated language mixing without prior
GSPR w/ Cold-start Qwen2.5-7B 34.10 0.06 0.22 >80% word reduction, optimal efficiency
RSafe Qwen3-8B 362.46 25.23 4.17 Severe language mixing on Qwen3 base
GSPR w/ Cold-start Qwen3-8B 77.73 0.04 1.00 Suppresses mixing and repetitive generation

Key Findings

  • Severe Inability of Prior Guardrails in Category Reasoning: Prior open-source models like LlamaGuard3 and GuardReasoner exhibit near-0% category accuracy when tasked with diverse taxonomies because their outputs are rigidly bound to static formats. RSafe, constrained by fixed prompt policies, achieves only 21%~30% category accuracy. GSPR surpasses RSafe by over 45% in C-Acc, reaching ~78% overall.
  • Cold-start SFT Enables Ultra-compact Reasoning Traces: Comparing GSPR w/o Cold-start with GSPR w/ Cold-start on Qwen2.5, cold-start SFT boosts in-domain C-Acc from 54.06% to 78.32% (+24.26%) while simultaneously slashing average response length from 93.18 to 34.10 words. This confirms that distilled seed traces prevent erratic RL exploration and suppress rambling.
  • Category Reward Enforces Concise Focus: While GSPR (safety only) improves over RSafe, its reasoning traces remain significantly longer (128.44 words). Adding the fine-grained category reward forces the policy to anchor its reasoning directly to the matching policy definition, eliminating extraneous text.
  • Persistent Gap in Out-of-Domain Fine-grained Attribution: While safety detection accuracy exceeds 95% on challenging out-of-domain benchmarks like HEx-PHI across all top models, category prediction accuracy peaks at only 58.28%. This highlights that fine-grained violation attribution across unseen policies remains substantially harder than binary detection.

Highlights & Insights

  • Decoupling Policies as Dynamic Prompt Variables: Rather than treating safety rules as static network parameters to be fitted during training, GSPR casts policy moderation as dynamic legal reasoning over provided guidelines, unlocking zero-shot adaptability to enterprise-specific or regional guidelines.
  • Length-Ratio Format Penalty Against RL Verbosity: The \(|o_i| \ge 2.5 |\hat{r}_i|\) penalty elegantly counteracts the common RL pathology where models inflate post-thinking tokens, yielding an exceptionally concise 34-word average explanation without degrading accuracy.
  • Transferable Distillation-to-RL Recipe: Distilling per-policy reasoning trajectories with a frontier teacher model as an initial cold-start SFT prior before executing multi-reward GRPO provides a clean, robust template for complex structured reasoning tasks such as legal compliance auditing and financial policy checking.

Limitations & Future Work

  • Author-admitted Limitations: GRPO alignment requires substantial training resources (8 NVIDIA H800 GPUs running for roughly 40 days of GPU hours). Furthermore, a >10% accuracy gap persists between binary safety detection and fine-grained category identification.
  • Input Context Overhead: Injecting extensive policy catalogs directly into prompt templates increases prefill context length (Prefill TTFT). Scaling to corporate handbooks containing hundreds of clauses would strain prompt windows and latency budgets.
  • Promising Next Steps: Integrating a dense retrieval front-end to dynamically retrieve top-\(K\) candidate policies before prompting, and designing hierarchical category rewards during RL to narrow the safety-category accuracy gap.
  • vs. Llama Guard Series / ShieldGemma: These baselines hardcode 4-14 predefined policies and collapse when exposed to novel rules. GSPR supports dynamic variable-based taxonomies trained on 167 distinct policies, achieving robust zero-shot policy transfer.
  • vs. GuardReasoner: GuardReasoner relies on complex preference optimization (HS-DPO) to produce verbose explanations but cannot output fine-grained categories. GSPR simultaneously balances binary safety, fine-grained categories, and concise justifications via GRPO.
  • vs. RSafe: RSafe only optimizes binary safety on fixed policies, suffering from severe language mixing (Mix %: 25.23%) and repetitions on newer foundations like Qwen3. GSPR introduces cold-start distillation and composite format/category rewards, ensuring cross-architecture robustness.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Transforming static guardrail classification into generalizable dynamic policy reasoning with variable prompt orchestration and targeted GRPO rewards.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across 8 in-domain and out-of-domain benchmarks with detailed response length and quality analyses.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clearly formulated motivation, solid mathematical foundations, and clean presentation.
  • Value: ⭐⭐⭐⭐⭐ Delivers an immediately applicable, lightweight, and explainable moderation solution for customizable enterprise AI safety policies.