Skip to content

TRACE: An Evidence-Grounded Benchmark for Safety Evaluation of Large Reasoning Models

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Safety
Keywords: Large Reasoning Model Safety, Guardrail Benchmark, Evidence Attribution, Reasoning Trace Safety, Jailbreak Robustness

TL;DR

Presents TRACE, the first safety evaluation benchmark covering the complete LRM inference pipeline (prompts, reasoning traces, and final responses) with continuous substring evidence grounding; benchmarking 18 guardrail models reveals that reasoning trace safety judgment is significantly harder than prompts/responses and that existing guardrails struggle severely with evidence attribution (TokenF1 below 15%).

Background & Motivation

Large Reasoning Models (LRMs) such as o1, DeepSeek-R1, Qwen, and Kimi have achieved state-of-the-art performance across demanding planning, mathematical reasoning, and code synthesis benchmarks. Their primary architectural catalyst is the generation of extensive intermediate reasoning traces (chains of thought) that allow step-by-step problem decomposition, multi-hypothesis exploration, and deliberate self-correction prior to delivering a final output. However, making these internal cognitive trajectories explicit introduces a critical and unprecedented systemic vulnerability: under adversarial prompts or jailbreak attempts, an LRM frequently elaborates actionable, forbidden operational details and circumvention strategies inside its reasoning trace, even when its final user-facing response appropriately issues a polite, safety-aligned refusal. This dangerous divergence between hazardous intermediate reasoning and seemingly benign final answers exposes a severe structural blind spot in contemporary AI safety architectures.

Existing content moderation and safety evaluation benchmarks (such as XSTest, ToxicChat, Aegis, WGTest, BeaverTails, and SafeRLHF) suffer from two foundational limitations that render them inadequate for LRMs. First, their scope is strictly confined to user prompts and model-generated final responses, completely ignoring the volatile intermediate reasoning traces where harmful logic and procedural hazards actually incubate. Second, prior benchmarks deliver only coarse binary safety labels (safe vs. unsafe) without providing verifiable textual evidence justifying those classifications. Lacking fine-grained evidence attribution prevents researchers from determining whether a guardrail model genuinely comprehends the underlying hazard or merely latches onto superficial lexical patterns, and makes it impossible to evaluate a guardrail's ability to localize unsafe fragments within lengthy, complex inference pipelines.

Constructing a full-pipeline, evidence-grounded safety benchmark requires addressing multiple interconnected complexities: curating an expansive taxonomy of safety risks and complex attack mechanisms, capturing diverse behavioral regimes spanning rigorous alignment to unconstrained generation, and enforcing objective, noise-free evidence extraction across thousands of tokens. To resolve these challenges, this work introduces a rigorous three-stage pipeline combining stratified prompt curation, unaligned and aligned LRM inference, multi-annotator cross-verification, and strict continuous substring matching. Core idea: Establish TRACE, the first safety evaluation benchmark spanning the entire LRM inference pipelineโ€”user prompts, reasoning traces, and final responsesโ€”with continuous substring evidence grounding, decoupling judgment accuracy from evidence attribution to uncover critical moderation blind spots in large reasoning models.

Method

Overall Architecture

The TRACE construction and evaluation framework follows a three-stage workflow: (1) Prompt Curation: Curating 1,993 English and Chinese prompts across 9 risk categories and 10 adversarial attack strategies via stratified sampling from S-Eval and WildChat; (2) LRM Inference: Generating reasoning traces and final responses using four diverse LRMsโ€”two safety-aligned models (Qwen3-8B and Gemma-4-E4B) and their two abliterated counterparts (Qwen3-8B-abliterated and Gemma-4-E4B-abliterated)โ€”producing 5,000 valid \((q, r_i, a_i)\) behavioral triples; and (3) Multi-Dimensional Safety Annotation & Verification: Deploying three frontier LRMs (DeepSeek-V3.2, Qwen3.5-Plus, and KIMI-K2.5) to independently annotate safety labels and extract supporting textual evidence, finalized via majority voting, strict continuous substring verification, and expert human adjudication.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Diverse Prompt Curation<br/>S-Eval / WildChat (1,993 Prompts)"] --> B["Full-Spectrum Triple Generation<br/>Safety-Aligned & Abliterated LRMs"]
    B --> C["Multi-Dimensional Annotation & Verification<br/>3 LRM Judges Majority Vote + Human Verification"]
    C --> D["Dual-Metric Guardrail Evaluation Protocol<br/>Judgment Correctness (F1/FPR/FNR) + Evidence Attribution (TokenF1)"]

Key Designs

1. Full-Spectrum Triple Generation: Capturing Diverse Behaviors from Safe Refusal to Unaligned Generation

Standard safety benchmarks rely exclusively on models tuned with conventional alignment, which predominantly yield uniform refusal responses that fail to reflect diverse real-world safety breakdowns. In contrast, under adversarial manipulation or targeted fine-tuning, safety boundaries can degrade. To mirror the complete spectrum of reasoning behaviors, TRACE employs four distinct models per prompt: two safety-aligned models (Qwen3-8B, Gemma-4-E4B) and their abliterated counterparts (Qwen3-8B-abliterated, Gemma-4-E4B-abliterated). Abliteration systematically neutralizes the internal refusal representations of safety-aligned models, allowing them to follow harmful instructions while preserving their core reasoning and instruction-following faculties. This design yields 5,000 valid \((q, r, a)\) triples that capture subtle shifts across inference stages, including prompts with unsafe traces followed by safe refusals, as well as cases of full compliance across both stages.

2. Multi-Dimensional Annotation & Verification: Ensuring High Consensus and Hallucination-Free Evidence Grounding

Evaluating lengthy reasoning traces spanning thousands of tokens introduces substantial risks of annotator fatigue and model hallucination if a single judge is used. TRACE resolves this by employing a multi-model cross-annotation protocol driven by three frontier LRMs: DeepSeek-V3.2, Qwen3.5-Plus, and KIMI-K2.5. Each annotator independently assigns binary safety labels \(y_c^{(j)} \in \{\text{safe}, \text{unsafe}\}\) to each component \(c \in \{q, r_i, a_i\}\) and extracts verbatim evidence \(e_c^{(j)}\). The final safety label is determined via majority vote: \(y_c = \text{MajorityVote}(\{y_c^{(j)}\}_{j=1}^3)\). To eliminate hallucinated or non-verifiable rationales, extracted evidence is retained only if provided by a majority-aligned judge and verified as an exact continuous substring of the original source text (\(e_c^{(j)} \sqsubseteq c\)). Any sample failing continuous substring verification (\(e_c = \emptyset\)) is escalated to human safety experts for gold-standard manual annotation, securing an inter-annotator Cohen's Kappa of 0.84.

3. Dual-Metric Guardrail Evaluation Protocol: Decoupling Macro Judgment Correctness from Micro Evidence Localization

Existing benchmarks treat safety moderation as a monolithic classification task, obscuring whether models make correct decisions based on genuine hazard understanding or surface heuristics. TRACE establishes a decoupled evaluation protocol assessing both Safety Judgment Correctness and Evidence Attribution Accuracy. In the correctness dimension, TRACE measures False Positive Rate (FPR, quantifying excessive refusal and usability damage), False Negative Rate (FNR, quantifying dangerous safety leakage), and overall F1-score across all three pipeline components. In the attribution dimension, for guardrail models that produce explanatory outputs (e.g., YuFeng-XGuard, GPTSafeGuard, Octopus), the predicted evidence \(\hat{e}_{c\ell}\) and ground-truth substring evidence \(e_{c\ell}\) are tokenized into token bags \(T(\cdot)\) to compute token-level overlap:

\[\text{TokenF1} = \frac{1}{|D|}\sum_{\ell=1}^{|D|} \frac{2 \cdot |T(\hat{e}_{c\ell}) \cap T(e_{c\ell})|}{|T(\hat{e}_{c\ell})| + |T(e_{c\ell})|}\]

This metric provides an exact quantitative diagnosis of how accurately guardrails pinpoint the concrete phrases that trigger moderation violations.

Benchmark & Evaluation Protocol

TRACE incorporates 1,993 distinct prompts spanning 9 standardized safety risk categories (Crimes and Illegal Activities, Hate Speech, Physical and Mental Health, Ethics and Morality, Data Privacy, Cybersecurity, Extremism, Risks Involving Minors, and Inappropriate Suggestions) and 10 adversarial attack strategies (Positive Induction, Reverse Induction, Code Injection, Instruction Jailbreak, Goal Hijacking, Instruction Encryption, DeepInception, In-Context Attack, Chain of Utterances, and Compositional Instruction Attack), with 57% English and 43% Chinese prompts. A total of 18 representative guardrail models across different model families and parameter scales are evaluated: LlamaGuard (1, 2, 3-1B, 3-8B, 4-12B), ShieldGemma (2B, 9B), WildGuard (7B), NemotronGuard (8B), Octopus (14B), GPTSafeGuard (20B), Qwen3Guard (0.6B, 4B, 8B), PolyGuard (0.5B, 8B), and YuFeng-XGuard (0.6B, 8B).

Key Experimental Results

Main Results

Binary safety judgment performance (F1-score, %) across the entire LRM inference pipeline on TRACE:

Guardrail Model Params Prompt (EN / ZH / All) Reasoning Trace (EN / ZH / All) Final Response (EN / ZH / All)
LlamaGuard-1 7B 39.64 / 40.83 / 39.91 22.69 / 29.14 / 24.34 34.18 / 36.19 / 34.69
LlamaGuard-2 8B 69.34 / 74.52 / 70.45 61.36 / 71.76 / 63.67 61.08 / 71.24 / 63.40
LlamaGuard-3 1B 47.10 / 53.03 / 48.37 51.37 / 58.15 / 52.85 48.96 / 58.13 / 51.01
LlamaGuard-3 8B 73.09 / 73.30 / 73.14 62.28 / 65.36 / 63.08 64.66 / 69.52 / 65.95
LlamaGuard-4 12B 69.13 / 57.40 / 66.52 50.48 / 51.48 / 50.73 63.98 / 59.89 / 62.92
ShieldGemma 2B 49.03 / 50.60 / 49.35 52.47 / 59.44 / 53.92 49.10 / 55.32 / 50.38
ShieldGemma 9B 59.43 / 64.49 / 60.47 56.13 / 69.54 / 59.02 55.06 / 68.16 / 57.88
WildGuard 7B 86.29 / 86.67 / 86.37 58.95 / 61.57 / 59.63 68.09 / 74.71 / 69.86
NemotronGuard 8B 84.72 / 90.12 / 85.99 76.58 / 81.02 / 77.74 79.12 / 85.44 / 80.78
Octopus 14B 80.93 / 87.75 / 82.59 77.94 / 86.37 / 80.10 80.91 / 87.07 / 82.46
GPTSafeGuard 20B 86.05 / 89.90 / 86.96 77.95 / 84.06 / 79.60 79.76 / 85.84 / 81.39
Qwen3Guard 0.6B 84.23 / 89.68 / 85.47 70.39 / 75.72 / 71.79 79.41 / 87.41 / 81.52
Qwen3Guard 4B 87.54 / 90.91 / 88.31 71.11 / 74.74 / 72.04 81.08 / 88.31 / 83.05
Qwen3Guard 8B 87.30 / 92.32 / 88.43 73.77 / 81.13 / 75.73 82.05 / 87.99 / 83.64
PolyGuard 0.5B 82.65 / 86.63 / 83.53 65.65 / 66.89 / 65.91 67.83 / 71.58 / 68.68
PolyGuard 8B 90.59 / 93.16 / 91.17 80.95 / 87.38 / 82.57 82.83 / 88.12 / 84.15
YuFeng-XGuard 0.6B 87.66 / 90.15 / 88.24 73.79 / 83.98 / 76.55 79.34 / 87.83 / 81.65
YuFeng-XGuard 8B 87.49 / 91.03 / 88.27 82.82 / 88.46 / 84.26 85.21 / 88.72 / 86.11

Auxiliary & Diagnostic Experiments

Evidence attribution accuracy (TokenF1, %) and prompt risk category classification F1-score (%):

Guardrail Model Params Prompt TokenF1 Reasoning Trace TokenF1 Final Response TokenF1 Risk Category F1 (All)
Octopus 14B 9.53 14.56 13.25 โ€”
GPTSafeGuard 20B 10.74 12.58 9.92 62.25 (EN 60.47 / ZH 70.20)
YuFeng-XGuard 0.6B 10.53 13.08 12.03 52.62 (EN 50.30 / ZH 62.38)
YuFeng-XGuard 8B 11.68 14.88 13.71 53.24 (EN 51.38 / ZH 62.24)
NemotronGuard 8B โ€” โ€” โ€” 27.13 (EN 25.37 / ZH 32.29)
PolyGuard 0.5B โ€” โ€” โ€” 7.68 (EN 8.08 / ZH 6.09)
PolyGuard 8B โ€” โ€” โ€” 16.63 (EN 16.28 / ZH 17.88)

Key Findings

  • Reasoning traces constitute the primary bottleneck in safety moderation: Across all 18 evaluated models, the average F1-scores drop steadily from 75.75% on Prompts to 70.53% on Final Responses, and bottom out at 66.31% on Reasoning Traces. For 14 out of 18 models, reasoning trace moderation is strictly the hardest task. For example, YuFeng-XGuard-8B achieves 88.27% on prompts but drops to 84.26% on reasoning traces; early guardrails such as LlamaGuard-1-7B collapse completely on traces, dropping from 39.91% to 24.34%.
  • Current guardrail models exhibit near-total failure in evidence attribution: Even among models explicitly trained to output natural language rationales, the extracted evidence achieves TokenF1 scores below 15% (YuFeng-XGuard-8B tops out at 14.88% on traces and 11.68% on prompts). This demonstrates that while modern guardrails can produce correct binary judgments macroscopically, they struggle to ground their decisions in the precise textual spans responsible for policy violations.
  • Extreme polarization between over-refusal and safety leakage: Evaluated models display sharp imbalances between FPR and FNR. ShieldGemma-9B and LlamaGuard-2 suffer from crippling over-refusal, posting FPRs of 99.40% and 89.57% on prompts and near-100% on reasoning traces. In contrast, models like LlamaGuard-1, WildGuard, NemotronGuard, and GPTSafeGuard display dangerously high FNRs (WildGuard-7B has a 55.78% FNR on reasoning traces), leaking substantial hazardous content. Only YuFeng-XGuard-8B maintains a balanced operational profile (FNR 13.65%, FPR 15.77% on traces).
  • Instruction Encryption (IE) renders guardrails virtually powerless: Adversarial encoding schemes such as Base64 and Caesar ciphers severely degrade guardrail effectiveness across all stages. On the TRACE-(IE) subset, Qwen3Guard-0.6B and 4B score 0.00% F1 on reasoning traces, and even top-tier models like PolyGuard-8B and YuFeng-XGuard-8B suffer catastrophic drops, scoring only 15.25% and 38.85% F1 on traces, respectively.

Highlights & Insights

  • Pioneering safety evaluation inside the internal cognitive trace: TRACE is the first benchmark to explicitly demonstrate that reasoning models can decouple intermediate procedural planning from final refusals, cementing intermediate trace monitoring as an indispensable requirement for deploying LRMs.
  • Strict continuous substring grounding to prevent judge hallucinations: By requiring extracted evidence to be verifiable continuous substrings agreed upon by majority-aligned judges (backed by human expert review), TRACE establishes an objective, hallucination-free evaluation ground truth for interpretable guardrails.
  • Architectural advantages of Qwen-derived guardrails: Guardrails fine-tuned from the Qwen family (YuFeng-XGuard, PolyGuard, Qwen3Guard) systematically outperform counterparts built on Llama, Gemma, or Mistral backbones of equivalent scale across all stages, underscoring the benefits of strong native bilingual understanding and robust general reasoning capabilities in safety moderation.

Limitations & Future Work

  • Linguistic scope limited to bilingual settings: The benchmark currently evaluates only English and Chinese prompts, leaving low-resource languages and diverse multilingual adversarial surfaces unexamined.
  • Human annotation coverage can be scaled: While the 1,000-instance human re-annotation confirmed substantial inter-annotator agreement (Cohen's Kappa = 0.84), expanding expert human review to the entire benchmark would further eliminate edge-case noise in long-form traces.
  • Static vs. dynamic multi-turn interactions: TRACE evaluates single-turn reasoning pipelines; extending the benchmark to dynamic, multi-turn iterative jailbreaks and adaptive agentic reasoning loops represents a vital direction for future research.
  • vs. XSTest / ToxicChat / BeaverTails / SafeRLHF: Traditional safety benchmarks treat LLM generation as a monolithic input-output black box; TRACE opens this black box to audit intermediate thinking processes as first-class citizens.
  • vs. Aegis / S-Eval: Prior benchmarks evaluate only categorical or binary labels without verifiable grounding; TRACE introduces token-level continuous substring evidence evaluation, establishing the TokenF1 metric for explainable guardrails.
  • vs. Conventional Guardrails (LlamaGuard / WildGuard): Demonstrates that guardrail architectures trained purely on conversational data fail to generalize to the syntax, length, and procedural density of LRM reasoning traces, signaling the need for dedicated trace-aware guardrail architectures.

Rating

  • Novelty: โญโญโญโญโญ First benchmark to systematically target LRM intermediate reasoning trace safety with continuous substring evidence attribution.
  • Experimental Thoroughness: โญโญโญโญโญ Evaluates 18 guardrail models across 9 risk categories, 10 attack strategies, dual languages, and fine-grained error metrics (FPR/FNR/TokenF1).
  • Writing Quality: โญโญโญโญโญ Exceptionally clear narrative, rigorous formal definitions, insightful failure analyses, and high-quality visualizations.
  • Value: โญโญโญโญโญ Provides crucial diagnostic methodologies and benchmark data for securing the next generation of reasoning-oriented foundation models.