StepGuard: Learning Step-Level Guardrails with Scalable Supervision and Safety–Utility Balancing¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://zheng977.github.io/StepGuard/
Area: LLM Reasoning
Keywords: Agent Guardrails, Step-Level Monitoring, Synthetic Data Engine, Reinforcement Learning Calibration, Safety–Utility Balance
TL;DR¶
Addressing tool execution vulnerabilities and defense bias in autonomous LLM agents, StepGuard combines a prefix-aligned synthetic data engine (StepGen) with accuracy-gap-driven on-policy reinforcement learning (Balance-GRPO) to build a proactive 4B step-level guardrail, cutting attack success rates by 77.3% with only a 2.8-point sacrifice in benign task utility.
Background & Motivation¶
Large Language Model (LLM)-based agents are rapidly transforming from conversational chatbots into autonomous decision-makers that plan complex task workflows and interact directly with external environments via tool invocation. In real-world enterprise and personal computing environments, agents read and manipulate local files, execute terminal shell commands, query sensitive databases, trigger financial transactions, and dispatch emails. However, this profound real-world operational agency exposes severe security and reliability vulnerabilities. Malicious user prompts, indirect prompt injections embedded in external web data, compromised tool APIs, and hallucinated reasoning steps can easily steer an agent toward unauthorized data exfiltration, system corruption, or privilege escalation. To safeguard agent behaviors without undergoing prohibitively expensive retraining or alignment of the large core agent model, deploying external runtime guardrails has emerged as the prevailing architectural defense.
Nevertheless, existing agent guardrail solutions face structural deficiencies and acute defense bias that undermine their practical utility. Conventional trajectory-level guards evaluate executions purely post-hoc, detecting security violations only after harmful, state-changing actions have already executed in the wild. Meanwhile, preemptive, pre-execution inspection of individual candidate tool actions remains constrained by a chronic lack of scalable, high-quality step-level supervision. Constructing authentic multi-step failure cases by hand is expensive, unscalable across diverse tools, and rarely provides context-matched safe and unsafe actions branching from the exact same execution point. Consequently, existing guard models exhibit stark defense bias: over-defensive guards reject benign uses of sensitive tools (e.g., legitimate shell calls or payments), severely degrading task completion rates, while under-defensive models fail to intercept subtle contextual injections, exposing systems to compromise.
This pervasive defense bias stems directly from an uncalibrated optimization paradigm. Standard supervised fine-tuning (SFT) and off-the-shelf on-policy reinforcement learning (such as standard GRPO) optimize uniform task objectives across the entire dataset without explicitly accounting for the empirical accuracy discrepancy between safe and unsafe classes. When an early-stage model learns a conservative bias, vanilla advantage normalization persistently rewards safe-class over-filtering or reinforces imbalanced policy updates. Core idea: construct prefix-aligned contrastive trajectory groups with designated risk anchors and benign tool reuse via an automated data engine StepGen, and couple it with Balance-GRPO, an on-policy reinforcement learning algorithm that dynamically reweights advantage estimates based on the observed safe–unsafe accuracy gap to eliminate defense bias and achieve optimal runtime safety–utility balance.
Method¶
Overall Architecture¶
StepGuard provides a compact (4B parameter) proactive guardrail model capable of operating in dual modes: online pre-execution candidate action auditing and offline complete-trajectory safety diagnosis. Given an execution context \(X\)—consisting of the initial user prompt, accessible tool specifications, multi-step execution history prefix, and a proposed candidate tool call—StepGuard generates a structured safety diagnosis \(( \hat{Y}, \hat{B}, \hat{R}, \hat{S} )\), comprising a binary safety verdict \(\hat{Y}\), risk source presence \(\hat{B}\), fine-grained risk category \(\hat{R}\) (spanning 8 agentic risk types and 1 benign category), and an optional unsafe step index \(\hat{S}\). The training pipeline proceeds through two disciplined phases: cold-start supervised fine-tuning on StepGen's synthetic prefix-aligned groups, followed by on-policy policy calibration via Balance-GRPO to resolve class-wise performance imbalances.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: User Request + Tools + History Prefix + Candidate Action"] --> B["StepGen: Risk-Anchored Planning & Prefix-Aligned Rollout"]
B --> C["StepGen: Dual-Stage Mechanical & Semantic Verification"]
C --> D["Cold-Start Supervised Fine-Tuning (SFT)"]
D --> E["Balance-GRPO: Accuracy-Gap-Driven Advantage Calibration"]
E --> F["Output: Structured Safety Verdict & Risk Localization"]
Key Designs¶
1. Risk-Anchored Planning & Prefix-Aligned Rollout: Isolating Action Risk from Context Bias Standard agent safety datasets often fail to pinpoint the exact step where an interaction turns malicious, encouraging guards to rely on superficial tool identity heuristics rather than contextual parameter provenance. StepGen circumvents this by sampling a risk condition \((r, f, h)\) consisting of a risk source, failure mode, and harm type alongside a tool subset \(\mathcal{T}\). A two-stage LLM planner \(G_{\text{plan}}\) constructs a plausible workflow and compiles it into a structured plan \(P = (P_1, \dots, P_L)\), enforcing a single, uniquely designated unsafe action anchor \(i^\star\). After rolling out an initial unsafe trajectory \(\tau^U\) within an environment simulator, StepGen freezes the preceding execution prefix \(\tau_{<i^\star}^U\) and regenerates the suffix under two counterfactual safe modes: $\(\tau^m = \tau_{<i^\star}^U \circ \text{ReRoll}(P_{\ge i^\star}^U, m), \quad m \in \{\text{Refuse}, \text{Aware}\}\)$ In the Refuse mode, the agent recognizes the risk at step \(i^\star\) and gracefully declines execution with an explanatory rationale. In the Aware mode, the agent identifies contextual tampering (such as an injected instruction in tool outputs), filters the malicious payload, or substitutes an authorized tool call to legitimately fulfill the user's initial objective. StepGen additionally synthesizes an independent benign trajectory utilizing the same tool inventory in an everyday, non-adversarial scenario, explicitly teaching the guard to decouple sensitive tool schemas from inherently malicious intent.
2. Dual-Stage Mechanical & Semantic Verification: Eliminating Hallucinations and Format Drift Because automated multi-step LLM rollouts risk hallucinated parameters and disjointed tool schemas, StepGen filters candidate trajectory groups through a rigorous two-tier quality control funnel. First, a deterministic mechanical validator \(\phi_{\text{mech}}\) verifies four invariant checks: strict JSON parseability, tool schema and parameter validity, explicit parameter lineage (ensuring parameters declared from user queries or preceding tool outputs are faithfully matched rather than fabricated), correct risk anchor execution, and verified placement of injected payloads strictly within environment return slots. Second, candidate groups passing \(\phi_{\text{mech}}\) are inspected by an LLM quality auditor \(\phi_{\text{qual}}\) across six semantic dimensions on a 0–3 scale (authorization match, scope coherence, anchor correctness, downstream propagation consistency, response authenticity, and semantic cohesion). Groups are retained only if the aggregate score meets \(\eta \ge 14\) with zero dimensional failures, resulting in a selective 31.0% end-to-end admission rate that guarantees dependable step-level supervision.
3. Balance-GRPO: Accuracy-Gap-Driven Advantage Calibration for Safety–Utility Equilibrium While cold-start SFT establishes proper output formatting and foundational risk reasoning, the resulting checkpoint displays severe over-defense, lagging substantially in safe-class accuracy relative to unsafe-class recall. Vanilla GRPO fails to remediate this disparity because intra-group advantage normalization does not account for global cross-class performance skew. Balance-GRPO maintains the standard group reward structure (\(r_{i,j} = 0.5 \cdot \mathbb{I}[\hat{Y}_{i,j} = Y_i] + 0.5 \cdot \mathbb{I}[\hat{Y}_{i,j} = Y_i] \cdot \mathbb{I}[\hat{R}_{i,j} = R_i]\)) and prompt sampling distribution, but scales the normalized advantage \(A_{i,j}\) via dual dynamic factors: $\(A'_{i,j} = c_i \cdot \omega_i \cdot A_{i,j}\)$ Here, the sample balance coefficient \(c_i = \text{clip}\left(\frac{|B|}{2 n_{Y_i}}, c_{\min}, c_{\max}\right)\) corrects stochastic safe/unsafe prompt representation in each rollout batch. Crucially, the calibration factor \(\omega_i\) monitors the empirical class accuracy gap \(\Delta_{\text{cls}} = \text{Acc}_{\text{safe}} - \text{Acc}_{\text{unsafe}}\) in the current rollout: $\(\omega_i = 1 + \lambda \max(-\bar{\Delta}_{\text{cls}}, 0) \quad (\text{if } Y_i = \text{Safe}), \qquad \omega_i = 1 + \lambda \max(\bar{\Delta}_{\text{cls}}, 0) \quad (\text{if } Y_i = \text{Unsafe})\)$ By scaling up the policy gradient update pressure exclusively on whichever class currently underperforms, Balance-GRPO smoothly and automatically closes the accuracy gap throughout training without altering raw environmental reward definitions.
Loss & Training¶
StepGuard is built upon Qwen3-4B-Instruct. The SFT stage fine-tunes all parameters across 3,000 StepGen-annotated examples for 2 epochs on 4 NVIDIA H200 GPUs using AdamW (learning rate \(2 \times 10^{-5}\), effective batch size 16, context length 16,384, requiring ~0.5 hours). The Balance-GRPO stage uses 4,000 additional balanced examples, running on-policy rollouts with 64 prompts per batch and 8 sampled completions per prompt (rollout temperature 1.0, maximum response length 1,024). Optimization is performed with PPO clipping range \([0.2, 0.28]\), actor learning rate \(5 \times 10^{-7}\), low-variance KL coefficient \(\beta = 0.001\), balance weight multiplier \(\lambda = 2.0\), deadband \(0.02\), and effective weight range \([0.75, 1.5]\), completing in ~1.5 hours (6 H200 GPU-hours total).
Key Experimental Results¶
Main Results¶
The model was extensively evaluated on static safety benchmarks across both trajectory-level datasets (ATBench, R-Judge, ASSE Security) and step-level benchmarks (TS-Bench-Dojo, TS-Bench-Harm), as well as interactive runtime environments (AgentDojo, AgentDyn, AgentHarm) paired with an independent agent backbone (Qwen3.6-35B-A3B).
Table 1 details the static safety evaluation results across both trajectory and step granularities:
| Model | Size | Traj Avg. Acc. | Traj Avg. F1 | Step Avg. Acc. | Step Avg. F1 |
|---|---|---|---|---|---|
| GPT-5.4 (Closed-Source Reference) | – | 83.0 | 84.4 | 81.3 | 83.3 |
| Qwen3.5-397B-A17B | 397B | 82.1 | 82.5 | 82.9 | 84.4 |
| Qwen2.5-7B-Instruct | 7B | 60.5 | 53.2 | 78.0 | 69.4 |
| Qwen3-4B-Instruct | 4B | 70.5 | 63.3 | 79.3 | 69.2 |
| LlamaGuard3-8B | 8B | 46.5 | 15.7 | 77.6 | 54.2 |
| AgentDoG-Qwen3-4B | 4B | 74.1 | 77.8 | 52.5 | 61.1 |
| ShieldAgent-THU | 7B | 76.4 | 79.4 | 60.8 | 65.5 |
| TS-Guard | 7B | 71.5 | 58.4 | 84.0 | 79.9 |
| StepGuard (Ours) | 4B | 83.0 | 83.3 | 84.8 | 84.1 |
Table 2 reports the guarded-agent runtime performance, evaluating Attack Success Rate (ASR %, lower is better) versus Benign Task Utility (Utility %, higher is better) under dynamic execution:
| Guard Configuration | AgentDojo ASR ↓ | AgentDojo Utility ↑ | AgentDyn ASR ↓ | AgentDyn Utility ↑ | AgentHarm Malicious ↓ | AgentHarm Completion ↑ |
|---|---|---|---|---|---|---|
| No Defense | 25.1 | 89.7 | 21.2 | 73.3 | 22.8 | 70.9 |
| MetaSecAlign-70B | 1.9 | 84.5 | 9.0 | 55.0 | – | – |
| ShieldAgent-THU (7B) | 18.1 | 85.3 | 20.0 | 70.0 | 1.3 | 18.7 |
| TS-Guard (7B) | 6.2 | 92.8 | 13.8 | 70.0 | 5.4 | 58.4 |
| StepGuard (4B) | 1.2 | 90.7 | 9.3 | 66.7 | 3.4 | 52.8 |
Ablation Study¶
Ablation experiments isolate the contributions of StepGen's data components and Balance-GRPO's dynamic weighting:
Table 3(a): StepGen Data Component Ablation (SFT stage with fixed sample counts) | Component Evaluated | Target Benchmark | Metric | Without (w/o) | With (w/) | Absolute Gain | |---|---|---|---|---|---| | Prefix Supervision | 3 Trajectory Benchmarks Avg. | Avg. Acc. / F1 | 80.4 / 82.0 | 83.8 / 83.4 | +3.4 / +1.4 | | Benign Tool Reuse | TS-Bench-Harm | F1 Score | 69.2 | 75.2 | +6.0 |
Table 3(b): Balance-GRPO vs. Alternative Balancing Strategies | Optimization Method | Static Acc. | Static F1 | Safe Acc. | Unsafe Acc. | |Δ| Gap | Dojo ASR / Utility | Dyn ASR / Utility | |---|---|---|---|---|---|---|---| | SFT Baseline | 79.7 | 81.0 | 69.3 | 91.1 | 21.8 | – | – | | Safe 70:30 Upsampling | 81.4 | 80.1 | 84.5 | 76.2 | 8.3 | – | – | | Vanilla GRPO | 81.5 | 81.9 | 75.3 | 88.3 | 13.0 | 0.9 / 85.3 | 9.0 / 60.0 | | GRPO + Fixed Weighting (1.5/0.5) | 81.8 | 80.4 | 85.4 | 77.5 | 7.9 | – | – | | Balance-GRPO (Ours) | 82.2 | 82.1 | 78.4 | 86.4 | 8.0 | 1.2 / 90.7 | 9.3 / 66.7 |
Key Findings¶
- Eliminating Defense Bias Restores Task Utility: Standard GRPO leaves a 13.0-point gap between safe and unsafe accuracy, leading to excessive false rejections. Balance-GRPO narrows this gap to 8.0, translating into major runtime utility rebounds of +5.4 points on AgentDojo and +6.7 points on AgentDyn with negligible ASR variation (+0.3 points).
- Minimal Runtime Overhead: Under an exhaustive pre-execution inspection regime on AgentDojo, StepGuard adds only 599.9 ms latency and 195.5 generated tokens per guard call. Guard inference represents merely 7.24% of the total agent interaction time—34.5% faster per call than TS-Guard.
- Robust Generalization to Unseen Threats and Human Data: When trained strictly on two risk sources, StepGuard still attains 74.9% Acc / 78.1% F1 across six held-out risk categories on ATBench. Furthermore, on 100 human-verified real-world execution traces from the Nemotron-AIQ dataset, StepGuard achieves an 82.0% accuracy, vastly outperforming AgentDoG (57.0%) and TS-Guard (56.0%).
Highlights & Insights¶
- Prefix-Aligned Counterfactual Branching: Rather than synthesizing disconnected positive and negative episodes, StepGen enforces an identical history prefix up to the critical decision anchor \(i^\star\). This rigorously disentangles contextual semantics from action-level culpability, preventing guards from taking superficial shortcuts based on prompt keywords.
- Dynamic Policy Gap Feedback as an Advantage Controller: Instead of tuning static sample mixtures, Balance-GRPO reads the empirical class accuracy gap directly from on-policy rollouts and recalibrates advantage updates on the fly, providing an automated mechanism to prevent over-defense without touching underlying task rewards.
Limitations & Future Work¶
- Reliance on Teacher LLM Priors: StepGen's scenario design and semantic auditing lean on GLM-5.1 and GPT-5.4, which can propagate subtle normative biases or coverage gaps in highly esoteric tool domains.
- Blind Spots in Institutional Norms and Academic Integrity: Qualitative error analysis indicates StepGuard occasionally fails to flag normative policy violations that cause no direct technical or code-level harm (e.g., student impersonation and automated assignment submission), requiring deeper grounding in social and ethical boundaries.
- Multi-Agent Collusion and Extended Horizons: The current evaluation is grounded in single-agent environments with trajectories under 10–20 steps; scaling proactive step-level guardrails to decentralized multi-agent settings and infinite-horizon autonomous execution remains an open challenge.
Related Work & Insights¶
- vs. TS-Guard: TS-Guard pioneered candidate action verification across 4 hand-curated risk patterns but suffers from severe defense bias and limited risk coverage. StepGuard covers 8 agentic risk sources via automated prefix-aligned branching in StepGen and maintains high benign task utility via Balance-GRPO.
- vs. Safiron & AgentDoG: Safiron inspects static plans, and AgentDoG operates predominantly as a post-hoc diagnostic tool over completed trajectories. StepGuard unifies pre-execution action interception with post-hoc trajectory audits, achieving superior step-level F1 at 4B scale compared to 7B/8B alternatives.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates an elegant prefix-frozen counterfactual data synthesis pipeline and an on-policy accuracy-gap-driven advantage reweighting algorithm.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously validated across 5 static benchmarks, 3 dynamic environments, 100 human-verified traces, latency profiling, and cross-risk generalization.
- Writing Quality: ⭐⭐⭐⭐⭐ Highly coherent technical narrative, transparent experimental reporting, and in-depth failure mode analysis.
- Value: ⭐⭐⭐⭐⭐ Offers a practical, drop-in, lightweight guardrail architecture that successfully reconciles proactive security with business task utility in production agent workflows.