Skip to content

SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/pppyb/SecOPD
Area: LLM Reasoning
Keywords: Prompt Injection, On-Policy Distillation, Safety Alignment, Token-Level Feedback, Adaptive Attack Defense

TL;DR

SecOPD proposes a secure on-policy distillation paradigm that leverages the unperturbed base model under clean context to provide fine-grained token-level advantage feedback on student rollouts under injected inputs, effectively resolving the credit assignment ambiguity of sequence-level rewards and slashing the PISmith adaptive prompt injection attack success rate on Qwen3.6-27B from 94.0% to 9.0% with virtually zero loss in general reasoning capability.

Background & Motivation

As large language models (LLMs) transition from constrained text-in-text-out chatbots to autonomous AI agents capable of browsing web pages, parsing complex external documents, invoking APIs, and executing multi-step tool calls, their threat landscape expands dramatically. When processing untrusted data from open environments (such as scraped web pages, email bodies, or shared file repositories), an adversary can embed indirect prompt injections designed to hijack the model's control flow—instructing it to ignore system guidelines and execute unauthorized actions. Prompt injection is now recognized as the top vulnerability facing LLM applications by OWASP, having directly delayed real-world agent deployments due to risks of arbitrary code execution, private credential exfiltration, and malicious tool invocations.

To establish intrinsic model-level defenses, prevailing paradigms encapsulate untrusted external data within designated message channels and train models to prioritize system or user prompts over external data via defensive fine-tuning techniques such as Direct Preference Optimization (DPO) or Group Relative Policy Optimization (GRPO). Unfortunately, these defenses fail catastrophically when subjected to state-of-the-art adaptive attacks—such as PISmith, an automated red-teaming framework that uses reinforcement learning to train a dedicated attacker model against the target defense. Under PISmith, prior defensive models suffer from attack success rates (ASRs) exceeding 90%, behaving almost as vulnerably as undefended models.

A fundamental technical bottleneck underlying this failure is that both DPO and GRPO rely strictly on "sequence-level feedback signals." When evaluating a generation, an external judge or implicit reward treats the entire response holistically as a single unit. In adversarial scenarios, LLMs frequently produce "hybrid responses" that partially address the legitimate user query while appending or executing the malicious injection at the end. Sequence-level rewards cannot differentiate between the benign and malicious spans: DPO fails to cleanly assign a binary preference label, while GRPO penalizes the entire response, inadvertently degrading the model's instruction-following competence on benign tasks. Core idea: Secure On-Policy Distillation (SecOPD) adapts on-policy distillation by scoring student rollouts generated under attacked inputs against the frozen base model evaluated on paired clean inputs, delivering dense token-level advantages that precisely encourage benign task tokens while penalizing injection-following tokens.

Method

Overall Architecture

The core premise of SecOPD is that while defenders cannot inspect or sanitize external inputs during inference, training datasets allow privileged construction of paired inputs: a clean input \(p_c = R(I, c)\) containing only the trusted instruction \(I\) and benign data \(c\), alongside an attacked input \(p_a = R(I, A(c, g))\) where malicious injection \(g\) is injected into the untrusted channel. During each training iteration, the student model generates an on-policy rollout \(z\) under the attacked input \(p_a\). Concurrently, a frozen teacher (the initialization base model) evaluates the identical token sequence \(z\) conditioned on the clean input \(p_c\). By comparing the student and teacher log probabilities token by token, SecOPD computes a token-level advantage signal to perform low-variance policy updates on the student adapter without requiring any external LLM judge.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Training Triplet<br/>(Trusted Instruction I, Benign Data c, Injection Goal g)"] --> B["Paired Context Construction"]
    B --> C["On-Policy Rollout Generation<br/>Student π_θ generates rollout z given attacked input p_a"]
    B --> D["Cross-Context Token Scoring<br/>Frozen teacher π_ref evaluates z_t given clean input p_c"]
    C --> D
    D --> E["Token-Level Advantage Update<br/>Compute A_t = sg[ℓ_ref,t − ℓ_θ,t] and perform policy step"]
    E --> F["Aligned Robust Foundation Model<br/>Deployable without teacher; intrinsically robust to adaptive attacks"]

Key Designs

1. Paired Cross-Context Construction: Establishing Clean Reference Baselines for Privilege Separation

To enforce the principle of instruction hierarchy—ensuring that privileged user prompts \(I\) strictly dictate model actions while external data channels \(c\) remain strictly contextual—SecOPD constructs paired training samples. Using a structured chat template, the trusted prompt is wrapped in <|im_start|>user, while untrusted external data is isolated inside a distinct <|im_start|>input block. The injection constructor \(A(c, g)\) blends the injection goal \(g\) using diverse patterns, including prefix insertion, suffix insertion, and delimiter-like completion attacks that mimic chat-template boundary tokens. Crucially, the clean input \(p_c\) strips all injection tokens while preserving the identical benign context. This setup allows the clean input to serve as a ground-truth distributional anchor: the objective is to teach the model that its predictive token distribution when observing \(p_a\) should mirror its uncorrupted behavior under \(p_c\).

2. Cross-Context Token-Level Advantage Scoring: Fine-Grained Credit Assignment for Hybrid Responses

To overcome the credit assignment failure inherent in sequence-level RL, SecOPD computes token-level advantages across the paired contexts. For a student rollout \(z = (z_1, \dots, z_T) \sim \pi_{\theta}(\cdot \mid p_a)\), let \(h_t^a = (p_a, z_{<t})\) represent the student's prefix condition and \(h_t^c = (p_c, z_{<t})\) denote the teacher's prefix condition. Both conditions share the exact same generated prefix tokens \(z_{<t}\), differing solely in whether the untrusted context contains the adversarial payload. The student log probability is \(\ell_{\theta,t} = \log \pi_\theta(z_t \mid h_t^a)\) and the frozen initialization teacher log probability is \(\ell_{\text{teacher},t} = \log \pi_{\text{ref}}(z_t \mid h_t^c)\). The reverse-KL divergence on the sampled token is estimated as \(\hat{D}_t = \ell_{\theta,t} - \ell_{\text{teacher},t}\), yielding the advantage score:

\[A_t = -\text{sg}[\hat{D}_t] = \text{sg}[\ell_{\text{teacher},t} - \ell_{\theta,t}], \quad t = 1, \dots, T\]

where \(\text{sg}[\cdot]\) denotes the stop-gradient operator. When \(z_t\) follows the legitimate user task, the teacher on clean context assigns it high likelihood, resulting in \(\ell_{\text{teacher},t} \ge \ell_{\theta,t}\) and a positive advantage \(A_t \ge 0\), reinforcing the benign capability. Conversely, when \(z_t\) drifts into executing the injected payload, the teacher (who never saw the injection) assigns negligible probability, producing \(\ell_{\text{teacher},t} \ll \ell_{\theta,t}\) and triggering a strong negative advantage \(A_t < 0\). This mechanism provides surgical, token-specific credit assignment along the trajectory.

3. Teacher-Free Deployment via On-Policy Policy Optimization: Mitigating Judge Bias and Retaining Utility

Unlike GRPO setups that necessitate training or invoking an auxiliary LLM judge (such as Gemini or GPT-4) to provide sparse scalar rewards, SecOPD derives all supervision natively from the unaligned base model itself evaluated under clean context. The student model and frozen teacher are initialized from the identical pre-trained weights (Qwen3.6-27B), and only the student's LoRA adapter parameters (rank 128) are trained. Because the advantage values \(A_t\) are treated as constant scalar weights via stop-gradient, gradients flow directly through the policy likelihood \(\nabla_\theta \log \pi_\theta(z_t \mid h_t^a)\). This eliminates noisy external judge bias, reduces optimization variance, and avoids catastrophic forgetting on downstream mathematical and scientific reasoning benchmarks.

Loss & Training

The training objective optimizes the student policy \(\pi_\theta\) using token-level policy gradients:

\[\mathcal{L}_{\text{SecOPD}}(\theta) = -\mathbb{E}_{(I, c) \sim \mathcal{D}, g \sim \mathcal{G}, z \sim \pi_{\theta^-}(\cdot \mid p_a)} \left[ \sum_{t=1}^T A_t \cdot \log \pi_\theta(z_t \mid p_a, z_{<t}) \right]\]

Training is performed on 19K samples derived from the Cleaned-Alpaca dataset with non-empty input fields. The student model utilizes Qwen3.6-27B with LoRA rank 128, optimized via AdamW with a learning rate of \(1 \times 10^{-4}\) and sampling temperature 1.0. Generation length is capped at 16K tokens. The defense requires no external security reward models or auxiliary verification agents during either training or deployment.

Key Experimental Results

Main Results

Evaluations are conducted on the SEP benchmark (9.1K samples for text-based instruction following under injection) and AgentDojo (949 injection-task pairs across 4 tool-calling domains). Security is measured by Attack Success Rate (ASR, lower is better), with particular emphasis on PISmith (a state-of-the-art adaptive attack using TRL GRPO to train dedicated attack policies). General utility is tracked across AlpacaEval2 (length-controlled win rate), SEP Utility, AgentDojo Benign Utility, MMLU-Pro, GPQA-Diamond, GSM8K, and Minerva-Math.

Defense Method SEP Static ASR (↓) SEP Basic Adaptive ASR (↓) SEP PISmith Adaptive ASR (↓) AgentDojo Static ASR (↓) Unweighted Mean Utility (↑)
Undefended 99.4% 99.0% 97.9% 26.7% 88.1%
Meta-SecAlign (DPO) 28.9% 5.5% 94.0% 5.5% 88.5%
GRPO (Sequence-level) 15.0% 2.3% 61.2% 0.7% 83.1%
SecOPD (Ours) 1.3% 0.2% 9.0% 4.7% 88.1%

Fine-Grained Robustness Breakdown & Utility Preservation

The table below breaks down the six static attack families on the SEP benchmark alongside performance across diverse downstream utility benchmarks.

Model / Defense Straightforward ASR Ignore ASR Completion ASR Combined Static ASR (↓) AlpacaEval2 (↑) MMLU-Pro (↑) GPQA Diam. (↑) Minerva Math (↑)
Undefended 82.2% 79.7% 97.1% 99.4% 81.4% 84.1% 79.8% 92.7%
Meta-SecAlign 23.5% 5.6% 7.4% 28.9% 82.3% 83.8% 78.3% 95.1%
GRPO 2.9% 0.3% 9.7% 15.0% 76.0% 83.0% 77.8% 85.1%
SecOPD (Ours) 0.3% 0.5% 0.0% 1.3% 80.1% 84.1% 81.3% 94.3%

Key Findings

  • Adaptive Attacks Invalidate Static Defense Claims: While Meta-SecAlign achieves an ostensibly strong 5.5% ASR against basic adaptive attacks, an adversary optimized via PISmith achieves a 94.0% ASR, demonstrating that static benchmarks drastically overestimate DPO-based alignment robustness.
  • Order-of-Magnitude Security Leap via Token Supervision: Under the formidable PISmith attack, SecOPD achieves an unprecedented 9.0% ASR—an order of magnitude lower than Meta-SecAlign (94.0%) and far superior to GRPO (61.2%). Across static attack families, SecOPD suppresses delimiter completion attacks to 0.0% and overall static ASR to 1.3%.
  • Seamless Generalization to Agentic Tool Calling: Although trained solely on synthetic Alpaca QA data, SecOPD's learned instruction hierarchy transfers out-of-distribution to the AgentDojo agentic tool execution benchmark, achieving a 4.7% ASR (vs. 26.7% undefended).
  • Zero Capability Tax on Advanced Reasoning: GRPO incurs severe utility degradation, dropping Minerva-Math from 92.7% to 85.1% and average utility to 83.1%. In contrast, SecOPD preserves an average utility of 88.1%, matching the undefended baseline (88.1%) and outperforming it on GPQA-Diamond (81.3% vs. 79.8%). Manual audit of 46 Minerva-Math discrepancies revealed that only 4 (8.7%) represented genuine reasoning degradation, with the remainder stemming from output formatting or early termination.

Highlights & Insights

  • The Uncorrupted Self as the Optimal Teacher: The conceptual core of SecOPD is recognizing that an unaligned base model conditioned on clean context provides an optimal, unbiased supervisory signal. There is no need for external judge LLMs, complex reward modeling, or human safety labeling.
  • Solving Credit Assignment for Hybrid Outputs: Rather than struggling with the intractable problem of assigning a scalar reward to a response that answers both benign and injected prompts, token-level distillation decouples credit dynamically token by token, providing exact gradients for both reinforcement and suppression.
  • Refuting the Security-Utility Incompatibility Myth: Prevailing consensus often posits that robust defenses necessarily impair general LLM intelligence. SecOPD disproves this by demonstrating that fine-grained on-policy distillation unlocks high adversarial robustness while preserving mathematical, scientific, and tool-use capabilities.

Limitations & Future Work

  • Restricted to Indirect Injections: SecOPD addresses scenarios where the user prompt is trusted and external data contains untrusted injections. It does not mitigate direct jailbreaks or adversarial user prompts targeting model safety filters.
  • Dependency on Structured Message Demarcation: The framework requires the system infrastructure to reliably identify and encapsulate external content within the <|im_start|>input role. If untrusted data is concatenated directly with user instructions without delimiters, paired clean inputs cannot be constructed.
  • Omission of Injection-Aware Reasoning Traces: SecOPD conditions the model to generate reasoning traces as if the injection never existed (without acknowledging or deliberating on the injection). While beneficial for output cleanliness, this deprives downstream runtime monitoring systems of explicit telemetry for attack detection.
  • vs Meta-SecAlign (DPO): Meta-SecAlign optimizes offline preference pairs \((y^+, y^-)\) using DPO. Because DPO operates at the sequence level, it cannot localize or penalize malicious tokens embedded within an otherwise correct response, resulting in 94.0% vulnerability to PISmith. SecOPD uses online rollouts with token-level advantage estimation, lowering ASR to 9.0%.
  • vs GRPO Defense Baseline: GRPO uses an external LLM judge to assign binary rewards to complete rollouts. It suffers from severe credit assignment ambiguity and degrades math reasoning by 7.6 percentage points. SecOPD replaces the external judge with internal base model log-likelihoods, preserving high utility (94.3% on Minerva-Math).
  • vs StruQ & Spotlighting: System-level and architectural defenses attempt to isolate untrusted data via syntax parsing or input transformations. SecOPD demonstrates that foundation LLMs can be natively fine-tuned to internalize instruction hierarchy without relying on cumbersome external wrappers.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering application of on-policy distillation to prompt-injection defense, substituting sequence-level rewards with clean-context token-level advantages.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous red-teaming using RL-trained adaptive attackers (PISmith), comprehensive evaluation across 6 static attack families, AgentDojo tool execution, and 4 advanced reasoning benchmarks with manual error audits.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally articulate narrative structure, crisp problem formalization, and clean architectural visualizations.
  • Value: ⭐⭐⭐⭐⭐ Provides a highly practical, generalizable, and robust recipe for securing frontier autonomous AI agents against their foremost security threat.