Skip to content

The Autonomy Tax: Defense Training Breaks LLM Agents

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Agent
Keywords: Prompt Injection Defense, LLM Agent, Shortcut Learning, Cascade Amplification, Autonomy Tax

TL;DR

This paper exposes a fundamental capability-alignment paradox termed the "autonomy tax," where defense fine-tuning against prompt injection overfits surface shortcuts, destroying multi-step agent execution through immediate Step-1 tool incompetence, unrecoverable retry cascade timeouts (up to 99%), and high bypass rates under sophisticated attacks.

Background & Motivation

Large language models are rapidly shifting from single-turn chat interfaces to autonomous multi-step agents that interact with tools, execute file and API operations, and conduct sequential decision-making. Operating under frameworks such as ReAct, these agents maintain continuous reasoning-action-observation cycles where each action's environmental observation directly seeds the subsequent reasoning context. However, this deep reliance on external environments exposes agents to indirect prompt injection attacks, wherein adversaries embed malicious instructions inside web search results, database records, or retrieved files to hijack task objectives or manipulate system privileges.

To counter these vulnerabilities, practitioners commonly fine-tune models using paired benign and adversarial datasets, implementing structured queries with XML delimiters (e.g., StruQ), preference optimization via DPO (e.g., SecAlign), or instruction hierarchy constraints. Evaluated on conventional single-turn benchmarks, these defense models report stellar numbersโ€”frequently boasting attack rejection rates above 90% and false positive rates under 5%. Crucially, however, this single-turn evaluation paradigm masks catastrophic vulnerabilities in multi-step agent deployment, where input-output dependencies propagate over time and where localized model failures trigger persistent retry dynamics.

The root cause behind this discrepancy lies in shortcut learning: instead of instilling genuine semantic understanding of adversarial intent, empirical risk minimization on defense datasets encourages gradient descent to latch onto superficial syntactic boundaries and trigger keywords. Core idea: defense training imposes a steep "autonomy tax" governed by three systemic agent biasesโ€”immediate tool execution breakdown at Step 1 before receiving external observations (agent incompetence bias), unrecoverable retry loops driving task timeouts up to 99% (cascade amplification bias), and keyword-overfitting that lets sophisticated attacks bypass defenses at 73%โ€“86% while causing 25%โ€“71% false refusals on benign technical documents (trigger bias).

Method

Overall Architecture

The authors formalize the execution of multi-step agents as an iterative loop where at step \(t\), the agent receives an accumulated context \(c_t = [s, u, h_{<t}]\) composed of system instructions \(s\), user task \(u\), and execution history \(h_{<t} = \{(a_1, o_1), \dots, (a_{t-1}, o_{t-1})\}\). The model generates an action \(a_t = f_\theta(c_t)\), collects environment observation \(o_t\), and repeats until it emits a finish token or exhausts the step budget \(T_{\max}\). When action execution fails or is refused, standard agent orchestrators inject a retry prompt to prompt recovery.

To evaluate how defense training disrupts this loop, the paper introduces a diagnostic evaluation framework spanning two controlled datasets: 97 realistic tasks across workspace, banking, travel, and communication domains from AgentDojo (instrumented with controlled trigger-phrase injections to analyze Step-1 execution and depth-stratified cascade dynamics), and a curated 350-sample challenging subset distinguishing social engineering, obfuscated encodings, instruction overrides, and benign technical documentation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["User Task u & System Instructions s"] --> B["Step 1: Zero External Observations"]
    B --> C["Agent Incompetence Diagnostic<br/>Valid Action Rate vs Immediate Refusal"]
    C -->|Malformed Output / Tool Refusal| D["Cascade Amplification Diagnostic<br/>Retry Loop Accumulates Error State"]
    D --> E["Max Depth Timeout Boundary<br/>CFR Climbs to 36%โ€“99%"]
    B --> F["Tool Returns Environment Observation o_t"]
    F --> G["Trigger Bias Diagnostic<br/>Surface Patterns vs Semantic Threat"]
    G -->|Obfuscation / Social Eng. No Keywords| H["High Attack Bypass Rate (73%โ€“86%)"]
    G -->|Benign Tech Docs with Trigger Words| I["High Benign Over-Refusal (25%โ€“71%)"]
    G --> J["Role Dependency Diagnostic<br/>Meta SecAlign Deployment Paradox"]

Key Designs

1. Agent Incompetence Bias: Step-1 Execution Breakdown Before Tool Observations

Standard safety assessments assume utility degradation only occurs when the model encounters potentially adversarial text. The authors refute this through temporal isolation: examining step \(c_1 = [s, u]\) where the agent receives solely the system prompt and the user task description, with zero tool observations. A defense-trained model exhibits a severely degraded probability of generating valid actions \(a \in \mathcal{A}\) within the schema: $\(P(f_{\theta'}(c_1) \in \mathcal{A} \mid c_1 = [s, u]) \ll P(f_\theta(c_1) \in \mathcal{A} \mid c_1 = [s, u])\)$ Empirical testing reveals that StruQ models produce malformed, unparseable outputs on 69%โ€“71% of benign tasks at Step 1, while SecAlign models exhibit a 46%โ€“47% immediate refusal rate, explicitly declaring an inability to access the file system or use tools despite clear system instructions granting permission. This demonstrates that defense training damages fundamental instruction-following and tool-invocation competence.

2. Cascade Amplification Bias: Retry Loops Perpetuating Timeout Concentration

Agent orchestrators treat execution failures and refusals as transient, recoverable errors, prompting the model to retry (e.g., "Previous attempt failed. Try alternative approach"). However, because defense-induced incompetence is a systematic behavioral defect rather than a stochastic token sampling fluke, subsequent retries encounter the identical refusal heuristic across all iterations: $\(\text{CFR}(f_{\theta'}) = P(\forall t \le T_{\max} : f_{\theta'}(c_t) \neq \text{finish} \mid u \in \mathcal{U}_{\text{benign}})\)$ The cascade amplification factor \(\text{CFR}(f_{\theta'}) / \text{CFR}(f_\theta) > 1\) reaches 2.0โ€“2.7ร— in practice. Trajectory analysis exhibits a sharp bimodal distribution: tasks either conclude smoothly in steps 1โ€“9 or cascade inexorably to timeout at maximum depth (\(T_{\max} = 10\)). A single localized false refusal is thus amplified by the agent loop into total, irreversible trajectory breakdown.

3. Trigger Bias & Security-Utility Inversion: Keyword Shortcuts Over Semantic Threat Modeling

When defense datasets contain spurious correlations between trigger words and attack labels, empirical risk minimization drives gradients toward lexical detection heuristics: $\(\nabla_\theta \mathcal{L} \approx \nabla_\theta \mathbb{E}[\ell(f_\theta(x), y) \mid w \in x]\)$ leading to \(P(f_{\theta'}(x) = \text{refuse} \mid w \in x) \gg P(f_\theta(x) = \text{refuse} \mid w \in x)\). This shortcut creates a paradoxical dual collapse: sophisticated attacks that strip plaintext trigger words via Base64, ROT13, URL encoding, or benign social framing ("QA testing account deletion") evade defense detection with bypass rates of 73%โ€“86%. Conversely, benign technical documentation containing legitimate security words ("bypass certificate validation", "override timeout") triggers widespread false refusals (FPR of 25%โ€“71%), simultaneously destroying both security and utility.

4. Real-World Deployment Conflicts: Circular Dependencies in Role-Delimited Defenses

Analyzing Meta's official SecAlign defense on Llama-3.1, which introduces a special {"role": "input"} marker to designate untrusted observations, the authors uncover a fatal circular dependency. In real agent deployments, leveraging this defense requires pre-identifying which external content is potentially adversarial in order to wrap it in the "input" roleโ€”the exact task the defense was built to automate. When deployed without clairvoyant pre-labeling, Meta SecAlign's attack detection drops to 68.0% (worse than the 79.6% baseline), proving that static structural boundaries fail to protect dynamic multi-step agent environments.

Key Experimental Results

Main Results

Evaluations were conducted across 97 multi-step tasks in AgentDojo (spanning workspace, banking, travel, and Slack domains with a depth budget of 10) and a curated 350-sample diagnostic set (289 adversarial, 61 benign).

Model Defense Completion Rate (CR) Change vs Base (ฮ”Base) Cascade Timeout (CFR) Attack Detection (TPR) False Refusal (FPR)
Llama-3-8B Base (Undefended) 86.6% โ€” 13.4% 82.7% 0.0%
Llama-3-8B StruQ (Delimiters) 80.4% -6.2pp 19.6% 37.4% (-45.3pp) 70.5% (+70.5pp)
Llama-3-8B SecAlign (DPO) 63.9% -22.7pp 36.1% 26.7% (-56.0pp) 24.6% (+24.6pp)
Llama-3.1-8B Meta SecAlign (Official) 85.6% -1.0pp 14.4% 68.0% (-11.6pp) 0.2% (-0.4pp)
Mistral-7B Base (Undefended) 50.5% โ€” 49.5% 12.8% 0.0%
Mistral-7B StruQ (Delimiters) 14.4% -36.1pp 85.6% 18.0% (+5.2pp) 54.5% (+54.5pp)
Mistral-7B SecAlign (DPO) 1.0% -49.5pp 99.0% 6.6% (-6.2pp) 36.1% (+36.1pp)

Ablation & Diagnostic Studies

Step-1 Execution Behavior & Cascade Amplification Metrics

Model & Defense Configuration Step-1 Valid Action Step-1 Explicit Refusal Step-1 Invalid Output Cascade Amplification Factor Avg Retry Depth
Llama-3-8B Base 96.9% 1.0% 2.1% 1.0ร— 5.32
Llama-3-8B StruQ 22.7% 8.2% 69.1% 1.5ร— 5.38
Llama-3-8B SecAlign 52.6% 47.4% 0.0% 2.7ร— 7.78
Llama-3.1-8B Meta SecAlign 94.8% 0.0% 5.2% 1.1ร— 5.45
Mistral-7B Base 97.9% 0.0% 2.1% 1.0ร— 6.14
Mistral-7B StruQ 23.7% 5.2% 71.1% 1.7ร— 8.70
Mistral-7B SecAlign 53.6% 46.4% 0.0% 2.0ร— 9.94

Bypass Rates Across Attack Categories & Benign FPR Breakdown

Model & Configuration Social Engineering Bypass Obfuscation Bypass Instruction Override Bypass General Trigger FPR Security Context FPR
Llama-3-8B Base 22.0% 14.0% 18.0% 0.0% (0/30) 0.0% (0/31)
Llama-3-8B StruQ 78.0% 86.0% 68.0% 70.0% (21/30) 71.0% (22/31)
Llama-3-8B SecAlign 73.0% 81.0% 75.0% 3.3% (1/30) 45.2% (14/31)
Llama-3.1-8B Meta SecAlign 53.4% (Direct Request) 2.8% (Constraint Removal) โ€” 0.0% 0.3%

Key Findings

  • Immediate Execution Breakdown: Without seeing any external tool observations, defense training causes Step-1 failures on 47%โ€“77% of benign tasks (compared to ~3% on base models), manifesting as severe action syntax corruption in StruQ and hallucinated tool incapacity in SecAlign.
  • Catastrophic Cascade Failures: While base models degrade gracefully across reasoning depths, defense models concentrate failures at the maximum depth boundary. On Mistral-7B, SecAlign causes completion rates to crash from 50.5% to 1.0%, with 99.0% of tasks timing out in unrecoverable retry loops.
  • Shortcut Reliance Masks True Insecurity: Obfuscation and social engineering bypass defense filters in 73%โ€“86% of cases, while benign technical terms trigger false refusals in up to 71% of queries, exposing how lexical heuristics fail under distribution shift.

Highlights & Insights

  • Agentic Capability-Alignment Paradox: Extends prompt injection evaluation from static single-turn question-answering into sequential agent trajectories, uncovering how single-turn metrics (>90% refusal, <5% FPR) disguise catastrophic system-level degradation.
  • Temporal Isolation Methodology: By isolating Step-1 behavior where context contains zero external observations, the authors provide indisputable empirical proof that execution failures originate from model weights rather than observation contamination.
  • Dissecting Circular Role Dependencies: Provides a rigorous critique of Meta SecAlign's input role mechanism, demonstrating that assuming trusted and untrusted inputs can be cleanly bifurcated beforehand is fundamentally incompatible with autonomous agent execution.

Limitations & Future Work

  • Model Scale Scope: Experiments primarily evaluate 7B and 8B model families; examining whether larger foundational models (e.g., Llama-3-70B/405B, GPT-4o, Claude 3.5 Sonnet) suffer similar magnitude autonomy taxes under safety fine-tuning remains an open direction.
  • Prescriptive Solution Implementation: Proposed architectural remedies (e.g., trajectory-level cascade-penalizing losses, chain-of-thought safety audits, position-invariant attention) serve as conceptual design principles and require dedicated empirical benchmarking.
  • vs StruQ & SecAlign: Whereas original defense proposals report high defense efficacy in isolated turn slices, this work demonstrates that they induce 73%โ€“86% bypass on non-keyword attacks and trigger up to 99% task-level timeouts.
  • vs AgentDojo & ToolBench: While existing agent benchmarks focus primarily on attack success rates (ASR), this paper is the first to rigorously quantify the collateral damage inflicted upon benign execution competence by alignment training itself.

Rating

  • Novelty: โญโญโญโญโญ Establishes the autonomy tax paradox in agent safety and formalizes three distinct agent-specific failure biases.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous multi-domain tasks, controlled trigger injections, temporal Step-1 isolation, and depth-stratified cascade measurements.
  • Writing Quality: โญโญโญโญโญ Exceptionally clear mathematical formalization, compelling prose, and vivid qualitative execution case studies.
  • Value: โญโญโญโญโญ Crucial wake-up call for the LLM agent and alignment communities, charting concrete principles for agent-aware safety architectures.