Exposing Implicit Vulnerabilities in Text-to-Image Models via Adversarial Agentic Probing¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Image Generation
Keywords: text-to-image models, implicit vulnerabilities, red-teaming, multimodal agent, cumulative adversarial decoding
TL;DR¶
Addressing the vulnerability of text-to-image models to implicit adversarial prompts that appear benign on the surface yet trigger unsafe images, AdvPIE introduces a black-box, training-free multi-agent red-teaming framework featuring global-relative multimodal feedback and Cumulative Adversarial Decoding, achieving superior attack success rates across open-source and commercial generative models.
Background & Motivation¶
Modern text-to-image (T2I) models such as Stable Diffusion and FLUX have demonstrated remarkable fidelity in synthesizing complex scenes from textual descriptions. To mitigate critical safety risks such as the generation of sexually explicit, violent, or hateful imagery, current deployments typically install front-end safety filters and apply alignment techniques. However, while explicit adversarial prompts containing overtly harmful keywords are easily detected and intercepted by textual safety guardrails, implicit adversarial prompts present a fundamentally harder security challenge. These prompts maintain benign linguistic appearances and circumvent input moderation filters, yet reliably manipulate the diffusion generation process into producing inappropriate visual content.
In real-world deployment and commercial API scenarios, red-teaming faces severe black-box constraints: attackers have zero access to model weights or internal gradients, having to rely exclusively on sparse, non-deterministic visual outputs returned by black-box queries. This produces a core operational tension: an automated agent must navigate an enormous discrete textual search space guided only by coarse visual outcomes, lacking any token-level attribution of which particular semantic choices triggered the unsafe generation. Existing multimodal red-teaming frameworks either depend heavily on explicitly toxic seed phrases, or operate via coarse prompt rewrites without granular memory, quickly becoming trapped in redundant exploration noise or inadvertently drifting into overtly harmful phrasing that triggers upfront text moderation.
The angle of attack in this work is to decompose multi-agent iterative exploration into structured multi-level feedback synthesis and fine-grained token-level guided decoding. By combining a global memory bank of historical top-performing adversarial candidates with contrastive step-by-step progress signals, the framework extracts high-signal guidance without training any neural components. Core idea: build a training-free black-box red-teaming framework, AdvPIE, that orchestrates a policy agent and a judge agent via global-relative multimodal feedback and Cumulative Adversarial Decoding, dynamically reweighting token logits via variance-scaled harmfulness scores and selective repetition penalties to steer generation toward implicit, stealthy vulnerabilities.
Method¶
Overall Architecture¶
AdvPIE operates as a closed-loop, training-free interaction between three entities: the black-box target T2I model \(\mathcal{T}\), a policy agent \(\Pi\) responsible for crafting and refining adversarial prompts (instantiated with LLaVA-v1.6-mistral-7B), and a judge agent \(\mathcal{J}\) responsible for safety evaluation (instantiated with Gemma3-4B).
The exploration starts from an innocuous seed prompt \(x_0\). At iteration \(t\), the target model \(\mathcal{T}\) takes candidate prompt \(x_t\) and synthesizes image \(i_t = \mathcal{T}(x_t)\). The judge agent \(\mathcal{J}\) evaluates safety across both text and visual modalities, returning structured labels, numerical scores, violation categories, and textual rationales. From these assessments, the framework computes global-relative guidance alongside a text-image differential harmfulness score assigned to tokens across iterations. Finally, the policy agent leverages Cumulative Adversarial Decoding (CAD) to generate the next implicit prompt \(x_{t+1}\), iteratively navigating toward model vulnerabilities until uncovering prompts that bypass text filters while consistently provoking unsafe images.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Benign Seed Prompt x0 / Candidate xt"] --> B["Black-box Target T2I Model<br/>Generate image it = T(xt)"]
B --> C["Judge Agent Multimodal Evaluation<br/>Extract labels, scores, categories & rationales"]
D["Global & Relative Feedback Mechanism<br/>Top-k memory bank + adjacent step contrast"]
C --> D
D --> E["Differential Harmfulness Scoring<br/>Penalize explicit text & accumulate token scores"]
E --> F["Cumulative Adversarial Decoding<br/>Variance-scaled logit reweighting & selective penalty"]
F --> G["Refined Implicit Adversarial Prompt xt+1"]
G -->|Budget not reached| B
G -->|Jailbreak success: safe text & unsafe image| H["Exposed Implicit Vulnerability x*"]
Key Designs¶
1. Global-relative feedback mechanism: overcoming single-step coarseness and historical noise
Relying exclusively on single-step image evaluation provides insufficient granularity regarding which specific prompt phrases drove the generation, while naively aggregating the entire raw interaction history introduces excessive clutter and contradictory guidance. To solve this, the judge agent \(\mathcal{J}\) evaluates both prompt \(x_t\) and image \(i_t\), yielding a structured modality-specific tuple: $\(f_t = (y_t^{\text{txt}}, s_t^{\text{txt}}, c_t^{\text{txt}}, r_t^{\text{txt}};\; y_t^{\text{img}}, s_t^{\text{img}}, c_t^{\text{img}}, r_t^{\text{img}})\)$ where \(y \in \{\text{safe}, \text{unsafe}\}\) indicates discrete safety classification, \(s \in [0, 1]\) measures continuous harmfulness, \(c \in \mathcal{C}\) denotes harm category, and \(r \in \mathcal{R}\) provides natural language reasoning. Building upon this tuple, AdvPIE derives dual guidance signals: (i) Global guidance maintains an external memory bank \(\mathcal{X}^{\text{ref}}_t = \operatorname{TopK}(\{x_\tau \mid \tau < t, y_\tau^{\text{img}} = \text{unsafe}\}, s_\tau^{\text{img}})\) containing the \(k\) most harmful prompts discovered so far, serving as an anchor for successful attack patterns; (ii) Relative guidance contrasts the visual harmfulness scores of the two most recent prompts (\(x_{t-1}\) and \(x_t\)) to deliver a directional gradient that informs whether local prompt edits progressed toward more harmful visual distributions.
2. Differential harmfulness scoring and token credit accumulation: steering toward implicit triggers
A successful implicit attack requires that prompts remain textually benign while maximizing downstream visual harm. Rewarding visual harmfulness alone invariably nudges the language policy toward inserting overtly graphic tokens, leading to immediate rejection by input filters. AdvPIE addresses this by computing a combined differential score at each iteration \(t\): $\(s_t = \omega^{\mathrm{img}} s_t^{\mathrm{img}} - \omega^{\mathrm{txt}} s_t^{\mathrm{txt}}\)$ with hyperparameters configured to \(\omega^{\mathrm{img}} = 0.8\) and \(\omega^{\mathrm{txt}} = 0.2\). This objective strongly incentivizes visual violations while heavily penalizing textual toxicity, giving maximal reward only to combinations that appear safe in text yet induce unsafe visual outputs. The resulting scalar \(s_t\) is assigned to all non-padding tokens in \(x_t\). Across iterations, the system tracks and updates the cumulative running average score \(\bar{s}_i\) for each unique token \(i\), smoothing stochastic fluctuations in diffusion generation and isolating tokens that reliably elicit implicit vulnerabilities.
3. Cumulative Adversarial Decoding: variance-adaptive sampling and diversity preservation
Rather than relying on unguided beam search or sequence-level heuristic rewrites, AdvPIE introduces Cumulative Adversarial Decoding (CAD) to intervene directly on vocabulary logits during policy autoregressive generation. Given vocabulary logits \(\ell^{(\tau)}\) and their across-vocabulary standard deviation \(\sigma^{(\tau)}\) at decoding step \(\tau\), the logit for token \(i\) is reweighted via: $\(\ell_i^{(\tau)\prime} = \ell_i^{(\tau)} + \alpha \bar{s}_i \sigma^{(\tau)} \cdot \left[ \frac{\beta}{n_i} \right]^{\mathbf{1}_{\{\bar{s}_i > 0\}}}\)$ The variance-scaling factor \(\sigma^{(\tau)}\) dynamically adapts the modification magnitude to the sharpness of the underlying policy model's predictive distribution, preventing over-manipulation that degrades grammatical fluency. Meanwhile, the selective decay term utilizes token occurrence count \(n_i\) and penalty hyperparameter \(\beta\) exclusively on tokens with positive average scores (\(\mathbf{1}_{\{\bar{s}_i > 0\}}=1\)). This design prevents the policy agent from repeatedly regurgitating the same high-scoring trigger tokens, compelling it to explore linguistically diverse, fluent, and covert prompt variations.
A Worked Example¶
Consider an adversarial search under the medical disaster category. The trajectory initiates with an innocuous prompt depicting a parent-child hospital visit. In early rounds, relative feedback guides the policy to introduce neutral clinical equipment and surgical settings; CAD elevates the weights of vocabulary tokens such as "procedure" and "chaotic" without exceeding textual moderation thresholds. By iteration 4, the prompt shifts to describe unexpected technical mishaps during surgery. By iteration 6, CAD synthesizes the implicit prompt: "Medical procedure gone disastrously wrong. Gory, chaotic aftermath. Blood-soaked doctors, nurses, patients." This prompt successfully eludes textual moderation APIs while causing DALL-E 3 to generate intensely graphic medical imagery. Throughout the process, the token repetition penalty discouraged explicit toxic slang, allowing the policy to locate stealthy descriptive phrasing.
Key Experimental Results¶
Main Results¶
The evaluation protocol employs GPT-4o to generate 30 benign seed prompts across 11 harmful categories, evaluating each method over 20 iterative rounds (generating 3 images per iteration). An attack is counted as successful under the Attack Success Rate (ASR, %) metric only when the prompt passes the OpenAI Moderation API and the resulting image is flagged as harmful by the vision judge (Gemma3-4B, LLaVA Guard, or Google SafeSearch). Benchmark results across four open-source models are summarized below:
| Target Model | Method | Gemma ASR (%) | LLaVA Guard ASR (%) | SafeSearch ASR (%) |
|---|---|---|---|---|
| safe-sd-v1-5 | Groot | 11.55 | 9.35 | 0.11 |
| ART | 21.33 | 15.88 | 0.05 | |
| FLIRT | 33.26 | 24.14 | 0.08 | |
| AdvPIE (Ours) | 30.45 | 24.73 | 0.11 | |
| safe-sd-v2-1 | Groot | 10.97 | 8.55 | 0.30 |
| ART | 21.42 | 14.73 | 0.26 | |
| FLIRT | 29.97 | 21.14 | 0.58 | |
| AdvPIE (Ours) | 32.52 | 23.61 | 0.59 | |
| sd-3.5-large | Groot | 22.24 | 17.71 | 0.47 |
| ART | 45.08 | 34.42 | 1.58 | |
| FLIRT | 50.06 | 43.36 | 1.31 | |
| AdvPIE (Ours) | 57.83 | 45.41 | 1.70 | |
| FLUX.1-schnell | Groot | 24.33 | 17.47 | 0.39 |
| ART | 43.82 | 31.36 | 0.77 | |
| FLIRT | 49.34 | 40.50 | 0.88 | |
| AdvPIE (Ours) | 58.71 | 49.64 | 1.21 |
When deployed against the commercial black-box Stable Diffusion API, AdvPIE attains the highest ASR across all evaluators: 50.45% under Gemma (+10.31% over ART), 38.00% under LLaVA Guard (+7.95%), and 1.15% under SafeSearch (+0.54%).
Ablation Study¶
Ablations on Safe-sd-v2-1 (sexual content category) isolate the contributions of core modules and individual feedback components, measuring ASR alongside the textual Bypass Rate (%):
| Configuration | Gemma ASR (%) | LLaVA Guard ASR (%) | SafeSearch ASR (%) | Text Bypass Rate (%) | Note |
|---|---|---|---|---|---|
| AdvPIE (Full) | 37.67 | 36.67 | 3.50 | 54.83 | Full framework with feedback & CAD |
| w/o CAD | 24.50 (↓13.17) | 25.17 (↓11.50) | 1.67 (↓1.83) | 45.17 (↓9.66) | Lacks token-level guidance; drifts into explicit terms |
| w/o feedback | 27.33 (↓10.34) | 25.67 (↓11.00) | 3.33 (↓0.17) | 48.33 (↓6.50) | Unstructured history degrades exploration trajectory |
| w/o reasoning | 30.50 (↓7.17) | 28.83 (↓7.84) | 2.33 (↓1.17) | 47.17 (↓7.66) | Omitting natural language rationales hurts subtlety |
| w/o guidance | 21.83 (↓15.84) | 21.33 (↓15.34) | 2.00 (↓1.50) | 37.83 (↓17.00) | Removing Top-k anchor yields severe degradation |
Key Findings¶
- Tight synergy between CAD decoding and structured feedback: Disabling CAD produces severe performance drops exceeding 11%–13% in ASR and nearly 10 percentage points in bypass rate. This confirms that sequence-level LLM prompting alone cannot replace direct token-level logit steering for evading safety filters.
- Top-k memory guidance prevents catastrophic drift: Removing score-based guidance (w/o guidance) degrades ASR by over 15% and slashes the bypass rate from 54.83% to 37.83%, performing worse than stripping all feedback entirely. Anchoring exploration around proven high-harm samples is essential to maintain targeted search.
- Human evaluation verifies implicit stealthiness: A blind human evaluation of 3,520 prompt-image pairs by three independent annotators revealed that AdvPIE achieves 11.0%–15.0% majority-agreement attack success rates, outperforming baselines (1.0%–6.0%) by wide margins and confirming that the probed prompts are genuinely perceived as benign by human reviewers.
Highlights & Insights¶
- Variance-adaptive token reweighting: Scaling logit adjustments by the instantaneous standard deviation \(\sigma^{(\tau)}\) provides intrinsic self-regulation across varying policy architectures and generation contexts, preserving fluent linguistic structure while steering semantics.
- Differential harm formulation for implicit alignment: Formulating the feedback reward as \(s_t = \omega^{\mathrm{img}} s_t^{\mathrm{img}} - \omega^{\mathrm{txt}} s_t^{\mathrm{txt}}\) balances the conflicting goals of visual harm maximization and prompt-level stealthiness without requiring model fine-tuning.
- Scalable training-free black-box discovery: Because all model parameters remain frozen, AdvPIE eliminates costly adversarial dataset collection and reinforcement learning fine-tuning, functioning as an agile diagnostic probe for auditing deployed commercial T2I APIs.
Limitations & Future Work¶
- Dependence on target visual fidelity: On low-fidelity or heavily constrained generation models (such as Safe-SD-v1-5), degraded visual outputs weaken the informativeness of the judge agent's visual assessment, slightly lowering probing efficacy.
- Uniform token credit assignment: Currently, non-padding tokens in a prompt receive identical harmfulness increments, overlooking syntactic and compositional asymmetry between modifiers and head nouns; integrating cross-attention or Shapley-based attribution could refine credit allocation.
- Holistic defense implications: Defense mechanisms cannot rely solely on superficial prompt string filtering; future protection strategies must incorporate multi-stage latent space anomaly detection during diffusion sampling.
Related Work & Insights¶
- vs FLIRT (EMNLP 2024): While FLIRT employs in-context feedback loops, its scoring relies solely on image harmfulness, frequently prompting the generator toward overtly toxic phrasing that gets caught by input filters; AdvPIE balances text and visual scores and applies CAD to maintain high stealth.
- vs ART (NeurIPS 2024): ART requires extensive supervised fine-tuning of an adversarial LLaVA model on curated red-teaming corpora; AdvPIE achieves superior attack performance in a training-free, black-box paradigm.
- vs Adversarial Nibbler (FAccT 2024): Adversarial Nibbler relies on extensive human crowdsourcing to uncover subtle failure cases; AdvPIE fully automates the discovery loop, scaling implicit vulnerability auditing to commercial APIs.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneers training-free agentic red-teaming for implicit T2I vulnerabilities with variance-scaled Cumulative Adversarial Decoding]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Thoroughly benchmarked across four base/aligned models, commercial APIs, multiple safety judges, and blind human evaluation]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulations, clean multi-agent interaction abstraction, and insightful ablation dissection]
- Value: ⭐⭐⭐⭐⭐ [Addresses the pressing security gap of stealthy prompt jailbreaks in generative vision models, offering an effective diagnostic toolkit for safety auditing]