Rethinking Robust Adversarial Concept Erasure in Diffusion Models¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/Qhong-522/S-GRACE
Area: Image Generation
Keywords: Diffusion Models / Concept Erasure / Adversarial Robustness / Semantic Prior / Text Encoder Tuning

TL;DR¶

Addressing the fundamental trade-off between robustness and computational efficiency in adversarial concept erasure, S-GRACE harnesses the rich image-text semantic prior of the CLIP text encoder to develop single-sample semantic-guided adversarial optimization and automatic surrogate mapping, robustly erasing sensitive concepts in just 4 minutes while maintaining high image generation utility.

Background & Motivation¶

Text-to-image diffusion models pre-trained on web-scale datasets tend to memorize and generate not-safe-for-work (NSFW) content, copyrighted artistic styles, and specific sensitive objects. While standard concept erasure aims to eliminate targeted representations without retraining the entire backbone, static erasure techniques remain vulnerable to adversarial prompt attacks (e.g., P4D, Ring-A-Bell, UnlearnDiff) that easily recover the suppressed concepts from residual parameter pathways.

Adversarial concept erasure mitigates this risk through a bi-level min-max game: an adversarial phase exposes residual target representations, followed by an erasure phase updating model weights. However, existing methods suffer from a severe trade-off between robustness and computational cost. Rooted in the zero-shot classifier property of diffusion models, current formulations approximate the adversarial loss via Monte Carlo sampling across random timesteps and noise vectors. Exact or multi-sample approximations produce high-quality adversarial embeddings that accurately capture the residual concept space, but performing reverse-process evaluations at every gradient step incurs prohibitive compute. Conversely, aggressive shortcuts using few or single samples yield degraded adversarial embeddings that deviate from the true target manifold, leaving dangerous blind spots un-erased.

The paper argues that instead of blindly exploring unconstrained parameter spaces with random samples, one can leverage the structured image-text semantic manifold embedded within the diffusion model's own text encoder. Core idea: S-GRACE introduces Semantic-Guided Adversarial Optimization, which injects a frozen CLIP text encoder prior into single-sample optimization to accurately identify residual concept spaces at minimal compute, paired with Semantic-Guided Concept Erasure, which fine-tunes the text encoder to dynamically project target representations onto neighboring benign surrogates while preserving generation utility via LLM-mined orthogonal anchors.

Method¶

Overall Architecture¶

S-GRACE operates within an alternating two-stage adversarial framework executed over 4 outer iterations, focusing parameter updates entirely on the CLIP text encoder \(\mathcal{T}_\theta\) while keeping the UNet denoiser frozen. In each iteration, the pipeline first conducts Semantic-Guided Adversarial Optimization: starting from the target concept embedding, it optimizes \(P\) adversarial embeddings using single-sample noise guided by frozen CLIP semantic similarity. Next, Semantic-Guided Concept Erasure fine-tunes \(\mathcal{T}_\theta\) by minimizing the similarity between adversarial embeddings and the original target concept, maximizing similarity to original adversarial features to establish an automatic benign surrogate, and penalizing drift on \(Q\) orthogonal anchor prompts mined by an LLM to safeguard non-target generation utility.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Target embedding c_tar and base diffusion model"] --> B["Semantic-Guided Adversarial Optimization<br/>Single-sample noise + frozen CLIP semantic prior yields c_adv"]
    B --> C["Semantic-Guided Concept Erasure<br/>Fine-tune T_theta: push away c_tar + map to benign surrogate"]
    C --> D["Anchor Preservation Mechanism<br/>LLM-mined orthogonal anchors c_anc protect non-target utility"]
    D -->|Check iteration budget| E{"Reached 4 iterations?"}
    E -->|No| B
    E -->|Yes| F["Output: Robustly erased diffusion model"]

Key Designs¶

1. Semantic-Guided Adversarial Optimization: Single-sample adversarial search anchored by textual semantic priors To overcome the severe semantic divergence caused by single-sample Monte Carlo approximations in existing adversarial optimization, this design integrates a cosine similarity regularization term derived from the frozen pre-trained text encoder \(\mathcal{T}\). In each step, only a single noise vector \(n \sim \mathcal{N}(0, I)\) and timestep \(t \sim \mathcal{U}(1, 1000)\) are sampled to generate noisy latent \(z_t\). The continuous adversarial token embedding \(c_{\mathrm{adv}}\) is optimized against:

\[\mathcal{L}_{\mathrm{adv}} = \|n - \Phi_\theta(z_t, t, c_{\mathrm{adv}})\|^2 + \lambda \left(1 - \operatorname{sim}(\mathcal{T}(c_{\mathrm{tar}}), \mathcal{T}(c_{\mathrm{adv}}))\right)\]

where \(\operatorname{sim}(a, b) = \frac{\langle a, b \rangle}{\|a\| \|b\|}\), \(\Phi_\theta\) represents the current victim diffusion model, and \(\lambda\) governs prior regularization strength. The first term prompts \(c_{\mathrm{adv}}\) to trick the diffusion denoiser into synthesizing target representations, while the semantic prior prevents the embedding from drifting into non-concept off-manifold noise. This enables single-sample optimization to achieve concept alignment comparable to multi-sample Monte Carlo methods at a fraction of the computational budget.

2. Semantic-Guided Concept Erasure: Automatic surrogate mapping via efficient text-encoder adaptation Rather than modifying the parameter-heavy UNet denoiser—which demands substantial backward-pass computation and offers poor cross-model transferability—S-GRACE restricts parameter adaptation entirely to the text encoder \(\mathcal{T}_\theta\). Furthermore, instead of manually specifying fixed surrogate concepts or leaving erasure unconstrained, the core erasure objective simultaneously pushes away the target concept while pulling toward the original adversarial embedding:

\[\mathcal{L}_{\mathrm{erase\_core}} = \frac{1}{P} \sum_{i=1}^P \left[ \operatorname{sim}(\mathcal{T}(c_{\mathrm{tar}}), \mathcal{T}_\theta(c_{\mathrm{adv}}^{(i)})) + \alpha \left(1 - \operatorname{sim}(\mathcal{T}(c_{\mathrm{adv}}^{(i)}), \mathcal{T}_\theta(c_{\mathrm{adv}}^{(i)}))\right) \right]\]

The first term erases target semantics by penalizing alignment with \(\mathcal{T}(c_{\mathrm{tar}})\). The second term, weighted by \(\alpha\), binds the updated embedding \(\mathcal{T}_\theta(c_{\mathrm{adv}}^{(i)})\) to its pre-trained representation \(\mathcal{T}(c_{\mathrm{adv}}^{(i)})\). This acts as an implicit spring, automatically sliding the erased concept toward the closest benign semantic neighbor on the pre-trained CLIP manifold (e.g., mapping adversarial nudity toward clothed subjects) without arbitrary human heuristics.

3. Anchor Preservation Mechanism: Orthogonal concept shielding against collateral utility degradation To prevent gradient updates on \(\mathcal{T}_\theta\) from distorting un-targeted visual concepts, S-GRACE prompts an LLM (GPT-4) to extract \(Q\) anchor concepts that co-occur frequently with the target concept in natural corpora yet remain conceptually orthogonal. During erasure optimization, an explicit preservation penalty fixes their embeddings:

\[\mathcal{L}_{\mathrm{anc}} = \beta \frac{1}{Q} \sum_{j=1}^Q \|\mathcal{T}(c_{\mathrm{anc}}^{(j)}) - \mathcal{T}_\theta(c_{\mathrm{anc}}^{(j)})\|^2\]

This enforces localized surgical modifications within the text encoder's representation space, preserving surrounding semantic knowledge and preventing catastrophic degradation of general text-to-image synthesis quality.

Loss & Training¶

The complete objective function for the concept erasure stage is formulated as:

\[\mathcal{L}_{\mathrm{era}} = \frac{1}{P} \sum_{i=1}^P \left[ \operatorname{sim}(\mathcal{T}(c_{\mathrm{tar}}), \mathcal{T}_\theta(c_{\mathrm{adv}}^{(i)})) + \alpha \left(1 - \operatorname{sim}(\mathcal{T}(c_{\mathrm{adv}}^{(i)}), \mathcal{T}_\theta(c_{\mathrm{adv}}^{(i)}))\right) \right] + \beta \frac{1}{Q} \sum_{j=1}^Q \|\mathcal{T}(c_{\mathrm{anc}}^{(j)}) - \mathcal{T}_\theta(c_{\mathrm{anc}}^{(j)})\|^2\]

Using Stable Diffusion v1.4 as the base backbone, S-GRACE runs for 4 bi-level iterations. The adversarial optimization phase executes \(N = 10\) steps with learning rate \(\eta = 1 \times 10^{-3}\) and prior weight \(\lambda = 0.1\). The concept erasure phase optimizes \(\mathcal{T}_\theta\) over 50 steps with learning rate \(1 \times 10^{-5}\), \(P = 8\) adversarial embeddings, \(Q = 16\) anchor prompts, and weights \(\alpha = 1.2, \beta = 1.2\). The complete erasure procedure finishes in approximately 4 minutes.

Key Experimental Results¶

Main Results¶

Evaluated on NSFW concepts ("Nudity", "Violence", "Illegal Activity") from the I2P benchmark, models are assessed using attack success rate (ASR % ↓) across standard natural prompts and three adversarial red-teaming attacks (P4D, RAB, UD), along with COCO-30K image-text alignment (CLIP-Score ↑) and visual quality (FID ↓).

Concept	Method	Prompt↓	P4D↓	RAB↓	UD↓	Avg. ASR↓	CLIP-Score↑	FID↓
Nudity	SDv1.4 (Original)	92.25	100.00	100.00	100.00	98.06	31.34	14.05
	ESD	14.00	75.00	26.06	80.00	48.77	30.12	14.36
	AdvUnlearn	7.75	19.72	16.90	21.13	16.38	29.30	15.04
	R.A.C.E	5.00	49.00	19.72	47.00	30.18	29.42	16.05
	RECE	15.49	64.79	13.38	65.49	39.79	30.95	14.45
	Receler	26.76	31.20	1.10	42.25	25.33	31.02	14.10
	CPE	3.52	37.32	0.00	30.28	17.78	31.19	13.89
	STEREO	3.52	29.58	7.75	30.99	17.96	30.23	15.70
	S-GRACE (Ours)	2.11	14.79	5.63	12.68	8.80	29.44	15.01
Violence	SDv1.4 (Original)	42.57	100.00	99.01	100.00	85.40	31.34	14.05
	ESD	27.00	84.00	88.12	79.00	69.53	30.19	15.15
	R.A.C.E	11.00	75.00	79.21	68.00	58.30	29.15	18.94
	Receler	30.69	89.11	59.20	86.14	66.29	30.77	15.24
	S-GRACE (Ours)	6.93	38.61	6.93	37.62	22.52	29.85	16.00
Illegal Activity	SDv1.4 (Original)	37.76	95.92	-	96.94	76.87	31.34	14.05
	ESD	29.00	89.00	-	85.00	67.67	30.36	14.69
	R.A.C.E	20.00	85.00	-	80.00	61.67	29.71	17.19
	S-GRACE (Ours)	12.24	66.33	-	73.47	50.68	29.64	17.28

In artistic style erasure ("Van Gogh", "Picasso") and object erasure ("Church", "Parachute", "Garbage Truck", "Tench"), S-GRACE matches or outperforms competing baselines. On "Van Gogh", S-GRACE achieves 0.00% average ASR (CLIP-Score 31.22, FID 13.87); on "Picasso", it achieves 1.00% average ASR (FID 13.43). For object categories, average ASR is 2.00% for Church, 1.00% for Parachute, 0.00% for Garbage Truck, and 0.50% for Tench.

Ablation Study¶

Ablation on Nudity erasure examining hyper-parameters \(\lambda\), \(\alpha\), \(\beta\), sample count \(P\), and anchor LLM source:

Config / Variant	\(\lambda\)	\(\alpha\)	\(\beta\)	\(P\)	Anchor Source	UD ASR↓	CLIP-Score↑	FID↓	Note
Full Model (Default)	0.1	1.2	1.2	8	GPT-4	12.68	29.44	15.01	Balanced optimal robustness and utility
w/o Semantic Prior	0.0	1.2	1.2	8	GPT-4	73.94	31.23	13.36	Degrades to naive single-sample; search diverges
Excessive Semantic Prior	0.5	1.2	1.2	8	GPT-4	37.32	30.97	11.98	Constrains adversarial exploration; leaves targets intact
w/o Surrogate Mapping	0.1	0.0	1.2	8	GPT-4	0.00	16.04	91.12	Unconstrained mapping destroys feature manifold
w/o Anchor Protection	0.1	1.2	0.0	8	GPT-4	0.00	16.70	87.27	Catastrophic forgetting on general concepts
Expanded Adversarial Prompts	0.1	1.2	1.2	16	GPT-4	11.97	29.21	15.96	Marginal robustness gain at slight utility cost
Open-source LLM Anchors	0.1	1.2	1.2	8	Llama-3	23.94	29.37	15.26	Slight drop vs GPT-4, still beats all baselines

Key Findings¶

Semantic guidance is indispensable for efficient adversarial optimization: Setting \(\lambda = 0\) causes the UD attack success rate to jump from 12.68% to 73.94%. Without prior regularization, single-sample gradient updates drift into off-manifold space, yielding low-quality adversarial prompts that fail to trigger deep residual concepts.
Surrogate constraint and anchor preservation prevent utility collapse: Removing either the surrogate alignment term (\(\alpha = 0\)) or the anchor loss (\(\beta = 0\)) causes FID to explode from 15.01 to over 87–91, despite driving attack success to 0.00%. This highlights the danger of trivial robustness achieved through model destruction.
Dramatically reduced compute: By avoiding UNet backpropagation and utilizing single-sample guided optimization, S-GRACE finishes concept erasure in 4 minutes, compared to >100 minutes for AdvUnlearn and tens of minutes for CPE and STEREO.

Highlights & Insights¶

Adversarial quality governs erasure depth: The paper reveals that adversarial concept erasure often fails not because the erasure mechanism is weak, but because low-cost adversarial optimization crafts pseudo-adversarial perturbations that do not genuinely belong to the target concept distribution.
Continuous automatic surrogate projection: In contrast to heuristic manual substitution (e.g., mapping Van Gogh to generic painting), S-GRACE allows the text encoder to smoothly migrate target embeddings to the nearest pre-existing manifold coordinates via cosine regularization.
Cross-architecture transferability: Because edits are made strictly to the CLIP text encoder, the unlearned encoder can be directly transplanted into other diffusion pipelines (such as SDXL or downstream LoRAs) sharing the same CLIP backbone without additional fine-tuning.

Limitations & Future Work¶

Limitations acknowledged by authors: The design relies on heuristic balance weights without formal causal unlearning guarantees; it inherits inherent biases from the underlying pre-trained CLIP model; exact boundary definitions of the mapped surrogates remain opaque; and the defense remains somewhat susceptible to continuous concept embedding (CCE) attacks.
Identified empirical challenges: Broad, multi-faceted concepts such as "Violence" and "Illegal Activity" exhibit higher residual attack success across all methods (S-GRACE sits at 22.52% and 50.68% average ASR, respectively), indicating that distributed semantic concepts are significantly harder to disentangle within textual embeddings than localized objects.
Future directions: Integrating multimodal LLMs to decompose abstract concepts into atomic visual primitives for hierarchical erasure, and expanding defense mechanisms from text token space into cross-attention and latent trajectory controls.

vs ESD (Erasing Concepts from Diffusion Models): ESD fine-tunes UNet parameters without adversarial exploration, making it highly vulnerable to prompt optimization attacks (48.77% ASR on nudity). S-GRACE incorporates an adversarial loop while tuning only the text encoder, achieving 8.80% ASR at substantially lower computational cost.
vs AdvUnlearn: AdvUnlearn introduced adversarial training to concept unlearning but re-samples noise at every step across multiple iterations, resulting in high training times (>100 min). S-GRACE demonstrates that a semantic-guided prior allows single-sample optimization to reach superior robustness in 4 minutes.
vs R.A.C.E / Receler / STEREO: R.A.C.E uses unconstrained single-sample optimization and struggles to eliminate residual concepts; Receler and STEREO require extra adapter networks or multi-stage pipelines. S-GRACE achieves a clean Pareto improvement across robustness, utility, and speed using direct loss formulation.

Rating¶

Novelty: ⭐⭐⭐⭐☆ [Astute analysis of the sample-efficiency bottleneck in adversarial erasure; elegant integration of CLIP semantic priors]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluation over 9 concepts across 3 domains, 4 red-teaming attack vectors, and full ablation]
Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous problem setup, clean mathematical definitions, and lucid visual illustrations]
Value: ⭐⭐⭐⭐⭐ [Brings robust adversarial concept erasure down to 4 minutes per concept with high practical deployment utility]