Universal Image Immunization against Diffusion-based Image Editing via Semantic Injection¶

Conference: ECCV 2026
arXiv: 2602.14679
Code: https://ChanhuiLee1111.github.io/Universal-Immunization
Area: Diffusion Models / Image Generation Security
Keywords: Image Immunization, Universal Adversarial Perturbations, Diffusion Editing Defense, Semantic Injection, Cross-attention

TL;DR¶

To defend against malicious editing of diffusion models, this paper trains an image-agnostic universal adversarial perturbation (UAP). Through dual losses of "injecting target semantics + suppressing source semantics", the diffusion model is forced to misidentify the protected image as a pre-defined target concept (e.g., misinterpreting a dog as "Ronaldo"). Consequently, during inference, a simple addition operation is sufficient to block unauthorized editing, incurring near-zero computational overhead.

Background & Motivation¶

Diffusion models have made text-guided image editing both fast and powerful, but they also open the door to deepfakes, identity forgery, and copyright infringement. The mainstream defense paradigm in academia is "image immunization"—embedding adversarial perturbations that are virtually imperceptible to the human eye into the original image to disrupt the underlying editing process, ensuring that even if malicious users obtain the image, they cannot generate usable edited results. From encoder/diffusion-level attacks like PhotoGuard to methods manipulating cross-attention or intermediate representations like Semantic Attack and AdvPaint, defensive strength is indeed improving. However, almost all of them are image-specific: for every new image, an expensive per-image optimization must be run at inference time, which often takes hundreds or thousands of seconds and consumes large GPU memory. This "deployment bottleneck" directly prevents immunization techniques from meeting real-world latency and computational constraints.

Some works attempt to bypass image-by-image optimization using optimization-free approaches (e.g., FastProtect, DiffVax), employing a pretrained perturbation generator or fusion module to generate perturbations in a single forward pass. While this saves computational time, it still relies on extra neural networks and GPU acceleration, resulting in significant memory overhead and remaining difficult to scale in resource-constrained scenarios. Moreover, a performance gap often remains compared to strong image-specific methods. Another path, validated as effective in classification tasks, is Universal Adversarial Perturbations (UAPs)—using a single perturbation for all inputs to amortize the cost. Yet, UAP has received little attention in diffusion editing defense. The prevailing intuition is that "universality implies weakness": since a single perturbation must simultaneously trick highly diverse inputs, the protection strength is bound to be compromised.

This paper's key insight is to transfer targeted UAPs from the classification domain to diffusion editing defense. The core idea of targeted UAP is to "push all inputs toward the same target label". Translating this into the diffusion context: instead of clumsily "destroying" the source image semantics, it is better to actively inject a target concept into the image. This causes the diffusion model, during editing, to misinterpret the image as the target from the outset (e.g., a dog is treated as "Ronaldo"), completely derailing the editing results and rendering them useless. Core Idea: Train an image-agnostic targeted UAP using a "target semantic injection loss" to write the target concept into the cross-attention output, while simultaneously erasing the original content using a "source semantic suppression loss". At test time, immunization is achieved via a single addition \(x+\delta\), realizing a near-zero-cost, cross-model transferrable, and scalable defense.

Method¶

Overall Architecture¶

The input is any image \(x\) to be protected, and the output is the immunized image \(x+\delta\), where \(\delta\) is a one-time-trained, dataset-shared universal perturbation. The core mechanism is to make the diffusion model "mislook" (misinterpret) the protected image as a pre-specified target concept (target) rather than its actual content. To this end, during the training phase, a surrogate diffusion model (Stable Diffusion V1.5) and a target image \(x_{tar}\) (e.g., generated from "a photo of Ronaldo") are used to manipulate the cross-attention layers, which serve as the key channel for "writing text semantics into images". The optimization pulls the attention response of the immunized image closer to that of the target image (injection) while pushing it away from the original image itself (suppression). Once \(\delta\) is jointly optimized under these two losses, it can be directly added to any new image at test time, requiring no image-wise optimization or additional networks.

The authors deliberately choose targeting the cross-attention output \(\text{CA}_l = A_l V_l\) instead of the attention map \(A_l\). Within the residual updates of the U-Net, the conditioning signal only affects latent variables through the term \(W_l^{CA}\text{CA}_l\); the actual semantic content "written" into the hidden states is carried by \(V_l\) and propagated via \(\text{CA}_l\). In contrast, \(A_l\) only dictates "where the attention maps onto which tokens" without encoding the injected semantic content themselves; operating solely on \(A_l\) can change token selection but is insufficient to control the injected semantics. The entire pipeline is "train once, apply everywhere," with the evaluation being purely additive.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Training Image x + Target Image x_tar<br/>(e.g., 'Ronaldo')"] --> B["Target Semantic Injection Loss<br/>Aligns Cross-Attention Outputs CA_l of<br/>Immunized and Target Images"]
    A --> C["Source Semantic Suppression Loss<br/>Separates Cross-Attention Outputs CA_l of<br/>Immunized and Source Images"]
    B --> D["Jointly Optimize Single UAP δ<br/>PGD-style Signed Gradient Update<br/>Constraint ‖δ‖∞ ≤ ε"]
    C --> D
    D --> E["Inference-time Immunization<br/>x_new + δ (Single Addition)"]
    E -->|Data Available| F["Data-dependent: LAION Image-Text Pairs"]
    E -->|No Data| G["Data-free (Ours_DF)<br/>Jigsaw Puzzle Noise + Injection Loss Only"]

Key Designs¶

1. Target Semantic Injection Loss: Make the diffusion model misidentify the source image as the target concept

Traditional immunization aims to "destroy" the original image semantics. This paper takes the opposite route—actively injecting a target concept into the image, forcing the diffusion model to perform text-to-image editing conditioned on this target, which naturally yields results irrelevant to the actual source content. Specifically: the attention response of the immunized image (perturbed by UAP and conditioned on the target prompt) is forced to approximate that of the "real target image" under the same prompt. To align across multiple spatial scales, the authors aggregate the cross-attention outputs of all intermediate blocks in the U-Net. The loss is defined as:

\[\mathcal{L}_{\text{inj}} = \sum_{\ell=1}^{L} \left\| \text{CA}_l\big(\Phi^\ell(\mathcal{E}(x+\delta)),\ t_{tar}\big) - \text{CA}_l\big(\Phi^\ell(\mathcal{E}(x_{tar})),\ t_{tar}\big) \right\|_2^2\]

where \(\Phi^\ell\) is the feature map after the \(\ell\)-th intermediate block of the U-Net, \(\mathcal{E}\) is the VAE encoder, and \(t_{tar}\) is the CLIP text embedding of the target prompt. The ingenuity of this approach is that it does not require the edit result to "look like" the target (which is too difficult and unnecessary); as long as the source image's semantics are scrambled enough to prevent faithful editing, the malicious image-to-image editing fails. The authors validate this via cross-attention visualization: the attention maps of the immunized image no longer focus on its actual content (such as "cow" or "people") but sharply focus on the target "Ronaldo", almost replicating the attention pattern of the target image.

2. Source Semantic Suppression Loss: Actively wiping original content from attention

Injecting the target alone is not enough; if the source image semantics are too stubborn, the injected target may be "diluted". Drawing inspiration from adversarial attacks that minimize cross-entropy for target labels while maximizing it for non-target labels, the authors design a loss in the opposite direction to maximize the difference between the cross-attention outputs of the immunized image and the original image:

\[\mathcal{L}_{\text{sup}} = -\sum_{\ell=1}^{L} \left\| \text{CA}_l\big(\Phi^\ell(\mathcal{E}(x+\delta)),\ t\big) - \text{CA}_l\big(\Phi^\ell(\mathcal{E}(x)),\ t\big) \right\|_2^2\]

Note that the conditioning input here is the original image's own prompt \(t\) (not the target prompt), and the negative sign guides the optimization to "pull apart" their distance, actively erasing the traces of the original image semantics in the eyes of the diffusion model. It works in tandem with the injection loss: \(\mathcal{L}_{\text{inj}}\) pulls the immunized image toward the target semantics, while \(\mathcal{L}_{\text{sup}}\) pushes it away from the source semantics. Jointly optimizing \(\delta^* = \arg\min_\delta \mathbb{E}_{(x,t)\sim\mathcal{D}_p}[\mathcal{L}_{\text{inj}}+\mathcal{L}_{\text{sup}}]\) subject to \(\|\delta\|_\infty\le\epsilon\) allows simultaneous push-pull targeting, securing more thorough targeted attacks. Ablations show that adding the suppression loss further improves black-box transferability and robustness.

3. Data-free Extension: Training a viable UAP with just one target image

In reality, access to real training images or target domain priors may not be feasible. The authors point out that the design of the injection loss is fundamentally independent of "what the input is"—it only cares about matching any image's attention pattern to the target—rendering it naturally suitable for data-free scenarios. Following data-free attacks like TRM-UAP, they use random jigsaw puzzle noise as synthetic training samples (applying mean filtering to smooth boundaries and incorporating curriculum learning to gradually increase complexity). Here, \(\delta\) is optimized only using the target semantic injection loss \(\mathcal{L}_{\text{inj}}\), dropping the suppression loss which requires real-image prompts. In this way, without touching any real data, a targeted UAP capable of protecting diverse inputs can be trained using only a single target image. In experiments, this data-free variant (\(\text{Ours}_{DF}\)) still achieves the second-best performance across most metrics, closely approaching or even exceeding some image-specific methods, highlighting its practicality under extremely constrained conditions.

Loss & Training¶

Training uses PGD-style signed gradient ascent. Initializing \(\delta\sim U(-\epsilon,\epsilon)\), for each (image, prompt) and across a set of timesteps \(K=\{5,10,15,20,25\}\), forward noise is added to the clean, immunized, and target images respectively. The gradients of \(\mathcal{L}_{\text{inj}}+\mathcal{L}_{\text{sup}}\) are accumulated, and \(\delta \leftarrow \delta - s\cdot\text{sign}(\text{grad})\) is updated and projected back into the perturbation budget. Key hyperparameters: attack step size \(s=1/255\), perturbation budget \(\epsilon=10/255\) (smaller than the commonly used \(16/255\) in image-specific methods, as one perturbation must generalize to all images and should be more constrained to prevent perceptible artifacts), training for 20 epochs, with 50 inference steps and a guidance scale of 7.5. Essentially, it extends the image-wise optimization of Semantic Attack into dataset-level single-perturbation learning—moving from "optimizing a separate perturbation for each test image" to "jointly training a single \(\delta\) across the entire dataset."

Key Experimental Results¶

Main Results¶

The surrogate model is SD V1.5. Evaluations are conducted on a self-built editing dataset \(D_E\) representing 500 images (10 categories × 50 images) and the real-world dataset ImageNet-Edit. Metrics include PSNR / SSIM / VIFp / FSIM (lower is better, indicating larger deviation of editing results) and LPIPS (higher is better), as well as CLIP feature similarity Feat. Sim.(C) (lower is better, indicating editing results depend less on the source image).

White-box comparison with "generalized baselines" (directly porting image-specific methods to the UAP setting) (Table 1):

Method	PSNR↓	SSIM↓	VIFp↓	LPIPS↑	Feat.Sim.(C)↓
Encoder (Generalized)	16.55	0.482	0.154	0.452	0.708
Embedding (Generalized)	15.80	0.378	0.117	0.548	0.696
Map (Generalized)	16.16	0.468	0.152	0.465	0.704
\(\text{Ours}_{DF}\) (Data-Free)	14.68	0.378	0.106	0.557	0.685
Ours (Full)	14.19	0.332	0.082	0.606	0.673

Comparison of efficiency and effectiveness with image-specific/optimization-free methods (Table 4, white-box SD V1.5): The proposed method outperforms EA/DA/SA (which require image-wise optimization) and FP (opt-free) on the majority of immunization metrics, while achieving comparable or better imperceptibility (DISTS/LPIPS)—even under the more challenging "universal + smaller budget" setting. The comparison of testing costs is highly telling:

Method	Input-adaptive	Per-image Latency CPU/GPU (s)	GPU VRAM (GB)
EA [42]	✓	315.71 / 8.01	6.51
DA [42]	✓	2706.84 / 212.46	30.84
SA [27]	✓	2216.13 / 55.66	9.08
FP [2] (opt-free)	✓	2.76 / 0.04	0.77
Ours	✗	≈0 / ≈0	0

Black-box transferability (Table 2/5): Using SD V1.5 as the surrogate, and transferring directly to SD V1.4 / V2.0 / InstructPix2Pix, the proposed method consistently outperforms generalized baselines and image-specific methods on most metrics, showing that semantic injection produces perturbations with high cross-model generalization.

Ablation Study¶

Ablation of individual losses (Table 9, black-box/white-box across 4 models; using SD V1.5* white-box as an example):

Configuration	PSNR↓	LPIPS↑	Feat.Sim.(C)↓	Description
\(\text{Inj}_{DF}\)	14.68	0.557	0.685	Injection loss only + Data-free
Inj	14.41	0.585	0.680	Injection loss + Real data
Inj+Sup	14.19	0.606	0.673	Full (with source semantic suppression)

Target independence (Table 10): Testing five highly distinct targets (Ronaldo / Tiger / Sunflower / Peacock / Mandala) yielded small standard deviations across all metrics (e.g., SSIM mean 0.332, std 0.029), proving that the method is insensitive to the choice of target concept. Robustness (Table 3): Against four purification methods (JPEG compression, GrIDPure, Conditional DiffPure, Noisy Upscaling; the latter two being adaptive defenses aware of the mechanism), the proposed method still consistently outperforms generalized baselines.

Key Findings¶

The suppression loss is the "icing on the cake" rather than "a necessity": The improvement from \(\text{Inj}_{DF}\) to Inj comes from using real data, whereas the further gain of Inj+Sup is attributed to erasing the residual source semantics, primarily enhancing black-box transferability and anti-purification robustness. The three factors (injection, real data, and semantic decoupling) act complementarily.
Targeting \(\text{CA}_l\) rather than \(A_l\) is theory-driven: Since only cross-attention outputs carry the injected semantic content, experiments confirm that manipulating \(\text{CA}_l\) yields significantly stronger semantic effects than manipulating \(A_l\).
Universality does not equate to weakness: Even when a single perturbation is applied to all images under a smaller budget (\(10/255\) vs \(16/255\)), the performance still surpasses image-wise optimization methods, overturning the common belief that "universal performance must degrade".

Highlights & Insights¶

Shifting from "destruction" to "injection" is the most clever idea: Traditional immunization techniques try to dismantle the original image. This work takes a different path, proactively writing in a fake identity and letting the model voluntarily edit it incorrectly. Shifting the objective from "making the edit fail" to "making it successfully generate the wrong target" proves to be more stable and transferrable.
Inference requires only a single addition, reducing the marginal cost of immunization to virtually zero, requiring zero VRAM, and no GPU. This represents a paradigm shift for real-world deployments aiming to protect massive volumes of user images.
The path for transferring targeted UAP from classification to generation is solidly built: Far from a naive application, the method precisely identifies "cross-attention output as the write-in point for semantics" as a specific characteristic of diffusion models, yielding self-consistent theoretical and empirical support.
The data-free variant can train a practical perturbation using only jigsaw puzzle noise and a single target image. This recipe of "synthetic priors + single loss" can be readily transferred to other adversarial generation defense tasks where target-domain data is inaccessible.

Limitations & Future Work¶

Dependency on surrogate models + accessible cross-attention: Training requires access to the internal cross-attention layers of a diffusion model (e.g., the SD series U-Net). For structurally distinct models like DiTs, how well cross-architecture transferability holds and whether the injection loss remains effective remains an open question, only briefly touched upon in the appendix.
"Static target" injection: A single UAP is tied to a specific target concept (or set of concepts). Although the authors demonstrate that target concepts can be chosen arbitrarily and that distinct prompts can generate diverse UAPs to resist reverse engineering, the defense boundary against adaptive adversaries with full knowledge of the target concept remains to be clarified.
Anti-purification is not a complete victory: Faced with adaptive purifications such as Conditional DiffPure / Noisy Upscaling that assume knowledge of the defense mechanism, the absolute immunization strength is weakened, though still surpassing baselines. The arms race between perturbation purification and immunization is far from over.
The smaller budget of \(10/255\) is a double-edged sword: while it yields imperceptibility and generalization, it also implies an upper bound on a single perturbation's "semantic coverage capacity." Performance on extremely complex or high-resolution images warrants further stress-testing.

vs PhotoGuard (EA/DA) [42]: PhotoGuard relies on image-specific encoder/diffusion-level attacks that require per-image optimization, making them slow and memory-intensive. This work converts this concept into a universal perturbation, yielding zero inference cost and outperforming it on most metrics.
vs Semantic Attack (SA) [27]: SA performs image-wise immunization by manipulating attention maps. This paper adapts its optimization strategy but upgrades it from image-wise to dataset-level single-perturbation learning, shifting the intervention point from the attention map \(A_l\) to the cross-attention output \(\text{CA}_l\), thereby upgrading "destruction" to "injection".
vs FastProtect (FP) / DiffVax [2,32]: These methods reduce costs using optimization-free pretrained generators/immunizers, but still require auxiliary networks and GPU execution. In contrast, our inference is a single addition, requiring no auxiliary network or VRAM, while frequently exceeding their performance.
vs Targeted UAP in Classification [59]: This work transfers the concept of driving all inputs toward a target label via a single perturbation to diffusion editing defense. The primary innovation lies in identifying cross-attention outputs as the semantic injection point, coupled with the source semantic suppression loss.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ The first work to apply targeted UAP to diffusion editing immunization. The transition from "destruction to injection" is exceptionally clean.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across white/black-box settings, real-world images, multiple target concepts, anti-purification, ablation of individual losses, and computational overhead. The appendix also addresses complex prompts, DiT, and inpainting.
Writing Quality: ⭐⭐⭐⭐ Clear closed-loop from motivation, method, theoretical justification, to experiments. The visualization of cross-attention maps is highly compelling; formulas contain minor typesetting noise but do not impede understanding.
Value: ⭐⭐⭐⭐⭐ Reduces testing costs to near-zero without requiring a GPU, holding immediate value for large-scale real-world deployment.