Push–Pull Attentional Anchoring for Diffusion Concept Erasure¶

Conference: ECCV 2026
Paper: ECCV Official Poster
Code: https://push-pull-concept-eraser.github.io
Area: Image Generation
Keywords: Concept Erasure, Diffusion Models, Cross-Attention, Semantic Preservation, Relative Cosine Ratio

TL;DR¶

Push–Pull Attentional Anchoring (PPAA) regulates weight modification in diffusion cross-attention key-value projections via bounded relative cosine-similarity margins, achieving Pareto-optimal concept removal without corrupting unobserved same-domain peer entities.

Background & Motivation¶

Text-to-image diffusion models have achieved unprecedented generative fidelity, yet their widespread deployment has sparked urgent societal concerns regarding privacy invasion, explicit content synthesis, and intellectual property infringement. Concept erasure techniques offer an efficient alternative to full model retraining by directly editing model parameters to suppress conditional generation of sensitive target concepts. While early guidance-based methods essentially conceal target concepts and remain vulnerable to adversarial prompt attacks, recent state-of-the-art approaches increasingly favor destruction-based parameter updates that permanently alter intermediate representations.

However, existing destruction-based paradigms suffer from a fundamental tension between adversarial robustness and semantic preservation: more aggressive erasure induces extensive collateral damage to non-target generation quality. While modern two-stage frameworks such as STEREO and SUMA leverage adversarial prompt discovery in Stage 1, their Stage 2 erasure objectives rely on unconstrained weight modifications or rigid absolute-distance mappings inherited from earlier literature. Consequently, when suppressing a specific target concept such as "Barack Obama", the model excessively warps the local semantic neighborhood, causing severe distortion to closely related same-domain peer entities like "Will Smith". While broad general-domain concepts (e.g., general objects in MS-COCO) remain relatively intact, preserving unobserved same-domain peers has remained an open challenge.

This paper tackles the preservation bottleneck by reconsidering the geometric properties of the erasure objective. Standard absolute distance metrics fail to accommodate varying semantic abstraction across network depths, and unbounded objectives push representations into arbitrary distortion. The core idea is to introduce Push–Pull Attentional Anchoring (PPAA) in cross-attention key and value projection spaces, enforcing bounded push and pull margins over the relative cosine similarity ratio to achieve concept-aware, layer-adaptive semantic displacement.

Method¶

Overall Architecture¶

PPAA intervenes directly on the cross-attention Key and Value projection matrices ($W_K$ and $W_V$), which govern text-conditioned semantic routing in diffusion models, while freezing all other network weights. Given a target concept token $p_e$ and a generic anchor concept token $p_g$ provided by the user (such as mapping "Barack Obama" to generic "Person"), the framework optimizes a compound objective composed of two directional modification losses (semantic push and pull) and two preservation regularizers (semantic anchor alignment and null-prompt denoising consistency).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Inputs: Target Token pe<br/>Generic Anchor Token pg"] --> B["Cross-Attention K/V Projections<br/>W·pe and W·pg"]
    B --> C["Semantic Push Loss Lpush<br/>Drive away from W0·pe via cosine lower bound"]
    B --> D["Semantic Pull Loss Lpull<br/>Anchor toward W0·pg via relative cosine ratio"]
    B --> E["Semantic Preservation Loss Lpreserve<br/>Align W·pg with W0·pg direction"]
    F["Terminal Timestep Null Prompt pnull"] --> G["Denoising Preservation Loss Ldenoise<br/>Align ε(xT, pnull) with ε0(xT, pnull)"]
    C --> H["Joint Loss Ltotal<br/>Update only WK and WV weights"]
    D --> H
    E --> H
    G --> H
    H --> I["Output: Erased Diffusion Model with Preserved Peer Concepts"]

Key Designs¶

1. Semantic Push Loss: Bounded Cosine Margin for Controllable Divergence

To suppress target concept generation, the updated projection $W p_e$ must sufficiently deviate from the original representation $W_0 p_e$. Unlike prior methods that employ unbounded $L_2$ penalties or negative guidance directions—which push representations arbitrarily far and distort neighboring semantics—PPAA defines a bounded one-sided threshold $m_{\text{push}} \in [0, 1]$ directly in cosine space: $$\mathcal{L}_{\text{push}}(W_0, W, p_e) = \max\left(\text{sim}(W p_e, W_0 p_e) - m_{\text{push}}, 0\right)$$ The optimization penalizes representations only while the cosine similarity exceeds $m_{\text{push}}$, halting gradient updates once sufficient separation is achieved. This prevents runaway semantic drift and transforms $m_{\text{push}}$ into an intuitive, monotonic control knob for erasure intensity.

2. Semantic Pull Loss: Adaptive Anchoring via Relative Cosine Similarity Ratio

A push loss alone provides no directional guidance for where the displaced representation should land, risking the destruction of shared categorical context (e.g., destroying "human", "suit", or "portrait" attributes when erasing "Barack Obama"). However, enforcing a static absolute distance toward the generic anchor representation $W_0 p_g$ is problematic because different network layers operate at distinct feature scales, and different concept-anchor pairs exhibit vastly divergent initial similarities. To overcome this, PPAA introduces a relative cosine ratio normalized by the initial baseline similarity: $$\mathcal{L}_{\text{pull}}(W_0, W, p_e, p_g) = \max\left(m_{\text{pull}} - \frac{\text{sim}(W p_e, W_0 p_g)}{\text{sim}(W_0 p_e, W_0 p_g)}, 0\right)$$ By normalizing by $\text{sim}(W_0 p_e, W_0 p_g)$, the margin parameter $m_{\text{pull}}$ acts as an interpretable semantic retention percentage (e.g., retaining at least 80% or 40% of baseline generic alignment). This formulation automatically adapts across layers and concepts, enabling a single scalar hyperparameter to be applied uniformly across the entire network.

3. Dual Preservation Regularization: Safeguarding Generic Manifolds and Denoising Capacity

To prevent unintended degradation of unrelated prompts and global diffusion dynamics, PPAA incorporates two complementary preservation constraints. First, a cosine-based semantic preservation loss aligns the updated anchor representation with its original state: $$\mathcal{L}_{\text{preserve}}(W_0, W, p_g) = 1 - \text{sim}(W p_g, W_0 p_g)$$ Second, terminal-timestep denoising preservation enforces consistency under the null prompt $p_{\text{null}}$: $$\mathcal{L}_{\text{denoise}} = \mathbb{E}_{x_T \sim \mathcal{N}(0, \mathbf{I})} \left\| \epsilon(x_T, p_{\text{null}}) - \epsilon_0(x_T, p_{\text{null}}) \right\|_2^2$$ Together, these terms anchor the global diffusion trajectory and preserve generic text-to-image synthesis capabilities.

Loss & Training¶

The joint objective aggregates layer-wise terms across all cross-attention layers $l \in [1, L]$ and projection types $j \in \{K, V\}$: $$\mathcal{L}_{\text{total}} = \frac{1}{2L} \sum_{l=1}^L \sum_{j \in \{K, V\}} \left( \lambda_{\text{push}} \mathcal{L}_{\text{push}}^{(l,j)} + \lambda_{\text{pull}} \mathcal{L}_{\text{pull}}^{(l,j)} + \lambda_{\text{preserve}} \mathcal{L}_{\text{preserve}}^{(l,j)} \right) + \lambda_{\text{denoise}} \mathcal{L}_{\text{denoise}}$$ For multi-token concepts, losses are computed independently per token plus the EOS token and averaged. Default hyperparameters are set to $\lambda_{\text{push}} = \lambda_{\text{pull}} = \lambda_{\text{denoise}} = 1$, $\lambda_{\text{preserve}} = 32$, and $m_{\text{pull}} = 0.40$. Models are trained on Stable Diffusion v1.4 using Adam for 1,000 steps while tuning only the $K$ and $V$ projection weights.

Key Experimental Results¶

Main Results¶

The method is evaluated across four benchmark categories: Celebrities (Barack Obama, Rihanna, Margot Robbie, David Beckham), Artistic Styles (Van Gogh, Picasso, Claude Monet, Jackson Pollock), Intellectual Property Entities (Mickey Mouse, R2D2, Grumpy Cat, MacBook), and Nudity. Target concept detection accuracy (lower is better) and CMMD ($\times 1000$, lower is better) assess erasure and preservation respectively.

Table below presents results from Table 2 of the original paper under CCE adversarial attacks on nudity erasure:

Method	ASR (Attack Success Rate) ↓	CMMD (Preservation) ↓	Note
ESD	0.69	13.94	Significant semantic disruption
UCE	0.79	4.05	Vulnerable to adversarial recovery
DUO	0.01	6.44	High robustness but higher visual drift
STEREO	0.04	139.47	Severe collapse in generative distribution
Ours (PPAA)	0.20	1.43	Superior preservation with strong non-adversarial defense

Table below transcribes results from Table 1 of the original paper under UnlearnDiff prompt optimization attacks on nudity:

Method	Pre-ASR ↓	ASR (Final Attack Success) ↓	Avg Attack Time (mins) ↑
ESD	0.26	0.92	9.89
UCE	0.41	0.92	8.35
MACE	0.55	0.92	7.11
AC	0.20	0.88	9.98
SUMA	0.16	0.83	11.80
Ours (PPAA)	0.08	0.83	11.46
DUO	0.02	0.47	13.45
STEREO	0.00	0.08	14.74

Furthermore, on modern flow-matching Transformer architecture FLUX (Fig. 4b), Ours achieves 0.01 target detection accuracy with a CMMD of 239.61, outperforming ESD (Acc 0.00, CMMD 1511.69) and UCE (Acc 0.33, CMMD 268.42).

Ablation Study¶

Ablation experiments reported in Fig. 6 systematically validate the design components:

Config	Performance Characteristics	Mechanistic Explanation
Full Model (Ours)	Pareto-optimal frontier across all categories	Normalized relative margin provides cross-layer self-adaptive balance
Absolute Margin Pull (Abs-Margin)	Noticeable increase in CMMD for same erasure	Static thresholds cause under-constraint in deep layers and over-fitting in shallow layers
$L_2$ Distance Metric ($L_2$-Push/Pull)	Substantially worse trade-off and sensitivity	Unbounded $L_2$ metrics distort embedding geometries unpredictably
w/o Preservation Losses ($\mathcal{L}_{\text{preserve}}, \mathcal{L}_{\text{denoise}}$)	Severe degradation of non-target sample quality	Unconstrained global drift collapses unrelated text-to-image mapping

Key Findings¶

Relative ratio eliminates per-layer manual calibration: Empirical baseline similarities between target and anchor tokens vary significantly across layers; normalizing by baseline cosine similarity allows a single $m_{\text{pull}}$ to function reliably network-wide.
Predictable progressive concept removal: As $m_{\text{push}}$ decreases from 0.80 to 0.20, generated outputs smoothly transition away from target identity attributes while preserving scene context, facial structure, and lighting fidelity.
Enhanced inherent resistance to optimization attacks: Without any adversarial training during optimization, PPAA substantially delays attack convergence (increasing UnlearnDiff search time to 11.46 minutes compared to 7.11 minutes for MACE).

Highlights & Insights¶

Relative Cosine Normalization Formulation: Formulating the pull loss as a relative ratio against the pre-trained anchor similarity provides an interpretable semantic retention percentage that scales cleanly across architectural depths.
Explicit Focus on Same-Domain Peer Preservation: Moves beyond coarse COCO preservation benchmarks to resolve the challenging problem of preserving adjacent domain entities (e.g., maintaining "Will Smith" while unlearning "Barack Obama").
Direct Portability to Modern Architectures: Restricting parameter modification strictly to cross-attention Key and Value projections allows immediate plug-and-play adoption on recent flow-based diffusion transformers like FLUX.

Limitations & Future Work¶

Lack of Integrated Adversarial Search in Optimization: The objective itself does not dynamically search for worst-case adversarial prompts during training; combining PPAA as a Stage 2 objective within iterative adversarial frameworks (e.g., STEREO's Stage 1) represents a promising direction for maximum robustness.
Manual Anchor Specification: Requires a user-defined generic concept for each target, which may introduce human bias or sub-optimal semantic clustering.

vs ESD (Gandikota et al., ICCV 2023): ESD uses negative guidance that alters denoising trajectories without explicit semantic bounds; PPAA enforces bounded cosine push-pull constraints to eliminate collateral distortion.
vs UCE / MACE (Gandikota et al., WACV 2024; Lu et al., CVPR 2024): UCE and MACE apply closed-form projection updates in representation space, leading to rigid mappings that harm same-domain peers; PPAA preserves fine-grained peer concepts through relative anchoring.
vs SUMA (Nguyen et al., ICCV 2025): SUMA uses early-step textual inversion embeddings as anchors which can prematurely collapse; PPAA uses explicit anchor concepts with normalized margins for greater numerical stability.

Rating¶

Novelty: ⭐⭐⭐⭐ [Elegant relative cosine ratio formulation that solves cross-layer calibration in cross-attention editing]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous evaluation across 4 domains, adversarial attack benchmarks, and FLUX scaling]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical definitions, self-contained narratives, and transparent ablation justifications]
Value: ⭐⭐⭐⭐⭐ [Highly practical for copyright and safety compliance with minimal impact on model generative quality]