CGCE: Classifier-Guided Concept Erasure in Generative Models

Conference: ECCV2026
Paper: Official page
PDF: Full paper
Authors: Viet Nguyen, Vishal M. Patel
Project: https://viettmab.github.io/cgce-page/
Area: Image Generation
Keywords: concept erasure, text embeddings, classifier guidance, utility preservation, cross-model safeguarding

TL;DR

CGCE places concept erasure before the generator: a learned concept-conditioned classifier applies weighted-gradient refinement only to flagged text embeddings, reducing MMAD attack success on SD-v1.4 from 63.60% to 1.90% without changing generator weights while retaining general generation utility close to the original model.

Background & Motivation

Concept erasure seeks to prevent a generator from producing a specified concept while preserving its other capabilities. Fine-tuning or editing generator weights changes model behavior but can also affect unrelated concepts. Extra guidance during sampling is constrained by the generation architecture: methods requiring negative prompts or standard classifier-free guidance do not transfer directly to FLUX or visual autoregressive models. This paper therefore investigates an upstream safeguard rather than retraining every image or video generator.

Text embeddings appear to offer a shared interface, but simple linear projection at that interface also has limitations. SAFREE selects and projects tokens according to their relationship with a target subspace. The paper argues that this approach is more likely to miss concept information distributed across context in T5. Even tokens that do not explicitly name the target can carry its semantics through bidirectional attention. The paper also evaluates SAFREE without negative prompting, distinguishing effective embedding intervention from suppression supplied by the subsequent sampling mechanism.

CGCE does not seek a fixed erasure direction that works for every encoder. Instead, it learns a differentiable decision boundary from prompt pairs with and without the target concept, then uses that boundary to refine the input. Core Idea: use the same concept classifier to decide which prompts need intervention and to supply token-level importance and gradients, placing erasure in the text conditioning rather than generator parameters.

Method

Overall Architecture

The input consists of a user prompt and a specified concept to erase; the output is the text embedding passed to the original generator. Offline, semantically similar prompt pairs differing in the target concept are constructed and used to train a concept-conditioned classifier. Online, the classifier scores the prompt embedding: unflagged embeddings pass through unchanged, while flagged embeddings undergo token-weighted refinement.

The process connects “Paired Prompt Construction,” “Concept-Conditioned Classifier,” and “Token-Weighted Refinement.” The generator in the diagram does not participate in classifier training, and its weights are not updated during refinement. Plug-and-play here means attaching a safeguard to a compatible text-embedding interface, not using one classifier with arbitrary encoders without adaptation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Paired Prompt Construction"] --> B["Concept-Conditioned Classifier"]
    P["Prompt and target concept<br/>Text encoding"] --> B
    B -->|Not detected| G["Frozen generator<br/>Image or video"]
    B -->|Detected| C["Token-Weighted Refinement"]
    C -->|Rescore| B
    C -->|Iteration limit reached| G

Key Designs

1. Paired Prompt Construction: center the decision boundary on the target concept rather than unrelated scenes

If positive and negative prompts describe entirely different scenes, a classifier can exploit backgrounds, people, or object categories as shortcuts. CGCE therefore uses a large language model to generate paired examples: one contains the target concept, while the other preserves similar content but removes that concept. The pairs span different scenarios. Each task uses 1000 prompt pairs and does not require synthesizing training images with the generator, concentrating training costs on text data and a lightweight classifier.

Pairing is a data-construction constraint rather than an additional pairwise loss. During training, the two prompts receive separate binary labels indicating concept presence or absence. The intended signal is that the scene remains similar while the target concept changes; synthetic data nevertheless cannot guarantee coverage of every implicit expression or out-of-distribution input.

2. Concept-Conditioned Classifier: preserve token-level associations before producing a sentence-level decision

The user prompt and target concept are encoded into their respective token embeddings and projected into a lower-dimensional space. Cross-attention uses prompt tokens as queries and concept tokens as keys and values. Each prompt token can therefore obtain its association with the target concept, rather than having a concept query immediately compress the entire sentence into one vector. This attention direction supports detection while retaining the information needed for localized refinement.

Let \(A\) denote the cross-attention weights. The classifier assigns each prompt token an importance score equal to its largest attention weight over concept tokens, \(s_i=\max_j A_{ij}\). A softmax over these scores supplies weights for aggregating the cross-attention outputs, after which an MLP and sigmoid produce a sentence-level concept probability. The two uses should be distinguished: classification pooling uses normalized weights, whereas the subsequent gradient weighting uses the token importance score \(s\) defined in the paper.

The default detection threshold is \(\tau=0.5\). If the probability does not exceed it, the original embedding goes to the generator unchanged. This bypass explains why many benign prompts retain their outputs, but “not modifying prompts classified as benign” does not imply “never modifying any benign prompt.” The reported false-positive rates on COCO-30K are 7.18% for CLIP-L/14 and 11.32% for T5-XXL; those false positives still enter refinement.

3. Token-Weighted Refinement: change conditioning along classifier gradients without updating the generator

When the classifier detects the target concept, CGCE freezes classifier and generator parameters and optimizes the prompt embedding itself. It first computes the gradient of the concept probability with respect to that embedding, then weights the gradient by token importance so that positions more strongly associated with the concept receive stronger updates. Compared with using raw gradients everywhere, this aims to reduce disruption to unrelated semantics. Importance is not a hard mask, so other tokens are not guaranteed to remain unchanged.

The update is then scaled by the ratio of the embedding norm to the weighted-gradient norm. Its magnitude is therefore controlled relative to the current embedding instead of being determined solely by the raw gradient scale. The following is a reconstruction based on the prose explaining Eq. (3); the original equation is corrupted in the local text extraction, so the original paper remains authoritative for the exact notation:

\[ g^{(k)}=s^{(k)}\odot\nabla_{\varepsilon^{(k)}}f_\theta(\varepsilon^{(k)},\varepsilon_c),\qquad \varepsilon^{(k+1)}=\varepsilon^{(k)}-\eta\frac{\|\varepsilon^{(k)}\|_2}{\|g^{(k)}\|_2}g^{(k)}. \]

Here, \(\varepsilon^{(k)}\) is the current prompt embedding, \(\varepsilon_c\) is the fixed concept embedding, \(\eta\) controls refinement strength, and \(\odot\) denotes elementwise multiplication broadcast over tokens. After each update, the classifier evaluates the embedding again. Refinement stops when the probability falls below the threshold or the iteration limit is reached, and the resulting embedding is passed to the generator. The paper states that one or two updates usually suffice, but the cached paper lacks the full algorithm in the appendix; a universal iteration cap or numerical-stability term cannot be supplied from this evidence.

The mechanism neither depends on diffusion denoising internals nor requires generating images before optimization, making it applicable to image and video models. The authors discuss classifier reuse within CLIP and T5 embedding interfaces separately. Transfer depends on text-representation compatibility, not deletion of concept knowledge inside the base model. Removing the upstream module leaves the original generator's capabilities intact.

A Worked Example

Consider the paper's “church” erasure task. Offline, construct prompt pairs with and without the target object while keeping the remaining scene similar, then train the classifier. At inference time, an architectural-scene prompt is encoded, and the classifier combines it with the target concept to calculate the concept probability and token importance.

If the probability does not exceed 0.5, the generator receives the original embedding. Otherwise, refinement weights the gradient by importance and uses the SD-v1.4 step size of 0.5 for this task. Once rescoring reaches a stopping condition, the original generator produces an image using the refined conditioning. This is an illustration of the mechanism: it does not prescribe a particular replacement building or treat passing the classifier as proof that erasure must succeed.

Loss & Training

The classifier uses binary cross-entropy, labeling prompts containing the target concept as 1 and corresponding prompts without it as 0. The low-dimensional projections, cross-attention module, and final classification layers are trained jointly. The text encoder supplies embeddings, and generator weights remain unchanged. “No generator-weight modification” should not be shortened to “no training at all.”

Each task uses 1000 synthetic prompt pairs and trains for 10 epochs with Adam, a learning rate of \(10^{-4}\), and a batch size of 32. On SD-v1.4, the refinement step sizes for content-safety, artistic-style, and object erasure are 1.0, 0.15, and 0.5, respectively. None should be treated as a universal cross-task default.

Key Experimental Results

Main Results

The main table evaluates content-safety concept erasure on SD-v1.4. ASR is the proportion of attack outputs that still contain the target content, so lower is better; this task uses NudeNet with a detection threshold of 0.45. On COCO-30K, lower FID and higher CLIP Score indicate better general generation utility, not stronger erasure. The following selection from Table 2 highlights the trade-off rather than treating ASRs from different benchmarks as equally difficult tests.

Method	I2P ASR ↓ (%)	SixCD ASR ↓ (%)	P4D ASR ↓ (%)	RAB ASR ↓ (%)	MMAD ASR ↓ (%)	UDA ASR ↓ (%)	FID ↓	CLIP ↑
Original SD-v1.4	36.41	82.13	75.74	97.19	63.60	97.18	16.79	31.31
STEREO	0.75	4.74	5.15	4.21	7.90	28.17	17.98	30.16
SLD-Max	15.68	46.46	40.81	64.21	35.60	21.83	26.63	29.20
SAFREE	12.35	39.18	48.90	74.74	44.70	80.99	19.55	30.69
CGCE	4.62	2.47	4.41	3.87	1.90	27.46	16.10	31.03

CGCE leads on SixCD, P4D, RAB, and MMAD, but not on every benchmark: STEREO is lower on I2P, and SLD-Max is lower on UDA. Relative to STEREO, CGCE lowers MMAD ASR by 6.00 percentage points, improves FID from 17.98 to 16.10, and raises CLIP from 30.16 to 31.03. This supports a better overall trade-off, not superiority against every attack.

Artistic-style erasure additionally measures changes to unrelated styles. The paper defines \(\mathrm{LPIPS}_d=\mathrm{LPIPS}_e-\mathrm{LPIPS}_u\): the first term measures changes to erased styles relative to the original SD output, and the second measures changes to unerased styles. CGCE obtains 0.43, 0.00, and 0.43 for the three measures, respectively, with 0.00% ASR on the ordinary style test. Under UDA, however, ASR remains 24.00% for Van Gogh style and 16.00% for the church object. A zero on the ordinary test does not establish irreversible erasure.

Across architectures, FLUX RAB ASR falls from 97.19% to 0.70%, versus 87.89% with SAFREE. Video evaluation uses frame-level ASR—the number of frames containing detected target content divided by all generated frames—not a whole-video failure rate. On Gen, Hunyuan scores 59.87% without intervention, 9.18% with T2VU, and 6.53% with CGCE. These results support interface transfer, but image ASR and video frame-level ASR are not directly comparable.

Ablation Study

The following results come from Tables 7 and 8, both using the SD-v1.4 content-safety task. The first two rows isolate token importance; the last two show the effect of step-size choices rather than the benefit of another independent module.

Config	P4D ASR ↓ (%)	RAB ASR ↓ (%)	MMAD ASR ↓ (%)	FID ↓	CLIP ↑
Without token importance	6.99	5.26	2.20	16.67	30.99
Full model, step size 1.0	4.41	3.87	1.90	16.10	31.03
Full model, step size 0.5	11.03	13.68	5.50	16.61	31.23
Full model, step size 1.5	1.10	0.00	0.10	16.96	30.64

Key Findings

• Token weighting does more than increase erasure strength. Removing it raises UDA ASR from 27.46% to 40.14% while worsening both FID and CLIP, indicating that localizing updates helps both erasure and utility.
• A larger step is not always better. A step size of 1.5 reduces RAB ASR to 0.00%, but relative to 1.0, FID rises from 16.10 to 16.96 and CLIP falls from 31.03 to 30.64.
• Detection thresholds are relatively insensitive, not numerically identical. In Table 9, moving the threshold from 0.25 to 0.75 changes I2P ASR from 4.62% to 4.73% and SixCD from 2.28% to 2.67%.

Highlights & Insights

• One classifier performs detection, token-association modeling, and refinement-direction computation, avoiding a separate embedding rewriter. The attention direction matters because it must retain prompt-token information for subsequent updates.
• The bypass gives utility preservation a concrete meaning: unflagged embeddings are not modified. This is easier to interpret than average quality alone, although its practical coverage still depends on false positives and false negatives.
• Intervening at the text interface gives diffusion, visual autoregressive, and video models a common treatment strategy. The experiments test different generation backends rather than establish permanent forgetting in internal representations.

Limitations & Future Work

• Erasure remains incomplete: UDA retains substantial residual ASR, and a low classifier score is not a formal output-safety guarantee. The method is an upstream safeguard, not proven unbypassable unlearning.
• Benign prompts may be falsely flagged and modified, particularly given the 11.32% COCO-30K false-positive rate for T5-XXL. Utility preservation has empirical support but does not mean every benign prompt retains its original output.
• Evidence primarily covers a single target concept per task and the listed attacks and generators. It does not establish effectiveness for arbitrary concept combinations, every encoder, or all new attacks. Video evaluation emphasizes frame-level safety detection without equally extensive quantification of temporal consistency and utility.
• The local cache contains the complete main paper and references but not the appendix cited in the text, and the extraction of Eqs. (1)–(3) is corrupted. Classifier width, attention-head count, iteration cap, and end-to-end runtime cannot be verified here; deployment reproduction still needs the complete implementation or appendix.

Related Work & Insights

• vs SAFREE: Both intervene on text conditioning outside the generator, but SAFREE relies on predefined-subspace projection, whereas CGCE learns a concept-conditioned classifier and refines embeddings iteratively. The latter adds classifier training and gradient computation in exchange for adapting to contextually distributed semantics; comparisons on SD-v3 must state the negative-prompting setting.
• vs STEREO / ESD: These methods change model weights, whereas CGCE leaves the generator itself unchanged. STEREO remains stronger on I2P, and CGCE offers a different trade-off between general utility and multiple attacks—not a universal replacement for earlier methods.
• vs T2VU: T2VU targets video concept erasure, while CGCE reuses a safeguard through the text-embedding interface. Current video ASR results support that transfer but do not establish superiority on every video-quality dimension.

Rating

• Novelty: 4/5. Concept-conditioned detection, token importance, and gradient refinement form a well-motivated upstream module, although its individual components are established.
• Experimental Thoroughness: 4/5. Multiple attacks, generation architectures, and key ablations are covered, but efficiency and video-utility details remain insufficiently verifiable in this cache.
• Writing Quality: 4/5. Motivation and ablations align clearly, although the claim that benign prompts remain unchanged requires the false-positive context, and corrupted equation extraction hinders offline checking.
• Value: 4/5. Useful where generator-weight modification is impractical, with residual attack success requiring cautious deployment boundaries.