Achieving Subcategorical Erasure in Text-to-Image Models¶

Conference: ECCV 2026
Authors: Pranav Singh Chib, Pravendra Singh
Paper: ECCV official page · PDF
Area: Image Generation / Diffusion Models / Concept Erasure
Keywords: Subcategorical unlearning, neutral preservation, adversarial concept retrieval, Lipschitz regularization

TL;DR¶

SURE starts from a single parent-category target, maps that category and retrieved related concepts toward an empty prompt, preserves neutral predictions, and regularizes local latent variation, reaching a 10.00% subcategory detection rate on Guns without establishing complete erasure of every subcategory.

Background & Motivation¶

Concept erasure fine-tunes a text-to-image model to suppress selected content without retraining it from scratch. However, removing the ability associated with one name does not necessarily remove all of its specific instances. A model trained to forget a parent category may still respond to prompts naming its children. The paper uses WordNet 3.1 hypernym and hyponym relations to distinguish categories from subcategories: flower is a parent of rose, tulip, and daisy. Covering synonyms is not the same as covering this semantic hierarchy.

Enumerating every child would turn the problem into multi-concept erasure, but such lists can be large and incomplete. More aggressive editing also risks damaging unrelated generation. Explicitly retaining a large collection of unrelated image-text pairs creates another enumeration problem, since a pretrained model's semantic coverage exceeds any convenient hand-built preservation set. The intended interface therefore avoids requiring users to specify either every unwanted child or every unrelated category that should survive.

SURE uses neutral conditioning as a shared reference: unwanted predictions move toward it, while neutral predictions themselves stay close to the original model. Adversarial retrieval and local smoothing then extend the effect beyond the parent label. Core idea: extend parent-category erasure by finding related concepts that remain distinct from the neutral reference and smoothing responses around target-generated images, while anchoring empty-prompt behavior to limit distribution drift.

Method¶

Overall Architecture¶

SURE takes a pretrained diffusion model, one parent-category target, and a candidate concept space, and produces an updated erasure model. The original model provides reference noise predictions. Four signals constrain the updated model: Parent-to-Neutral Mapping, Neutral Prediction Preservation, Adversarial Subcategory Retrieval with an associated erasure objective, and Lipschitz Local Smoothing.

These are not four separately trained models chained at inference. They jointly constrain one model during fine-tuning. The diagram shows their dependencies: retrieval seeks candidates whose current predictions still differ from the neutral reference, whereas model updating reduces that difference. The smoothing branch starts from a target-conditioned generated image and obtains its regularization signal through perturbation and fixed VAE encoding.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Parent label, original model<br/>and candidate concept space"] --> Erase["Parent-to-Neutral Mapping"]
    Input --> Preserve["Neutral Prediction Preservation"]
    Erase --> Retrieve["Adversarial Subcategory Retrieval"]
    Erase --> Smooth["Lipschitz Local Smoothing"]
    Preserve --> Update["Joint erasure-model update"]
    Retrieve --> Update
    Smooth --> Update
    Update --> Output["Evaluate child residuals<br/>and unrelated capabilities"]

Key Designs¶

1. Parent-to-Neutral Mapping: give unwanted predictions a concrete destination

Asking a model not to generate a category does not by itself define a learning direction. SURE supplies one using an empty string as the neutral concept. At the same diffusion state and timestep, the updated model's parent-conditioned noise prediction is trained toward the original model's empty-prompt prediction. The first loss measures their squared L2 distance.

The alignment acts on denoiser predictions; it does not replace final pixels with a blank image or merely remove a keyword from the input. Empty conditioning represents the original model's behavior without the target text guidance, so the objective weakens the parent-specific generation direction. Keeping the original model as a reference also prevents the updated model from satisfying the objective simply by changing what its own neutral prediction means.

2. Neutral Prediction Preservation: constrain drift without enumerating retained categories

The parent-mapping loss alone could be satisfied through broad changes that also damage non-target content. The second loss therefore matches the updated and original models' noise predictions under the same empty prompt. Its conditioning differs from the first loss: the first compares the new model's parent prediction with the old model's neutral prediction; the second compares both models' neutral predictions.

This shared reference avoids explicitly listing a large preservation set and limits distribution drift during parameter updates. It is a practical regularizer, not a mathematical guarantee that every non-target concept survives. Consequently, the paper still measures non-target object detection, a COCO distributional metric, and similarity for retained artistic styles. A small neutral-prediction loss cannot substitute for those behavioral checks.

3. Adversarial Subcategory Retrieval: target candidates that resist neutralization

A parent label is an entry point rather than an exhaustive child list. SURE searches the candidate concept space outside the designated erasure set for related concepts whose current predictions differ strongly from the original model's neutral output. Candidate selection maximizes this discrepancy. Updating the erasure model minimizes the discrepancy for the selected candidate, removing another residual generation direction. The opposite optimization directions concern different variables and should not be described as the model maximizing and minimizing the same objective simultaneously.

To optimize discrete selection, the method learns an embedding distribution and uses low-temperature Gumbel-Softmax to approximate a hard choice. Pursuing one difficult candidate can provide a stronger erasure signal than averaging several candidates, and Table 5 supports this operator choice empirically. Nevertheless, a single parent as user input does not remove dependence on a candidate space. The main text does not fully specify its construction, relatedness constraints, or search schedule, so maximum discrepancy alone cannot establish exhaustive retrieval of genuine hyponyms.

4. Lipschitz Local Smoothing: reduce latent variation near target-generated images

Mapping a finite set of text conditions can leave nearby generation variants intact. SURE adds small Gaussian perturbations to an image generated under the target category, encodes the original and perturbed images with the first-stage VAE, and penalizes latent change relative to input perturbation size. Multiple perturbation samples estimate this local sensitivity. Reducing it extends the constraint beyond one exact input rather than relying only on isolated prompt mappings.

The perturbations are applied to images, not by replacing prompt words with synonyms. Nearby latent representations also do not enumerate a semantic hierarchy. The paper treats the VAE encoder as fixed, but the available main-text cache lacks sufficient sampling and gradient-path details to reconstruct the implementation fully. Smoothing alone does not identify what must be forgotten: it must work with neutral mapping, otherwise a model could become insensitive to perturbations while still generating the target.

A Worked Example¶

Consider the paper's Musical Instruments category. During fine-tuning, the user supplies that parent rather than explicitly listing the five evaluation children: Trumpet, Guitar, Drums, Saxophone, and Piano. Parent-conditioned predictions move toward the original empty-prompt predictions while the updated empty-prompt behavior is preserved. Candidate retrieval adds related concepts that remain far from neutral, and the image-perturbation branch discourages local sensitivity around the target.

Evaluation uses 50 prompts for each of the five children, totaling 250 prompts, and checks non-target behavior on ten CIFAR-10 classes. SURE reaches an 8.40% residual target detection rate and 66.40% locality accuracy for this parent. Those results show suppression of the evaluated children, not that training explicitly recovered these five names, and certainly not zero residual generation.

Loss & Training¶

The joint objective combines parent mapping, neutral preservation, related-concept erasure, and Lipschitz regularization. Their comparison targets and optimization directions are described above. Equations (5) through (9) are damaged in the extracted cache, so this note does not reconstruct their full expressions or invent missing coefficients, optimizer settings, or training steps.

The best tested Guns configuration uses five perturbation samples, Gaussian standard deviation 0.10, Lipschitz strength 0.01, and Gumbel-Softmax. These are empirical settings for that evaluation, not universally optimal values. Performance is non-monotonic across 1, 3, 5, and 7 samples. The paper attributes deterioration at larger sample counts to oversmoothing, but also describes the term as a Monte Carlo estimate of an expectation; its actual normalization and weighting need verification from supplementary material or code.

Key Experimental Results¶

Main Results¶

Object evaluation covers four parents with five children each and 50 prompts per child. Preservation uses ten non-target CIFAR-10 classes with 50 paraphrased prompts per class. GroundingDINO checks whether the requested object appears. \(\mathrm{Acc}_E\) is the percentage of images retaining a target subcategory, so lower is better; \(\mathrm{Acc}_L\) is non-target accuracy, so higher is better. The reported harmonic mean combines erasure and preservation, with both inputs expressed as percentages:

\[ H=\frac{2(100-\mathrm{Acc}_E)\mathrm{Acc}_L}{(100-\mathrm{Acc}_E)+\mathrm{Acc}_L}. \]

The following selection from Table 1 uses ESD as a consistent comparator alongside SURE rather than selecting the weakest baseline for each column. H retains the source table's rounding.

Parent	Method	AccE, lower better	AccL, higher better	H, higher better
Guns	ESD	52.80	66.20	55.11
Guns	SURE	10.00	67.20	76.94
Blade Weapons	ESD	18.80	68.60	74.37
Blade Weapons	SURE	2.40	64.60	77.74
Musical Instruments	ESD	51.60	68.80	56.82
Musical Instruments	SURE	8.40	66.40	76.99
Toys	ESD	49.20	66.20	57.48
Toys	SURE	14.80	66.40	74.63

Ablation & Sensitivity Analysis¶

These Guns results come from Tables 4 through 7. Each group is a separate component ablation or parameter sweep; rows from different groups must not be interpreted as cumulative removals.

Group	Configuration	AccE, lower better	AccL, higher better
Full configuration	SURE	10.00	67.20
Component ablation	Without Lipschitz	25.20	65.80
Component ablation	Without preservation loss	30.00	64.80
Perturbation samples	n = 1	19.60	65.40
Perturbation samples	n = 3	30.40	67.00
Perturbation samples	n = 7	27.60	66.60
Retrieval operator	Straight-Through Estimator	44.80	67.20
Retrieval operator	Entmax	38.40	67.00
Retrieval operator	Sparsemax	46.80	64.20
Perturbation standard deviation	sigma = 0.01	20.00	63.60
Perturbation standard deviation	sigma = 0.30	11.20	65.00
Regularization strength	lambda = 0.1	41.60	66.40
Regularization strength	lambda = 1	22.32	65.80

Cross-Task Analysis¶

The style experiment uses Stable Diffusion v1.4 and five artist names. Detailed prompts and five random seeds produce 200 images per artist. This selection from Table 3 reports lower target CLIP as weaker agreement with the target prompt, higher target LPIPS as greater change from original-model images, and higher retained CLIP as stronger prompt agreement for non-erased artists. Higher LPIPS alone does not establish higher visual quality.

Method	Target CLIP, lower better	Target LPIPS, higher better	Retained CLIP, higher better
ESD	23.56 ± 4.73	0.72 ± 0.11	29.63 ± 3.57
MACE	27.96 ± 4.22	0.60 ± 0.10	31.52 ± 2.91
AP	21.57 ± 5.46	0.78 ± 0.10	30.13 ± 3.44
SURE	18.02 ± 5.96	0.85 ± 0.10	30.91 ± 3.45

The paper also evaluates content-safety erasure on 4,703 I2P prompts using a detector-thresholded exposure rate as its residual metric. On COCO-30K, SURE obtains FID 14.02 versus AGE's 14.20. This extends evaluation beyond objects, but its task definition differs from WordNet subcategory coverage and should not be treated as identical generalization evidence.

Key Findings¶

SURE has the highest H for all four parents in the full main table, but preservation is not uniformly best: Blade Weapons AccL falls from ESD's 68.60 to 64.60. The improvement concerns the combined trade-off, not strict dominance on both axes.
Removing Lipschitz increases AccE by 15.20 percentage points; removing preservation increases it by 20.00 points. Preservation can also support erasure, plausibly by stabilizing the reference and optimization, although the ablation does not establish a unique causal explanation.
Retrieval operators and regularization strength materially affect results. Some prose claims about relative gains cannot be reproduced from the tables using conventional baseline-relative changes. This note therefore uses table values and explicit percentage-point differences instead.

Highlights & Insights¶

The evaluation shifts from forgetting a supplied name to covering its semantic children. That change makes semantic coverage, rather than adaptation to a training word, central to the task.
Neutral predictions serve as both an erasure destination and a preservation reference, but the two losses compare different conditions. This distinction is more transferable to model editing than a generic instruction to add preservation regularization.
Adversarial retrieval addresses residual concept conditions, whereas image-based local smoothing addresses an input neighborhood. They are complementary empirical mechanisms, not interchangeable proofs of semantic completeness.

Limitations & Future Work¶

The authors identify stronger non-target preservation as future work. The tables likewise show that winning a combined metric does not imply the best preservation score in every setting.
This note's assessment: four parents with five children each cannot establish removal of every child under an arbitrary parent. Residual target rates remain nonzero, and the detector can miss or misidentify objects. Deeper hierarchies, long-tail children, and human verification are needed.
The candidate-space source, relatedness constraints, search updates, and gradient path for regularization through a fixed VAE cannot be fully reconstructed from this main-text cache. Mentioned supplementary experiments on Stable Diffusion 2 and multi-subcategory erasure are not present, so their results are not reported here.
Lower style CLIP and higher LPIPS can reflect content drift or general image changes, not only removal of style identity. Content-preservation evaluation, blinded human assessment, and confidence intervals across seeds would strengthen the evidence.
The Lipschitz term is a finite-sample local smoothness regularizer, not an established global Lipschitz bound, guarantee of irreversible forgetting, or complete adversarial robustness result. Clear gradient implementation and dedicated attack evaluation are appropriate next steps.

Versus ESD / TIME: SURE inherits the idea of mapping a target toward a neutral concept. Its emphasis is parent-to-child coverage with retrieval and smoothing, not the first use of neutral mapping.
Versus UCE / MACE: The main text discusses explicit preservation, batch concept editing, and synonym-based evaluation in connection with these methods. SURE reduces user enumeration but still depends on a candidate space; avoiding a hand-written child list does not mean requiring no additional knowledge.
Versus AGE / AP: Adversarial concept search builds on these output-based erasure approaches, combined here with category-level evaluation and Lipschitz regularization. The Gumbel-Softmax ablation supports the selection mechanism but does not establish independence from candidate quality.
Versus RACE / AdvUnlearn: These methods directly address adversarially robust erasure. SURE's subcategory tests and local image perturbations are not equivalent to a comprehensive prompt-attack benchmark, so they cannot establish that robustness is solved.

Rating¶

Novelty: 4/5. Parent-driven subcategory erasure is a useful target, with a focused combination of existing neutral mapping, adversarial retrieval, and local regularization.
Experimental Thoroughness: 3/5. Three task families and several ablations provide evidence, but category coverage, detector dependence, and incomplete implementation details constrain the conclusions.
Writing Quality: 3/5. The motivation and component roles are clear, while some gain claims and smoothing explanations need scrutiny; damaged cached equations are a separate source-readability issue.
Value: 4/5. A useful baseline for hierarchical semantic unlearning and erasure-preservation trade-offs, not a guarantee of complete deletion or deployment safety.