Obliviate: Erasing Concepts from Autoregressive Image Generation Models¶

Conference: ECCV 2026
arXiv: 2606.28643
Code: None (the paper mentions an appendix but provides no repository link)
Area: Image Generation / AI Safety / Concept Erasure
Keywords: Concept Erasure, Autoregressive Image Generation, KL Distribution Supervision, Full-Trajectory Training, Teacher Guidance

TL;DR¶

This paper adapts the mature "negative guidance concept erasure" from diffusion models to autoregressive image generation models. By aligning conditional and pseudo-unconditional predictions via shared visual prefixes and applying KL distribution supervision over the entire token trajectory, it thoroughly erases concepts such as nudity, gore, and brand logos while causing minimal damage to model utility (e.g., reducing the nudity detection rate of RAB on Liquid from 91.58% to 3.15%).

Background & Motivation¶

As generative models become increasingly realistic, the barrier to misuse lowers, escalating the risk of generating harmful content such as nudity, gore, and copyrighted brands. A major response in the community is concept erasure: instead of retraining the entire model, the model weights are directly updated post-training to erase the ability to generate a specific harmful concept while preserving the generation quality of other normal content. This pipeline has matured significantly for diffusion models, where approaches like ESD, Ablating Concepts, and MACE have established the "frozen teacher + fine-tuned student" paradigm, which has even been incorporated into the release of cutting-edge models like FLUX.2.

However, text-to-image synthesis is recently undergoing an architectural resurgence. The autoregressive (AR) pipeline has regained popularity due to the demand for "vision-language unification." Models like Janus-Pro, Emu3, and Liquid use a single Transformer backbone to treat image synthesis as next-token prediction, naturally inheriting the scalability of large language models. The issue is that erasure methods are almost exclusively designed for diffusion models, leaving the autoregressive side virtually blank, despite being equally vulnerable to red-teaming prompts. Worse, diffusion paradigms cannot be directly transferred: the temporal momentum and global denoising features that diffusion erasure relies on are fundamentally different from the "token-by-token generation, token-sequential spatial encoding" mechanism of autoregressive models. The few existing attempts (e.g., EAR) compute the conditional and unconditional paths on different sampling trajectories. Consequently, since the two distributions diverge from the beginning, the image quality typically degrades before the concept is even erased. Furthermore, EAR only updates non-overlapping token windows and requires preparing a separate dataset for each concept.

The key insight of this paper is: since autoregressive generation is inherently a trajectory and "mismatched trajectory prefixes" are the root cause of divergence, one can let the teacher generate a target concept trajectory, feed this same trajectory simultaneously to both the conditional and pseudo-unconditional paths, use their difference to construct the erasure target, and track distribution-level supervision over the entire trajectory. The core idea of this paper is to align the conditional and pseudo-unconditional paths with a shared visual prefix and perform full-trajectory KL distribution supervision over the complete autoregressive rollout, stably migrating diffusion negative guidance to concept erasure in autoregressive image generation.

Method¶

Overall Architecture¶

Obliviate follows the "frozen teacher + trainable student" erasure framework but adapts three components for autoregressive image generation. Given a target concept prompt to be erased (e.g., "Coca-Cola logo"), the workflow is: ① The frozen base model (e.g., Liquid) acts as the teacher to first sample a complete harmful image token trajectory \(\hat{\mathbf{x}}=\{\hat{x}_1,\dots,\hat{x}_N\}\) using the target prompt. ② Using this same trajectory as a shared prefix, the teacher predicts the logits at each position under both "conditional" (with target prompt \(\mathbf{c}\)) and "pseudo-unconditional" (with empty prompt \(\varnothing\)) setups. The difference between the two is subtracted and scaled by negative guidance to construct a target distribution \(p_{\mathrm{tgt}}\) that suppresses the concept. ③ The student (a LoRA copy of the teacher) predicts the distribution at the same positions under the target prompt, and KL divergence is used to pull the student distribution toward the target distribution over the entire trajectory. These three adaptations correspond to three key designs: visual prefix alignment, full-trajectory updates, and KL distribution supervision.

These three adaptations are necessary because "translating diffusion ESD directly to autoregressive models" fails. In diffusion, the negative guidance target compares the conditional and unconditional noise predictions at a certain denoising step \(t\) (Eq. 3). If copied directly to autoregressive models, the most natural way is to compare conditional/unconditional next-token predictions at position \(k\)—but if the two paths are sampled independently without aligned prefixes (denoting conditional prefix as \(\hat{\mathbf{x}}_{<k}\) and unconditional prefix as \(\bar{\mathbf{x}}_{<k}\)), the two logit distributions will severely diverge (red line in paper Fig 2a), damaging the model's generation capability before the concept is even erased. Obliviate is designed precisely around eliminating this divergence.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Target concept prompt c<br/>(e.g., Coca-Cola logo)"] --> B["Visual Prefix Alignment<br/>Teacher samples a harmful trajectory with c<br/>to serve as the shared prefix for both paths"]
    B --> C["Conditional prediction z(x̂,c)<br/>Pseudo-unconditional prediction z(x̂,∅)"]
    C -->|Subtracted + Negative Guidance η| D["Full-Trajectory Update<br/>Construct target distribution p_tgt along the complete rollout"]
    D --> E["KL Distribution Supervision<br/>Align student distribution to p_tgt"]
    E --> F["Erased Student Model<br/>(LoRA weights)"]

Key Designs¶

1. Visual Prefix Alignment: Comparing conditional and pseudo-unconditional paths under the same visual context

This is the foundation of the paper, directly addressing the failure mode where independent sampling paths cause distribution divergence and image degradation. In prior work (EAR), the conditional and unconditional prefixes originate from two independent rollouts, so they are not in the same visual context. Consequently, the two logit distributions do not align from the start, and the contrastive signal of negative guidance gets contaminated with noise unrelated to the target concept (simply representing differences between the two trajectories). Obliviate solves this directly: the teacher first samples a harmful trajectory \(\hat{\mathbf{x}}\) under the target prompt \(\mathbf{c}\), and then reuses this trajectory as the shared prefix for both conditional and pseudo-unconditional paths. The "pseudo-unconditional" path feeds empty prompt \(\varnothing\) to predict the next token given this prefix (which already contains harmful structures). Therefore, the target signal at position \(k\) is formulated as:

\[z_{\mathrm{tgt}}^{(k)}=z_{\theta^{\ast}}(\hat{\mathbf{x}}_{<k},\varnothing)-\eta\big(z_{\theta^{\ast}}(\hat{\mathbf{x}}_{<k},\mathbf{c})-z_{\theta^{\ast}}(\hat{\mathbf{x}}_{<k},\varnothing)\big),\qquad p_{\mathrm{tgt}}^{(k)}=\mathrm{softmax}(z_{\mathrm{tgt}}^{(k)})\]

where \(\eta>0\) is the negative guidance strength (guidance scale), and \(\theta^{\ast}\) is the frozen teacher. Comparing this to independent sampling reveals the key difference: the prefix in both paths is now the same \(\hat{\mathbf{x}}_{<k}\), rather than splitting into \(\hat{\mathbf{x}}_{<k}\) and \(\bar{\mathbf{x}}_{<k}\). The shared prefix brings two major benefits: first, stability, since the predictions are compared within the same visual context, reducing the token-by-token distribution discrepancy (blue line in Fig 2a) compared to the split paths (red line); second, accuracy, because when feeding the empty prompt, the teacher tends to drift towards neutral content even if early harmful structures exist in the prefix. On the other hand, feeding \(\mathbf{c}\) continues to reinforce the harmful trajectory. The difference between the two precisely highlights the tokens that actually maintain the target concept, allowing the weight updates to focus on concept-related parts without disrupting other semantic aspects.

2. Full-Trajectory Update: Supervising every position in a single rollout instead of targeting single tokens

Diffusion ESD updates on a single sampled timestep at a time (local supervision). However, autoregressive generation is inherently a trajectory, where harmful concepts accumulate through a chain of dependent token predictions rather than appearing at an isolated position. Additionally, the causal mask naturally allows parallelized supervision across all positions in a single forward pass. Therefore, Obliviate expands the single-step objective to a full-trajectory objective, which conceptually averages the token-wise KL divergence over the entire rollout of length \(N\) (as in Eq. 9). This is more aligned with the nature of concepts unfolding along the generation chain and is more efficient: a single sampled trajectory can provide training signals for every token position. The paper's ablation is intuitive: when updating only isolated tokens, almost no erasure occurs in the first 40 steps, whereas full-trajectory updates successfully erase the concept within the first 20 steps (Fig 2b middle/bottom rows).

3. KL Distribution Supervision: Matching the entire prediction distribution instead of strictly targeting single tokens

Full-trajectory updates have a side effect: most tokens in a generation do not actually carry the target concept. Using standard cross-entropy to strictly fit a single target token at each position would lead to aggressive updates on a large number of unrelated general visual tokens, destroying model utility (which is why prior works avoided full-trajectory updates). Here, the authors exploit a key difference between image tokens and language tokens: while a language token strongly constrains subsequent outputs, image token prediction is often multimodal—there are many "visually synonymous" token continuations that share similar local semantics but differ in precise token values. Thus, supervising the "entire distribution" is more appropriate than supervising "single tokens": it automatically redistributes the probability mass more evenly and downweights low-information regions, preventing over-penalization of isolated target tokens. Specifically, the student is trained to match the teacher's induced target distribution using KL divergence:

\[\mathcal{L}_{\textsc{Obliviate}}=\frac{1}{N}\sum_{k=1}^{N}D_{\mathrm{KL}}\big(p_{\mathrm{tgt}}^{(k)}\,\|\,p_{\theta}^{(k)}\big),\qquad p_{\theta}^{(k)}=\mathrm{softmax}\big(z_{\theta}(\hat{\mathbf{x}}_{<k},\mathbf{c})\big)\]

In terms of effect, the student learns to shift probability mass away from continuations related to the harmful concept and redistribute it to safer alternatives. This is particularly advantageous for brand erasure: brand logos are typically composed of robust local visual patterns (simple color schemes, basic shapes) where multiple token combinations can render similar logos. Suppressing only the most probable token leaves synonymous tokens that still leak the brand. By acting on the entire distribution combined with negative guidance, KL suppresses both the main brand tokens and nearby alternative tokens that produce similar symbols. Consequently, the model does not just slightly perturb the Coca-Cola logo but completely avoids the classic red-and-white design to generate a generic can instead.

Loss & Training¶

The final training objective is the full-trajectory KL loss \(\mathcal{L}_{\textsc{Obliviate}}\) (Eq. 9) shown above, which combines the negative-guidance target distribution (Eq. 7), trajectory averaging, and KL distribution matching. Implementation-wise, training is conducted entirely via LoRA fine-tuning (rank 32, \(\alpha=16\), 5% dropout) where all three models only update their respective LoRA-compatible layers (excluding modules like visual encoders and projection layers). The negative guidance strength \(\eta\) is the main hyperparameter: Liquid is fixed at \(\eta=2\), Emu3-Gen uses \(\eta=1\), and Janus-Pro dynamically adjusts based on the scenario (\(\eta=10\) is a robust default). Training for nudity/gore scenarios takes around 400–1000 steps, while the brand scenario converges extremely fast (only 30 steps on Liquid), suggesting that localized concepts are much easier to erase than diffuse concepts.

Key Experimental Results¶

Evaluation is performed on three autoregressive text-to-image models: Liquid-7B, Emu3-Gen, and Janus-Pro, covering three concept categories: nudity, gore, and brand (Coca-Cola). The core metric is the Concept Detection Rate (CDR, ↓): a specialized classifier determines whether the target concept appears in the generated images, and CDR represents the proportion of images containing the target concept. Nudes are detected using NudeNet, gore using Q16, and brand using a majority vote of three open-source VLMs (Qwen2.5-VL, LLaVA-1.5, Phi-3.5-Vision). Utility is measured by FID (↓) and CLIP-Score (↑).

Main Results¶

Nudity erasure (Table 1, showing snippets of Liquid and Janus-Pro). Obliviate suppresses the CDR to the lowest levels while keeping FID intact, whereas baseline methods either fail to erase thoroughly or severely damage model utility:

Model	Method	T2I-RP↓	RAB↓	MMA-Diff↓	FID↓	CLIP↑
Liquid	Original	45.82	91.58	20.30	14.24	13.06
Liquid	Negative Prompt	17.67	35.79	7.60	16.24	13.13
Liquid	SFT	27.26	45.26	16.20	14.60	13.05
Liquid	Obliviate	3.73	3.15	2.80	15.41	13.10
Janus-Pro	Original	62.08	55.79	18.70	12.39	13.16
Janus-Pro	EAR	28.33	11.58	0.80	31.63	13.16
Janus-Pro	Obliviate	18.11	1.05	1.00	12.31	13.35

Brand erasure (Table 2b) represents Obliviate's most outstanding scenario, reducing CDR almost to zero: Liquid drops from 94.60 to 5.22, Emu3-Gen from 98.74 to 4.14, and Janus-Pro from 87.77 to 0.18, with Janus-Pro's FID actually improving compared to the original model (12.39 → 11.76). Gore erasure (Table 2a) is the hardest task—only dropping from 94.74 to 77.83 on Janus-Pro. The authors acknowledge that gore is much harder to erase than nudity, yet Obliviate still achieves the largest drops across all three models.

Ablation Study¶

The paper conducts three sets of ablations. The most representative ones justifying the design choices are "KL vs Cross-Entropy" (Table 4, Liquid nudity scenario) and "Guidance Strength \(\eta\)" (Table 3a):

Configuration	RAB↓	MMA↓	FID↓	Description
CE (Cross-Entropy token supervision)	5.26	4.40	23.24	Reasonable erasure, but full-trajectory training causes severe distribution drift, hurting FID to 23.24
KL (Distribution supervision, full)	3.15	2.80	15.41	Better erasure with significantly lower FID; KL acts as an implicit regularizer
\(\eta=1.0\) (Liquid)	1.05	5.20	14.72	Weak guidance
\(\eta=2.0\) (Liquid, Chosen)	3.15	2.80	15.41	Erasure-utility trade-off point
\(\eta=4.0\) (Liquid)	4.21	3.20	16.48	Overly strong guidance, raising FID

Key Findings¶

The primary value of KL distribution supervision is preserving utility rather than harsher erasure: Table 4 shows that while KL is only slightly better at erasure than CE on some benchmarks, it pulls the FID back from 23.24 to 15.41. This is because KL acts as an implicit regularizer over non-concept regions, preventing over-updating on general tokens during full-trajectory training, directly validating the motivation for Design 3.
Prefix alignment is key to preventing collapse: Fig. 2b reveals that unaligned prefixes destroy generation capability before the concept (e.g., Coca-Cola) can be erased. Unifying prefix alignment with full-trajectory updates is crucial for "fast erasure without collapse." EAR's FID spikes to 31.63 on Janus-Pro while Obliviate remains at 12.31, which the authors attribute to the more faithful guidance signals provided by the shared visual prefix.
Concept type (diffuse vs. local) dictates how prompts should be written (Table 3b): For semantically diffuse concepts like nudity, using a "detailed prompt" is more effective as it covers a wider boundary of related semantics. Conversely, localized concepts like Coca-Cola are much better handled by a "simple and precise prompt" (calling it rawly "Coca-Cola logo") than detailed descriptions. On Liquid, a detailed prompt for brands causes the CDR to bounce back to 54.50.
Gore is the hardest to erase, and multi-concept joint erasure degrades performance: Gore is highly challenging to suppress due to its visual diversity and lack of fixed structures. For multi-concept settings, erasing 1–2 concepts can push CDR near 0, but as the number of concepts reaches 3/4/5, the average CDR increases to 12.20/25.05/47.56, and GenEval drops from 79.38 to around 74–75, revealing a loss in combinatorial generation capability.

Highlights & Insights¶

"Shared visual prefix" is the most clever design choice: The root cause of failure when migrating negative guidance to autoregressive models is precisely identified as "mismatched trajectory prefixes leading to distribution divergence." The solution is simple: "reusing the same teacher trajectory as the shared prefix for both paths." With virtually zero extra cost, it addresses both "stability" and "concept localization" simultaneously, serving as a general trick transferrable to any autoregressive erasure/editing task.
Deducing KL's superiority over CE from the "multimodal nature of image tokens": Instead of making a generic claim that "distribution supervision is smoother," the authors point out that image token prediction features numerous visually synonymous continuations. Relying strictly on single-token targets leaves out synonymous alternatives (especially in brand scenarios). This observation explains the superiority of KL from a mechanistic perspective.
A strong dedication to preserving utility: While many erasure works only report erasure metrics, this paper continuously emphasizes FID/CLIP against a baseline like EAR (with FID=31.63), reminding the community that "effective erasure" and "utility preservation" are two dimensions that must be evaluated simultaneously.

Limitations & Future Work¶

Robustness is only measured against existing red-teaming benchmarks: The authors acknowledge that they did not test against adaptive white-box attacks (which have access to model parameters/gradients), so the robustness of the erasure against targeted counter-erasure attacks remains unconfirmed.
Performance degrades under multi-concept scaling: Relying on adapter fusion for joint erasure harms performance as the number of concepts scales up (with an average CDR of 47.56 at 5 concepts) and damages combinatorial generation capabilities (GenEval). An elegant multi-concept solution is yet to be developed.
Erasure of diffuse concepts like gore remains incomplete: The gore CDR on Janus-Pro is still at 77.83, far from "completely clean."
Value is tied to the choice of model architecture: The authors candidly state that if diffusion remains the dominant paradigm for text-to-image synthesis, the demand for specialized autoregressive erasure methods will narrow. The practical significance of this method depends on whether autoregressive/unified multimodal architectures become mainstream.

vs. EAR (the closest competitor in autoregressive erasure): EAR translates diffusion guidance distillation to autoregressive models, but computes the conditional/unconditional paths on unaligned, independent trajectories, updates only on non-overlapping token windows, and requires a dedicated dataset for each concept. Obliviate aligns both paths using a shared prefix, performs parallel full-trajectory updates, and requires no specialized datasets. Empirically, Obliviate vastly outperforms EAR in utility preservation (Janus-Pro FID 12.31 vs. 31.63).
vs. ESD (the source of the diffusion erasure paradigm): ESD applies negative guidance updates to a single sampled denoising step (with inherently local supervision). Obliviate retains the negative guidance objective (Eq. 7) but substitutes "single-step" with "full-trajectory", and "noise MSE loss" with "logits KL loss," adapting to the discrete tokens and trajectory-based generation of autoregressive models.
vs. EraseFlow (the inspiration for full-trajectory training): EraseFlow demonstrated that in diffusion models, erasing along the complete denoising trajectory is superior to point-wise updates. Obliviate takes this trajectory perspective and migrates it to autoregressive models—where the autoregressive generation process itself is inherently trajectory-based and the causal mask naturally permits parallelized supervision across all positions, making this migration even more natural and efficient than in diffusion models.
vs. SLD / Negative prompts (inference-time guidance): These methods only modify logits during inference without updating weights, making them easy to bypass. Obliviate permanently alters the weights, providing stronger robustness. The paper also adapts SLD to autoregressive models (\(SLD^{\ast}\)) as a baseline, finding that warm-up and momentum must be removed (since autoregressive models generate spatially token-by-token and lack a shared temporal state), and the guidance scale must be scaled down considerably.

Rating¶

Novelty: ⭐⭐⭐⭐ Among the first to systematically tackle concept erasure in autoregressive image generation. The three adaptations are highly targeted, though the core concept (negative guidance + full-trajectory) is migrated from mature diffusion paradigms.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across three models, three concepts, and multiple red-teaming benchmarks, with comprehensive ablations on KL/CE, \(\eta\), prompts, multi-concept scenarios, Van Gogh style, and VLM evaluator reliability.
Writing Quality: ⭐⭐⭐⭐⭐ Clear motivational steps tracing "why naive migration fails" down to the three design choices, with highly convincing failure mode visualization in Figure 2.
Value: ⭐⭐⭐⭐ Fills a crucial gap in autoregressive concept erasure with a plug-and-play approach (LoRA), though its ultimate impact heavily depends on whether autoregressive/unified multimodal architectures become mainstream in the future.