IREU: Identity-Related Encoder-Only Unlearning for Customized Portrait Generation

Conference: ECCV2026
arXiv: 2606.29880
Code: To be confirmed
Area: AI Safety
Keywords: Identity Unlearning, Customized Portrait Generation, Encoder Fine-Tuning, Face-Swap, Privacy Protection

TL;DR

IREU introduces the identity unlearning problem for Customized Portrait Generation (CPG) for the first time. By leveraging Face-Swap to localize identity-related dimensions in the embedding space and executing feature perturbations strictly along these dimensions, it establishes an unlearning pipeline that only updates the image encoder to erase target identity generation capabilities while preserving the fidelity of other identities. Furthermore, the unlearned encoder can be transferred to different CPG generators with zero fine-tuning.

Background & Motivation

Customized Portrait Generation (CPG) technology has advanced rapidly in recent years. Pipelines represented by PhotoMaker and FastComposer utilize pretrained image encoders to extract identity embeddings from input portraits, which are then fed into diffusion models like Stable Diffusion to accomplish text-guided high-fidelity portrait editing (e.g., attribute modification, identity blending, and scene recontextualization). However, this convenience also introduces severe privacy risks: malicious users can leverage publicly accessible CPG generators to forge portraits of others for impersonation, deepfakes, and even social trust disruption.

Existing defenses against this risk mainly operate at the image level, which inject human-imperceptible adversarial perturbations into user images to prevent models from generating high-quality counterfeits. However, the premise of these methods is that all images of the target identity must be protected, which is almost impossible in reality because uncontrolled sources like already circulated old photos or candid shots cannot be covered. More importantly, according to regulations such as GDPR, model service providers are obligated to erase a user's "generative capability" upon request. This necessitates unlearning within the model rather than merely adding noise on the input side.

The authors thus focus on generative unlearning at the model level. However, existing generative unlearning methods (e.g., BIA) fine-tune the U-Net or denoising network, closely coupling unlearning with a specific generator (denoiser) and requiring re-training whenever a generator changes. Moreover, the parameter size of U-Net is massive, containing rich generative prior knowledge; erasing identity information within its parameter space easily triggers catastrophic forgetting, leading to degraded quality across all generated images. The authors observe that mainstream CPG pipelines share the same encoder family (CLIP ViT) despite using different diffusion backbones. If unlearning occurs solely on the encoder side, an "unlearned" encoder can be plug-and-play across any generator sharing this encoder. Core Idea: Update only the image encoder (encoder-only). By leveraging Face-Swap to localize the embedding space dimensions that specifically carry identity information and performing feature perturbations exclusively along these dimensions, the target identity is precisely erased while preserving the fidelity of other identities and enabling cross-generator transferability.

Method

Overall Architecture

IREU is an encoder-only identity unlearning framework whose workflow consists of two main stages. First stage (offline localization): Face-Swap is used to generate a face-swapped portrait for the target identity image, where only the identity is replaced. By computing the difference between their embeddings from the original encoder, sorting the components by their absolute values in descending order, and selecting the top-k dimensions, a binary mask is constructed to isolate identity-related features. Second stage (online unlearning with frozen diffusion models): For target identity images, controllable directional shifts are applied along the dimensions selected by the mask to construct "virtual identity" embeddings as optimization targets, minimizing the distance between the unlearned encoder's output and these virtual embeddings. For other identities to be retained, the output discrepancy between the unburned encoder and the original encoder is minimized. During inference, the original encoder is directly replaced with the unlearned encoder, keeping the diffusion model completely frozen.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Target Identity Image x_-"] --> B["Face-Swap<br/>x_f = F(x_-, x_a)"]
    B --> C["Encoder ε Extracts Embeddings<br/>z_- vs z_f"]
    C --> D["Compute Difference Δz<br/>Sort by ∣Δz_i∣"]
    D --> E["Build Binary Mask M<br/>(Top-k dim = 1)"]
    E --> F["Construct Virtual Identity<br/>z̃ = z + α(M⊙Δz)"]
    F --> G["[Training Stage]<br/>ε_u Output → min ‖z_- - z̃‖₂"]
    H["Other Identities x_+"] --> I["[Training Stage]<br/>min ‖ε_u(x_+) - ε(x_+)‖₂"]
    G --> J["Replace ε with ε_u<br/>Freeze Diffusion Model D"]
    I --> J

Key Designs

1. Face-Swap for Localizing Identity-Related Feature Dimension Masks

The core insight here is that identity information is not uniformly distributed in the CLIP image embedding space—some dimensions are highly sensitive to identity changes while others remain mostly unchanged. To identify these sensitive dimensions, the authors utilize an off-the-shelf Face-Swap tool on the target identity image \(x_-\) and a randomly sampled other-identity image \(x_a\) (where only the identity differs and context remains consistent) to generate a face-swapped image \(x_f = F(x_-, x_a)\). The embedding difference \(\Delta z = z_- - z_f\) is computed under the original encoder \(\varepsilon\). The absolute value of each component \(|\Delta z_i|\) reflects its sensitivity to identity changes. Statistical validation across 1000 Face-Swap image pairs proves that this bias indeed exists, with high-response dimensions consistently appearing in specific locations. Thus, all dimensions are sorted in descending order of \(|\Delta z_i|\), and the top \(k\) ratio is selected to construct a binary mask \(M\) (\(M_i = 1\) indicates that the dimension is identity-related and will be perturbed). This mask essentially acts as an identity-related dimension selector that is independent of positive or negative directions, ensuring that subsequent perturbation attacks only act on "identity neurons".

2. Identity-Related Transformations to Construct Virtual Identities

After obtaining the mask, an "unlearning target" needs to be explicitly defined during training for the target identity images—i.e., what identity the generated images should transform into. Instead of randomly generating a stranger's embedding, the authors apply controlled shifts along the direction of \(\Delta z\) restricted to the dimensions selected by the mask: \(\tilde{z}_- = z_- + \alpha(M \odot \Delta z)\), where \(\alpha\) regulates the perturbation magnitude (saturated at \(\alpha=100\) in experiments). This has the advantage that only the identity-related dimensions selected by the mask are modified, while identity-irrelevant features, such as pose, background, and lighting, are preserved at their original values, preventing them from being collaterally destroyed in the process of pushing the identity away. The subsequent unlearning loss forces the unlearned encoder \(\varepsilon_u\) to map \(x_-\) as close as possible to this virtual identity: \(\mathcal{L}_f = \|\varepsilon_u(x_-) - \tilde{z}_-\|_2\). Since the identity information in the virtual identity \(\tilde{z}_-\) is significantly weakened (shifted a great distance along the identity-sensitive direction), minimizing this loss is equivalent to forcing the encoder to map the identity of \(x_-\) to a "de-identified" region.

3. Consistency Preservation Loss for Retained Identities

For the identities \(x_+\) that do not need to be unlearned, the retention loss in IREU is consistent with the baseline: \(\mathcal{L}_r = \|\varepsilon_u(x_+) - \varepsilon(x_+)\|_2\), which keeps the output of the unlearned encoder for these identities as close as possible to the original encoder. The key difference is that the global perturbations of the baseline jumble the entire feature space when unlearning multiple identities—shifting the embeddings of all images (including retained identities) and causing the fidelity of retained identities to collapse in the fiveID setting (\(ID_{other}^o\) dropped from 0.51 to 0.01). IREU's localized perturbations process identity-related and identity-irrelevant dimensions separately. Even when unlearning 5 identities, the embeddings of the retained identities are almost unaffected in identity-irrelevant dimensions, maintaining high fidelity (\(ID_{other}^o = 0.53\), PSNR 28.25).

4. Encoder-Only Cross-Generator Zero-Shot Transfer

This is the most practical design of the entire framework: unlearning only modifies the image encoder \(\varepsilon_u\), leaving the diffusion model \(D\) and text encoder completely frozen. This means that once training is complete, \(\varepsilon_u\) can be directly plugged into any CPG generator (e.g., FastComposer, PhotoMaker) that shares the same encoder family without any additional fine-tuning. Experiments validate this advantage: directly inserting \(\varepsilon_u\) into PhotoMaker achieves a \(\Delta ID\) of 0.7007 (compared to only 0.1108 for the baseline), demonstrating that identity-related perturbations are more robust in cross-generator scenarios. This plug-and-play nature is extremely valuable for real-world deployment, as service providers only need to maintain a set of unlearned encoders without needing to execute unlearning for each generator individually.

Loss & Training

The total loss is a weighted sum of the unlearning loss and the retention loss:

• Unlearning loss: \(\mathcal{L}_f = \|\varepsilon_u(x_-) - \tilde{z}_-\|_2\), where \(\tilde{z}_- = z_- + \alpha(M \odot \Delta z)\)
• Retention loss: \(\mathcal{L}_r = \|\varepsilon_u(x_+) - \varepsilon(x_+)\|_2\)
• Total loss: \(\mathcal{L}_u = \lambda_1 \mathcal{L}_f + \mathcal{L}_r\), where \(\lambda_1 = 0.1\) is used in experiments.

Training configuration: Adam optimizer, lr=1e-5, batch size=1, single RTX 3090 GPU. The oneID setting is trained for 1000 steps, and the fiveID setting is trained for 12000 steps. The mask ratio is \(k=0.4\) (i.e., top 40% dimensions are selected as identity-related), and the shift magnitude is \(\alpha=100\) (saturated value).

Key Experimental Results

Main Results

Evaluated on CelebA-HQ. The following are the results under the single-identity (oneID) unlearning setting on FastComposer:

Metric	Meaning	BIA	Baseline	IREU
\(ID_{target}^o\) (↓)	Generator identity similarity between generated and input images	0.05	0.03	0.01
\(ID_{target}^g\) (↓)	Gen-similarity before and after unlearning	0.30	0.03	0.21
\(ID_{other}^o\) (↑)	Fidelity of retained identities	0.55	0.14	0.55
\(ID_{other}^g\) (↑)	Gen-similarity of retained identities before and after	0.82	0.51	0.85
\(\Delta ID\) (↑)	Discrepancy between retained and target identities	0.52	0.48	0.64
PSNR (↑)	Fidelity of retained identities	20.51	22.82	28.88
SSIM (↑)	Structural similarity of retained identities	0.70	0.79	0.92
\(\Delta FID\) (↓)	FID degradation of retained identities	4.14	4.33	1.02

Ablation Study

Configuration	Key Findings
Global perturbation (Baseline)	\(ID_{other}^o=0.14\) in oneID (obvious degradation); collapses to 0.01 under fiveID, failing to unlearn multiple identities simultaneously.
IREU (\(k=0.4, \alpha=100\))	\(\Delta ID=0.64\) in oneID, \(\Delta ID=0.55\) in fiveID, achieving the best trade-off.
Mask ratio \(k\)	When \(k > 0.4\), \(\Delta FID\) rises rapidly and SSIM variance increases, indicating that perturbations spread to identity-irrelevant dimensions.
Shift magnitude \(\alpha\)	Unlearning is insufficient when \(\alpha < 100\); \(ID_{target}^o\) saturates when \(\alpha \geq 100\).
Stability across different \(x_a\)	The average mask overlap rate across 10 different \(x_a\) is 70.46%, demonstrating that the selected identity-related dimensions are stable.
Cross-generator transfer (PhotoMaker)	IREU \(\Delta ID=0.7007\), baseline \(\Delta ID=0.1108\). IREU shows a greater advantage in cross-generator scenarios.

Key Findings

• The primary contribution is the identity-related dimension mask: removing it (using global perturbations) drops the fidelity of retained identities in fiveID from 0.53 to 0.01, rendering it almost completely ineffective. This indicates that without separating identity and identity-irrelevant features, the global feature space will be thoroughly corrupted once multiple identities need to be unlearned.
• An unexpected practical advantage of IREU is that during transfer to PhotoMaker, the baseline's retention identity \(\Delta ID\) almost vanishes (0.1108), whereas IREU maintains a high value (0.7007). This is because the baseline's global perturbations excessively alter the output distribution of the encoder, causing the new generator to struggle with the shift. In contrast, IREU's localized perturbations preserve most of the output distribution, thereby maintaining transfer robustness.
• Unlearning is also effective for unseen images of the same target identity (\(ID_{target}^{o, unseen}\) stays at 0.15 vs. above 0.55 for the original encoder), demonstrating that the unlearned encoder indeed alters the identity mapping at the feature space level, rather than merely overfitting to the training set images.

Highlights & Insights

• Using Face-Swap to identify identity-related dimensions is highly clever: paired portraits (differing only in identity) naturally decouple the identity factor. The high-response dimensions in the difference vector represent identity feature dimensions without requiring any annotation or prior knowledge. This offline-localized mask can be directly reused in subsequent training.
• Decoupling localized versus global perturbations is a core insight: baseline methods (global perturbation) cause a notable decrease in the fidelity of retained identities in the oneID setting and trigger complete collapse under fiveID. IREU proves that most encoder dimensions are identity-irrelevant—perturbing them contributes nothing but fidelity degradation.
• The encoder-only design provides high practical value: the training overhead of the unlearning pipeline is minimal (single 3090, 1k-12k steps), and inference only requires replacing an encoder weight file. For real-world deployment, this means CPG service providers can maintain a pool of unlearned encoders and directly deploy them when a user requests deletion, without needing to redeploy the entire generation pipeline.
• The stability verification of the mask is solid: ablation results show an average mask overlap of 70.46% ± 5.19% across different \(x_a\), far exceeding random expectation (40%), proving that the identity-related dimensions are indeed a stable structural property in the embedding space.

Limitations & Future Work

• It assumes that the Face-Swap tool itself does not leak identity information—if the face-swapping quality is insufficient (leaving residual textures of the original identity), the difference vector \(\Delta z\) will be corrupted by non-identity noise, reducing mask precision.
• Offline mask localization requires access to the target identity image \(x_-\) and another identity image \(x_a\), which may fail in edge-case scenarios where only a single public image is available and an appropriate \(x_a\) cannot be found.
• The encoder-only design relies on various CPG pipelines sharing the same encoder family (CLIP ViT). If mainstream CPG shifts toward end-to-end learning and abandons CLIP encoders in the future, the plug-and-play advantage of this method may be diminished.

Related Work & Insights

• vs BIA (CVPR 2025): BIA executes a "black hole" absorption in the U-Net bottleneck (identity representations are sucked into the bottleneck), which is limited to specific generators. IREU performs directional perturbations in the encoder embedding space, enabling cross-generator transfer. While BIA still retains an \(ID_{target}^o\) of 0.05 after identity unlearning (retaining too many identity traces), IREU reduces it to 0.01.
• vs GUIDE: GUIDE is a GAN-based identity unlearning method that is inapplicable to mainstream diffusion CPG pipelines, whereas IREU natively adapts to the CLIP + Stable Diffusion architecture.
• vs traditional concept erasure: Concept erasure suppresses concepts in the cross-attention layers, adapted for "style" or "object category" levels. IREU targets fine-grained identity-level unlearning and proves that dimension-level localization in the encoder embedding space is feasible for fine-grained tasks (distinguishing different identities of human faces).

Rating

• Novelty: ⭐⭐⭐⭐⭐ Proposes the identity unlearning problem in CPG scenarios for the first time, with a clean and effective Face-Swap mechanism for localizing identity-related dimensions.
• Experimental Thoroughness: ⭐⭐⭐⭐ Main results, ablations, cross-generator transfer, and unseen image tests cover a comprehensive range, though validation is confined to a single dataset (CelebA-HQ).
• Writing Quality: ⭐⭐⭐⭐⭐ Clear motivational logic, self-consistent methodology description, and ablation studies target every key design point.
• Value: ⭐⭐⭐⭐⭐ The encoder-only design delivers practical transferability and minimal training costs, showing direct value for real-world deployment.

Related Papers

• [CVPR 2026] Bridging Privacy and Provenance: Traceable Virtual Identity Generation
• [CVPR 2026] Reinforcement-Guided Synthetic Data Generation for Privacy-Sensitive Identity Recognition
• [CVPR 2026] No Way To Steal My Face: Proactive Defense Against Identity-Preserving Personalized Generation
• [ECCV 2026] Moiré Video Authentication: A Physical Signature Against AI Video Generation
• [ICCV 2025] Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answering