title: >- [Paper Note] R-ESC: Robustly Erasing Space Concepts via Stochastic Feature Remapping description: >- [ECCV 2026][Interpretability][Concept Erasure] R-ESC achieves training-free, irreversible feature-level concept erasure via prototype-orthogonal projection and multi-expert stochastic remapping. tags: - ECCV 2026 - Interpretability - Concept Erasure - Machine Unlearning - Feature Remapping date: 2026-09-19 content_hash: 8f6978430b069297
R-ESC: Robustly Erasing Space Concepts via Stochastic Feature Remapping¶
Conference: ECCV 2026
Paper: ECCV 2026 Poster
Area: Interpretability
Keywords: Concept Erasure, Machine Unlearning, Feature Remapping, Prototype Orthogonalization, Diffusion Safety
TL;DR¶
Addressing the vulnerability of prior concept erasure methods that degrade remain-data utility and leave cohesive latent clusters recoverable by fine-tuning, R-ESC introduces a training-free framework that combines prototype-orthogonal projection with multi-expert stochastic remapping to achieve robust, irreversible feature-level erasure.
Background & Motivation¶
With the rapid proliferation of foundation models and generative AI systems, ensuring model safety and regulatory compliance has become paramount. Developers and regulators increasingly require mechanisms to selectively ablate unwanted, biased, or copyrighted concepts (such as specific artist styles or proprietary categories) from trained representations. However, deep neural networks are inherently monolithic and polysemantic, with conceptual features deeply entangled across latent dimensions. Most existing machine unlearning and concept erasure approaches focus strictly on output-centric evaluations, verifying merely that the target concept is absent from final generation outputs or classification logits. Recent literature reveals that this creates an illusion of deletion: underlying latent representations frequently remain intact, allowing malicious actors to trivially reconstruct the forgotten concepts via lightweight fine-tuning or linear probing.
To enforce genuine feature-level erasure, Erasing Space Concept (ESC, CVPR 2025) pioneered the extraction of principal directions from forget activations via singular value decomposition (SVD) and inserted a projection layer to zero out the top-\(k\) forget prototypes. Yet, subspace pruning approaches suffer from three severe limitations. First, due to high correlation and entanglement between forget and remain prototypes, naive subspace removal inflicts significant collateral damage on benign representations, causing steep utility degradation. Second, even after zeroing out the targeted subspace, representations of the forget concept remain cohesive and linearly separable in latent space, leaving them highly vulnerable to recovery under probing attacks. Third, collecting full activation matrices to perform SVD incurs \(O(N_f d)\) memory overhead, triggering out-of-memory bottlenecks on large datasets.
The key insight of this paper is to shift the paradigm from passive subspace pruning to active manifold remapping and dispersion. Rather than creating an artificial null space that damages correlated features and leaves behind distinct clusters, the model decorrelates concept axes mathematically and dynamically blends the forget energy into the benign remain distributions. The core idea is to project features into a prototype-orthogonal space to decouple concept interactions, and deploy a multi-expert stochastic router to disperse forget activations across multiple remain prototypes, achieving linear-time, irreversible feature-level erasure.
Method¶
Overall Architecture¶
R-ESC (Robustly Erasing Space Concepts) is a completely training-free framework that can be inserted directly between the feature extractor and classifier head of a vision backbone, or into the cross-attention layers of text-to-image diffusion models. The end-to-end pipeline operates in three sequential phases: first, class prototypes are extracted as simple activation means and converted into an orthogonal basis via the Moore-Penrose pseudoinverse; second, each incoming feature vector is assigned uniformly at random to one of several remapping experts by a parameter-free stochastic router; third, the selected expert subtracts the forget subspace component and actively redirects its energy toward a designated remain prototype, completely dispersing the forget manifold across benign clusters.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Feature z"] --> B["Prototype-Orthogonal Projection<br/>Decouple concept axes via pseudoinverse"]
B --> C["Stochastic Multi-Expert Routing<br/>Uniformly assign remapping expert"]
C --> D["Feature Erasure & Multi-Target Remapping<br/>Subtract forget basis & fold into remain prototypes"]
D --> E["Output Unlearned Feature ẑ"]
Key Designs¶
1. Prototype-Orthogonal Projection: Eliminating Concept Entanglement and Collateral Damage In standard latent feature spaces, the cosine similarity between forget prototypes and remain prototypes is substantial, averaging ~0.5 and peaking at 0.77. Consequently, naive erasure collapses the self-correlation of remain prototypes from 1.0 down to ~0.52. To decouple these interactions, R-ESC constructs a prototype-orthogonal (PO) space. Given the prototype matrix \(P = [p_1, \dots, p_k] \in \mathbb{R}^{d \times k}\), its Moore-Penrose pseudoinverse is computed via singular value decomposition \(P = U \Sigma V^\top\) as:
Since \(P^+ P = I_k\), projecting features via \(P^+\) maps each class prototype to an independent coordinate axis. To preserve general features lying outside the low-rank prototype span, R-ESC introduces a complement-space projection term:
where \(s \in \{0, 1\}^k\) denotes the binary selector vector indicating forget classes, and \(P_f = P \mathrm{diag}(s)\). The term \(P_f P^+\) serves as a surgical linear operator that extracts and subtracts components aligned with forget concepts while leaving all remaining directions unaltered.
2. Feature Erasure and Remapping: Active Spatial Folding to Destroy Separability Pure orthogonal suppression zeros out the forget dimensions, but samples from the forget set still cluster together under non-concept residual dimensions. To eliminate this cluster structure, R-ESC replaces passive suppression with active spatial remapping:
where \(P_m\) is a remapping matrix whose \(i\)-th column is assigned to a designated remain prototype \(p_j\) whenever \(s_i = 1\), and zero otherwise. When a forget feature triggers activation in the PO space, the operator redirects its representation directly into the center of a benign remain class, merging the two distributions and rendering linear probing ineffective.
3. Stochastic Multi-Expert Routing: Breaking Cohesion Against Adversarial Recovery Redirecting a forget concept to a single remain prototype still leaves a localized sub-cluster that finely tuned linear probes might exploit. To systematically dismantle this cohesion, R-ESC introduces a Mixture-of-Experts architecture where multiple remapping experts map the same forget prototype to different remain prototypes. An input-independent stochastic router assigns each sample uniformly at random across experts during inference. This stochastic routing scatters the forget data points across several distinct remain clusters, destroying the geometric manifold of the erased concept and preventing recovery even under aggressive fine-tuning attacks.
Loss & Training¶
R-ESC is entirely training-free and requires no optimization, gradient backpropagation, or iterative loss minimization: - Prototype Computation: Concept prototypes are assembled simply by taking the activation mean over class samples: \(p_i = \frac{1}{N_i} \sum_{x \in \mathcal{D}_i} h_\psi(x)\). This eliminates the prohibitive \(O(N_f d^2)\) SVD computation of prior work, achieving \(O(N d)\) linear time complexity. - Memory Footprint: Only \(k\) prototype vectors need to be stored, resulting in a negligible \(O(d k)\) memory overhead (typically under a few megabytes), ensuring seamless scalability to ImageNet-scale datasets.
Key Experimental Results¶
Main Results¶
R-ESC was evaluated on image classification benchmarks (CIFAR-10 with AllCNN, CIFAR-100 with ResNet-18, Tiny-ImageNet with ViT, and ImageNet) and text-to-image diffusion models (Stable Diffusion v1.4, SD 2.1, and SDXL) across artistic style erasure and object erasure tasks. Both standard unlearning and Knowledge Retention (KR; learning rate = 0.1 probe attack) settings were evaluated (mean of 3 runs):
| Dataset / Setting | Method | Forget Test Acc \(D_{ft}\) (%) (↓) | Remain Test Acc \(D_{rt}\) (%) (↑) | Harmonic Mean \(HM_t\) (↑) |
|---|---|---|---|---|
| CIFAR-10 (Standard) | Retrain (Gold Standard) | 0.00 | 92.04 | 95.86 |
| CIFAR-10 (Standard) | ESC (CVPR'25) | 10.70 ± 0.87 | 81.27 ± 0.50 | 85.10 ± 0.65 |
| CIFAR-10 (Standard) | R-ESC (Ours) | 0.00 ± 0.00 | 91.02 ± 0.11 | 95.30 ± 0.06 |
| CIFAR-10 (KR Setting; lr=0.1) | Retrain (Gold Standard) | 72.90 | 88.01 | 41.44 |
| CIFAR-10 (KR Setting; lr=0.1) | ESC (CVPR'25) | 73.60 ± 0.17 | 84.98 ± 0.16 | 40.28 ± 0.19 |
| CIFAR-10 (KR Setting; lr=0.1) | R-ESC (Ours) | 12.87 ± 8.96 | 86.82 ± 3.30 | 86.91 ± 6.17 |
| CIFAR-100 (KR Setting; lr=0.1) | Retrain (Gold Standard) | 58.00 | 71.66 | 52.96 |
| CIFAR-100 (KR Setting; lr=0.1) | ESC (CVPR'25) | 59.65 ± 1.56 | 75.90 ± 0.01 | 52.65 ± 1.15 |
| CIFAR-100 (KR Setting; lr=0.1) | R-ESC (Ours) | 1.27 ± 0.72 | 77.01 ± 0.06 | 86.53 ± 0.31 |
| Tiny-ImageNet (KR Setting; lr=0.1) | ESC (CVPR'25) | 13.67 ± 1.06 | 90.54 ± 0.08 | 88.38 ± 0.52 |
| Tiny-ImageNet (KR Setting; lr=0.1) | R-ESC (Ours) | 0.43 ± 0.25 | 89.94 ± 0.07 | 94.51 ± 0.14 |
On Stable Diffusion v1.4: - Artistic Style Erasure (Van Gogh): R-ESC achieved an erasure LPIPS score \(L_f = 0.33\) and retain distortion \(L_r = 0.08\), yielding the highest overall trade-off score \(L_d = 0.25\) (outperforming UCE at 0.20 and RECE at 0.23). - Imagenette Object Erasure: The accuracy of the erased classes under a ResNet-50 classifier dropped to 0.4% (compared to 78.2% for base SD, 12.6% for ESD-u, and 2.6% for UCE), while retain-class accuracy remained at 76.6%.
Ablation Study¶
The component-wise ablation on CIFAR-10 demonstrates the distinct contributions of PO projection, feature remapping, and multi-expert stochastic routing (Table 2):
| Configuration | PO Applied | Forget Test Acc \(D_{ft}\) (%) (↓) | Remain Test Acc \(D_{rt}\) (%) (↑) | Harmonic Mean \(HM_t\) (↑) | KR Setting \(D_{ft}\) (%) (↓) | KR Setting \(HM_t\) (↑) |
|---|---|---|---|---|---|---|
| Erase (subspace suppression) | ✗ | 13.47 | 91.24 | 88.82 | 88.27 | 20.79 |
| Remap (single-expert) | ✗ | 0.00 | 79.64 | 88.67 | 91.27 | 15.94 |
| Erase (subspace suppression) | ✓ | 0.00 | 91.67 | 95.65 | 88.27 | 20.79 |
| Remap (single-expert) | ✓ | 0.00 | 91.16 | 95.38 | 29.83 | 69.78 |
| R-ESC (full multi-expert) | ✓ | 0.00 | 90.90 | 95.23 | 8.93 | 89.61 |
Key Findings¶
- PO Projection prevents collateral utility collapse: Without PO projection, single-expert remapping severely degrades remain-test accuracy from 91.67% to 79.64% due to concept entanglement. Incorporating PO projection insulates remain prototypes, keeping \(D_{rt}\) above 90.9% across configurations.
- Stochastic scattering is crucial for irreversibility: While both single-expert Remap and multi-expert R-ESC achieve 0.00% forget accuracy in standard testing, fine-tuning probe attacks (KR setting) recover 29.83% forget accuracy on single-expert Remap (\(HM_t = 69.78\)). R-ESC restricts recovery to 8.93% on CIFAR-10 and 1.27% on CIFAR-100, convincingly outperforming the Retrain model (which suffers 58.00% recovery on CIFAR-100).
- Extreme compute and memory efficiency: R-ESC completes erasure on CIFAR-10/100 in under 10 seconds with less than 200 MB GPU memory. On ImageNet, it finishes in approximately 10 minutes with only 6 GB of memory, whereas optimization-based baselines consume over 80 GB and take several hours.
Highlights & Insights¶
- From Subspace Pruning to Manifold Dispersion: R-ESC transitions concept unlearning from naive zeroing to active manifold dispersion. By scattering forget representations into multiple benign classes, it eliminates the cohesive geometric clustering that attackers exploit.
- Super-Retrain Irreversibility: A striking revelation of the KR evaluation is that a retrained model retains rich visual priors that allow rapid re-learning of erased classes under fine-tuning. By actively trapping forget features inside remain distributions, R-ESC creates a deceptive manifold that misleads gradient updates, rendering it more resistant to recovery than retraining from scratch.
- Closed-Form Plug-and-Play Generality: Utilizing simple activation-mean prototypes enables R-ESC to scale without retraining to diverse architectures, ranging from AllCNN and ResNet to Vision Transformers (ViT) and billion-parameter diffusion backbones (SDXL).
Limitations & Future Work¶
- Layer-Depth Sensitivity: As shown in Table 9, applying R-ESC at shallower layers degrades efficacy (forget accuracy rebounds to 19.47% at the third-last layer on CIFAR-10), indicating that early representations contain entangled low-level spatial features that resist linear prototype orthogonalization.
- Target Selection Heuristics: Sensitivity analysis (Table 7) indicates that mapping target classes affect recovery resistance (KR forget accuracy varies between 29.8% and 80.9% across different targets on CIFAR-10). Future work could develop automated semantic distance metrics to optimize target allocations.
- Simultaneous Multi-Concept Scalability: When unlearning a large proportion of concepts simultaneously (e.g., >50%), preserving numerical stability in the pseudoinverse and providing sufficient remain targets for scattering poses an open challenge.
Related Work & Insights¶
- vs ESC (Lee et al., CVPR 2025): ESC directly applies SVD to prune the forget subspace, which incurs substantial remain utility loss due to prototype entanglement and leaves behind a coherent cluster vulnerable to probing. R-ESC introduces PO projection to safeguard utility and multi-expert remapping to break cluster cohesion.
- vs Closed-Form Diffusion Erasure (UCE / RECE / ESD): Traditional closed-form diffusion editing alters cross-attention projection matrices globally, often wiping out surrounding scene context or collapsing on large models like SDXL. R-ESC preserves prompt adherence (e.g., starry nights over towns) while selectively erasing artist-specific textures, demonstrating superior stability on SDXL.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ [Novel perspective shifting concept erasure from subspace zeroing to prototype orthogonalization and stochastic multi-expert remapping]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across classification and generation backbones under standard and aggressive fine-tuning probe attacks]
- Writing Quality: ⭐⭐⭐⭐⭐ [Well-structured narrative with rigorous mathematical formulation and clean empirical visualizations]
- Value: ⭐⭐⭐⭐⭐ [Provides a training-free, practically deployable solution for model alignment, privacy compliance, and copyright protection]