Measuring the Depth of LLM Unlearning via Activation Patching¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/gnueaj/unlearning-depth-score
Area: LLM Safety
Keywords: machine unlearning, activation patching, unlearning depth score, causal intervention, membership inference attack
TL;DR¶
The authors introduce the Unlearning Depth Score (UDS), a training-free, causal, and dataset-invariant metric using two-stage cross-model activation patching to quantify mechanistic unlearning depth, ranking first in both faithfulness and symmetric robustness across 150 models from 8 methods.
Background & Motivation¶
Large language models memorize substantial portions of their pre-training and fine-tuning corpora, introducing severe privacy and safety risks when sensitive personal identifiers or hazardous knowledge become embedded in parametric memory. Machine unlearning seeks to erase targeted data while preserving general model utility, aiming to produce a model computationally indistinguishable from one trained solely on retained data. However, verifying whether sensitive knowledge is genuinely expunged remains a fundamental challenge. Current benchmarks like TOFU and MUSE overwhelmingly rely on observational output metricsโsuch as cross-entropy loss, ROUGE overlap, and membership inference attacks (MIAs). These metrics only probe final-layer distributions and are easily deceived when models are superficial refusers (e.g., fine-tuned to say "I don't know") or undergo global representational shifts, allowing adversaries to rapidly restore ostensibly erased knowledge via lightweight fine-tuning or activation steering.
To address the blindness of black-box metrics, recent studies explore white-box diagnostic probes, such as Centered Kernel Alignment (CKA) for layer-wise representational similarity, Logit Lens for reading intermediate hidden states through frozen decoders, and linear probes. Nonetheless, existing white-box diagnostics suffer from critical drawbacks: geometric similarity metrics (CKA) and gradient-sensitivity metrics (Fisher Information) capture optimization trajectories rather than factual knowledge retention, collapsing under minor downstream fine-tuning; observational readout mechanisms like Logit Lens rely on a fixed unembedding matrix, which falsely signals complete unlearning whenever internal representations are rotated or linearly transformed; and probe-based techniques require supervised auxiliary training tied to specific dataset splits, precluding a standardized, dataset-invariant score.
These failures expose a core tension: passive observation cannot separate genuine knowledge deletion from representational obfuscation. A reliable metric must causally intervene on model representations to test whether target facts remain decodable by intact downstream computational pathways. Core idea: propose the Unlearning Depth Score (UDS), a training-free causal metric that employs two-stage activation patching between the unlearned, retain, and full models to identify factual knowledge-encoding layers and quantify the exact depth of knowledge erasure on a normalized 0-to-1 scale.
Method¶
Overall Architecture¶
UDS defines unlearning depth as how deeply knowledge erasure penetrates internal representations, rather than whether final logits are masked. The framework operates on triple-model configurations: a full model \(M_{\text{full}}\) trained on all data, a gold-standard retain model \(M_{\text{ret}}\) trained without the forget set \(D_f\), and the unlearned candidate \(M_{\text{unl}}\). In Stage 1 (Baselining), residual stream activations from \(M_{\text{ret}}\) are patched into \(M_{\text{full}}\) across individual layers during teacher-forced evaluation over factual entity spans, establishing the layer-wise upper bound of knowledge degradation and filtering out noise to identify Knowledge-Encoding (KE) layers. In Stage 2 (Quantification), the candidate model \(M_{\text{unl}}\) replaces \(M_{\text{ret}}\) as the source, patching its hidden states into \(M_{\text{full}}\) to measure actual prediction degradation. Finally, per-layer erasure ratios are bounded and weighted by baseline layer importance to aggregate into a single normalized score spanning from 0 (knowledge fully intact) to 1 (knowledge erased to the retain model baseline).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Forget Set Input & Entity Span<br/>Teacher-forced Context + Target Tokens"] --> B["Stage 1: Baselining & KE Layer Identification<br/>Patch M_ret into M_full to measure baseline drop"]
B --> C["Stage 2: Erasure Quantification<br/>Patch M_unl into M_full to measure actual drop"]
C --> D["Layer Erasure Ratio & Weighted Aggregation<br/>Clip LER in 0-1 and weight across KE layers"]
D --> E["Final UDS Score & Privacy Axis Integration<br/>Normalized 0-1 depth and harmonic evaluation"]
Key Designs¶
1. Stage 1: Baselining & KE Layer Identification: isolating factual circuits and bounding baseline degradation
Comparing arbitrary model activations without calibration introduces cross-model representation drift. Stage 1 establishes a rigorous factual baseline. Given an example \(i\) partitioned into context prefix \(x_i\) and factual entity span \(y_i = (y_{i,1}, \dots, y_{i,T_i})\), teacher forcing feeds the full sequence into \(M_{\text{full}}\) in a single forward pass, recording reference log-probabilities \(s^{\text{full}}_{i,t} = \log P_{M_{\text{full}}}(y_{i,t} \mid x_i, y_{i,<t})\). For each layer \(l\), the residual stream activations of \(M_{\text{ret}}\) at the entity token prediction positions are patched into \(M_{\text{full}}\), producing patched log-probabilities \(s^{S1}_{i,t}\). The baseline drop is computed as: $$ \Delta S1_{i,l} = \frac{1}{T_i} \sum_{t=1}^{T_i} \left( s^{\text{full}}{i,t} - s^{S1} \right) $$ A large \(\Delta S1_{i,l}\) proves that layer \(l\) in \(M_{\text{full}}\) encodes knowledge of \(D_f\) that \(M_{\text{ret}}\) lacks. Layers with negligible degradation represent noisy, non-specific activations; thus, a threshold \(\tau = 0.05\) defines the set of Knowledge-Encoding layers \(\text{KE}_i = \{l \mid \Delta S1_{i,l} > \tau\}\), which also bounds the denominator away from zero.
2. Stage 2: Erasure Quantification: causal testing of recoverable latent knowledge
With the knowledge-encoding layers identified, Stage 2 quantifies how much of this target information persists in \(M_{\text{unl}}\). The residual stream hidden states of \(M_{\text{unl}}\) at the entity positions are patched into the exact same target model \(M_{\text{full}}\) at layer \(l\), yielding patched predictions \(s^{S2}_{i,t}\) and degradation: $$ \Delta S2_{i,l} = \frac{1}{T_i} \sum_{t=1}^{T_i} \left( s^{\text{full}}{i,t} - s^{S2} \right) $$ Because \(M_{\text{full}}\) possesses the intact decoding parameters and downstream non-linear transformations required to interpret forget set facts, it can successfully extract and output the entity if the information remains encoded within \(M_{\text{unl}}\)'s representations (yielding \(\Delta S2_{i,l} \approx 0\)). Conversely, if \(M_{\text{unl}}\) genuinely purged the knowledge, its patched hidden state degrades \(M_{\text{full}}\)'s prediction to the same degree as \(M_{\text{ret}}\) (yielding \(\Delta S2_{i,l} \approx \Delta S1_{i,l}\)).
3. Layer Erasure Ratio & Weighted Aggregation: constructing a calibrated, bounded depth metric
To standardize layer-wise performance across diverse architectures and forget sets, the Layer Erasure Ratio (LER) normalizes the measured degradation against the retain baseline and applies clipping: $$ \text{LER}{i,l} = \text{clip}\left( \frac{\Delta S2, 0, 1 \right) $$ Clipping to }}{\Delta S1_{i,l}\([0, 1]\) prevents over-unlearning (where corrupted representations cause catastrophic drops exceeding \(M_{\text{ret}}\)) from inflating the score. The per-example score \(\text{UDS}_i\) aggregates LER across all identified \(\text{KE}_i\) layers, weighting each layer proportionally by its baseline knowledge-encoding strength \(\Delta S1_{i,l}\): $$ \text{UDS}i = \frac{\sumi} \Delta S1} \cdot \text{LER{i,l}}{\sumi} \Delta S1 $$ The overall model-level score is the sample mean }\(\text{UDS} = \frac{1}{N} \sum_{i=1}^N \text{UDS}_i\). A score of 1 indicates erasure matching the gold-standard retain model, while 0 reflects complete parametric retention.
4. Final UDS Score & Privacy Axis Integration: redefining evaluation protocols and streamlining pipelines
In comprehensive benchmarking frameworks, unlearning efficacy is partitioned across Memorization, Utility, and Privacy. Conventional setups restrict Privacy to output-level membership inference attacks (\(\text{MIA}_{\text{agg}}\)). UDS extends this axis via a harmonic formulation: $$ \text{Privacy} = \text{HM}(\text{MIA}_{\text{agg}}, \text{UDS}) $$ This formulation heavily penalizes models that mask outputs while harboring latent knowledge. Furthermore, because Stage 1 baselines (\(\Delta S1\), reference probabilities, and KE sets) depend solely on \(M_{\text{full}}\) and \(M_{\text{ret}}\), they are computed once and cached. Auditing new unlearned models requires only a single teacher-forced forward pass in Stage 2, enabling evaluation speed comparable to output metrics while offering strong pre-perturbation robustness that removes the need for costly post-quantization or post-relearning sweeps.
Key Experimental Results¶
Main Results¶
The authors conduct a large-scale meta-evaluation across 20 unlearning metrics on the TOFU forget10 benchmark using the Llama-3.2-1B-Instruct architecture. The study investigates 150 unlearned models generated via hyperparameter sweeps across 8 methods (GradDiff, NPO, SimNPO, IdkNLL, IdkDPO, AltPO, RMU, and UNDIAL). Faithfulness is evaluated via AUC-ROC on separating a 30-model P-pool (trained with \(D_f\)) from a 30-model N-pool (trained without \(D_f\)). Robustness evaluates stability under 4-bit NF4 quantization (\(Q\)) and 1-epoch relearning (\(R\)), using a newly introduced symmetric penalty formulation that penalizes instability in either direction.
The table below summarizes the meta-evaluation results (selected from Table 3 of the main paper):
| Metric Group | Metric | Scope | Overall โ | Faithfulness โ | Agg. Robustness โ | Quant. Robustness (\(Q\)) โ | Relearn Robustness (\(R\)) โ |
|---|---|---|---|---|---|---|---|
| Memorization | Truth Ratio | Output (O) | 0.542 | 0.947 | 0.379 | 0.996 | 0.234 |
| Generation | ROUGE | Output (O) | 0.456 | 0.722 | 0.333 | 0.934 | 0.203 |
| Generation | Jailbreak ROUGE | Output (O) | 0.438 | 0.757 | 0.308 | 0.971 | 0.183 |
| Privacy (MIA) | MIA-LOSS | Output (O) | 0.767 | 0.902 | 0.668 | 0.935 | 0.519 |
| Normalized MIA | sLOSS | Output + Retain (O, R) | 0.778 | 0.891 | 0.690 | 0.719 | 0.663 |
| White-box | CKA | Retain + Internal (R, I) | 0.051 | 0.648 | 0.026 | 0.997 | 0.013 |
| White-box | Fisher (0.1% mask) | Retain + Internal (R, I) | 0.716 | 0.712 | 0.721 | 0.583 | 0.946 |
| White-box | Logit Lens | Output + Retain + Internal (O, R, I) | 0.902 | 0.927 | 0.879 | 0.959 | 0.812 |
| Ours | UDS (Ours) | Full Scope (O, R, I) | 0.951 | 0.971 | 0.932 | 0.968 | 0.900 |
Across all 20 evaluated metrics, UDS achieves the highest Overall score (0.951), highest Faithfulness (AUC-ROC 0.971), and highest Aggregate Robustness (HM 0.932).
Ablation Study & Depth Analysis¶
To pinpoint why causal activation patching succeeds where observational decoding fails, the authors present a layer-wise audit of an IdkDPO unlearned model predicting the entity "historical fiction" (Table 4 in the paper):
| Layer | Logit Lens: \(\Delta S1\) | Logit Lens: \(\Delta S2\) | Logit Lens: LER | UDS: \(\Delta S1\) | UDS: \(\Delta S2\) | UDS: LER | Mechanistic Analysis |
|---|---|---|---|---|---|---|---|
| Layer 0โ4 | - | - | not KE | - | - | not KE | Early representations carry no factual specialization |
| Layer 5 | 0.375 | 0.250 | 0.667 | - | - | not KE | UDS threshold \(\tau\) filters noisy low-delta layer |
| Layer 7 | 0.312 | 0.812 | 1.000 | 0.053 | -0.059 | 0.000 | Logit Lens fooled by representation rotation; UDS detects full retention |
| Layer 9 | 1.375 | 1.375 | 1.000 | 0.346 | 0.039 | 0.113 | Downstream causal pathway decodes entity; UDS LER is only 0.113 |
| Layer 11 | 1.250 | 2.062 | 1.000 | 0.838 | 0.088 | 0.105 | Fixed unembedding fails; causal patching proves knowledge persists |
| Layer 13 | 0.926 | 2.465 | 1.000 | 1.299 | 0.299 | 0.230 | Internal representations retain clear decodable factual signals |
| Layer 15 (Final) | 1.713 | 0.436 | 0.254 | 1.713 | 0.436 | 0.254 | Metrics converge as representations directly govern output logits |
| Aggregate Score | - | - | 0.801 (Judged Erased) | - | - | 0.209 (Revealed Intact) | Logit Lens produces false confidence; UDS uncovers residual truth |
In prompt-type breakdown analysis for an IdkNLL model, Yes/No questions achieve an average UDS of 0.624, whereas person names (0.025), book titles (0.038), and biographical details (0.049) show near-zero unlearning depth (overall average 0.076). This demonstrates that refusal-based unlearning penetrates deeply only for boolean assertions, while factual entities remain entirely intact beneath superficial output shifts.
Key Findings¶
- Representational shifts fool observational decoders: As demonstrated in Table 4, Logit Lens projects intermediate hidden states through a frozen unembedding matrix. When unlearning optimization rotates the latent feature space, the projection fails, erroneously reporting complete erasure (\(\text{LER} = 1.000\) from Layer 7 to 13, aggregate 0.801). UDS patches hidden states into \(M_{\text{full}}\), allowing subsequent nonlinear transformer blocks to realign features and causally decode the entity, proving that latent knowledge is almost entirely intact (\(\text{UDS} = 0.209\)).
- Symmetric robustness exposes output-level fragility: Prior evaluations only penalized knowledge recovery after attacks. By penalizing both recovery and artificial degradation (such as precision loss from 4-bit quantization that ruins ROUGE independent of knowledge), output metrics collapse (ROUGE relearning robustness is just 0.203). UDS maintains balanced stability (\(Q=0.968, R=0.900\)) by reading directly from intact internal pathways.
- Internal depth alters unlearning method rankings: Incorporating UDS into the privacy axis swaps the relative ranking of NPO and SimNPO. While NPO achieves high output MIA privacy (0.875) but low internal depth (\(\text{UDS} = 0.619\)), SimNPO's reference-free, length-normalized objective applies unlearning pressure into deeper residual streams (\(\text{UDS} = 0.739\)), demonstrating that internal auditing is required to steer algorithmic progress.
Highlights & Insights¶
- Causal activation patching as a quantitative unlearning metric: Transforms diagnostic interpretability tooling into a standardized, dataset-invariant evaluation metric. By combining Stage 1 baselining with Stage 2 quantification, UDS provides the first calibrated 0-to-1 score of internal unlearning depth without requiring probe retraining.
- Exposing the "superficial refusal" illusion: The paper provides concrete empirical proof that current unlearning methods primarily act as localized output suppressors. Factual associations in mid-to-late transformer layers remain functionally intact and readily exploitable by adversaries.
- Retain-free ranking preservation: Although UDS relies on a retain model for 0-to-1 scale calibration, the authors prove that running Stage 2 alone yields an average degradation score (\(\Delta S2\)) whose model rankings correlate with full UDS at Spearman \(\rho = 0.994\), providing a viable fallback for production auditing.
Limitations & Future Work¶
- Dependency on full and retain checkpoints: The complete two-stage formulation requires access to both \(M_{\text{full}}\) and \(M_{\text{ret}}\) weights, restricting direct applicability to closed-source proprietary APIs where intermediate activations cannot be extracted or injected.
- Restriction to isolated entity spans under teacher forcing: The current metric relies on automated prefix/entity span partitioning evaluated in single forward passes. Extending UDS to long-form open generation, multi-hop reasoning chains, and broad conceptual knowledge remains an open research question.
- Indistinguishability of over-unlearning: Because LER is clipped to 1.0, severe representational corruption that causes performance degradation beyond \(M_{\text{ret}}\) cannot be separated from optimal unlearning using UDS alone, necessitating joint auditing alongside general utility metrics.
Related Work & Insights¶
- vs Logit Lens & Linear Probes (Lynch et al., 2024; Patil et al., 2024): Observational readouts rely on static decoding matrices or supervised probe training, making them vulnerable to coordinate rotations or dataset overfitting. UDS uses causal intervention through \(M_{\text{full}}\)'s native computational hierarchy, achieving higher faithfulness (0.971 vs 0.927).
- vs CKA & Fisher Diagnostics (Xu et al., 2026): CKA tracks global representational geometry while Fisher measures parameter curvature along optimization paths, both collapsing under slight downstream fine-tuning (\(R=0.013\) and \(Q=0.583\)). UDS targets factual token prediction directly, delivering superior aggregate robustness (0.932).
- vs Standard Benchmarks (TOFU, MUSE, OpenUnlearning): While existing suites prioritize output generation similarity and black-box membership inference, UDS introduces a robust internal metric and symmetric meta-evaluation protocols, establishing a more rigorous defense against extraction attacks.
Rating¶
- Novelty: โญโญโญโญโญ Pioneering adaptation of cross-model activation patching to formulate a calibrated, causal metric for mechanistic unlearning depth.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive meta-evaluation across 20 metrics and 150 models over 8 methods, backed by layer-wise and prompt-type mechanistic case studies.
- Writing Quality: โญโญโญโญโญ Clear problem formulation, rigorous mathematical grounding, and compelling narrative structure contrasting observational vs. causal paradigms.
- Value: โญโญโญโญโญ Provides an indispensable auditing tool for privacy compliance, mechanistic interpretability, and the development of robust unlearning algorithms.