Skip to content

Measuring the Depth of LLM Unlearning via Activation Steering

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/gnueaj/unlearning-depth-score
Area: Knowledge Editing
Keywords: machine unlearning, activation patching, mechanistic depth, recoverability evaluation, representation auditing

TL;DR

Proposes the Unlearning Depth Score (UDS), a training-free, causal, and dataset-invariant metric that evaluates the recoverability of forgotten knowledge across internal layers via two-stage activation patching, outperforming observational probes and output-level metrics in detecting residual knowledge obscured by representation shifts.

Background & Motivation

Large language models memorize significant fractions of their pretraining corpus, creating urgent privacy and safety vulnerabilities when this data involves personal identifiable information, copyrighted texts, or hazardous knowledge. Machine unlearning seeks to resolve this post-hoc by eliminating the influence of a designated forget set while maintaining general language capabilities, ideally driving the model to mirror a gold-standard retain model trained entirely without the forget data. However, rigorously auditing whether target knowledge is genuinely expunged remains a critical challenge, as standard evaluation protocols overwhelmingly depend on output-level heuristics like generation ROUGE, extraction strength, and membership inference attacks that are easily circumvented by surface-level refusals.

The central tension lies in the gap between output behavior and internal representation: lightweight unlearning interventions (such as negative preference optimization, refusal fine-tuning, or logit steering) often merely rotate downstream vocabulary projections or induce refusal prefixes, leaving the underlying factual circuits intact. Adversaries can subsequently recover ostensibly erased knowledge via lightweight parameter tuning or subtle activation steering. While existing white-box diagnostics attempt to probe latent representations using Centered Kernel Alignment, Logit Lens, or auxiliary linear probes, they are fundamentally limited: trained probes lack dataset invariance and require auxiliary supervision, whereas observational readouts like Logit Lens rely on static unembedding matrices that produce false-positive erasure calls whenever representations undergo simple geometric shifts.

This paper tackles the challenge by shifting from passive observation to causal intervention, testing whether internal representations still carry functional agency to decode factual knowledge through intact computational circuits. Core idea: quantify the mechanistic penetration of unlearning via the Unlearning Depth Score (UDS), which establishes a baseline of knowledge-encoding layers by patching hidden states from a retain model into a full model, and then causally measures residual knowledge by patching the unlearned model's states to yield a calibrated 0โ€“1 erasure score.

Method

Overall Architecture

The UDS evaluation framework considers three models: a full model \(M_{\text{full}}\) trained on all data including the target forget set, a retain model \(M_{\text{ret}}\) trained solely on retained data, and the unlearned model \(M_{\text{unl}}\) under audit. The evaluation unfolds in two primary phases: Stage 1 (Baselining) patches hidden states from \(M_{\text{ret}}\) into \(M_{\text{full}}\) across layers under teacher forcing on targeted entity spans, isolating knowledge-encoding (KE) layers where \(M_{\text{full}}\) demonstrates significant factual loss compared to \(M_{\text{ret}}\). Stage 2 (Quantification) patches hidden states from \(M_{\text{unl}}\) into \(M_{\text{full}}\) to gauge how closely the resulting degradation mirrors the retain baseline. Individual layer erasure ratios are then clipped and aggregated via baseline sensitivity weighting, producing an interpretable metric from 0 (knowledge fully intact) to 1 (knowledge erased to the retain model standard).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Context & Entity Span<br/>(Teacher Forcing)"] --> B["Stage 1: Retain Baseline Patching<br/>M_ret into M_full to identify KE layers"]
    B --> C["Stage 2: Unlearned Model Quantification<br/>M_unl into M_full to probe residual states"]
    C --> D["Layer Erasure Ratio Computation<br/>LER clipped to [0, 1]"]
    D --> E["Sensitivity-Weighted Aggregation<br/>Compute example- & model-level UDS"]

Key Designs

1. Entity-Span Teacher-Forcing Activation Patching: Eliminating Generation Noise and Autoregressive Overhead Traditional generation-based auditing suffers from high autoregressive latency and prompt template confounding. UDS partitions each forget QA pair into a contextual prompt and an exact factual entity span (e.g., isolating biographical titles from leading copulas). In a single forward pass under teacher forcing, the hidden states of the source model at the pre-entity prediction positions are patched into the residual stream of the target \(M_{\text{full}}\). Because \(M_{\text{full}}\) possesses the full parametric capacity to decode target facts, its subsequent nonlinear layers naturally realign and decode the patched representations if the underlying knowledge remains present, completely bypassing the vulnerability of static output heads to geometric coordinate rotations.

2. Knowledge-Encoding Layer Filtering & Layer Erasure Ratio: Isolating Causal Circuits with Normalized Baselines In Stage 1, the baseline log-probability degradation \(\Delta S_{1, i, l}\) induced by patching \(M_{\text{ret}}\) into \(M_{\text{full}}\) is quantified: $\(\Delta S_{1, i, l} = \frac{1}{T_i} \sum_{t=1}^{T_i} \left( s_{i,t}^{\text{full}} - s_{i,t}^{S1} \right)\)$ Layers satisfying \(\Delta S_{1, i, l} > \tau\) (with default \(\tau=0.05\)) are designated as Knowledge-Encoding (\(\text{KE}_i\)) layers, effectively filtering out uninformative noise in early representation layers and bounding the denominator away from zero. In Stage 2, patching \(M_{\text{unl}}\) yields degradation \(\Delta S_{2, i, l}\). The Layer Erasure Ratio (LER) compares these two shifts: $\(\text{LER}_{i, l} = \text{clip}\left(\frac{\Delta S_{2, i, l}}{\Delta S_{1, i, l}}, 0, 1\right)\)$ When unlearning genuinely eradicates target representations, \(\Delta S_2 \approx \Delta S_1\), driving \(\text{LER} \to 1\); conversely, if latent knowledge remains decipherable, \(M_{\text{full}}\) smoothly reconstructs the tokens, causing \(\Delta S_2 \approx 0\) and \(\text{LER} \to 0\).

3. Baseline-Weighted Metric Aggregation & Cache-Optimized Inference: Ensuring Scale Invariance and High Throughput To integrate layerwise observations into a robust holistic score, UDS computes a normalized weighted average of LER across all validated KE layers, using the Stage 1 baseline degradation \(\Delta S_{1, i, l}\) as the weighting coefficient: $\(\text{UDS}_i = \frac{\sum_{l \in \text{KE}_i} \Delta S_{1, i, l} \cdot \text{LER}_{i, l}}{\sum_{l \in \text{KE}_i} \Delta S_{1, i, l}}\)$ This formulation guarantees that layers carrying the primary factual burden dictate the final score, preventing inert peripheral layers from distorting evaluation. Crucially, because Stage 1 depends strictly on \(M_{\text{ret}}\) and \(M_{\text{full}}\), all reference log-probabilities, baseline deltas, and KE sets can be computed once and permanently cached. Auditing any novel unlearned checkpoint requires merely extracting its hidden states and running Stage 2 forward passes, ensuring computational efficiency on par with standard output-level benchmarks.

A Worked Example

Consider an example from the TOFU forget split: "What is the profession of Hsiao Yun-Hwa's father?" targeting the entity "civil engineer". During Stage 1 baselining, patching \(M_{\text{ret}}\) into \(M_{\text{full}}\) produces minimal log-probability shifts across layers 0โ€“5 (\(\Delta S_1 \le 0.05\), tagged as non-KE), whereas mid-to-late layers exhibit substantial drops (e.g., layer 15 reaches \(\Delta S_1 = 1.713\)). When evaluating an unlearned IdkDPO checkpoint, the observational Logit Lens detects zero decodable target tokens across layers 7โ€“13, falsely outputting \(\text{LER} = 1.0\) and an aggregate score of 0.801 (judging the entity erased). In contrast, UDS reveals that injecting the IdkDPO representations into \(M_{\text{full}}\) yields a minor degradation of only \(\Delta S_2 = 0.039\) at layer 9 (\(\text{LER} = 0.113\)), proving that \(M_{\text{full}}\)'s deeper layers effortlessly recover the rotated vectors, culminating in an authentic unlearning score of only 0.209.

Key Experimental Results

Main Results

Evaluation was conducted on an expanded meta-evaluation suite following OpenUnlearning with Llama-3.2-1B-Instruct on the TOFU forget10 benchmark. Faithfulness was validated using balanced pools of 30 models with forget knowledge (P-pool) and 30 retain models (N-pool). Robustness was appraised across 150 unlearned models generated via hyperparameter sweeps across 8 methods (GradDiff, NPO, SimNPO, IdkNLL, IdkDPO, AltPO, RMU, UNDIAL) under 4-bit NF4 quantization and 1-epoch relearning attacks. UDS was systematically benchmarked against 19 competitive metrics.

Metric Group Metric Scope Overall (HM) โ†‘ Faithfulness (AUC) โ†‘ Agg. Robustness (HM) โ†‘ Quant. Robustness \(Q\) โ†‘ Relearn Robustness \(R\) โ†‘
Proposed UDS (Ours) Output/Retain/Internal (O+R+I) 0.951 0.971 0.932 0.968 0.900
White-box Logit Lens Output/Retain/Internal (O+R+I) 0.902 0.927 0.879 0.959 0.812
White-box Fisher Masked (0.1%) Retain/Internal (R+I) 0.716 0.712 0.721 0.583 0.946
White-box CKA Retain/Internal (R+I) 0.051 0.648 0.026 0.997 0.013
Memorization Extraction Strength Output (O) 0.875 0.891 0.859 0.970 0.770
Memorization Exact Memorization Output (O) 0.782 0.817 0.750 0.984 0.605
Memorization Truth Ratio Output (O) 0.542 0.947 0.379 0.996 0.234
Generation ROUGE Output (O) 0.456 0.722 0.333 0.934 0.203
Generation Paraphrased ROUGE Output (O) 0.209 0.832 0.119 0.951 0.064
Normalized MIA sZLib Output/Retain (O+R) 0.790 0.870 0.724 0.704 0.745
Raw MIA MIA-Min-K Output (O) 0.774 0.907 0.675 0.923 0.532

Ablation Study

Extensive ablations examined component patching sites, knowledge-encoding layer filtering thresholds \(\tau\), and cross-model parameter scaling.

Configuration / Ablated Component Key Metric / Performance Experimental Findings & Mechanism
Patching Location: Full Layer Output (\(h_l\)) Mean \(\Delta S_1 \approx 2.5 \sim 4.5\) The comprehensive residual stream carries the dominant share of causal knowledge (chosen default)
Patching Location: Attn + Residual (\(m_l\)) Mean \(\Delta S_1 \approx 1.5 \sim 3.0\) Captures intermediate attention contributions, though divergence widens in deeper transformer layers
Patching Location: MLP Isolated Output Mean \(\Delta S_1 \approx 0.8 \sim 1.8\) Isolating MLP output vectors yields lower causal degradation, capturing only partial factual signals
Patching Location: Attention Isolated Output Mean \(\Delta S_1 \approx 0.2 \sim 0.8\) Isolated attention projections display negligible standalone factual causal control
KE Threshold \(\tau = 0.05\) Mean |KE| 10.9 (1.5% skipped) Balances noise suppression and numerical stability of LER denominators (chosen default)
KE Threshold \(\tau = 0.01\) Mean |KE| 13.2 (0.5% skipped) Includes marginal early layers with minor deltas, reducing signal-to-noise ratio
KE Threshold \(\tau = 0.10\) Mean |KE| 9.6 (3.5% skipped) Overly aggressive filtering, dropping 14 examples due to empty KE candidate sets
Model Scale: Llama-3.2-1B (full / 99 / 95 / 90) UDS: 0.002 / 0.153 / 0.496 / 1.000 Perfect monotonic scaling directly proportional to the unseen data fraction
Model Scale: Llama-3.2-3B (full / 99 / 95 / 90) UDS: 0.008 / 0.151 / 0.482 / 1.000 Preserves strict monotonicity across intermediate scales with consistent relative shifts
Model Scale: Llama-3.1-8B (full / 99 / 95 / 90) UDS: 0.000 / 0.101 / 0.455 / 1.000 Validates robustness on 8B architectures where larger model capacities dampen minor representational perturbations

Key Findings

  • Causal Intervention Resolves Representation-Shift False Positives: On IdkDPO checkpoints, Logit Lens mistakenly reports near-complete erasure (0.801) across layers 7โ€“13, while UDS causally detects that the factual knowledge remains recoverable (0.209), exposing internal preservation that static decoders miss.
  • Output Metrics Suffer Severe Degradation Under Relearning: Output-based metrics like ROUGE and Truth Ratio collapse under a single epoch of relearning (\(R=0.203\) and \(R=0.234\)). Models rapidly recover fluent text generation despite minimal true parametric restructuring. In contrast, UDS directly interrogates latent representations and maintains exceptional stability (\(R=0.900\)).
  • Unlearning Depth Shows Severe Prompt-Type Heterogeneity: Under IdkNLL (which maps answers to "I don't know"), normalized MIA metrics uniformly assign a score of 0.0. UDS reveals that Yes/No questions achieve an unlearning depth of 0.624 (where refusal functions as factual negation), whereas entity names (0.025) and titles (0.038) exhibit virtually no mechanistic unlearning.
  • Viability of Fallback Evaluation Without Retain Models: When the gold-standard retain model \(M_{\text{ret}}\) is unavailable in deployment scenarios, running Stage 2 alone against \(M_{\text{full}}\) yields a Spearman rank correlation of \(\rho = 0.994\) against full UDS. Stage 1 primarily serves to provide scale calibration onto \([0, 1]\), while Stage 2 alone suffices for comparative method ranking.

Highlights & Insights

  • Causal Decoding via Intact Computational Circuits: Intelligently employs the undisturbed full model \(M_{\text{full}}\) as a live causal interpreter, circumventing the vulnerability of Logit Lens and trained linear probes to geometric space rotation.
  • Symmetric Robustness Formulation: Uncovers an evaluation blind spot where existing one-directional robustness benchmarks reward metrics for drops caused by quantization-induced generation failure, establishing symmetric stability criteria.
  • Reshaping Method Rankings and Hyperparameter Dynamics: Integrating UDS into the privacy evaluation axis reverses the competitive ranking between NPO and SimNPO, incentivizing practitioners toward larger learning rates and longer training to achieve true representation-level unlearning.

Limitations & Future Work

  • Dependency on Retain Model for Absolute Scaling: Precise calibration onto \([0, 1]\) relies on training a gold-standard retain model, which imposes significant computational overhead in large foundation model workflows.
  • Saturation Under Over-Unlearning: The \([0, 1]\) clipping operation masks potential over-unlearning (where latent representations drift past the retain model into corrupted states), necessitating joint monitoring with utility metrics.
  • Entity-Span Constraints: The current methodology assumes localized factual entity spans under teacher forcing, requiring further extension to assess long-form generation, dialogic reasoning, and procedural logic unlearning.
  • vs Logit Lens & Diagnostic Probes (Lynch et al., 2024; Patil et al., 2024): Logit Lens relies on fixed unembedding heads that misdiagnose rotated representations as erased; diagnostic probes require auxiliary supervision and lack dataset invariance. UDS leverages dynamic non-linear causal decoding in \(M_{\text{full}}\), operating entirely training-free.
  • vs OpenUnlearning Benchmark (Dorna et al., 2025): OpenUnlearning focuses on observational output metrics whose asymmetric robustness rewards generation degradation. UDS delivers the first causally verified white-box metric and standardizes symmetric robustness auditing.

Rating

  • Novelty: โญโญโญโญโญ Pioneering framework applying two-stage activation patching for causal auditing of machine unlearning depth.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive meta-evaluation covering 150 models, 8 unlearning methods, 20 comparative metrics, and dual perturbation attacks.
  • Writing Quality: โญโญโญโญโญ Elegant mathematical formulation, lucid conceptual framing, and insightful micro-to-macro case analyses.
  • Value: โญโญโญโญโญ Provides a foundational, manipulation-resistant auditing standard for AI safety, privacy compliance, and knowledge editing.