SMI: Statistical Membership Inference for Reliable Unlearned Model Auditing

Conference: NeurIPS 2026
arXiv: 2602.01150
OpenReview: https://openreview.net/forum?id=LZfoK7J0Q5
Code: https://github.com/Alen-Su/SMI
Area: Safety
Keywords: machine unlearning, membership inference attacks, model auditing, mixture proportion estimation, maximum mean discrepancy

TL;DR

Reveals that conventional membership inference attacks suffer from inherent alignment bias and illusory forgetting due to opposing gradient directions in machine unlearning, and proposes SMI/SMI-M, a training-free statistical auditing framework that formulates auditing as mixture proportion estimation in feature space with bootstrap reference intervals.

Background & Motivation

Machine unlearning (MU) has emerged as an essential mechanism for upholding data privacy regulations such as the GDPR, which legally enshrines the "right to be forgotten." When user consent is revoked, unlearning algorithms aim to erase the influence of specific training instances from trained models without degrading overall utility on retained data. However, establishing a rigorous and dependable auditing mechanism to verify whether requested data has genuinely been forgotten remains an outstanding open problem. While backdoor and adversarial poisoning methods have been explored as auditing tools, defensive techniques such as adversarial training can independently suppress attack success rates without actual forgetting, and retraining target models interferes with the baseline task. Consequently, Membership Inference Attacks (MIA) have become the preeminent paradigm for unlearned model auditing.

Despite widespread adoption, the foundational premise of MIA-based auditing—that unlearning can be certified whenever an attacker fails to distinguish unlearned instances from non-members—is fundamentally flawed and prone to illusory forgetting. In standard MIA, a binary classifier separates member representations associated with positive training gradients from non-member representations that induce no gradient. In stark contrast, unlearning procedures apply negative gradient updates to unlearned samples. This opposing optimization dynamic pushes unlearned samples into regions of the feature representation space that are fundamentally distinct from natural non-member distributions. Using PAC-Bayesian analysis, the authors prove that conventional MIA auditing incurs an unavoidable and unobservable alignment bias bounded by the Rényi divergence between the training distribution and the auditing distribution. Moreover, state-of-the-art MIA variants require training dozens of auxiliary shadow models, resulting in prohibitive computational overhead.

Addressing both the alignment bias and computational bottlenecks requires abandoning sample-wise classification hyperplanes that assume matched distributions. The core idea is to reformulate unlearned model auditing from sample-level classification into distribution mixture proportion estimation in the feature space, relaxing the parameter-space Monge optimal transport problem into a convex combination and deriving training-free closed-form solutions (SMI/SMI-M) with resampling-based confidence intervals.

Method

Overall Architecture

Rather than predicting a binary membership label for each audited instance, SMI evaluates the audited population from a distributional perspective. Conceptually, ideal unlearning corresponds to transporting the parameter distribution of the initial model to that of an ideally retrained model under a Monge optimal transport formulation. Because verifying parameter distributions is intractable in high-dimensional deep networks with functional redundancies, SMI shifts the transport objective into the feature representation space. It then relaxes the transport problem into a distribution mixture model, treating the unlearned feature distribution as a convex combination of retained member and non-member distributions. By minimizing the statistical discrepancy between empirical distributions, SMI directly solves for the non-member mixture proportion as the unlearning score.

The end-to-end auditing pipeline consists of feature extraction, moment matching or kernel mean embedding optimization, and bootstrap uncertainty quantification, as illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Data & Audited Model<br/>Member / Non-member / Audited Unlearned Sets"] --> B["Feature Extraction & Objective Relaxation<br/>Extract features and model as distribution mixture"]
    B --> C["Moment Matching & Convex Optimization<br/>Low-order moments (SMI) or Kernel Embeddings (SMI-M)"]
    C --> D["Feature Resampling Uncertainty Estimation<br/>Bootstrap sampling for 90% reference range"]
    D --> E["Auditing Output<br/>Point estimate forgetting ratio and confidence bounds"]

Key Designs

1. Objective Relaxation and Mixture Proportion Modeling: Resolving Feature Alignment Bias

Standard MIA naively assumes that the unlearned feature distribution \(\mathcal{F}_u\) matches the test non-member distribution \(\mathcal{F}_n\). Due to negative gradient updates, the distribution divergence \(\mathcal{D}_\infty(\mathcal{F}_t \| \mathcal{F}_u)\) introduces systematic auditing error into MIA classifiers. To eliminate this mismatch, SMI moves away from hard sample-level decision boundaries and relaxes the feature-space Monge problem into a convex combination of member and non-member distributions:

\[\mathcal{F}_u \approx (1 - \rho)\mathcal{F}_m + \rho\mathcal{F}_n, \quad \rho \in [0, 1]\]

Here, the mixture coefficient \(\rho\) quantifies the proportion of non-member characteristics exhibited by the unlearned population. An unlearned model corresponds to \(\rho = 0\) when untouched, and reaches \(\rho = 1\) when achieving ideal retraining. Consequently, the auditing task reduces to finding the optimal proportion \(\rho^* = \arg\min_{\rho \in [0,1]} D(\mathcal{F}_u, (1-\rho)\mathcal{F}_m + \rho\mathcal{F}_n)\).

2. Low-Order Moment Matching (SMI): Closed-Form Estimation via Cubic Optimization

To bypass classifier training and hyperparameter tuning, SMI exploits the first- and second-order statistical moments of the mixture distribution. Under the mixture model, the theoretical mean and covariance satisfy \(\mu_u = \rho \mu_n + (1-\rho)\mu_m\) and \(\Sigma_u = \rho \Sigma_n + (1-\rho)\Sigma_m + (\rho - \rho^2)(\mu_n - \mu_m)(\mu_n - \mu_m)^\top\). Setting \(d = \mu_n - \mu_m\), \(B = dd^\top\), \(A = \Sigma_u - \Sigma_m\), and \(G = \Sigma_n - \Sigma_m + B\), matching the empirical covariance under the Frobenius norm reduces to:

\[\min_{0 \le \rho \le 1} \| A - \rho G + \rho^2 B \|_F^2\]

Differentiating with respect to the scalar \(\rho\) yields a univariate cubic equation: \(2\|B\|_F^2 \rho^3 - 3\langle G, B\rangle_F \rho^2 + (\|G\|_F^2 + 2\langle A, B\rangle_F)\rho - \langle A, G\rangle_F = 0\). The candidate solution set \(\mathcal{C}\) is restricted to \(\{0, 1\}\) and the real roots falling in \([0, 1]\). Evaluating the objective at these candidate roots yields the exact global minimizer in microseconds, requiring neither back-propagation nor iterative training.

3. RKHS Kernel Mean Embedding (SMI-M): High-Dimensional Statistical Stability

In small-sample or high-dimensional feature settings, empirical covariance estimation can exhibit high variance, degrading the accuracy of raw moment matching. SMI-M addresses this by embedding feature representations into a Reproducing Kernel Hilbert Space (RKHS) via a kernel function \(k(\cdot, \cdot)\). This maps second-order moment matching into first-order kernel mean embeddings. Measuring distribution divergence via Maximum Mean Discrepancy (MMD) transforms the optimization into a convex quadratic program:

\[\min_{0 \le \rho \le 1} \rho^2 \|\mu_v^{(k)} - \mu_t^{(k)}\|_\mathcal{H}^2 - 2\rho \langle \mu_f^{(k)} - \mu_t^{(k)}, \mu_v^{(k)} - \mu_t^{(k)} \rangle_\mathcal{H}\]

Because the quadratic objective is strictly convex, it can be resolved analytically or via standard one-dimensional convex solvers. Kernel mean embeddings effectively smooth high-frequency estimation noise, granting SMI-M superior stability across architectures and sample scales.

4. Feature Resampling Uncertainty Estimation: Shadow-Model-Free Confidence Intervals

Conventional white-box MIA relies on training ensembles of auxiliary shadow models to quantify decision variance, creating extreme computational bottlenecks. SMI avoids multi-model training entirely by performing Bootstrap resampling on the unlearned feature representations \(D_u\). By resampling \(D_u\) with replacement for \(B\) iterations (e.g., \(B=200\)) and re-solving the scalar convex program on each bootstrap replicate, SMI produces an empirical distribution of estimates \(\{\rho^*_b\}_{b=1}^B\). This yields an empirical 90% reference range \([\rho^*_{5\%}, \rho^*_{95\%}]\) at negligible computational expense, providing a rigorous statistical uncertainty metric for audit decisions.

Loss & Training

SMI is completely training-free and model-agnostic. It does not train any neural network weights, shadow models, or auxiliary discriminators. The audited model weights \(w\) remain strictly frozen during feature extraction \(g(x; w)\). The default kernel for SMI-M is the Gaussian RBF kernel, whose bandwidth parameter is chosen via the standard median heuristic computed over pairwise Euclidean distances across member, non-member, and unlearning samples.

Key Experimental Results

Main Results

Experiments were conducted on CIFAR-10, CIFAR-100, and CINIC-10 with ResNet-18, ResNet-50, and ViT backbones under 5% and 10% random unlearning splits, using exact retraining as the golden standard. Baselines include white-box implementations of LiRA, RMIA, IAM, and RULI. Representative results are summarized below:

Auditing Setting (Model - Dataset - Unlearn% - Shadows)	Metric	SMI	SMI-M	RULI (SOTA)	IAM	LiRA	Unleak	Random
ResNet18 - CIFAR10 - 5% - 64	Forgetting Ratio \(\rho^*\)	88.89%	92.44%	87.94%	88.67%	87.84%	62.58%	50.06%
ResNet18 - CIFAR10 - 5% - 64	False Negative Rate (FNR)	–	–	13.28%	12.72%	13.80%	41.24%	49.52%
ResNet18 - CIFAR10 - 5% - 64	False Positive Rate (FPR)	–	–	0.00%	13.08%	0.00%	43.48%	50.88%
ResNet18 - CIFAR100 - 5% - 16	Forgetting Ratio \(\rho^*\)	79.64%	89.84%	83.00%	89.36%	86.64%	28.52%	50.88%
ResNet50 - CIFAR100 - 5% - 32	Forgetting Ratio \(\rho^*\)	81.22%	92.64%	73.08%	63.56%	87.04%	10.40%	54.68%
ResNet50 - CIFAR100 - 10% - 32	Forgetting Ratio \(\rho^*\)	83.46%	93.95%	74.44%	64.98%	87.38%	4.68%	54.14%
ResNet50 - CINIC10 - 10% - 16	Forgetting Ratio \(\rho^*\)	90.89%	92.27%	89.88%	85.35%	90.47%	78.02%	49.76%
ViT - CINIC10 - 10% - 16	Forgetting Ratio \(\rho^*\)	91.44%	93.62%	89.84%	41.01%	83.51%	55.71%	51.04%

Ablation Study

The table below contrasts the kernel choice and bandwidth sensitivity of SMI-M on CIFAR-100 against the computational runtime across auditing paradigms:

Kernel & Hyperparameter Configuration (CIFAR-100)	5% Unlearn \(\rho^*\)	10% Unlearn \(\rho^*\)	Auditing Method / Stage	Computation Time (ms)	Shadow Models Required
RBF (Median Bandwidth 1.0X)	89.84%	91.84%	Train Single Shadow Model (ResNet-18)	\(2.46 \times 10^6\)	–
RBF (Narrow Bandwidth 0.25X)	24.39%	26.03%	Single Target Model Forward Pass	\(4.47 \times 10^4\)	–
RBF (Narrow Bandwidth 0.5X)	77.70%	79.22%	LiRA (White-box with Shadows)	1,464.51	8-64
RBF (Wide Bandwidth 2.0X)	86.88%	89.88%	RULI (White-box with Shadows)	5,672.83	8-64
Laplacian Kernel	80.05%	82.41%	SMI (Moments + 200 Replicates)	930.71	0
Linear Kernel	86.65%	89.88%	SMI-M (Kernel MMD + 200 Replicates)	4,301.66	0

When evaluating practical approximate unlearning methods (FineTune, Fisher, Scrub), SMI-M accurately identified varying degrees of real forgetting: Scrub achieved the highest verified forgetting rates (72%–77%), FineTune hovered around 60%, and Fisher reached only 35%–45%, demonstrating consistent sensitivity to incomplete unlearning.

Key Findings

• In the ResNet18-CIFAR10-5% complete unlearning setting, SMI-M achieved 92.44% forgetting estimation, aligning closest with the ideal retrained model benchmark without requiring shadow training.
• Conventional MIA methods exhibit severe alignment degradation. On Vision Transformers (ViT), IAM and RMIA collapsed to 41.01% and 15.33% respectively, whereas SMI-M preserved 93.62% fidelity.
• Computational efficiency improved by orders of magnitude: training a single ResNet-18 shadow model requires over \(2.46 \times 10^6\) ms (~41 minutes), whereas SMI-M completed the entire audit in 4,301 ms and SMI in 930 ms.

Highlights & Insights

• From Individual Classification to Population Mixture: Conventional MIA struggles because unlearned samples subjected to negative gradients occupy an anomalous subspace where individual samples are indistinguishable. Shifting the problem to macroscopic mixture proportion estimation naturally circumvents this alignment mismatch.
• Closed-Form Elegance Without Shadow Models: By mapping low-order moments to a cubic root-finding problem or kernel embeddings to a 1D convex quadratic program, SMI guarantees global optima analytically without gradient descent.
• Anomaly Detection for Forged Unlearning: Under adversarial attacks designed to fool unlearning evaluations, SMI returns unconstrained estimates of \(\rho^* \gg 10\), naturally flagging fabricated forgetting attempts as distribution anomalies.

Limitations & Future Work

• Finite Sample Sensitivity: As a statistical estimator, SMI requires a sufficient number of unlearning samples to establish stable empirical distributions; the authors note a minimum empirical threshold of roughly 20 samples per class.
• Reference Distribution Requirement: SMI requires access to a slice of retained training data (members) and test data (non-members) to construct baseline anchors, which may be constrained in strict zero-knowledge environments.
• Expansion to Generative and Large Language Models: While thoroughly validated on convolutional networks and vision transformers, extending statistical mixture estimation to discrete token sequences in LLMs or latent spaces of diffusion models remains an open frontier.

Related Work & Insights

• vs Conventional MIA Auditing (LiRA / RMIA / RULI): Existing methods depend heavily on ensembles of expensive shadow models and assume unlearned representations match test non-members. SMI eliminates shadow models entirely, cuts runtime from hours to seconds, and resolves theoretical alignment bias.
• vs Optimal Transport Wasserstein Distance (SMI-W): In their author discussion, the authors note that Wasserstein distance assumes geodesic transport paths, whereas real gradient trajectories in neural networks are highly non-geodesic. Relaxing optimal transport to mixture proportion estimation (SMI/SMI-M) proved empirically far superior to Wasserstein-based metrics.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Identifies structural alignment flaws in MIA unlearning auditing and reformulates the problem as mixture proportion estimation]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Covers multiple datasets, architectures, unlearning ratios, approximate unlearning algorithms, and rigorous runtime benchmarks]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical exposition connecting PAC-Bayesian theory, Monge transport, and kernel embeddings into a unified narrative]
• Value: ⭐⭐⭐⭐⭐ [Provides an efficient, shadow-free, and mathematically sound auditing standard for machine unlearning compliance]

Related Papers

• [NeurIPS 2026] Sequential Membership Inference Attacks
• [NeurIPS 2026] USAD: Uncertainty-aware Statistical Adversarial Detection
• [NeurIPS 2026] Persona-Model Collapse in Emergent Misalignment
• [NeurIPS 2026] Patch Rebirth: Fast and Transferable Model Inversion of Vision Transformers
• [NeurIPS 2026] Identified-Set Geometry of Distributional Model Extraction under Top-K Censored API Access