Enhancing DPSGD via Per-Sample Momentum and Low-Pass Filtering

Conference: AAAI2026
arXiv: 2511.08841
Code: To be confirmed
Area: AI Safety
Keywords: differential privacy, DPSGD, Per-Sample Momentum, Low-Pass Filtering, Privacy-Utility Trade-off

TL;DR

Proposed DP-PMLF, which reduces clipping bias via per-sample momentum while simultaneously suppressing high-frequency DP noise using a low-pass filter, mitigating the accuracy degradation of DPSGD from both directions for the first time.

Background & Motivation

Differentially Private Stochastic Gradient Descent (DPSGD) provides formal privacy guarantees for deep learning through gradient clipping and noise injection, but suffers from severe accuracy loss. The root cause lies in two conflicting factors:

1. DP Noise: Protecting privacy requires injecting calibrated Gaussian noise into the aggregated gradients, where the noise scale is proportional to the clipping threshold \(C\)—the larger \(C\) is, the more noise is introduced.
2. Clipping Bias: Clipping the norm of per-sample gradients introduces bias—the smaller \(C\) is, the larger the bias.

Most existing methods only address one of these: - LP-DPSGD (Zhang et al.) uses a low-pass filter to reduce DP noise, but introduces an additional bias term, leading to worse performance when clipping bias dominates. - InnerOuter (Xiao et al.) uses inner and outer momentum to reduce clipping bias, but lacks suppression of DP noise, and the lack of normalization in the outer momentum accumulates noise, resulting in severe performance degradation under \(\epsilon=1\) (strong privacy).

The authors observe that clipping bias is not only related to the threshold \(C\) but also proportional to the sampling variance \(\sigma_{SGD}\), suggesting room to simultaneously reduce both noise and bias.

Core Problem

How to simultaneously mitigate both DP noise and clipping bias in DPSGD without consuming additional privacy budget?

Method

Overall Architecture: DP-PMLF

DP-PMLF consists of two complementary modules that sequentially operate on the gradient processing pipeline:

1. Per-Sample Momentum

Maintains a momentum term for each sample \(\xi\), performing an exponentially decaying weighted average over the gradients from the past \(k\) steps:

\[v_t^{(\xi)} = \sum_{i=t-k+1}^{t} \hat{\beta}^{t-i} \nabla f^{(\xi)}(x_i)\]

where \(\hat{\beta}^{t-i} = \beta^{t-i}/c_\beta\), and \(c_\beta\) is a normalization constant ensuring the sum of coefficients equals 1.

Function: Smooths gradient estimation before clipping, reducing the sampling variance \(\sigma_{SGD}\). Theoretically, the variance can be reduced by a factor of \(\rho^2\), where:

\[\rho = \sqrt{\frac{(1+\beta)(1-\beta^k)}{(1-\beta)(1+\beta^k)}}\]

As \(\beta \to 1\), \(\rho^2 \to k\), which corresponds to simple average. The normalization also avoids the noise accumulation problem caused by excessively large momentum coefficients (a drawback of InnerOuter).

2. Low-Pass Filter

After clipping, aggregated gradients are added with Gaussian noise, and then a linear low-pass filter is applied:

\[m_t = -\sum_{r=1}^{n_a} a_r m_{t-r} + \sum_{r=0}^{n_b} b_r \bar{v}_{t-r}\]

The filter coefficients satisfy \(-\sum a_r + \sum b_r = 1\) to guarantee that the mean of the signal remains unchanged.

Mechanism: DP noise is uniformly distributed across all frequencies, whereas the true gradient signal is concentrated in the low-frequency band. The low-pass filter retains the gradient signal while suppressing high-frequency noise. Since the filter is applied only as post-processing on the already-noised output, according to the DP post-processing lemma, it consumes no additional privacy budget.

3. Initialization Bias Correction

A normalization constant \(c_{m,t}\) is computed recursively to output \(\hat{m}_t = m_t / c_{m,t}\), which corrects the transient bias in the initial stage of the filter.

Algorithmic Pipeline

1. Sample mini-batch \(\mathcal{B}_t\)
2. Compute per-sample momentum \(v_t^{(\xi)}\) for each sample
3. Clip \(\tilde{v}_t^{(\xi)} = \text{clip}(v_t^{(\xi)}, C)\)
4. Aggregate and add noise: \(\bar{v}_t = \frac{1}{B}\sum \tilde{v}_t^{(\xi)} + w_t\)
5. Low-pass filtering + bias correction \(\to\) \(\hat{m}_t\)
6. Update model: \(x_{t+1} = x_t - \eta \hat{m}_t\)

Theoretical Guarantees

• Convergence: Under standard assumptions such as \(L\)-smoothness, bounded variance, and bounded gradients, the convergence upper bound is

\[\mathcal{O}\!\left(\frac{f(x_0)-f^*}{\eta T} + L\eta C^2 + \frac{L\eta \, d\sigma_{DP}^2}{\Gamma_{DP}} + \frac{\sigma_{SGD}^2}{\rho^2 \Gamma_{SGD}}\right)\]

where \(\Gamma_{DP}\) and \(\Gamma_{SGD}\) represent the suppression factors of the low-pass filter on DP noise and clipping bias, respectively. Compared to vanilla DPSGD, the clipping bias term is additionally divided by \(\rho^2\), and the DP noise term is additionally divided by \(\Gamma_{DP}\).

• Privacy: Using the Gaussian mechanism + subsampled privacy amplification + moments accountant, it satisfies \((\epsilon, \delta)\)-DP.

Key Experimental Results

Image Classification (ViT, No Pre-training)

Method	CIFAR-10 (\(\epsilon\)=1)	CIFAR-10 (\(\epsilon\)=8)	CIFAR-100 (\(\epsilon\)=1)	CIFAR-100 (\(\epsilon\)=8)
DPSGD	35.74	47.74	7.52	18.27
LP-DPSGD	35.84	48.37	7.55	18.52
InnerOuter	11.55	33.53	1.13	13.93
DP-PMLF	40.96	51.47	11.40	23.15

• Outperforms the best baseline by approximately 5% on CIFAR-10 under \(\epsilon=1\), and by approximately 4% on CIFAR-100.
• InnerOuter severely degrades under \(\epsilon=1\) due to noise accumulation (reaching only 11.55% on CIFAR-10).

Sentence Classification (RoBERTa-base Fine-tuning, GLUE)

Method	MNLI (\(\epsilon\)=1)	QNLI (\(\epsilon\)=1)	QQP (\(\epsilon\)=8)	SST-2 (\(\epsilon\)=8)
DPSGD	51.36	65.59	80.38	90.83
DP-PMLF	56.81	72.38	83.42	90.39

• Outperforms the baseline by over 4% on MNLI, and by nearly 3% on QNLI (\(\epsilon=1\)).

Multiple Model Architectures (CIFAR-10, \(\epsilon=1\))

• CNN-5: DP-PMLF achieves ~47%, outperforming the best baseline by approximately 9%
• ResNet-18: DP-PMLF achieves ~50%, outperforming by approximately 1-2%
• ViT: DP-PMLF achieves ~31%, outperforming by approximately 8%

Ablation Study

• Removing per-sample momentum \(\to\) leads to consistent performance degradation, validating its effectiveness in reducing clipping bias.
• Removing the low-pass filter \(\to\) leads to performance degradation when \(\epsilon \leq 6\) (where noise is large); however, when \(\epsilon > 6\), over-smoothing conversely results in a minor loss of true gradient information (roughly 0.5-0.7%).

Highlights & Insights

1. Simultaneously addressing both major degradation sources for the first time: Elegantly combines per-sample momentum (reducing variance/bias) and a low-pass filter (reducing noise), covering the scenarios where LP-DPSGD and InnerOuter respectively fail.
2. Post-processing noise reduction with zero privacy cost: Utilizing the post-processing lemma of DP, the low-pass filter consumes no privacy budget.
3. Rigorous theoretical guarantees: Provides complete convergence analysis and privacy proof, with the convergence upper bound clearly showing the individual contribution of each module.
4. Cross-modal generalization: Demonstrates effectiveness in both image classification (CNN/ResNet/ViT) and sentence classification (RoBERTa).

Limitations & Future Work

1. Hyperparameter sensitivity: \(\beta\), \(k\), and filter coefficients \(\{a_r\}, \{b_r\}\) require manual tuning; the authors do not propose an adaptive selection method.
2. Per-sample history storage overhead: Requires maintaining gradients of the last \(k\) steps for each sample, with the memory overhead scaling with the dataset size and \(k\).
3. Low absolute accuracy under strong privacy: Under \(\epsilon=1\), the accuracy on CIFAR-10 reaches at most 40.96%, which is still far from practical utility.
4. Strong theoretical assumptions: Requires assumptions such as bounded gradients (Assumption 3) and gradient autocorrelation (Assumption 4); it has not been extended to more general non-convex conditions (e.g., PL condition, \((L_0, L_1)\)-smoothness).
5. Risk of over-smoothing: The ablation study shows that when DP noise is small (\(\epsilon > 6\)), the low-pass filter slightly degrades performance instead.

Related Work & Insights

Method	Reduces DP Noise	Reduces Clipping Bias	Theoretical Guarantees	Extra Privacy Cost
LP-DPSGD	✓	✗ (increases bias instead)	Yes (contains extra bias term)	None
InnerOuter	✗ (accumulates noise)	✓	None	None
DiceSGD	✗ (requires more noise)	✓ (error feedback)	Yes	Yes
Clipless DPSGD	✓	✓ (no clipping)	Yes	None
DP-PMLF	✓	✓	Yes	None

Inspirations & Connections

• Variance reduction as a general tool: The core of per-sample momentum is reducing sampling variance, an idea that can be generalized to other DP optimizers (e.g., DP-Adam).
• The frequency-domain perspective is worth further exploration: The low-pass filter exploits the spectral difference between the gradient signal and the noise. High-order or adaptive filters might yield further improvements.
• Integration with Federated Learning: Federated learning also suffers from accuracy loss due to noise and gradient compression. Similar strategies of momentum smoothing and frequency-domain filtering could be effective.

Rating

• Novelty: 7/10 — A clever combination of two known components. The core contribution lies in the insight and theoretical analysis of "simultaneous treatment".
• Experimental Thoroughness: 7/10 — Covers multiple datasets, models, and modalities, but lacks evaluation on large-scale models and comparison with more DPSGD variants.
• Writing Quality: 8/10 — Clear motivation, well-organized theory and experiments.
• Value: 7/10 — Provides a practical and theoretically-backed improvement in the field of DP training.

Related Papers

• [ICLR 2026] Video Unlearning via Low-Rank Refusal Vector
• [CVPR 2026] FedMOP: Achieving Enhanced Privacy and Performance in Federated Learning via Momentum Orthogonal Projection
• [ICLR 2026] Prior-based Noisy Text Data Filtering: Fast and Strong Alternative for Perplexity
• [ICLR 2026] Memorization Through the Lens of Sample Gradients
• [CVPR 2025] Joint Out-of-Distribution Filtering and Data Discovery Active Learning