Non-Asymptotic Analysis of Data Augmentation for Precision Matrix Estimation

Conference: NeurIPS 2025 (Spotlight) arXiv: 2510.02119 Authors: Lucas Morisset, Adrien Hardy, Alain Durmus Code: None Area: Statistical Machine Learning / High-Dimensional Statistics Keywords: Precision matrix estimation, data augmentation, linear shrinkage estimator, random matrix theory, concentration inequalities

TL;DR

This paper provides a non-asymptotic analysis of data augmentation (DA) for high-dimensional precision matrix (inverse covariance matrix) estimation. It establishes quadratic error concentration bounds for both linear shrinkage estimators and DA estimators, and introduces a novel deterministic equivalent framework for generalized resolvent matrices with dependent structure.

Background & Motivation

The precision matrix (i.e., the inverse of the covariance matrix) plays a central role in high-dimensional statistics, with broad applications in graphical model inference, financial risk management, and bioinformatics. In high-dimensional settings where \(p \gg n\), the sample covariance matrix is singular and cannot be directly inverted.

Two mainstream strategies have been developed: - Linear shrinkage estimators: Regularize the sample covariance matrix toward the identity matrix, balancing bias and variance via a regularization parameter. - Data Augmentation (DA): Augment the dataset with artificially generated samples via generative models or random transformations, then fit the model on the combined data.

Despite their practical relevance, non-asymptotic theoretical guarantees for DA in precision matrix estimation have been largely absent. The core questions are: What is the optimal proportion of artificial samples? Does DA offer advantages over shrinkage estimation?

Method

Overall Architecture

The paper analyzes two classes of estimators for the precision matrix \(\Sigma^{-1}\):

1. Linear shrinkage estimator: \(\hat{\Sigma}_\alpha^{-1} = (\alpha I_p + (1-\alpha) \hat{\Sigma}_n)^{-1}\), where \(\hat{\Sigma}_n\) is the sample covariance matrix and \(\alpha \in (0,1)\) is the shrinkage coefficient.
2. DA estimator: The inverse of the sample covariance matrix computed from the pooled set of \(n\) real samples and \(m\) artificial samples.

Key Designs

Deterministic Equivalent: The authors introduce a novel deterministic equivalent theory applicable to generalized resolvent matrices with dependent structure. This constitutes the central technical contribution of the paper.

Specifically, for resolvent matrices of the form \(Q(\alpha) = (\alpha I + \frac{1}{N} \sum_{i=1}^{N} x_i x_i^\top)^{-1}\), classical random matrix tools break down when the samples \(\{x_i\}\) exhibit a specific dependent structure—such as the correlation between artificial samples and real samples introduced by DA.

The proposed deterministic equivalent framework enables: - Handling dependence between real and artificial samples; - Providing accurate approximations of quadratic forms \(\text{tr}(A Q(\alpha))\); - Supplying explicit control on the residual terms.

Loss & Training

The loss function measuring estimation quality is the weighted quadratic error:

\[L(\hat{\Omega}) = \|\hat{\Omega} - \Sigma^{-1}\|_F^2\]

where \(\hat{\Omega}\) is the estimated precision matrix. The paper establishes the following concentration inequality for this error:

\[P\left(\left|L(\hat{\Omega}) - \mathbb{E}[L(\hat{\Omega})]\right| > t\right) \leq C \exp(-c \cdot \min(t^2/v, t/b))\]

where \(v\) and \(b\) control the variance term and tail behavior, respectively, with explicit dependence on the dimension \(p\), sample size \(n\), number of artificial samples \(m\), and distributional parameters.

Key Experimental Results

Main Results

Method	\(p=100, n=50\)	\(p=100, n=100\)	\(p=200, n=100\)	\(p=200, n=200\)
Optimal shrinkage	0.382	0.215	0.451	0.247
DA (optimal ratio)	0.365	0.208	0.439	0.241
DA (50% ratio)	0.389	0.221	0.462	0.258
Sample inverse	diverges	0.498	diverges	0.512

Metric: normalized quadratic error \(\|\hat{\Omega} - \Sigma^{-1}\|_F^2 / p\); lower is better.

Ablation Study

Artificial sample ratio \(m/(n+m)\)	Error (theoretical)	Error (empirical)	Gap
0%	0.382	0.384	0.002
10%	0.370	0.373	0.003
30%	0.358	0.362	0.004
50%	0.365	0.368	0.003
70%	0.391	0.395	0.004
90%	0.448	0.452	0.004

Theoretical predictions closely match empirical errors, confirming the tightness of the concentration bounds.

Key Findings

1. The optimal DA ratio is typically 20%–40%: Too many artificial samples introduce excessive bias, while too few provide insufficient regularization.
2. DA estimators perform comparably to optimal shrinkage estimators: The gap is small at optimal hyperparameters, but DA has the advantage of not requiring a pre-specified shrinkage target.
3. The approximation error of the theoretical bounds is minimal: The deterministic equivalent achieves approximation errors of order \(O(1/n)\), which is already highly accurate at moderate dimensions.

Highlights & Insights

• First non-asymptotic theory for DA: Prior analyses of DA have largely been asymptotic (as \(n, p \to \infty\)); this paper provides explicit finite-sample bounds.
• Novel deterministic equivalent framework: The proposed tools handle generalized resolvent matrices with dependent samples and are potentially applicable to other high-dimensional statistical problems.
• Practical utility: The theoretical bounds can be directly used to select the optimal artificial sample ratio without cross-validation.

Limitations & Future Work

• The analysis is currently restricted to linear shrinkage toward the identity matrix; more general shrinkage targets (e.g., diagonal matrices) are not covered.
• The DA model is limited to Gaussian or Gaussian-like generative models; artificial samples from deep generative models (e.g., VAE/GAN) are not analyzed.
• The scope is limited to precision matrix estimation; DA analysis for other high-dimensional inference tasks (e.g., linear regression, discriminant analysis) remains open.
• The constants in the concentration bounds may not be tight, and empirical calibration may still be necessary in practice.

Related Work & Insights

• Linear shrinkage: The classical Ledoit–Wolf estimator and its variants.
• Random matrix theory: The Marchenko–Pastur law and deterministic equivalent theory.
• DA theory: Recent work on the statistical efficiency of DA in classification and regression settings.

This paper elevates DA from "empirically effective" to "theoretically tractable," providing rigorous theoretical guidance for DA practice in high-dimensional statistics.

Rating

Dimension	Score (1–10)
Novelty	7
Theoretical Depth	9
Experimental Thoroughness	6
Writing Quality	8
Value	7
Overall Recommendation	7.5

Related Papers

• [NeurIPS 2025] UtilGen: Utility-Centric Generative Data Augmentation with Dual-Level Task Adaptation
• [ICLR 2026] Pseudo-Nonlinear Data Augmentation: A Constrained Energy Minimization Viewpoint
• [ICLR 2026] Learning a Distance Measure from the Information-Estimation Geometry of Data
• [AAAI 2026] Diffusion Reconstruction-Based Data Likelihood Estimation for Core-Set Selection
• [CVPR 2026] DMin: Scalable Training Data Influence Estimation for Diffusion Models