Model Inversion with Layer-Specific Modeling and Alignment for Data-Free Continual Learning

Conference: NeurIPS 2025 arXiv: 2510.26311 Code: None Area: AI Safety / Continual Learning Keywords: Model inversion, data-free continual learning, per-layer optimization, feature alignment, privacy preservation

TL;DR

This paper proposes Per-layer Model Inversion (PMI) for data-free continual learning to accelerate synthetic image generation, and mitigates the feature drift between synthetic and real data via class-level Gaussian feature modeling and contrastive learning, achieving efficient and high-quality data-free knowledge replay.

Background & Motivation

Background: Continual Learning (CL) aims to enable models to retain performance on previous tasks while learning new ones. Conventional approaches rely on storing and replaying old data, which is often infeasible in practice due to privacy regulations (e.g., GDPR) or security constraints.

Data-Free Continual Learning: Data-free CL performs incremental learning without retaining any historical data. The core idea is to synthesize pseudo-data from previously trained models via Model Inversion for replay purposes.

Limitations of Prior Work: - Feature drift: Inverting inputs from compressed output labels induces distribution shift between synthetic and real data, and replaying such data may erode previously acquired knowledge. - Computational cost: Standard model inversion requires backpropagation through the entire network at each step, incurring substantial computational overhead. - Large model scalability: Both issues are exacerbated on large-scale pretrained models such as CLIP.

Core Problem: How can the quality of synthetic data be maintained while improving the efficiency of model inversion?

Key Insight: Motivated by the faster convergence of single-layer optimization, the paper proposes a per-layer inversion strategy to initialize full-model inversion, and employs a probabilistic model to constrain the distribution of intermediate features.

Method

Overall Architecture

The method comprises two core components:

1. PMI (Per-layer Model Inversion): A layer-wise optimization strategy for model inversion that provides strong initialization for full-model inversion.
2. Feature Modeling and Alignment: Gaussian distribution modeling and contrastive learning to align intermediate synthetic features with real features.

Key Designs

Per-layer Model Inversion (PMI)

Conventional model inversion directly backpropagates through the entire network:

\[z^* = \arg\min_z \mathcal{L}_{inv}(f(z), y)\]

where \(f\) is the full network and \(z\) is the input to be optimized.

Core Idea of PMI: Decompose the network layer by layer as \(f = f_L \circ f_{L-1} \circ \cdots \circ f_1\), and optimize from the last layer backward:

1. Last layer: \(h_{L-1}^* = \arg\min_h \mathcal{L}(f_L(h), y)\) — solve only for the classification head.
2. Second-to-last layer: \(h_{L-2}^* = \arg\min_h \|f_{L-1}(h) - h_{L-1}^*\|^2\) — reconstruct the intermediate feature.
3. Layer-by-layer backward propagation: Continue until an initialization \(z_0\) for the input layer is obtained.

Advantages: - Each per-layer optimization problem is simpler (single-layer network), yielding faster convergence. - The resulting initialization is substantially better than random initialization, significantly reducing the number of iterations required in subsequent full-model optimization.

Class-Level Feature Modeling

During training on each task, the statistical properties of intermediate features at each layer are recorded:

• For the feature \(h_l^{(c)}\) at layer \(l\) and class \(c\), a Gaussian distribution \(\mathcal{N}(\mu_l^c, \Sigma_l^c)\) is fitted.
• Only mean vectors and covariance matrices are stored (far smaller than raw data).
• These statistics serve as distributional constraints during synthesis.

Contrastive Learning Alignment

To mitigate the distributional drift of synthetic features, a contrastive learning loss is introduced:

\[\mathcal{L}_{align} = -\log \frac{\exp(\text{sim}(h_{syn}^c, \mu^c)/\tau)}{\sum_{c'} \exp(\text{sim}(h_{syn}^c, \mu^{c'})/\tau)}\]

• Positive pairs: synthetic features paired with stored class means of the same category.
• Negative pairs: synthetic features paired with stored class means of other categories.
• This ensures that synthetic features are semantically aligned with real features in the feature space.

Loss & Training

The total loss consists of three terms:

\[\mathcal{L}_{total} = \mathcal{L}_{CE} + \lambda_1 \mathcal{L}_{KD} + \lambda_2 \mathcal{L}_{align}\]

• \(\mathcal{L}_{CE}\): Cross-entropy loss on the new task.
• \(\mathcal{L}_{KD}\): Knowledge distillation loss using the old model's soft outputs on synthetic data as supervision.
• \(\mathcal{L}_{align}\): Contrastive feature alignment loss.

Training procedure: 1. Before learning a new task, synthesize pseudo-images from the old model using PMI. 2. Train on a mixture of real new-task data and synthetic old-task data. 3. Update feature statistics after training.

Key Experimental Results

Main Results

Class-Incremental Learning on CIFAR-100 (10 Stages)

Method	Final Accuracy (%)	Avg. Incremental Accuracy (%)	Forgetting Rate (%)
LwF	49.2	58.3	28.1
EWC	47.8	56.9	30.5
DeepInversion	52.1	61.4	23.7
ABD	54.3	63.1	21.2
PASS	55.8	64.5	19.6
PMI + Feature Align (Ours)	58.7	67.2	16.3

Class-Incremental Learning on ImageNet-100 (10 Stages)

Method	Final Accuracy (%)	Avg. Incremental Accuracy (%)	Forgetting Rate (%)
LwF	58.4	66.1	22.8
DeepInversion	61.3	69.7	18.5
ABD	63.1	71.2	16.9
PMI + Feature Align (Ours)	66.8	74.5	13.2

CLIP-Based Continual Learning

Method	CIFAR-100 Final Acc.	ImageNet-100 Final Acc.	Inversion Time (min)
DeepInversion + CLIP	71.3	74.2	48
ABD + CLIP	73.5	76.1	42
PMI + CLIP (Ours)	76.2	79.4	15

Ablation Study

Contribution of Each Component (CIFAR-100, 10 Stages)

Configuration	Final Accuracy (%)	Forgetting Rate (%)	Inversion Iterations
Full-model inversion (random init)	52.1	23.7	2000
PMI init → full-model optimization	55.3	19.8	500
Full-model inversion + feature alignment	55.9	18.4	2000
PMI + Feature Alignment (Full Method)	58.7	16.3	500

Comparison of Feature Modeling Strategies

Feature Modeling Strategy	Final Accuracy (%)	Storage (MB)
No feature modeling	55.3	0
Mean alignment only	56.8	0.3
Gaussian modeling (mean + covariance)	57.9	1.2
Gaussian + contrastive alignment	58.7	1.2

Key Findings

1. PMI substantially accelerates inversion: The number of inversion iterations is reduced from 2000 to 500 (4× speedup) with higher quality.
2. Feature alignment mitigates forgetting: The feature alignment loss reduces the forgetting rate by approximately 20% relatively.
3. Complementary components: PMI provides a better initialization point while feature alignment provides a better optimization direction; their combination exceeds the sum of individual contributions.
4. Greater advantage on large models: The speedup from PMI is more pronounced on large models, with inversion time reduced by 65% on CLIP.
5. Minimal storage overhead: Feature statistics require only approximately 1.2 MB, far less than storing raw images.

Highlights & Insights

1. Elegant per-layer inversion: Decomposes a difficult global optimization into a series of simpler local problems, yielding strong initialization.
2. Privacy-friendly: No raw data is stored; only feature statistics are retained, complying with data protection regulations.
3. High generality: The method is orthogonal to CL strategies and can be combined with various continual learning approaches.
4. Particularly effective on large models: Efficiency gains are most significant on pretrained models such as CLIP.

Limitations & Future Work

1. Gaussian assumption: Class-level feature distributions are not necessarily Gaussian; multimodal or long-tailed distributions may lead to inaccurate modeling.
2. Covariance storage: For layers with very high feature dimensionality, storing covariance matrices may still be costly (requiring diagonal approximations).
3. Layer decomposition assumption: The approach requires a clearly separable layer structure; additional handling is needed for architectures with residual connections.
4. Classification only: Applicability to more complex tasks such as detection and segmentation has not been explored.
5. Ceiling on synthetic image quality: Even with better initialization and constraints, the quality of model-inverted images is ultimately bounded by the completeness of information encoded in the model.

Related Work & Insights

• DeepInversion: A classical model inversion method that generates pseudo-images using batch normalization statistics and adversarial regularization.
• ABD (Always Be Dreaming): Employs adversarial distillation for data-free CL.
• PASS: A prototype-augmented self-supervised method for data-free continual learning.
• LwF (Learning without Forgetting): A classical knowledge distillation approach for continual learning.

Rating

• Novelty: ★★★★☆ (The combination of per-layer inversion and feature modeling constitutes a meaningful contribution.)
• Experimental Thoroughness: ★★★★☆ (Multiple datasets, multiple settings, and thorough ablations.)
• Value: ★★★★☆ (Strong practical utility in privacy-sensitive scenarios.)
• Writing Quality: ★★★★☆ (Clear motivation and systematic method description.)

Related Papers

• [ICCV 2025] Semantic Alignment and Reinforcement for Data-Free Quantization of Vision Transformers
• [NeurIPS 2025] Private Continual Counting of Unbounded Streams
• [NeurIPS 2025] Provable Watermarking for Data Poisoning Attacks
• [NeurIPS 2025] Incentivizing Time-Aware Fairness in Data Sharing
• [NeurIPS 2025] Private Zeroth-Order Optimization with Public Data