When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution

Conference: ICML 2025
arXiv: 2507.04119
Code: None
Area: Model Compression/Knowledge Distillation
Keywords: Data-Free Knowledge Distillation, Non-Transferable Teacher, Out-of-Distribution, Data Privacy, Model Compression

TL;DR

This paper investigates the challenges faced by Data-Free Knowledge Distillation (DFKD) when the teacher model is designed to be "non-transferable." Since synthetic samples tend to fall into out-of-distribution (OOD) regions which leads to distillation failure, this work proposes an "escaping OOD" approach to achieve effective distillation.

Background & Motivation

Background

Background: Data-Free Knowledge Distillation (DFKD) synthesizes training data via a generator to train the student model without accessing the original training data, which is highly valuable in data privacy scenarios. Meanwhile, Non-Transferable Learning aims to protect intellectual property by degrading model performance outside authorized domains.

Limitations of Prior Work: When the teacher model undergoes non-transferable training, it yields misleading outputs in out-of-distribution (OOD) regions. Consequently, the generator in traditional DFKD may produce OOD samples, causing these misleading insights to be transferred to the student model.

Key Challenge: DFKD requires synthetic data to explore the teacher's knowledge, yet a non-transferable teacher provides "poison" instead of beneficial "knowledge" in OOD regions.

Goal: To enable effective data-free knowledge distillation when the teacher model is designed with non-transferability.

Key Insight: Modeling the problem as "escaping OOD" to ensure that the synthetic samples generated by the generator fall within the reliable regions of the teacher model as much as possible.

Core Idea: To make DFKD feasible under a non-transferable teacher by detecting and avoiding out-of-distribution regions.

Method

Overall Architecture

• Input: Non-transferable teacher model (without original training data)
• Core Module: OOD-aware sample generation
• Output: A well-performing student model

Key Designs

1. OOD Detection Mechanism:

   • Leverages the output characteristics of the teacher model (e.g., confidence patterns, feature statistics) to determine whether synthetic samples are in-distribution.
   • Non-transferable teachers exhibit characteristic output patterns (e.g., excessively low confidence or random outputs) in OOD regions.
   • Design Motivation: To identify "untrustworthy" regions of the teacher and avoid learning from them.

2. OOD-Aware Generator Training:

   • Modifies the generator's objective function by incorporating an OOD penalty term.
   • Guides the generator to produce synthetic samples closer to the in-distribution area.
   • Potentially employs contrastive learning or feature matching to constrain the generative distribution.
   • Design Motivation: To reduce the generation of OOD samples at the source.

3. Adaptive Distillation Strategy:

   • Assigns varying weights to the teacher's outputs based on confidence.
   • Ensures higher weights are given to higher-confidence outputs, which are more likely to be in-distribution.
   • Design Motivation: To extract knowledge from reliable outputs even if the generator is imperfect.

Loss & Training

• Generator Loss = Traditional DFKD loss + OOD penalty term
• Student Loss = Weighted KL divergence (weights based on OOD detection scores)
• Alternating training of the generator and the student model

Key Experimental Results

Main Results

Method	CIFAR-10	CIFAR-100	Gain
Standard DFKD (Normal Teacher)	High Baseline	High Baseline	-
Standard DFKD (Non-Transferable Teacher)	Significant Drop	Significant Drop	Issue Exposed
Ours (Non-Transferable Teacher)	Recovered	Recovered	Effectively Alleviated

Ablation Study

Configuration	Student Performance	Description
W/o OOD Detection	Poor	Learned misleading knowledge
+ OOD Detection	Improved	Filters unreliable outputs
+ OOD-Aware Generation	Further Improved	Reduces OOD samples from the source
+ Adaptive Weights	Best	Comprehensive protection

Key Findings

• Non-transferable teachers indeed severely undermine the performance of standard DFKD.
• OOD detection and OOD-aware generation are complementary and highly effective strategies.
• The stronger the non-transferability of the teacher, the more noticeable the performance drop of standard DFKD.
• The proposed method achieves effective distillation while maintaining intellectual property protection.

Highlights & Insights

• This work is the first to investigate the interaction between non-transferable teachers and DFKD, representing an intersection of two critical research directions.
• The problem formulation is practically relevant: model providers may use non-transferability techniques to protect models, while users still wish to perform distillation.
• The "escaping OOD" framework can be extended to other KD scenarios where the teacher's quality is non-uniform.
• It reveals the vulnerability of DFKD regarding the quality of the teacher model.

Limitations & Future Work

• The accuracy of OOD detection directly impacts the performance of the proposed method.
• If the non-transferable regions of the teacher mix with the in-distribution regions, detection becomes more challenging.
• The method can be scaled to NLP and large language model distillation scenarios.
• The interaction with other IP protection techniques like model watermarking represents an interesting avenue for research.

Related Work & Insights

• Combines two research domains: DFKD (e.g., DeepInversion, CMI) and non-transferable learning.
• Closely related to research in OOD detection (e.g., ODIN, Energy-based).
• Insight: Knowledge distillation needs to consider the "quality distribution" of the teacher model rather than assuming the teacher is reliable everywhere.

Rating

• Novelty: ⭐⭐⭐⭐ First study at the intersection of two critical directions
• Experimental Thoroughness: ⭐⭐⭐⭐ Validated across multiple datasets and through comprehensive ablations
• Writing Quality: ⭐⭐⭐⭐ Clear problem definition and strong motivation
• Value: ⭐⭐⭐⭐ Practical significance for both knowledge distillation and model protection

Related Papers

• [CVPR 2025] DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture
• [ICCV 2025] A Good Teacher Adapts Their Knowledge for Distillation
• [ICLR 2026] Q&C: When Quantization Meets Cache in Efficient Generation
• [ICML 2025] A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features
• [ICCV 2025] Gradient Short-Circuit: Efficient Out-of-Distribution Detection via Feature Intervention