Flash-DD: An Ultra Parameter-Efficient Approach to Dataset Distillation

Conference: ECCV 2026
Paper: ECCV Official
Area: Model Compression
Keywords: Dataset Distillation, Soft-Label Generation, Parameter-Efficient, Model Pruning, Model Ensemble

TL;DR

To resolve the excessive storage, slow online inference, and suboptimal generalization caused by full-sized teacher models in large-scale dataset distillation, Flash-DD introduces dataset-distillation-oriented model parameter reduction and adaptive ensemble label generation, cutting extra storage down to 0.03% with an 843.81× speedup, while improving accuracy by up to 13.4% under a 1.8% storage budget.

Background & Motivation

Dataset distillation (DD) aims to synthesize a compact dataset encapsulating the essential knowledge of massive original datasets, dramatically reducing storage/transmission burdens and accelerating downstream task training. While near-lossless compression has been achieved on small-scale, low-resolution benchmarks such as CIFAR-10/100, scaling to high-resolution, complex datasets like ImageNet-1K and Places365 causes acute performance degradation at high compression ratios. To combat this, state-of-the-art large-scale DD frameworks (e.g., SRe2L, G_VBSM, RDED) couple aggressive data augmentations with soft-label supervision produced by well-trained teacher models during downstream training, providing informative probabilistic targets to compensate for missing sample diversity.

However, this prevailing paradigm introduces three critical drawbacks in practical applications. First, retaining full-sized teacher models (such as ResNet-18 requiring 44.7MB) throughout downstream training imposes substantial extra storage overhead that frequently dwarfs the size of the synthetic images themselves (e.g., only a few megabytes at IPC 1 or 10), directly violating the fundamental storage-efficiency premise of dataset distillation. Second, evaluating augmented instances through teacher models at every epoch incurs severe GPU computational latency, drastically slowing downstream training. Most crucially, theoretical and empirical analyses reveal that guidance from full-sized teachers yields suboptimal downstream generalization: because the synthetic dataset supports only a restricted hypothesis space, downstream student networks struggle to absorb overly complex teacher representations, leading to severe optimization friction and negative transfer.

The core tension lies in the fundamental capacity mismatch between uncompressed teacher models and highly compressed synthetic datasets. Core idea: automatically search for and prune a compact teacher model matched to the learnable capacity of the synthetic dataset, matching the generalization lower bound with ultra-low parameter budgets, and deploying a constrained ensemble strategy across expanded storage spaces to maximize supervisory efficiency.

Method

Overall Architecture

Flash-DD introduces a plug-and-play, storage-adaptive label generation pipeline designed specifically for the online training phase of large-scale dataset distillation. The system encompasses a single-teacher extreme compression stage and an adaptive multi-teacher ensemble stage. Given a pre-trained full-sized teacher model and a synthetic dataset at a target IPC, Flash-DD first applies structured pruning guided by early downstream training loss dynamics and teacher accuracy to locate the optimal teacher capacity. When additional storage budget is permitted, Flash-DD restructures the available parameter space across multiple sub-teachers governed by strict capacity symmetry and lower-bound constraints, feeding lightweight, highly discriminative soft-label streams to downstream students.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Pretrained Teacher + Distilled Dataset"] --> B["DD-Oriented Parameter Reduction & Capacity Search<br/>Structured pruning + dynamic loss drop tracking"]
    B --> C["Knowledge Capacity Alignment & Stopping Bound<br/>Spectral lower bound + feature clustering audit"]
    C -->|Constrained single-teacher budget| D["Optimal Compact Single Teacher<br/>Ultra-lightweight inference"]
    C -->|Expanded storage budget| E["Adaptive Ensemble Soft-Label Generation<br/>Multi-sub-teacher complementary weighted ensemble"]
    D --> F["Downstream Efficient Online Soft-Label Training"]
    E --> F

Key Designs

1. DD-Oriented Parameter Reduction & Capacity Search: matching the representation ceiling of synthetic data

Standard model compression optimizes solely for the teacher's own test accuracy, whereas teachers in dataset distillation must directly serve downstream transfer on highly compressed synthetic datasets. An overly complex teacher presents an excessively difficult imitation objective, causing sluggish early loss descent for the student, while an over-simplified teacher strips away essential inter-class semantic structures. Flash-DD adopts \(L_1\)-norm structured channel pruning as the base parameter reduction operator. Within the tolerance window \(\delta\) where the teacher's initial accuracy remains intact, redundant parameters are rapidly eliminated. Subsequently, the method tracks the student's early-stage downstream loss reduction \(\mathcal{D}is(\theta_T)\):

\[\mathcal{D}is(\theta_T) = \mathbb{E}_{\theta_S^{(0)} \sim \mathcal{P}_\theta}\left[\mathcal{L}(\mathcal{D}_S; \theta_S^{(t)}, \theta_T) - \mathcal{L}(\mathcal{D}_S; \theta_S^{(0)}, \theta_T)\right]\]

where \(t\) denotes early training epochs (e.g., 50 epochs for ImageNet-1K/Places365, 20 epochs for ImageNet-100). As parameter reduction proceeds, the cumulative loss reduction curve plateaus; the finite difference ratio \(\Delta \mathcal{D}is / \Delta |\theta_T|\) identifies the threshold \(\zeta\) where incremental gains diminish. The optimal single teacher \(\theta_T^*\) is jointly determined by maximizing both early loss reduction dynamics and the teacher's intrinsic accuracy:

\[\theta_T^* = \arg\max_{\theta_T \in \Theta_T} \left( \mathcal{D}is(\theta_T) + \gamma \mathcal{A}cc(\theta_T) \right)\]

where \(\gamma\) scales the accuracy term and the accumulated-loss term onto comparable magnitudes (\(\gamma = 0.1\) for ImageNet-1K and Places365; \(\gamma = 0.05\) for ImageNet-100). This selection criterion accurately pinpoints the critical capacity sweet spot where students absorb dark knowledge most effectively.

2. Knowledge Capacity Alignment & Stopping Bound: theoretical lower bounds via covariance spectral decomposition

To prevent excessive pruning from catastrophic supervisory collapse, Flash-DD establishes both theoretical and empirical stopping criteria. Theoretically, decomposing the global data covariance matrix via spectral decomposition \(\Sigma = V_k \Lambda_k V_k^T + V_\perp \Lambda_\perp V_\perp^T\), full-sized teacher parameters split into principal subspace components \(\theta_{T_1}\) and orthogonal residual components \(\theta_\delta\). When the compact teacher retains core predictive rank, the residual eigenvalues \(\Lambda_\perp\) approach zero, ensuring that true generalization errors between students distilled from full and pruned teachers satisfy \(\varepsilon(S_0) \approx \varepsilon(S_1)\).

Empirically, the downstream accuracy achieved by the original uncompressed teacher on the synthetic dataset, denoted as \(\tau(\mathcal{D}_S)\), serves as an absolute lower bound. When pruned teacher accuracy falls below \(\tau(\mathcal{D}_S)\), student performance drops irreversibly. Furthermore, t-SNE pre-softmax logit visualizations confirm that capacity-aligned teachers produce well-separated class clusters with sharp boundaries while preserving non-zero, informative cross-class probability distributions, avoiding the noisy manifold clutter of full-sized teachers and the cluster collapse of over-pruned models.

3. Adaptive Ensemble Soft-Label Generation: complementary multi-view constraints without capacity waste

When extra storage headroom is available (e.g., 20MB to 45MB), reverting to a single larger teacher reintroduces capacity mismatch and suboptimality. Instead, Flash-DD partitions the total storage budget \(M = \sum \mathcal{C}(\theta_{T_i})\) among multiple compact sub-teachers, taking their convex combination at epoch \(t\):

\[\bar{f}(x; \bar{\Theta}_T^{(t)}) = \sum_{\theta_{T_i} \in \Theta_T} \mu_i^{(t)} f(x; \theta_{T_i})\]

To prevent conflicting predictions and supervisory degradation, Flash-DD enforces rigorous geometric ensemble guidelines: the capacity range \(\max \mathcal{C}(\theta_{T_i}) - \min \mathcal{C}(\theta_{T_i}) < \epsilon\) and cumulative pairwise differences must stay strictly bounded; all individual sub-teachers must maintain classification accuracy above \(\tau(\mathcal{D}_S)\); and each sub-teacher's individual capacity must remain proximal to the optimal single teacher \(|\mathcal{C}(\theta_{T_i}) - \mathcal{C}(\theta_T^*)| < \epsilon\). Empirical results confirm that the sum of parameter disparities across sub-teachers is strongly negatively correlated with downstream accuracy. Enforcing capacity symmetry across sub-teachers provides rich, multi-perspective dark knowledge that breaks through the single-teacher performance ceiling.

Loss & Training

During downstream student training, synthetic images undergo augmentation \(X_S^{(t)\prime} = \mathcal{A}ug(X_S, t; \phi)\) to produce epoch-specific inputs, while targets \(Y_S^{(t)\prime}\) are generated on-the-fly by the compact single teacher or the balanced ensemble. The downstream objective balances MSE/KL soft-label matching with hard ground-truth supervision:

\[\mathcal{L}(\mathcal{D}_S; \theta_S, \theta_T) = \mathbb{E}_{(x,y) \sim \mathcal{D}_S} \left[ \|f(x, \theta_T) - f(x, \theta_S)\|_2^2 + \eta \|f(x, \theta_S) - y\|_2^2 \right]\]

Adjusting \(\eta\) regulates gradient flow between calibrated smooth posteriors and exact class anchors.

Key Experimental Results

Main Results

Under a single-teacher constrained budget, Flash-DD is integrated into state-of-the-art large-scale DD baselines (SRe2L, G_VBSM, RDED) across ImageNet-100, Places365-Standard, and ImageNet-1K using ResNet-18 backbones.

Dataset	Method Variant	IPC 1 Acc. (%)	Extra Mem. (MB)	IPC 10 Acc. (%)	Extra Mem. (MB)	IPC 50 Acc. (%)	Extra Mem. (MB)
ImageNet-1K	SRe2L Baseline	0.1 ± 0.1	44.7	21.3 ± 0.6	44.7	46.8 ± 0.2	44.7
	SRe2L + Flash-DD	18.9 ± 0.7 (↑18.8)	1.1 (2.5%)	44.5 ± 0.1 (↑23.2)	4.7 (10.5%)	58.4 ± 0.1 (↑11.6)	9.5 (21.3%)
	G_VBSM Baseline	1.7 ± 0.1	84.1	31.4 ± 0.5	84.1	51.8 ± 0.4	84.1
	G_VBSM + Flash-DD	15.9 ± 0.1 (↑14.2)	1.1 (1.3%)	46.6 ± 0.2 (↑15.2)	9.5 (11.3%)	60.3 ± 0.1 (↑8.5)	12.1 (14.4%)
	RDED Baseline	6.6 ± 0.2	44.7	42.0 ± 0.1	44.7	56.5 ± 0.1	44.7
	RDED + Flash-DD	20.0 ± 0.1 (↑13.4)	0.8 (1.8%)	47.5 ± 0.2 (↑5.5)	7.2 (16.1%)	58.5 ± 0.1 (↑2.0)	10.7 (23.9%)
Places365	RDED Baseline	4.6 ± 0.2	43.3	29.1 ± 0.3	43.3	44.2 ± 0.2	43.3
	RDED + Flash-DD	13.8 ± 0.1 (↑9.2)	0.1 (0.3%)	36.9 ± 0.2 (↑7.7)	2.0 (4.6%)	45.9 ± 0.1 (↑1.7)	8.9 (20.6%)
ImageNet-100	RDED Baseline	8.1 ± 0.3	42.8	36.0 ± 0.3	42.8	61.6 ± 0.1	42.8
	RDED + Flash-DD	10.9 ± 0.3 (↑2.8)	0.04 (0.1%)	48.9 ± 0.3 (↑12.9)	1.4 (3.3%)	71.0 ± 0.2 (↑9.4)	6.5 (15.2%)

Ultra-Low Memory & Generation Speedup Evaluation

Minimal extra storage and acceleration required to match or exceed baseline RDED performance:

Dataset	IPC	#Params (Teacher)	Extra Mem. (MB)	GFLOPs	Label Gen. Speedup	Teacher Acc. (%)	Downstream Acc. (%)
ImageNet-100	1	<0.006M (0.05%)	<0.02 (0.05%)	<0.003 (0.14%)	>729.31×	<16.26	9.5 ± 0.3
	10	0.04M (0.37%)	0.16 (0.37%)	0.01 (0.60%)	166.15×	36.22	35.8 ± 0.1
	50	0.26M (2.28%)	0.98 (2.28%)	0.05 (2.96%)	33.73×	63.10	61.6 ± 0.3
Places365	1	0.003M (0.03%)	0.01 (0.03%)	0.002 (0.12%)	843.81×	4.09	4.1 ± 0.1
	10	0.08M (0.74%)	0.32 (0.74%)	0.02 (1.11%)	90.12×	29.00	29.2 ± 0.1
	50	0.75M (6.59%)	2.86 (6.59%)	0.14 (7.55%)	13.32×	45.66	44.3 ± 0.1
ImageNet-1K	1	0.03M (0.22%)	0.10 (0.22%)	0.004 (0.16%)	609.94×	6.49	6.6 ± 0.1
	10	0.55M (4.68%)	2.09 (4.68%)	0.09 (4.84%)	20.66×	43.85	41.8 ± 0.1
	50	1.88M (16.12%)	7.19 (16.12%)	0.30 (16.24%)	6.16×	57.73	57.0 ± 0.1

Cross-Architecture Generalization & Pruning Ablation

Evaluating robustness across pruning metrics on ImageNet-1K (RDED base) and cross-architecture transfer on ImageNet-1K (IPC 10):

Pruning Metric (IN-1K)	IPC 1 Acc. / Mem	IPC 10 Acc. / Mem	IPC 50 Acc. / Mem	Architecture (IN-1K IPC 10)	RDED Baseline (%)	RDED + Flash-DD (%)
\(L_1\)-norm (Default)	20.0 ± 0.1% / 0.8MB	47.5 ± 0.2% / 7.2MB	58.5 ± 0.1% / 10.7MB	ShuffleNet-V2	23.6 ± 0.5	30.7 ± 0.1 (↑7.1)
\(L_2\)-norm	19.6 ± 0.1% / 0.8MB	47.4 ± 0.1% / 7.2MB	59.1 ± 0.1% / 10.7MB	MobileNet-V2	34.4 ± 0.2	41.5 ± 0.5 (↑7.1)
Taylor Expansion	19.7 ± 0.2% / 0.8MB	47.3 ± 0.2% / 7.2MB	59.2 ± 0.1% / 10.7MB	EfficientNet-B0	42.8 ± 0.5	47.4 ± 0.1 (↑4.6)
-	-	-	-	Swin-V2-Tiny	17.8 ± 0.1	27.5 ± 0.7 (↑9.7)
-	-	-	-	RegNet-Y-400MF	38.5 ± 0.5	42.9 ± 0.5 (↑4.4)
-	-	-	-	AlexNet	11.9 ± 0.1	14.4 ± 0.1 (↑2.5)

Key Findings

• Capacity alignment outclasses brute-force teacher scale: Uncompressed full-sized ResNet-18 teachers exhibit strong standalone accuracy but poor transferability on tiny synthetic subsets. Pruning 80%~95% of parameters significantly lifts student accuracy (e.g., from 6.6% to 20.0% on ImageNet-1K IPC 1).
• Extreme label generation acceleration: Under equivalent accuracy targets, soft-label generation speed increases by up to 843.81× on Places365 IPC 1, requiring only 0.01MB (0.03%) extra memory, completely resolving the training latency bottleneck.
• Inverted U-curve in ensemble scaling: Under a fixed 21.65MB budget, moving from 1 to 2 or 3 sub-teachers boosts performance to 48.8% (beating the best single teacher at 47.5% and a raw single 21.65MB model at 45.0%). However, packing excessive sub-teachers degrades individual capacities below the critical threshold \(\tau(\mathcal{D}_S)\), causing ensemble collapse.

Highlights & Insights

• The "Pruning as Enhancement" counter-intuitive finding: Uncovers the fundamental capacity mismatch between teachers and compact datasets, demonstrating that pruned models output smoother, noise-free predictive manifolds that align naturally with small synthetic budgets.
• Reusable transferability proxy \(\mathcal{D}is(\theta_T)\): Exploits early-epoch student loss degradation dynamics as a rapid surrogate metric, evaluating teacher suitability without requiring full student training runs.
• Symmetry principles in soft-label ensembling: Formulates practical rules showing that parameter variance across ensemble members negatively impacts performance, guiding the design of balanced, edge-friendly distillation ensembles.

Limitations & Future Work

• Reliance on pretrained teacher checkpoints: The framework requires a pretrained base teacher as the pruning origin, rather than performing zero-shot teacher synthesis from scratch alongside distilled images.
• Heuristic discrete ensemble partition: Sub-teacher configurations currently rely on grid-based discrete search; exploring continuous differentiable neural architecture search (NAS) could automate optimal subspace partitioning.
• Broadening beyond classification: Validations are currently focused on visual classification datasets (ImageNet, Places365); extending to dense prediction (detection, segmentation) and multimodal foundation models remains an open frontier.

Related Work & Insights

• vs SRe2L / G_VBSM / RDED: Prior large-scale DD works established the "synthetic image recovery + teacher relabeling" paradigm but naively coupled it with cumbersome, uncompressed teachers. Flash-DD acts as an orthogonal plug-and-play enhancement, cutting costs by orders of magnitude while substantially boosting downstream accuracy.
• vs Standard Model Pruning (Slimming, Taylor): Traditional pruning targets test set accuracy or FLOPs reduction. Flash-DD repurposes pruning as a fine-grained knob for teacher-student capacity alignment in data-constrained distillation.

Rating

• Novelty: ⭐⭐⭐⭐☆ Pinpoints the critical yet overlooked storage and computation overhead of soft-label teachers in large-scale dataset distillation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluations across ImageNet-1K, Places365, and ImageNet-100 across IPC 1/10/50, accompanied by extensive cross-architecture and continual learning validations.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, elegant theoretical framing via covariance spectral analysis, and highly informative visualizations.
• Value: ⭐⭐⭐⭐⭐ Extremely high practical value for democratizing large-scale dataset distillation on resource-constrained hardware.