Towards Consistent and Efficient Dataset Distillation via Diffusion-Driven Selection

Conference: ECCV 2026
Paper: ECCV 2026
Area: Model Compression (Dataset Distillation)
Keywords: dataset distillation, diffusion model, patch selection, diffusion features, intra-class clustering

TL;DR

This work turns a pre-trained diffusion model from an image generator into a real-image patch selector, combining class-conditioned localization with DIFT clustering to build reusable compressed training sets and achieving \(61.0\pm0.3\%\) classification accuracy on ImageNet-1K at IPC=50 in the main experiment.

Background & Motivation

Dataset distillation compresses training data rather than network parameters: the goal is to train models that still generalize well from only a few images per class. IPC stands for images per class, the number of images retained for each category in the compressed dataset. Early approaches optimize synthetic pixels through gradient, distribution, or training-trajectory matching, but large, high-resolution datasets increase both the optimization space and computational cost. SRe2L decouples the process and uses a pre-trained classifier for image recovery and relabeling, enabling distillation at ImageNet scale. RDED instead selects and concatenates real-image patches, avoiding continuous pixel optimization, but its selection criterion still depends on a classifier trained for the target data. This couples the quality of the resulting training data to the choice of a proxy classifier that scores candidate samples.

Diffusion generators offer another route: directly synthesize training images using an already learned visual prior. However, DiT pre-trained on ImageNet and Stable Diffusion pre-trained on other image-text data impose different distributional constraints on the target task. The former often requires task-specific fine-tuning, while the latter can produce class-inaccurate, noisy, or stylistically mismatched samples. Repeated changes to class combinations or IPC further accumulate the cost of fine-tuning models or optimizing prototypes. The paper therefore considers not only the best accuracy at one fixed compression ratio, but also whether data-processing results can be reused across configurations. That objective requires attention to class relevance, diversity, and construction cost, rather than visual realism alone.

The proposed approach retains pixels from real images and uses diffusion only to identify regions representative of a known class. The diffusion prior determines where to select rather than freely deciding what to generate, reducing opportunities for additional distribution shift during generation. Diffusion features then organize patches within each class to avoid retaining many similar samples simply because they all receive high relevance scores. Here, distributional consistency is an empirical goal relative to generative distillation, not a mathematical claim that cropping and selection leave the data distribution unchanged. Core Idea: use a frozen diffusion model to localize class-relevant regions in real images, then apply intra-class clustering and ranking to create a candidate pool reusable across class combinations and IPC settings, replacing repeated generation with analysis followed by on-demand selection.

Method

Overall Architecture

The inputs are real training images and their class texts; the output is a compressed dataset containing a specified number of images per class. The first stage, Class-Difference Localization, compares diffusion responses with and without class text to score candidate regions in each original image. The second stage, Diffusion Feature Aggregation, extracts DIFT from the selected patches and clusters and ranks them within each class. Candidate Pool Reuse then selects existing patches according to the class subset and IPC, with concatenation also used at small IPC. The first two stages are the core of Figure 2; candidate pool reuse organizes their outputs and explains how one distillation process supports multiple settings. The diffusion image encoder, text encoder, and U-Net remain frozen during selection, without optimizing newly generated images for the target compressed set. The teacher classifier does not participate in these two core selection stages, but a pre-trained teacher still supplies soft labels when training the student afterward. Training-free dataset construction is therefore different from a learning system that requires no training at all.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Real images and class texts"] --> B["Class-Difference Localization"]
    B --> C["Diffusion Feature Aggregation"]
    C --> D["Candidate Pool Reuse"]
    D --> E["Compressed training set"]
    E --> F["Student training and evaluation"]
    G["Pre-trained teacher soft labels"] --> F

Key Designs

1. Class-Difference Localization: use the diffusion prior to guide cropping, not generate new pixels

High-resolution images are resized while preserving aspect ratio so that the shorter edge is 256 pixels, then encoded into the diffusion latent space. Noise prediction is performed for the same image using a class prompt and a prompt with the class removed, exemplified by An image of c and An image of . in the paper. The class text does not trigger a search over all possible categories; the input data already provide the image's category. Unlike diffusion-based zero-shot classification that compares multiple candidate classes, this method contrasts the class condition with a class-free condition. The intended signal identifies regions that the model can better explain once it knows the class. Responses across multiple noise samples and time steps are averaged to reduce instability from individual perturbations. Localization uses the normalized time-step interval \([0.1,0.7]\), rather than just one fixed denoising step. The latent difference map is interpolated to the original image resolution and averaged across channels to obtain a spatial score map.

A \(224\times224\) average-pooling window then scores patches, and the highest-scoring region represents the original image. This is more direct than randomly cropping first and asking a classifier to score the resulting crops, because a dense score map determines the location. For fixed-area windows, the region-sum formulation identifiable in Equation (7) and Algorithm 1 can be organized as:

\[ R(p\mid c)=\sum_{(i,j)\in p}\widehat M(x\mid c)[i,j],\qquad p^{*}=\underset{p}{\arg\max}\;R(p\mid c). \]

Here, \(\widehat M(x\mid c)\) is the interpolated, channel-aggregated class-difference map, and \(p\) is a fixed-size candidate patch. Because all windows have the same area, summation and average pooling differ only by a constant and select the same optimal region. This formula only organizes the recoverable region-aggregation step; it does not add a noise-loss direction that the source leaves unclear. Equations (1) through (6) are damaged in the local text extraction, and the prose switches between descriptions of noise differences and loss differences. They are therefore not silently repaired into an apparently exact conditional likelihood, subtraction convention, or complete training loss. The recoverable procedure consists of two text conditions, averaging over samples, spatial aggregation, and maximum-score cropping. The retained output is a crop from the original image, not a new image produced by reversing diffusion from random noise. This limits generative shift but cannot guarantee preservation of all context, especially when an object extends beyond the crop window.

2. Diffusion Feature Aggregation: separate intra-class mode coverage from sample typicality

Selecting only high-scoring patches can still favor one pose, texture, or background, since class relevance does not imply complementarity between samples. The second stage uses DIFT, intermediate features extracted from a diffusion network, to describe each selected patch. Each patch is encoded, corrupted at a specified time step, and passed through the U-Net with class-text conditioning to obtain intermediate activations. This stage uses the fixed time step \(t=160\), serving a different purpose from interval sampling in the first stage. It constructs visual representations for clustering rather than searching for crop locations again. K-means is performed within each class with 32 cluster centers, prioritizing samples from the top 10 ranked clusters. Classes are not mixed during clustering, preventing large or similar categories from competing for the same compression budget. The procedure still depends on the quality of diffusion representations and should not be interpreted as unbiased sampling of the real distribution.

Ranking operates at two scales: patches within a cluster are ranked by distance to its centroid, while clusters are ranked by the median representativeness score of their patches. Prioritizing patches close to a centroid preserves typical members of that visual mode rather than merely retaining high-scoring outliers. Inter-cluster ranking reintroduces class relevance from the first stage, avoiding retention of weakly relevant content simply because it forms a cluster. The median reflects the score level of a cluster rather than its single highest-scoring patch. Choosing several strong clusters and then typical members from each provides the concrete constraints on representativeness and diversity. Here, diversity means coverage of different visual modes within a class, not increased randomness in diffusion generation. The paper provides no theoretical coverage guarantee for cluster selection; support primarily comes from clustering ablations and visualizations. The CLIP comparison also shows that clustering alone need not improve accuracy: its feature space must suit the selection objective.

3. Candidate Pool Reuse: decouple expensive analysis from the compression budget

Once patches, features, clusters, and rankings have been computed, different IPC settings can draw different numbers of samples from the same intra-class pool. Changing the class combination likewise allows reuse of processed categories rather than requiring optimization of a new set of generative prototypes. For IPC=100, the paper selects 10 patches from each of the top 10 clusters. If a cluster contains too few patches, high-scoring patches supplement the output so that the requested count is met. Equal sampling from clusters is therefore a preferred rule rather than a constraint guaranteed for every data distribution. When IPC exceeds 10, each patch is treated as an individual training image instead of forcing multiple local regions together. When IPC is at most 10, multiple patches are concatenated into one image, following the RDED experimental setup. The main text does not specify every concatenation layout or patch count here, so no universal collage configuration is added.

Low-resolution data are handled separately: complete images are scored and clustered without the local cropping described above. This distinction matters; the ImageNet \(224\times224\) search procedure cannot simply be transferred to CIFAR. The paper's one-step claim means that one set of analysis results can serve multiple distillation settings, not that the process uses one U-Net forward pass. Nor does it mean that the initial processing cost is identical for datasets of every size. Each candidate image still incurs diffusion-response computation, and selected patches require feature extraction and clustering. Section 4.6 explicitly considers reducing the number of candidate originals to lower this overhead, showing that initial construction cost remains relevant. New classes or images not covered by the pool still require their own analysis. The strongest reuse case is therefore repeated changes to class combinations and compression budgets on the same data source.

A Worked Example

Consider the paper's ImageNet-1K configuration with IPC=100: each class ultimately needs 100 training images. The system first computes class-difference maps for candidate originals in that class, with each high-resolution original contributing its highest-scoring patch. After DIFT encoding, these patches are assigned to 32 clusters, and the top 10 clusters are chosen by median patch score. Each cluster preferentially contributes its 10 members closest to the centroid; high-scoring supplemental selection is used only when necessary. Because IPC=100 exceeds 10, the resulting patches remain individual images rather than being concatenated into a canvas. If a later experiment uses a subset of classes, the existing patches and rankings for those classes can be reused directly. For another IPC, the selection count changes, but the student must still be trained again on the new training set. The example clarifies that the reusable asset is data analysis, not a single student that automatically adapts to every experiment.

Loss & Training

Selection does not update the diffusion model, and there is no iterative distillation optimization of synthetic pixels or latent variables. The difference response is an evaluation signal, not the objective for a newly trained diffusion model introduced by this work. During student training, a fixed pre-trained classifier supplies soft labels for the compressed data, with evaluation hyperparameters following EDC. Five randomly initialized networks are trained for each compressed dataset and their average performance is reported; the \(\pm\) entries are retained as printed. The main text does not fully specify the student soft-label loss formula, and the material read does not clearly define the statistic represented by every \(\pm\) entry. High-resolution main experiments use ResNet-18, while low-resolution experiments report separate ConvNet and ResNet-18 results. Experiments use PyTorch and NVIDIA RTX-3090 GPUs, but this does not remove the cost of teacher pretraining or downstream student training.

Key Experimental Results

Main Results

The following entries are selected from Table 1 on page 9, reporting classification accuracy (%) with ResNet-18 in the high-resolution main evaluation. RDED and IGD are existing comparison methods; only Ours refers to diffusion-driven selection, and these columns are not variants of this paper's modules.

Dataset	IPC	RDED (existing baseline)	IGD (existing baseline)	Ours
ImageNette	50	\(80.4\pm0.4\)	\(85.5\pm0.3\)	\(86.4\pm0.2\)
ImageWoof	10	\(38.5\pm2.1\)	\(39.8\pm0.1\)	\(44.5\pm0.4\)
ImageNet-100	50	\(61.6\pm0.1\)	\(64.2\pm0.3\)	\(67.5\pm0.3\)
ImageNet-1K	10	\(42.0\pm0.1\)	\(45.5\pm0.5\)	\(46.1\pm0.3\)
ImageNet-1K	50	\(56.5\pm0.1\)	\(60.3\pm0.4\)	\(61.0\pm0.3\)
ImageNet-1K	100	\(59.5\pm0.8\)	\(61.4\pm0.3\)	\(63.0\pm0.2\)

On ImageNet-1K at IPC=50, the method exceeds RDED by 4.5 percentage points and IGD by 0.7 percentage points; these are distinct comparisons. The original table contains additional methods, so the three selected method columns are not the complete baseline set. The method does not dominate low-resolution evaluation: Table 2 on the same page reports \(48.8\pm0.4\%\) for Ours on CIFAR-10 at IPC=10 with ConvNet, below RDED at \(50.2\pm0.3\%\) and TESLA at \(66.4\pm0.8\%\). These counterexamples restrict the scope of its advantages and prevent extending the high-resolution findings to every dataset and backbone.

Ablation Study

The following entries come from Table 5 on page 13, fixing IPC=50 and reporting accuracy (%) for different clustering features and no clustering.

Dataset	No clustering	CLIP-feature clustering	DIFT-feature clustering (Ours)
ImageWoof	70.4	69.5	73.7
ImageNet-1K	58.1	59.0	61.0

In this ImageWoof ablation, DIFT improves over no clustering by 3.3 percentage points, whereas CLIP clustering reduces accuracy by 0.9 percentage points. For ImageNet-1K, the source annotates DIFT with +1.9, but the raw values give \(61.0-58.1=2.9\); this note retains the raw values and explicitly flags the arithmetic inconsistency. The same ablation table reports 73.7 for ImageWoof at IPC=50 versus 73.5 in main Table 1, and 63.4 for ImageNet-1K at IPC=100 versus 63.0 in Table 1. The paper does not clearly explain these differences, so each table is cited independently rather than substituting ablation numbers for main results.

Table 6 on the same page tests the sampling interval used for localization; these are also ImageNet-1K accuracies (%) at IPC=50, not an ablation of the DIFT feature-extraction time step.

Normalized time interval	Accuracy	Interpretation
\([0.1,0.3]\)	58.9	Narrower interval
\([0.1,0.7]\)	61.0	Interval adopted in this work
\([0.1,0.9]\)	59.7	Includes higher-noise time steps
\([0.7,0.9]\)	57.9	High-noise interval only

Key Findings

• Table 6 supports retaining low time steps while covering an intermediate range, but wider is not always better: accuracy falls when higher-noise steps are included.
• Table 3 on page 11 shows that teacher selection still matters: at IPC=50, a ResNet-18 student scores 61.0% with a ResNet-18 teacher and 31.2% with a ViT-B teacher.
• In the same table, a ResNet-18 teacher with a ResNet-101 student reaches 65.7%, showing that real patches can support a deeper student; this is not a matched-architecture comparison against every existing distillation method.
• Figure 5 on page 12 compares construction time across tasks and supports reuse under repeated settings; its constant-cost description should not be interpreted as end-to-end constant time including teacher and student training.

Highlights & Insights

• A generative model need not generate new samples to be useful. Using its prior for localization while retaining target-data pixels is the main mechanism for reducing generative shift.
• Patch representativeness and set diversity are handled separately. The former determines where to look within each image, while the latter determines which visual modes the compressed set covers within a class.
• Changes in class combinations or per-class budgets need not always trigger reoptimization. Retaining features, clusters, and rankings by class allows one relatively expensive analysis to serve multiple experimental configurations.

Limitations & Future Work

• Real patches can still lose context, and concatenation can disrupt global structure, so real provenance does not imply an unbiased distribution or complete class features.
• The frozen diffusion prior retains pretraining biases; avoiding generative shift during selection does not eliminate possible semantic localization errors.
• Student evaluation still relies on teacher soft labels, and the substantial differences in Table 3 rule out claiming that the overall pipeline is independent of proxy models.
• The available cache contains pages 1 through 17, comprising the main text and references, but not the appendix mentioned in the text; the U-Net feature layer, complete evaluation details, and equation typesetting require further verification from original materials.
• Cross-table discrepancies and the gain annotation in Table 5 reduce reproducibility clarity; explicit experimental configuration records would be more useful than silently merging these results.

Related Work & Insights

• Compared with RDED: both use real patches, but RDED scores random crops with a classifier, while this work localizes regions through diffusion-condition differences and then selects with DIFT clustering.
• Compared with Minimax and IGD: these methods fine-tune a DiT-based generation process; this work does not optimize the generator and uses target images themselves as the source of output pixels.
• Compared with D4M and MGD3: these methods organize generation through Stable Diffusion and prototypes; this work reuses existing patches and their rankings, reducing the need to recalculate prototypes for different IPC settings.
• Connection to diffusion zero-shot classification and DIFT: the former motivates class-condition contrasts and the latter supplies intermediate visual representations; both are existing techniques, while this paper contributes their organization into a distillation-selection pipeline.

Rating

• Novelty: 4/5. Using the diffusion prior for patch selection instead of generation provides a clear reformulation, while relying on existing conditional-comparison and feature-extraction tools.
• Experimental Thoroughness: 4/5. Evaluation covers high and low resolutions, multiple IPC settings, architecture combinations, and selection strategies, but includes low-resolution counterexamples and cross-table discrepancies.
• Writing Quality: 3/5. The two-stage structure is understandable, but the scope of one-step processing, arithmetic annotations, and several reproducibility details need more precise treatment.
• Value: 4/5. Useful for repeatedly building training sets with different budgets from one visual data source; its primary value is reusable data selection rather than a new generator.