model -> image" dual compression process in mainstream large-scale dataset distillation (SRe2L-based family) causes severe information loss and shifts distilled images away from the real distribution, thereby undermining Relabeling. To address this, the authors propose CIM, which"> [Paper Note] Condensing Large-Scale Datasets Directly with Minimal Information Loss
Skip to content

Condensing Large-Scale Datasets Directly with Minimal Information Loss

Conference: ECCV 2026
arXiv: 2607.00916 ⚠️ Suspected placeholder/future date (2026-07), subject to the original text.
Code: https://github.com/LINs-lab/CIM
Area: Model Compression / Dataset Distillation
Keywords: Dataset Distillation, Information Loss, Distribution Alignment, Relabel, Feature Matching

TL;DR

This paper points out that the "data -> model -> image" dual compression process in mainstream large-scale dataset distillation (SRe2L-based family) causes severe information loss and shifts distilled images away from the real distribution, thereby undermining Relabeling. To address this, the authors propose CIM, which uses a computable "effective information gap" metric to directly minimize the information discrepancy between the synthetic and real sets on the original images, bypassing the expensive recovery stage. Consequently, CIM achieves 48.7% Top-1 accuracy (ResNet-18) on ImageNet-1K with IPC=10 in just 80 minutes on a single GPU.

Background & Motivation

Dataset distillation aims to condense the knowledge of a massive training set into an extremely small synthetic set, enabling models trained on the synthetic set to approximate the performance of those trained on the full dataset, thereby significantly reducing training and storage costs. Traditional matching-based methods (matching gradients, features, distributions, or training trajectories) exhibit impressive performance at low IPC (images per class). However, they require repeatedly computing differences and iterating until convergence between the synthetic and original sets at each step. This causes computational and memory overhead to explode with data scale, making scaling to ImageNet-1K extremely difficult. To overcome this bottleneck, SRe2L pioneered a decoupled three-stage paradigm: first, the data information is Squeezed into a pre-trained model; second, it is Recovered from the model parameters back into the image space to form synthetic images; finally, a pre-trained model is used to Relabel the synthetic images to inject label-space knowledge. By avoiding expensive unrolled optimization, SRe2L became the first scheme capable of scaling efficiently to ImageNet-1K, inspiring numerous subsequent works (such as G-VBSM, RDED, DELT, NRR-DD).

However, this extraction pipeline suffers from two persistent issues: poor cross-architecture generalization and massive computational overhead during the Recovery stage. This paper attributes the root cause to an overlooked implicit dual compression process: information is first squeezed from the data into model parameters (Squeeze), and then decompressed from the model back into synthetic images (Recover). The cascading of these two information bottlenecks naturally leads to severe information loss, resulting in synthetic images with sparse content and unrealistic textures that are overfitted to specific networks. More fatally, the authors employ theory and experiments to expose a hidden weakness of Relabeling: its effectiveness strictly depends on distribution alignment. When a labeler model trained only on real data is used to label distilled images, once dual compression pushes the synthetic samples away from the real distribution, the labeler becomes unreliable and produces suboptimal labels. These erroneous labels accumulate over training epochs, degrading the performance of downstream student models (experiments show that turning off Relabeling causes SRe2L's performance on ImageNet-1K to plummet to 1.1%).

Consequently, this paper abandons the flawed dual compression paradigm. Core Idea: Since the root cause is the information loss incurred by the roundabout "data -> model -> image" loop, the detour should be bypassed. The proposed method directly defines a computable "effective information gap" to measure the information discrepancy between the synthetic and original images. By directly minimizing this gap on the original images, high-fidelity information retention and natural distribution alignment are achieved. This not only eliminates the entire recovery process but also inherently satisfies the prerequisite for Relabeling to function effectively.

Method

Overall Architecture

CIM addresses the following challenge: how to directly inject the salient information of a batch of original images into a more compact synthetic image without going through the detour of "compressing into a model and then decompressing back into images", while ensuring that the synthetic image does not deviate from the real distribution. The overall pipeline consists of three sequential stages: first, select \(N \times \text{IPC}\) "easy-to-label" key samples from each class to form IPC subsets; second, for each subset, iteratively compress an initial synthetic image (created by resizing and stitching the images in the subset) by minimizing the effective information gap, squeezing the information of \(N\) original images into this single image; finally, relabel the synthetic images with cross-transformation soft labels for downstream training. The entire pipeline no longer requires any model inversion. Synthetic images can be optimized independently one by one, which makes memory usage highly manageable and allows dynamic adjustment of batch sizes.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Original full dataset T"] --> B["Key Sample Selection<br/>Select samples easiest to label<br/>based on a loss-based score"]
    B --> C["Effective Information Gap Minimization<br/>Compress N original images in a subset<br/>into one synthetic image x̃"]
    C --> D["Intermediate Feature Alignment<br/>Align intermediate features instead of logits<br/>to balance semantics and textures"]
    D --> E["Cross-Transformation Relabeling<br/>Generate soft labels for<br/>each transformed view"]
    E --> F["Small Synthetic Set S<br/>Downstream Training / Continual Learning"]

Key Designs

1. Effective Information Gap: Defining a computable metric for "the information discrepancy between two images"

Why extraction-based methods lose information and how much they lose has lacked a clear metric; researchers have had to rely on indirect visual judgments of whether "reconstructed images look authentic". The first step of CIM is to formalize "information". The authors define an observer set \(\mathcal{R}=\{\xi_j\}\) (where each observer is a function capable of extracting features from an image, e.g., the forward pass of a pre-trained model under a particular transformation). The effective information of a sample \(\mathbf{x}_i\) is defined as the distribution \(\mathcal{P}_{\mathbf{x}_i|\mathcal{R}}\) formed by the set of features extracted by all observers in the group. The information discrepancy between two images is modeled as the KL divergence between these two feature distributions:

\[I_G(\mathbf{x}_i,\mathbf{x}_j;\mathcal{R})=\mathrm{D}_{\mathrm{KL}}\!\left(\mathcal{P}_{\mathbf{x}_i|\mathcal{R}}\,\|\,\mathcal{P}_{\mathbf{x}_j|\mathcal{R}}\right)\]

The intuition is that if a group of observers extracts almost identical features from two images, the images are deemed to carry the same effective information. The value of this definition is that it transforms "information fidelity" from a vague concept into an optimizable objective—as long as the distilled image "looks consistent in information" to these observers compared to the original, the information is successfully retained.

2. KL Upper Bound Relaxation: Transforming uncomputable divergence into computable feature L2 distance

The aforementioned KL divergence cannot be computed directly (as it requires estimating the density of the feature distribution), and it measures pairwise sample discrepancies, making it inapplicable directly to a "set vs. a single synthetic image" scenario. CIM addresses this in two steps. First, it relaxes the set-level objective into an expectation: with the help of data augmentation \(\mathcal{A}\), the \(N\) augmented views of a single synthetic image are used to align individually with the \(N\) original images in the subset, minimizing the expected information gap of the pairs. Second, it establishes a computable upper bound for the KL divergence—the authors prove (Thm. 4.1) that the effective information gap is upper-bounded by the expected squared Euclidean distance of the features output by the observers:

\[I_G(\mathbf{x}_i,\mathbf{x}_j;\mathcal{R})\le \mathbb{E}_{\xi_k\sim\mathcal{R}}\left\|\xi_k(\mathbf{x}_i)-\xi_k(\mathbf{x}_j)\right\|^2\]

This step represents the most crucial simplification of the paper: optimizing the uncomputable KL divergence is replaced with feature matching, which simply forces the features of the synthetic and original images under each observer to be as close as possible. In practice, the authors set the observer group as a single pre-trained model wrapped with various transformations (\(\xi_k=\zeta_k\circ\phi_{\theta_{\mathcal{T}}}\), where \(\zeta_k\) is a transformation), achieving a "multi-observer" effect without maintaining multiple models. The synthetic image is initialized with resized and stitched real images, and augmented views are generated via RandomCrop. Then, gradient descent is performed for \(M\) steps on the perturbation \(\Delta\tilde{\mathbf{x}}\) to minimize the feature distance. Unlike SRe2L which relies on BN statistics alignment and complex inversion, this approach is independent of BN and requires no model inversion, directly optimizing a feature-matching loss in pixel space.

3. Intermediate Feature Alignment: Align intermediate features instead of logits to preserve texture

If the output of the model's final layer (logits) is directly used for alignment, a pitfall emerges: models naturally tend to extract semantics while discarding textural details. Consequently, while the synthetic images capture semantics, their textures degrade, hurting generalization. CIM addresses this by placing the observer/alignment target at the intermediate representation space instead of the final layer. Since shallow layers are rich in texturing and deep layers/logits favor semantics, aligning intermediate layers strikes a balance, achieving an optimal trade-off between semantic richness and texture fidelity. Ablation studies demonstrate that intermediate layer alignment is both robust and optimal—this is the primary reason why CIM's cross-architecture generalization significantly outperforms SRe2L (by preserving both texture and semantics, the synthetic images do not overfit to the preference of any specific network).

4. Cross-Transformation Relabeling: Equipping one image with multiple "transformation-dependent" soft labels

Standard Relabeling is a one-shot process (one label per image). However, random cropping might capture objects that differ from the original label, making the label inaccurate. Since CIM guarantees distribution alignment of synthetic images and ensures the labeler's reliability, the authors enrich the labels accordingly: they apply a series of transformations \(\zeta_k\) to the synthetic image and generate a soft label for each transformed view: \(\tilde{y}_k = \phi_{\theta_{\mathcal{T}}}(\zeta_k(\tilde{\mathbf{x}}))\). The downstream student model is then trained on these "view-label" pairs. This allows the label space to carry finer and more diverse knowledge than the standard one-shot strategy. It is worth emphasizing that this design is tightly coupled with the previous ones: because Designs 1–3 pull the distribution of the synthetic images back to the real domain, making the pre-trained model a reliable labeler again, the cross-transformation soft labels become meaningful. If the distribution were still shifted, generating more labels would only produce more errors.

Key Sample Selection (Reusing RDED)

CIM itself is selection-agnostic. By default, it inherits the mechanism of RDED alongside a loss-based importance score: each sample is scored as \(s = -\ell(\phi_{\theta_{\mathcal{T}}}(\mathbf{x}), y)\) (where smaller cross-entropy loss indicates easier correct recognition by the pre-trained model, yielding a higher score). The top-\((N \times \text{IPC})\) "easiest-to-label" samples are selected for each class. This aligns with the conclusion of Prop. 1: only samples correctly recognized by the labeler should be placed into the distilled set. To save compute and preserve diversity, a proxy subset of size 300 is first uniformly and randomly sampled from each class before scoring. Ablation studies (Tab. 6) reveal that even with random selection, CIM remains highly competitive, demonstrating that the performance gain primarily stems from minimizing the effective information gap rather than sample selection heuristics.

Loss & Training

During the distillation stage, the intermediate feature matching loss (Eq. 9) is minimized, and the perturbation on the synthetic image is optimized via AdamW (lr 0.01) for \(M=200\) steps. Each synthetic image compresses \(N=4\) original images, with a proxy subset size \(|\mathcal{T}'|=300\). Downstream training employs the MSE objective on cross-transformation soft labels (Eq. 10) using AdamW + MultiStepLR, for 300 epochs on ImageNet-1K and 1000 epochs on other datasets, with DSA utilized for augmentation. The authors emphasize that all hyperparameters are general and insensitive across different datasets and architectures.

Key Experimental Results

Main Results

On large-scale datasets (ResNet-18), CIM refreshes the SOTA in most settings, demonstrating a massive advantage especially at low IPCs:

Dataset IPC SRe2L RDED NRR-DD DELT CIM (Ours)
Tiny-ImageNet 1 13.5 15.4 13.5 9.3 25.1
Tiny-ImageNet 10 43.6 48.4 45.2 43.0 53.3
ImageNet-1K 10 31.1 41.1 46.1 45.8 48.7
ImageNet-1K 50 49.5 55.3 60.1 59.2 60.4

CIM likewise leads on smaller datasets (CIFAR-10/100), exposing the weaknesses of extraction-based methods—SRe2L struggles significantly on CIFAR-10 across all IPCs and on CIFAR-100 with IPC=1 (e.g., ResNet-18 on CIFAR-100, IPC=1 yields only 11.5% for SRe2L and 4.6% for RDED, whereas CIM reaches 31.1%). This confirms the premise that "information loss is most detrimental at low IPCs". CIM's 48.7% accuracy on ImageNet-1K at IPC=10 exceeds the previous SOTA methods NRR-DD and DELT by 2.6% and 2.9%, respectively.

The dependence of Relabeling effectiveness on distribution alignment (Tab. 7, ResNet-18/IPC=10) yields the most compelling evidence—the degree of performance collapse when turning off Relabeling differs starkly across methods:

Dataset Method Without Relabel With Relabel
ImageNet-1K SRe2L 1.1 31.1
ImageNet-1K RDED 19.7 41.1
ImageNet-1K CIM 22.0 48.7
Tiny-ImageNet SRe2L 0.6 43.6
Tiny-ImageNet CIM 27.0 53.3

Without Relabeling, SRe2L collapses to nearly zero (1.1%, 0.6%), showing that its synthetic images deviate severely from the real distribution and rely almost entirely on the labeler to salvage performance. Conversely, CIM maintains 22.0%/27.0% without Relabeling, proving that its synthetic images themselves retain high-fidelity and remain aligned with the real distribution—directly validating the core design goal.

Ablation Study

Configuration Phenomenon Explanation
Compression iterations \(M\) \(M=200\) is optimal; \(>200\) yields marginal gains Quality-speed trade-off point, and robust to \(M\)
Compressed images per synthetic image \(N\) \(N=4\) is verified as optimal across four datasets Larger \(N\) increases feature diversity but retains less information from each original image
Alignment layer Intermediate layers are optimal (shallow layers favor texture, deep layers/logits favor semantics) Intermediate layers balance semantics and textures
Information gap iterations \(K\) Increasing \(K\) effectively narrows the gap, saturating around 200 Balance between fidelity and computation
Selection strategy Small gaps among Random/K-means/Herding vs. RDED The framework is insensitive to selection methods (e.g., ImageNet-1K: Random 45.3% vs. RDED 48.6%)

In terms of efficiency (Tab. 5, Conv-4 on Tiny-ImageNet), generating 100 images with CIM takes only 13.02s with a peak memory of 0.65GB, which is much faster than SRe2L (51.68s/1.36GB) and G-VBSM (259.84s/4.94GB), and only slightly slower than the optimization-free RDED (1.68s). Relative to matching-based methods like DREAM (33906s) and DATM (12470s), CIM is over three orders of magnitude faster (Tab. 14).

Key Findings

  • Minimization of the information gap (rather than selection techniques) is the primary source of performance: CIM remains competitive even with random selection, indicating that the contribution comes from getting the "information fidelity" right.
  • Intermediate layer alignment is key to cross-architecture generalization: CIM leads comprehensively across 6 unseen architectures (e.g., 10.8% on ViT-T/16 vs. RDED's 8.5% and SRe2L's 3.2%) because it preserves both texture and semantics.
  • The "without Relabeling" comparison best illustrates the fundamental difference: SRe2L collapses close to 0%, while CIM remains steady, directly proving the fidelity and distribution alignment of CIM's synthetic images.
  • The advantage is largest in low IPC scenarios, where information loss is most harmful, aligning with the theoretical assessment.

Highlights & Insights

  • The paper attributes "why extraction-based distillation is suboptimal" to implicit dual compression (the two information bottlenecks of data -> model -> image). This diagnostic perspective is clear and falsifiable—using "whether performance collapses when Relabeling is turned off" to directly verify if synthetic images deviate from the distribution is an elegant experimental design.
  • The insight that Relabeling strictly depends on distribution alignment is backed by both theory (Prop. 1 uses Gaussian binary classification to prove that the optimal classifier on a shifted distribution is inherently suboptimal on the original distribution) and empirical evidence (Fig. 1 shows that Relabeling is only beneficial in early distillation stages when the images still resemble real ones). This thoroughly explains the operating limits of what has otherwise been treated as an omnipotent plugin.
  • The relaxation of the KL divergence to the feature L2 distance upper bound (Thm. 4.1) is the critical leap that translates the abstract goal of "information fidelity" into an optimizable loss, and this idea can be transferred to other scenarios requiring a metric for the information discrepancy between two samples or sets.
  • Synthetic images can be optimized independently one by one (unlike matching-based methods that require cooperative synthesis of a batch of images), yielding engineering advantages such as manageable GPU memory and dynamically adjustable batch sizes.

Limitations & Future Work

  • The primary limitation acknowledged by the authors: in terms of generation efficiency, CIM cannot outperform the optimization-free RDED (which requires no gradient optimization, running in 1.68s vs. CIM's 13.02s), limiting its use in extremely speed-sensitive scenarios.
  • The definition of effective information depends on the "observer group," which in practice degenerates into a "single pre-trained model + several transformations." Whether this is sufficient to capture the true information distribution, or whether multi-model observers would perform better, is not fully explored.
  • The method still requires a pre-trained model trained on the full dataset (serving as both the labeler and the feature observer). This prerequisite might itself be a bottleneck in truly large-scale or privacy-restricted settings.
  • The derivation of Thm. 4.1 involves kernel density estimation approximations and omits certain \(o(1)\)/constant terms (Appendix D); the tightness of the upper bound and the impact of approximation errors on the results warrant further evaluation.
  • vs SRe2L: SRe2L follows the decoupled three-stage pipeline "Squeeze -> Recover -> Relabel" and relies on model inversion to recover synthetic images. CIM discards the recovery phase and directly minimizes the information gap on the original images. The difference is that CIM eliminates the information loss caused by dual compression, resulting in high-fidelity synthetic images and aligned distributions, which leads to stronger cross-architecture generalization, higher efficiency, and more reliable Relabeling.
  • vs RDED: RDED is an optimization-free selection method that directly crops realistic patches from original images. CIM naturally reuses its selection mechanism but introduces pixel-level optimization to minimize the effective information gap. The cost is being slower than RDED, but the gain is significantly higher accuracy across various IPCs and datasets (e.g., ImageNet-1K IPC=10: 48.7% vs. 41.1%).
  • vs Matching-based (gradient/trajectory/distribution matching, e.g., DREAM/DATM/IDM): Matching-based methods repeatedly compute differences between the synthetic and original sets and iterate until convergence. This leads to computational explosions that hinder scaling and often overfit to specific architectures. CIM simplifies the objective into a computable feature distance and optimizes each image independently, running over three orders of magnitude faster with superior generalization.
  • vs NRR-DD / DELT: These methods improve SRe2L's instance/class feature capturing and intra-class diversity, respectively, but still operate within the dual compression paradigm. CIM replaces the information extraction paradigm entirely, leading by 2.6% and 2.9% on ImageNet-1K IPC=10, respectively.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ The diagnosis of "dual compression causing information loss," the effective information gap metric, and the theory behind Relabeling's dependence on distribution alignment form a complete, on-target package.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage with 4 datasets × multiple IPCs × 6 unseen architectures + efficiency analyses + ablations + continual learning + without-Relabeling comparisons.
  • Writing Quality: ⭐⭐⭐⭐ Smooth logic flow from diagnosis -> theory -> methodology, although some notation (Eq. 6/9) in the main text is dense, and parts of the derivations are relegated to the appendix.
  • Value: ⭐⭐⭐⭐⭐ Compresses ImageNet-1K in 80 minutes on a single GPU while establishing a new SOTA, combining both theoretical insight and practical utility.