Structural Assessment for Understanding and Guiding Dataset Distillation in Discrete Token Space¶

Conference: ECCV 2026
arXiv: 2606.21705
Code: None
Area: Model Compression / Dataset Distillation
Keywords: Dataset Distillation, Discrete Visual Tokens, Structural Score, Diffusion Model Guidance, Visual Token Statistical Analysis

TL;DR¶

This work proposes a Structural Score to evaluate the quality of distilled datasets through statistical analysis in the discrete visual token space. Based on this, a Token-Guided Dataset Distillation (TGDD) framework is developed, which utilizes the Structural Score to guide diffusion models for generating high-quality distilled data.

Background & Motivation¶

The goal of dataset distillation is to replace a large-scale training set with a small set of synthetic data, maintaining comparable model performance when trained on this subset. In recent years, numerous works have aligned distilled data with real data distributions in continuous feature spaces via feature matching, optimal transport, and gradient matching. However, such distribution-level similarity does not guarantee semantic structural effectiveness—a distilled dataset might align closely in the embedding space while losing rare concepts, over-representing trivial structures, or failing to cover discriminative patterns. In practice, it has even been observed that datasets most similar to the original data distribution do not always yield the best training performance. This raises a core question: besides distribution similarity, what fundamental characteristics determine the effectiveness of distilled datasets?

This work addresses this question from the perspective of discrete visual tokens. Discrete tokenizers (such as VQ-VAE) map each image to a sequence of indices in a finite codebook, making the composition of semantic primitives statistically analyzable. The authors observe plane images from ImageNet versus remote sensing plane images: although these two domains differ greatly in perspective, background, and style, they share high-frequency tokens (corresponding to shared visual primitives of planes), whereas their overall token distributions diverge due to contextual differences. This observation suggests shifting from global distribution matching to explicitly characterizing the token composition of the dataset.

Specifically, the authors introduce three complementary metrics to characterize the token distribution of each image: Jensen-Shannon Divergence (JSD) to measure the contextual fit between a sample's token distribution and the class center, the Herfindahl-Hirschman Index (HHI) to measure token usage balance, and TF-IDF coverage to measure the reservation of class-discriminative tokens. The weighted combination of these three metrics yields the Structural Score. Core Idea: Statistical analysis in the discrete token space reveals that token composition balance (HHI) of distilled datasets is the strongest predictor of validation accuracy, and distribution similarity does not equate to effectiveness. This structural score is then used to guide diffusion models to generate higher quality distilled data.

Method¶

Overall Architecture¶

The proposed method consists of two parts. The first part is diagnostic analysis: images are mapped to discrete token distributions using a multi-scale VQ-VAE. After cross-scale weighted fusion, three token-level statistics (JSD, HHI, COV) are calculated to fit a Structural Score via linear regression, predicting the validation accuracy of the distilled dataset. The second part is guided generation (TGDD): k-means clustering is applied to each class in the discrete token space. Within each cluster, candidate samples are ranked by their Structural Scores, and the Top-M anchors are selected. Their average embeddings serve as the pattern guidance signal to steer the DiT diffusion denoising process, generating the final distilled dataset.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Original Dataset"] --> B["Multi-scale VQ-VAE<br/>Encoding to Discrete Tokens"]
    B --> C["Cross-scale Weighted Fusion<br/>to Obtain Token Distribution per Image"]
    C --> D1["Calculate JSD/HHI/COV"]
    C --> D2["PCA Dimensionality Reduction +<br/>L2 Normalization"]
    D1 --> E1["Linear Regression<br/>→ Structural Score"]
    D2 --> E2["k-means Clustering<br/>C Clusters per Class"]
    E1 --> F2["Sort by Structural Score<br/>Select Top-M Anchors"]
    E2 --> F2
    F2 --> G["Average Anchor Embeddings<br/>→ Pattern Guidance Signal"]
    G --> H["DiT Diffusion Denoising<br/>Apply Pattern Guidance"]
    H --> I["Generate Distilled Dataset"]

Key Designs¶

1. Multi-Scale Discrete Token Statistical Representation: Converting Continuous Images into Statistically Analyzable Discrete Distributions

After passing through a multi-scale VQ-VAE, an image produces a token index map at each of the \(L\) scales. The token occurrence frequency at each scale is normalized into a probability vector \(\bm{p}_i^{(\ell)}\). Since the number of tokens at higher resolutions scales quadratically with resolution (\(L=10\) scales), direct summation would cause the fine scales to dominate the fused representation. To address this, the authors divide the 10 scales into low (1-3), mid (4-7), and high (8-10) frequency bands, assigning weights of [3, 1, 0.5] respectively to perform weighted fusion, compensating for the sparsity at lower resolutions:

\[\bm{p}_i = \sum_{\ell=1}^{L} w_\ell \, \bm{p}_i^{(\ell)}\]

The fused \(\bm{p}_i\) represents the overall token distribution of image \(i\) over the codebook, supporting all subsequent statistical analyses.

2. Three Metrics of Structural Score: Evaluating Data Quality across Contextual Fit, Compositional Richness, and Class Discriminativeness

Representing each image as a discrete distribution over a codebook, three complementary statistics characterize its token compositional structure:

JSD (Jensen-Shannon Divergence) measures the difference between a sample's token distribution \(\bm{p}_i\) and the class average token distribution \(\bm{\mu}_c\). A lower JSD indicates that the sample's token composition pattern is more representative of that class (i.e., higher contextual fit).

HHI (Herfindahl-Hirschman Index) measures the concentration of the token distribution: \(\mathrm{HHI}(\bm{p}_i)=\sum_k (p_i(k))^2\). A lower HHI indicates more balanced token usage and richer visual primitives. Experiments show that HHI is the single metric most strongly correlated with validation accuracy among the three—balanced token composition is more important than merely fitting the original distribution.

COV (Coverage) measures the retention of class-discriminative tokens. A set of top discriminative tokens \(\mathcal{T}_c\) is extracted for each class from the original data using TF-IDF, and the sum of probabilities of the sample's tokens belonging to this set is calculated. A higher COV implies that the sample retains more class-specific visual primitives.

The weighted combination of these three metrics yields the Structural Score, which reliably predicts the validation accuracy of distilled datasets (with a regression MSE of only ~\(6\) on ImageWoof). Notably, the three metrics do not contribute equally to distillation: regression coefficients show that the weight of HHI is \(-10.99\) (negative correlation, lower HHI \(\to\) higher accuracy), far exceeding those of JSD (\(2.96\)) and COV (\(2.07\)), indicating that the balance of token composition is significantly more critical than other factors.

3. Structural Score-Based Anchor Selection: Using Token Statistics Instead of Continuous Features to Filter Most Representative Guidance Samples

The key to converting the Structural Score from a diagnostic tool to a generative guidance tool lies in anchor selection. TGDD applies k-means clustering in the discrete token space with \(K_c\) clusters per class. Unlike MGD3, which directly uses cluster centroids for guidance, TGDD ranks candidate samples within each cluster by their Structural Scores and selects only the Top-M structurally optimal samples as anchors (\(M=20\) for IPC < 50, and \(M=10\) otherwise). This filtering avoids introducing noisy samples via mean pooling—especially at low IPCs where the small size makes casual noise more likely to pollute the cluster centers. Samples ranked bottom in Structural Score often exhibit unbalanced token usage, lack crucial class-discriminative tokens, or deviate from the class center. Excluding them from the guidance signal significantly improves generation quality.

4. Token-Guided Diffusion Generation: Guiding DiT Denoising with Anchor Centroid Signals

Once anchors are selected, the embeddings of all anchors within the same cluster are averaged to obtain the guidance signal \(\bar{\bm{z}}_{c,m}\) for that pattern. During the reverse denoising process of DiT, a pattern guidance of strength \(\lambda\) is applied at all timesteps \(t > t_{\text{stop}}\):

\[\bm{z}_{t-1} = \bm{z}_{t-1}^{\text{base}} + \lambda \, (\bar{\bm{z}}_{c,m} - \hat{\bm{z}}_0)\]

where \(\hat{\bm{z}}_0\) is the predicted clean latent variable at the current timestep. Guidance is terminated in early timesteps (\(t \leq t_{\text{stop}}\)) to preserve sample diversity. Compared to clustering directly in continuous feature spaces and using cluster centroids (e.g., MGD3), the compositional semantics in the token space are clearer, and anchor filtering is more precise, producing higher-quality synthetic data under the same diffusion backbone. Extensive ablations verify that the full TGDD significantly outperforms random anchor selection (43.1% vs baseline 45.2%) and Structural Score filtering on continuous features (44.5%), showing that discrete clustering and Structural Score selection are highly complementary.

Loss & Training¶

TGDD utilizes the same diffusion generation pipeline as MGD3 without requiring additional training. After anchor filtering, it directly guides the 50-step sampling of a pre-trained DiT, with the stop guidance step set to timestep 25. For the distilled synthetic dataset, a downstream classifier (ConvNet-6 / ResNetAP-10 / ResNet-18) is trained using SGD for 1500-2000 epochs, fully consistent with the evaluation protocols of prior methods such as MGD3.

Key Experimental Results¶

Main Results¶

Dataset	Downstream Model	IPC	SOTA (MGD3)	TGDD	Gain
ImageWoof	ResNetAP-10	50	56.5	60.3	+3.8
ImageWoof	ResNetAP-10	70	60.2	63.4	+3.2
ImageWoof	ResNetAP-10	100	66.5	67.3	+0.8
ImageNette	ResNetAP-10	10	66.4	67.8	+1.4
ImageNette	ResNetAP-10	50	79.5	81.3	+1.8
ImageNet-1k	ResNet-18	10	45.6	45.8	+0.2
ImageNet-1k	ResNet-18	50	60.2	60.3	+0.1

TGDD exhibits the most pronounced advantage on fine-grained datasets with high inter-class similarity like ImageWoof (where global distribution matching is less effective, and token composition analysis excels at capturing subtle differences). The gain on ImageNet-1k is relatively modest but still achieves state-of-the-art.

Ablation Study¶

Configuration	ImageWoof IPC=50	Description
MGD3 baseline (continuous space clustering + centroid guidance)	56.5	No discrete space, no PCA, no anchor selection
+ Discrete space clustering	57.4	Only switching clustering space from continuous to discrete token
+ PCA dimensionality reduction	58.9	Add 512-dim PCA
+ Anchor selection (Complete TGDD)	60.3	Use Structural Score to filter Top anchors

Individual and combined ablation of each Structural Score metric (ImageWoof IPC=10, ResNetAP-10):

Configuration	JSD	HHI	COV	Accuracy
Single metric (JSD)	✓			38.2
Single metric (HHI)		✓		37.8
Single metric (COV)			✓	37.1
Pairwise combination (JSD+HHI)	✓	✓		38.7
Complete three metrics	✓	✓	✓	41.2

Single-metric performance is limited; pairwise combinations show some complementarity, while the full three-metric combination performs the best.

Key Findings¶

HHI is the strongest single predictor of accuracy: The regression coefficient is \(w_{\mathrm{HHI}} = -10.99\), which is far larger in magnitude than those of JSD (\(2.96\)) and COV (\(2.07\)). This indicates that the balance of token composition affects training performance significantly more than other factors. Utilizing various visual primitives in the codebook in a more balanced manner is much more important than just fitting the original token distribution.
Distribution similarity does not equate to effectiveness: The JSD values of DM and RDED are higher than those of samples generated by Stable Diffusion (meaning they deviate more from the original distribution), yet their training accuracies are actually higher. This directly refutes the intuition that "closer distribution equals better performance."
Good cross-domain generalization: The regression coefficients fitted on ImageWoof generalize directly to the EuroSAT remote sensing dataset, where the Structural Score rises monotonically with the distillation process, aligning with the validation accuracy trend. Different discrete tokenizers (VQGAN, BEiTv2, VQ-VAE) all surpass continuous models (VAE, CLIP, DINOv2) in guiding generation quality.
Low IPC requires more anchors: At IPC=10, 20 anchors yield optimal performance; as IPC increases, fewer anchors are required (10 is sufficient), as excessive anchors may introduce noise and degrade performance.

Highlights & Insights¶

Closed loop from diagnosis to design: The most valuable contribution of this study is establishing a closed-loop of "understanding why distilled data is effective \(\to\) using the insight to guide generation." Regression analysis first reveals that token composition balance (HHI) is key; in turn, the Structural Score (incorporating HHI) is used to select anchors rather than directly using centroid averages as in MGD3, ensuring that analysis directly serves generation.
Statistical advantages of discrete token space: In contrast to continuous feature spaces where semantics are entangled and hard to analyze finely, the finite vocabulary of discrete tokens makes statistics (JSD/HHI/COV) naturally suited for distributional analysis. Since each token corresponds to specific visual primitives, the results are highly interpretable. This provides a general evaluation framework independent of specific distillation methods.
Zero additional training cost: Computing the Structural Score is an inference-time operation (a single VQ-VAE forward pass + statistic calculations), and TGDD does not require fine-tuning the diffusion model. The extra preprocessing overhead is only about 0.054 seconds per image, which is negligible compared to the 1.7 seconds per image for diffusion generation. The total distillation time (0.43 hours) is comparable to the highly efficient MGD3 (0.32 hours) and only 1/5 of training-heavy methods like Minimax (2.02 hours).

Limitations & Future Work¶

Structural Score depends on the quality of discrete tokenizers: If a tokenizer fails to capture fine-grained semantic differences (e.g., minor lesions in medical images), the score reliability degrades. This work only validates on 4 tokenizers, and generalization to broader scenarios requires further testing.
Limitations of the linear regression assumption: The Structural Score uses a linear model to fit accuracy, but the true relationship may be more complex (e.g., non-linear interactions or tight coupling among metrics). Based on the cross-domain results of EuroSAT, the absolute value of the score is not directly comparable across different domains and can only be used for relative ranking.
TGDD's guidance framework depends on MGD3: The guidance mechanism is inherited from MGD3, validated only on DiT and LDM, and the stop guidance hyperparameter (\(t_{\text{stop}}=25\)) might require re-tuning on different datasets.
Label dependency of datasets: The COV and JSD metrics in the Structural Score require class labels and class-wise TF-IDF calculations, making them inapplicable to unlabeled dataset distillation evaluations.

vs MGD3: MGD3 clusters in the continuous feature space and uses the average of all cluster members as the guidance signal. TGDD, instead, clusters in the discrete token space and uses the Structural Score to select the Top anchors. Even though they share the same diffusion guidance framework, the difference in anchor selection yields up to a 5.4% improvement for TGDD over MGD3 on ImageWoof.
vs GLaD / D4M: These works leverage the generative priors of GANs or diffusion models to assist distillation optimization but lack quantitative assessment tools for the distilled data itself. The proposed Structural Score fills this gap and can be used independently of generative methods.
vs DM (Distribution Matching): DM emphasizes global distribution alignment. The experiments in this paper directly challenge the intuition that "closer distributions are more effective"—while DM has a higher JSD (meaning less similarity to the original distribution), it achieves high accuracy, indicating that the balance of token composition is more critical than mere distribution proximity.

Rating¶

Novelty: ⭐⭐⭐⭐ Introducing discrete token statistical analysis into dataset distillation evaluation is a fresh perspective; the closed-loop design from diagnosis to generation is elegant.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Thoroughly ablated across 5 datasets, multiple IPC settings, 3 downstream architectures, multiple tokenizers, and diffusion architectures; the experimental design is highly rigorous.
Writing Quality: ⭐⭐⭐⭐ Clear motivation with a complete logical chain (discovery \(\to\) understanding \(\to\) guidance); high-quality figures and tables.
Value: ⭐⭐⭐⭐⭐ Provides the first interpretable quantitative evaluation tool for the dataset distillation field and directly improves generation quality, offering high practical value.