Skip to content

FD\(^2\): A Dedicated Framework for Fine-Grained Dataset Distillation

Conference: ECCV 2026
arXiv: 2603.25144
Code: https://github.com/Guang000/FD2
Area: Dataset Distillation / Fine-Grained Recognition
Keywords: Dataset Distillation, Fine-Grained Recognition, Counterfactual Attention Learning, Decoupled Distillation, Category Prototypes

TL;DR

Addressing the issue that existing decoupled dataset distillation methods ignore discriminative local regions on fine-grained datasets, FD\(^2\) introduces Counterfactual Attention Learning (CAL) to extract discriminative attention maps and class prototypes. Integrating fine-grained feature constraints and intra-class sample similarity constraints during the distillation phase, it serves as a plug-and-play module that significantly improves the distillation quality of SRe2L++ and FADRM+ on fine-grained datasets such as CUB-200-2011, FGVC-Aircraft, and Stanford Cars, achieving a maximum improvement of +15.1% at IPC=1.

Background & Motivation

Background: Dataset Distillation (DD) aims to compress large-scale training sets into tiny synthetic sets, enabling models trained on the synthetic sets to achieve performance close to those trained on the full dataset. Existing methods are mainly divided into five categories: gradient matching, distribution matching, trajectory matching, decoupled distillation, and generative distillation. Among them, decoupled distillation (e.g., SRe2L, SRe2L++, FADRM+) splits the pipeline into three stages: model pre-training, sample distillation, and soft label generation, balancing efficiency and accuracy on general benchmarks.

Limitations of Prior Work: When directly applying decoupled distillation methods to fine-grained datasets (such as CUB-200-2011 for bird recognition), two issues arise: (1) the distilled synthetic sets inherit the fine-grained characteristics of the original data, such as "large intra-class variance and small inter-class variance," making it difficult for student models to learn discriminative representations; (2) multiple distilled samples within the same category are overly similar, with attention heatmaps showing they focus on highly consistent regions, lacking coverage of different discriminative parts, which leads to a single type of discriminative clue available to student models.

Key Challenge: Decoupled distillation relies solely on coarse-grained category label supervision when optimizing synthetic samplesโ€”it enforces category-level semantic consistency but does not explicitly encourage fine-grained intra-class compactness and inter-class separability. Meanwhile, the sample-by-sample iterative synthesis process within the same class shares almost identical optimization paths, naturally leading synthetic samples to converge to similar solutions in the solution space.

Goal: Without changing the overall pipeline of decoupled distillation, introduce dedicated fine-grained supervision signals for fine-grained dataset distillation to (1) fabricates synthetic sets with better fine-grained characteristics (improving intra-class compactness and inter-class separability), and (2) enhance attention diversity of synthetic samples within the same category (covering more discriminative regions).

Key Insight: The authors observe that a highly effective approach in fine-grained recognition is to utilize attention mechanisms to locate discriminative local regions (such as bird beaks, wing textures) and quantify the contribution of attention through counterfactual intervention. Introducing this approach to dataset distillation, using high-quality attention maps and class prototypes constructed thereof as additional supervision during distillation, makes it possible to address the specific problems of fine-grained scenarios while maintaining the efficiency of the decoupled framework.

Core Idea: Use discriminative attention maps and category prototypes extracted by Counterfactual Attention Learning (CAL) as fine-grained supervision signals, appending two plug-and-play constraints to the decoupled distillation objective functionโ€”fine-grained feature constraints (pulling towards intra-class prototypes, pushing away from inter-class prototypes) and similarity constraints (maximizing the distance between attention maps of intra-class samples). This forces the synthetic set to cluster intra-class and separate inter-class in the representation space, while covering different discriminative regions in the attention space.

Method

Overall Architecture

FD\(^2\) follows the three-stage paradigm of decoupled distillation, embedding as an additional module into existing decoupled methods such as SRe2L++ or FADRM+. The core innovations of the framework occur in the pre-training and distillation stages: in the pre-training stage, a dual-branch Backbone+CAL joint training is used to optimize the teacher model and maintain the fine-grained prototypes of each category online; in the distillation stage, synthetic samples are grouped by category (each group of size \(N_S\)), introducing a fine-grained feature constraint and a similarity constraint to be jointly optimized with the original distillation targets; in the soft label generation stage, only the Backbone branch is used to avoid bias introduced by the architecture mismatch between the CAL branch and the downstream Backbone students.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Original Training Set"] --> B["Backbone+CAL Joint Pre-training<br/>Maintain Category Prototypes"]
    B --> C["Teacher Model + Class Prototypes"]
    C --> D["Grouped Distillation of Same Category<br/>N_S Samples per Group"]
    D --> E["Fine-Grained Feature Constraint<br/>Pull to Intra-Class Prototype / Push from Inter-Class Prototype"]
    D --> F["Similarity Constraint<br/>Maximize Attention Differences of Intra-Class Samples"]
    E --> G["Jointly Optimize Synthetic Samples<br/>L = L_other + L_cls + ฮปL_F + (1-ฮป)L_S"]
    F --> G
    G --> H["Backbone Branch Generates Soft Labels"]
    H --> I["Downstream Student Model Training & Evaluation"]

Key Designs

1. Counterfactual Attention Learning (CAL) Pre-training: Locating Discriminative Regions and Constructing Fine-Grained Class Prototypes

Teacher models in decoupled distillation typically use only a standard Backbone classifier, lacking the capability to explicitly model fine-grained discriminative regions. FD\(^2\) introduces the CAL branch during the pre-training stage to be jointly optimized with the Backbone classifier. Given an input image \(x\), the Backbone extracts a feature map \(F\), and the CAL attention predictor \(g(\cdot)\) generates \(M\) attention maps \(A=\{A_m\}_{m=1}^M\). Bilinear attention pooling is applied to obtain the factual representation \(z=\Phi(F,A)\). The core mechanism of CAL is counterfactual intervention: the attention map is replaced with a perturbed version \(\bar{A}\) to obtain the counterfactual representation \(\hat{z}=\Phi(F,\bar{A})\), and the effect prediction is calculated as the factual logit minus the counterfactual logit, \(p_{\text{eff}}=p_{\text{raw}}-W\hat{z}\). If the attention indeed focuses on discriminative regions, replacement weakens the category evidence, making \(p_{\text{eff}}\) more discriminative. During training, the cross-entropy losses of both \(p_{\text{raw}}\) and \(p_{\text{eff}}\) are optimized.

Simultaneously, CAL online maintains the feature centers of each category as class prototypes using momentum updates: \(c_y \leftarrow (1-\mu)c_y + \mu\,\text{Norm}(z)\), and uses a center regularization term \(\mathcal{L}_{\text{center}}=\|z-\text{Norm}(c_y)\|_2^2\) to encourage factual representations to align with their respective class prototypes. The overall pre-training loss is \(\mathcal{L}_{\text{pre}}=(1-\alpha)\mathcal{L}_{\text{ce}}(p_{\text{bb}},y)+\alpha\mathcal{L}_{\text{CAL}}\), where \(\alpha\) controls the contribution of the CAL branch. This dual-branch joint optimization ensures that the Backbone teacher retains the ability to generate stable soft labels while gaining the benefits of discriminative representations and prototype aggregation from CAL.

2. Fine-Grained Feature Constraint: Simultaneously Improving Intra-Class Compactness and Inter-Class Separability

The first core constraint of the distillation stage directly addresses the issue of "dispersed intra-class, close inter-class" in fine-grained data. For the \(i\)-th synthetic sample \(\tilde{x}_{y,i}\) of class \(y\), the teacher model extracts its feature representation \(z_{y,i}\), and the fine-grained feature constraint is defined as:

\[\mathcal{L}_F(\tilde{x}_{y,i}) = \beta\,\ell_2(z_{y,i},c_y) + (1-\beta)\left(1-\mathbb{E}_{k\neq y}\big[\ell_2(z_{y,i},c_k)\big]\right)\]

where \(\ell_2(u,v)=\|u-v\|_2/(\|u\|_2+\|v\|_2+\varepsilon)\) is the symmetrically normalized Euclidean distance, \(c_y\) and \(c_k\) represent the prototype of the current class and other classes respectively, and \(\beta\in[0,1]\) controls the relative weight of "pulling towards the current class" and "pushing away from other classes." The first term minimizes the distance between the sample and its class prototype (enhancing intra-class compactness), while the second term maximizes the distance to other class prototypes (enhancing inter-class separability). This constraint is added to the distillation objective as a plug-and-play term without altering the original optimization pipeline. The authors provide a theoretical analysis in the supplementary materials: under the assumption that prototypes approximate discriminative class representatives, minimizing \(\mathcal{L}_F\) is equivalent to maximizing the discriminative score in the normalized prototype space, aligning with the classical discriminant analysis principle of "minimizing intra-class variance, maximizing distance between class centers."

3. Similarity Constraint: Enhancing Attention Diversity of Intra-Class Synthetic Samples

The second constraint resolves the issue of intra-class synthetic samples being too similar and having highly overlapping attention regions. In decoupled distillation, the sample-by-sample iterative synthesis of intra-class samples shares the same optimization process and similar initialization images (randomly selected from original data), leading their attention maps to naturally converge. The similarity constraint maximizes the distance between the attention map of the \(i\)-th (\(i>1\)) synthetic sample and the attention maps of previous samples in the same group (of size \(N_S\)):

\[\mathcal{L}_S(\tilde{x}_{y,i}) = 1 - \mathbb{E}_{j<i}\big[\ell_2(A_{y,i}, A_{y,j})\big],\quad 1 < i \leq N_S\]

For \(i=1\), this constraint is zero (as there is no prior sample to compare). By minimizing \(\mathcal{L}_S\), the current sample is forced to allocate attention to different discriminative regions compared to existing samples in the same group, thereby allowing multiple synthetic samples of the same class to cover richer local discriminative cues. Theoretical analysis in the supplementary materials shows that minimizing \(\mathcal{L}_S\) expands the lower bound of \(\text{tr}(\Sigma_A^{(y)})\), enhancing the overall variance of attention maps for same-class samples, which propagates to the representation space via bilinear attention pooling, improving representation diversity.

Loss & Training

In the distillation stage, each synthetic sample also receives category supervision from both classifiers (Backbone + CAL):

\[\mathcal{L}_{\text{cls}}(\tilde{x}_{y,i}) = (1-\alpha)\mathcal{L}_{\text{ce}}(p_{\text{bb}}^{y,i}, y) + \alpha\mathcal{L}_{\text{ce}}(p_{\text{cal}}^{y,i}, y)\]

The final optimization objective for each synthetic sample is:

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{other}} + \mathcal{L}_{\text{cls}} + \lambda\mathcal{L}_F + (1-\lambda)\mathcal{L}_S\]

where \(\mathcal{L}_{\text{other}}\) is the original loss of the underlying decoupled distillation method (e.g., BN statistics matching loss in SRe2L++), and \(\lambda\) controls the relative weight of the two new constraints. A key implementation detail is the grouped distillation strategy: the IPC synthetic samples of each category are divided into \(G = \lceil\text{IPC}/N_S\rceil\) groups, with the first \(G-1\) groups containing \(N_S\) samples each, and the last group containing the remaining samples. This prevents the similarity constraint from introducing weakly discriminative regions when the number of samples is too large. The default configuration is set to \(N_S=4\), \(\lambda=0.8\), and \(\beta=0.5\), with the same CAL ratio \(\alpha\) used in both pre-training and distillation stages to maintain feature distribution consistency.

Key Experimental Results

Main Results

FD\(^2\) is evaluated on three fine-grained datasets (CUB-200-2011, FGVC-Aircraft, Stanford Cars) and two general datasets (ImageNette, ImageWoof), integrated as an additional module into SRe2L++ and FADRM+. The table below shows the Top-1 accuracy comparison (ResNet18 student) with IPC=1/3/5 on fine-grained datasets.

Dataset IPC RDED SRe2L++ SRe2L++\(_{\text{FD}^2}\) FADRM+ FADRM+\(_{\text{FD}^2}\)
CUB-200-2011 1 38.3 53.4 56.4 (+3.0) 54.8 55.0 (+0.2)
CUB-200-2011 3 52.6 60.0 64.9 (+4.9) 64.0 64.6 (+0.6)
CUB-200-2011 5 63.9 63.5 67.0 (+3.5) 66.4 67.5 (+1.1)
FGVC-Aircraft 1 22.1 52.6 58.2 (+5.6) 55.0 60.5 (+5.5)
FGVC-Aircraft 3 36.4 66.6 76.1 (+9.5) 72.9 75.1 (+2.2)
FGVC-Aircraft 5 38.6 68.3 80.0 (+11.7) 74.0 77.6 (+3.6)
Stanford Cars 1 33.0 52.4 64.5 (+12.1) 60.3 74.1 (+13.8)
Stanford Cars 3 69.4 68.2 75.2 (+7.0) 75.0 84.8 (+9.8)
Stanford Cars 5 76.1 70.9 81.4 (+10.5) 77.7 86.6 (+8.9)

When using ResNet50 as the student, the performance gains are even more pronounced. For instance, on CUB-200-2011 at IPC=1, SRe2L++\(_{\text{FD}^2}\) reaches 70.1% (+9.0%), and on Stanford Cars at IPC=1, SRe2L++\(_{\text{FD}^2}\) reaches 80.7% (+15.1%). On general datasets, FD\(^2\) shows noticeable improvements under high-capacity student models (ResNet50) (e.g., FADRM+\(_{\text{FD}^2}\) reaches 80.2% on ImageWoof with IPC=50, a +8.5% gain), but provides limited gain under smaller-capacity students (ResNet18)โ€”indicating that the fine-grained discriminative cues introduced by FD\(^2\) depend more on model capacity to be effectively utilized.

In cross-architecture generalization experiments (CUB-200-2011, IPC=3), SRe2L++\(_{\text{FD}^2}\) consistently yields improvements on ShuffleNetV2 (+9.1%), MobileNetV2 (+5.8%), DenseNet121 (+3.8%), and ConvNeXt-Tiny (+1.5%), demonstrating strong transferability of the distilled synthetic sets across different student architectures.

Ablation Study

The table below shows the impact of different constraint combinations on CUB-200-2011 with IPC=3:

Configuration \(\mathcal{L}_F\) \(\mathcal{L}_S\) Top-1 Acc
Baseline (Only \(\mathcal{L}_{\text{other}}+\mathcal{L}_{\text{cls}}\)) -- -- 63.4
+ Fine-Grained Feature Constraint โœ“ -- 64.8
+ Similarity Constraint -- โœ“ 64.6
+ Both Constraints โœ“ โœ“ 64.9

Both constraints individually bring performance gains, and their joint use yields the highest accuracy (64.9%), verifying their complementary natureโ€”\(\mathcal{L}_F\) improves intra-class compactness and inter-class separability in the feature space, while \(\mathcal{L}_S\) enhances diversity in the attention space.

Other key ablation conclusions: (1) \(\beta=0.5\) (the balance of pushing and pulling in \(\mathcal{L}_F\)) is optimal; (2) \(N_S=4\) (group size for similarity constraint) achieves the optimal 66.1%, while \(N_S=5\) drops to 64.6% due to limited discriminative cues; (3) \(\lambda=0.8\) (relative weight of \(\mathcal{L}_F\) and \(\mathcal{L}_S\)) achieves the optimal 67.0%, showing that the fine-grained feature constraint contributes more to distillation quality; (4) using the same CAL ratio \(\alpha\) in both pre-training and distillation stages reduces feature distribution mismatch, with \(\alpha=0.5\) being optimal on CUB-200-2011 (63.4%).

Key Findings

  • The contribution of the fine-grained feature constraint is slightly larger than that of the similarity constraint (+1.4% vs. +1.2%), but their joint integration yields a total gain of +1.5% due to complementary effects, indicating that simply improving the feature space or attention space alone is insufficient; their synergy works best.
  • The improvement is most significant at IPC=1: With only a single synthetic sample, the fine-grained supervision signals of FD\(^2\) help the sample retain discriminative cues to the maximum extentโ€”on Stanford Cars, SRe2L++\(_{\text{FD}^2}\) outperforms SRe2L++ by 12.1% (ResNet18) / 15.1% (ResNet50).
  • t-SNE visualization and nearest-neighbor distance analysis of class centers confirm that the samples synthesized by FD\(^2\) are more compact within classes and more separated between classes; attention heatmap comparisons show that after introducing \(\mathcal{L}_S\), intra-class samples focus on different discriminative regions, and the MPCS (Mean Pairwise Cosine Similarity) drops significantly.
  • RDED (a method using cropped real images) consistently performs worst, indicating that coarse-grained cropping cannot reliably capture fine-grained discriminative regions, further highlighting the value of FD\(^2\) utilizing CAL to precisely locate discriminative regions.
  • Efficiency overhead is acceptable: SRe2L++\(_{\text{FD}^2}\) takes 96.8ms per iteration (vs. 64.6ms for SRe2L++), with a peak GPU memory of 5.3GB (vs. 4.8GB for SRe2L++), which is still highly efficient compared to FADRM+ (12.2GB) and DSA (68.4GB).

Highlights & Insights

  • Migrating counterfactual attention from classification tasks to dataset distillation to locate discriminative regions as fine-grained supervision signals: This is not a simple "add an attention module" operation, but uses CAL's fact-counterfactual comparison mechanism to ensure attention maps truly focus on discriminative parts, then uses these high-quality attention maps and the resulting category prototypes to guide the optimization of synthetic samples, equivalent to introducing prior knowledge of "what constitutes key discriminative cues" during the distillation process.
  • The two constraints target the feature space and attention space respectively, complementing each other with different focuses: \(\mathcal{L}_F\) governs "which prototype each sample resembles" (pulling towards the same-class prototype and pushing away from alternative prototypes in the feature space), and \(\mathcal{L}_S\) ensures that "same-class samples differ from one another" (pushing each other away in the attention space to cover diverse discriminative regions)โ€”the former guarantees discrimination accuracy while the latter guarantees discrimination richness. This dual-space constraint design can be transferred to other tasks requiring the synthesis of diverse, high-quality data.
  • The grouped distillation strategy elegantly balances the diversity constraint and computational overhead: Dividing the IPC synthetic samples of each category into small groups of \(N_S=4\) ensures the constraint only applies intra-group. This avoids redundant attention repetition from overly large groups, keeps the computation of intra-group diversity constraints manageable, and naturally supports multi-process/multi-GPU parallelism. This "local diversity + global grouping" strategy serves as a useful reference for any scenario requiring the synthesis of diverse samples.
  • The counterfactual intervention mechanism of CAL essentially quantifies the causal effect of attention: Checking "how much the prediction degrades after replacing attention" to evaluate whether the attention region is genuinely discriminativeโ€”this causal attribution approach is not limited to fine-grained recognition but can be applied to any scenario that needs to evaluate feature importance (e.g., interpretability, feature selection, knowledge distillation soft target construction).

Limitations & Future Work

  • Limited gain on small-capacity student models: On ImageNette, ResNet18 + FADRM+\(_{\text{FD}^2}\) shows almost no gain or even a slight decrease at IPC=1 (-0.6%), showing that the fine-grained discriminative clues introduced by FD\(^2\) require sufficient model capacity to be effectively utilizedโ€”further research is needed on how to compress or distill these clues for lightweight deployment scenarios.
  • Dependence on the attention quality of the CAL teacher: The effectiveness of FD\(^2\) is built on the premise that CAL can accurately locate discriminative regions. On extremely lightweight backbones like ShuffleNetV2, CAL struggles to converge due to insufficient feature discriminatibility (only usable at \(\alpha=0.1\)), limiting the applicability of FD\(^2\) in resource-constrained environments.
  • Validation limited to the decoupled distillation paradigm: The design of FD\(^2\) is closely tied to the pretraining-distillation-soft label three-stage pipeline of decoupled distillation. The authors propose extending it to ViT-based models and larger-scale datasets (such as ImageNet-1K), as well as broader distillation paradigms.
  • The choice of group size \(N_S\) lacks an adaptive mechanism: \(N_S\) is sensitive to the final performance (2/3/4/5 correspond to 64.1/64.9/66.1/64.6), but is selected empirically via grid search; automatically determining the group size based on dataset characteristics is a meaningful direction for improvement.
  • Generalizability on general datasets still needs to be strengthened: The authors acknowledge that the improvements of FD\(^2\) on general datasets are not as significant as those on fine-grained datasets. How to make FD\(^2\) bring consistent benefits in general scenarios remains an open question.
  • vs. SRe2L / SRe2L++ / FADRM+: These decoupled distillation methods serve as the foundation of FD\(^2\). They perform excellently on general benchmarks using mechanisms like BN statistics matching and multi-scale residual connections, but rely strictly on coarse-grained category labels for supervision. FD\(^2\) does not change their workflow but appends two fine-grained constraints as plug-and-play terms. Experiments show that FD\(^2\) brings a more pronounced boost to SRe2L++ than to FADRM+ (the latter already possesses stronger multi-model ensemble capability, leaving less room for FD\(^2\) to improve).
  • vs. CAL: FD\(^2\) directly adopts CAL as the attention extraction and prototype construction module, but the innovation lies in repurposing CAL's output from classification tasks to serve as supervision signals for dataset distillation. CAL was originally used purely to enhance classification accuracy, whereas FD\(^2\) extends it as a source of fine-grained constraints during distillation, showcasing a new application of counterfactual attention in data synthesis.
  • vs. RDED: RDED constructs synthetic sets by cropping real images, essentially utilizing local regions of real images. However, its cropping is coarse-grained (based on image coordinates) and cannot guarantee that the cropped regions contain discriminative parts. The attention-guided synthesis of FD\(^2\) is more precise in locating discriminative regions, and experiments confirm a massive performance gap between the two (CUB-200 IPC=1: 38.3% vs 56.4%).

Rating

  • Novelty: โญโญโญโญ Introducing counterfactual attention learning to dataset distillation to solve the issues of intra-class dispersion/inter-class proximity and identical intra-class synthetic samples. The logic is clear and the target is precise. Instead of simply "adding modules", it designs two complementary constraints reflecting the core nature of the problem.
  • Experimental Thoroughness: โญโญโญโญโญ Covers 3 fine-grained + 2 general datasets, 2 deep distillation baselines, 5 student architectures, multiple IPC settings. Ablation studies comprehensively cover CAL ratio, \(\beta\), \(N_S\), joint constraints, \(\lambda\), and \(\mathcal{L}_{\text{cls}}\). Features multi-dimensional visualizations including t-SNE, attention heatmaps, and MPCS, alongside rigorous mathematical proofs and efficiency comparisons in the supplementary materials.
  • Writing Quality: โญโญโญโญ The problem and motivation are introduced through quantitative visualization in Fig. 1 (intra-class dispersion + inter-class distance) and graphical attention heatmaps. The method description is clear, accompanied by complete algorithm pseudocode, and mathematical arguments in the supplementary material provide rigorous support for the efficacy of both constraints.
  • Value: โญโญโญโญ Fine-grained dataset distillation is a meaningful but previously neglected problem. As the first targeted framework, and designed as a plug-and-play module that directly enhances existing decoupled methods, FD\(^2\) holds direct value for practical application scenarios requiring the efficient processing of fine-grained data (such as species identification, vehicle type recognition).