Skip to content

From Predictions to Embeddings: Dual Knowledge Distillation for Instance-Dependent Partial Label Learning

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/ayunxiaobao/DKDP
Area: Model Compression
Keywords: partial label learning, instance-dependent, knowledge distillation, label disambiguation, prototype learning

TL;DR

Addressing candidate label semantic confusion and dark knowledge distortion in instance-dependent partial label learning (IDPLL), DKDP introduces a dual distillation framework that structurally rectifies teacher predictions via minimal-perturbation closed-form constrained projection and extends reliable supervision into the embedding space using online class prototypes.

Background & Motivation

Partial label learning (PLL) serves as an effective weakly supervised learning paradigm where each training instance is annotated with a candidate label set containing the concealed ground-truth label, significantly mitigating manual annotation bottlenecks. Most conventional PLL methods rely on the instance-independent assumption (IIPLL), presuming that non-ground-truth candidate labels are selected randomly and uniformly from the label space without correlation to instance features. In real-world scenarios, however, annotators are predominantly confused by visually, semantically, or structurally similar classes. This generates instance-dependent partial labels (IDPLL), where ambiguous candidates correlate strongly with sample features. Under IDPLL, discriminative boundaries are blurred, causing conventional self-training, manifold propagation, and contrastive disambiguation techniques to suffer drastic performance degradation.

Overcoming IDPLL demands not only isolating the true class from confusing candidate sets, but also properly preserving and exploiting the informative dark knowledge embedded within non-candidate labels. A core structural prior states that for any valid sample, the minimum prediction confidence among candidate labels must exceed the maximum confidence among non-candidate labels. Although recent approaches like DIRK employ self-distillation to deliver soft supervision, they exhibit two critical flaws: first, they rely on heuristic piecewise scaling to depress non-candidate probabilities, lacking minimal-perturbation guarantees and inevitably distorting the fine-grained relative relationships among non-candidate labels; second, supervision remains strictly confined to the output logit layer, failing to prevent representation collapse in the feature space where sample representations interlock near decision boundaries.

This paper tackles these challenges by formalizing teacher distribution rectification as a minimal-perturbation projection optimization problem under prior inequality constraints, deriving an exact closed-form solution that preserves relative distributions within candidate and non-candidate groups. Simultaneously, it constructs an online prototype alignment bridge to propagate rectified soft supervisory signals directly into the representation space. Core idea: structurally rectify teacher predictions via minimal-perturbation closed-form constrained projection enforcing partial label priors, and couple logit distillation with online class prototype alignment to achieve joint output-and-embedding dual distillation (DKDP).

Method

Overall Architecture

DKDP establishes a dual distillation pipeline consisting of a teacher network and a student network. During each training iteration, the teacher model outputs an initial prediction distribution. This raw distribution is passed through a constrained projection operator to yield a rectified soft distribution that strictly satisfies the partial label prior with minimal structural modification. The student network is optimized to fit this rectified teacher prediction at the logit level. Simultaneously, the student maintains an online set of class prototypes updated via moving averages, and aligns its sample-to-prototype similarity distribution with a temporally smoothed candidate confidence distribution.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input instance & candidate set (x, S)"] --> B["Teacher prediction distribution p^T"]
    B --> C["1. Minimal-Perturbation Constrained Projection<br/>Closed-form scaling factor r*"]
    C --> D["Rectified teacher distribution q*"]
    D --> E["2. Logits Output Distillation<br/>Cross-entropy fitting q*"]
    D --> F["Smoothed candidate confidence π"]
    G["Student embeddings e & class prototypes ω_c"] --> H["Prototype similarity distribution ν"]
    F --> I["3. Fine-Grained Semantic Distillation<br/>Feature alignment via KL divergence"]
    H --> I
    E --> J["Joint objective L = L_kd + β L_fd"]
    I --> J

Key Designs

1. Minimal-Perturbation Constrained Projection: Closed-form distribution rectification Directly utilizing raw predictions from an EMA-updated teacher often leads to distillation collapse because early model predictions may assign non-candidate classes higher probabilities than candidate ones, violating the fundamental PLL condition. To rectify this without corrupting the dark knowledge ratios among non-candidate classes, the paper formalizes the correction as finding a distribution \(q^*\) on the probability simplex \(\Delta^C\) that minimizes the KL divergence \(\mathrm{KL}(q \parallel p^T)\) subject to \(\min_{i \in S} q_i \ge \max_{j \notin S} q_j + \rho\), where \(\rho > 0\) is a small margin constant.

To avoid solving a full numerical optimization on the simplex per iteration, the authors define a candidate re-scaling family. Letting \(P_S = \sum_{i \in S} p_i^T\) and \(P_{\bar{S}} = 1 - P_S\), the scaled distribution parameterized by \(r > 0\) is defined as \(q_i(r) = \frac{r p_i^T}{r P_S + P_{\bar{S}}}\) for \(i \in S\) and \(q_i(r) = \frac{p_i^T}{r P_S + P_{\bar{S}}}\) for \(i \notin S\). Under the condition \(p_{\min} - \rho P_S > 0\) (where \(p_{\min} = \min_{i \in S} p_i^T\) and \(p_{\max} = \max_{j \notin S} p_j^T\)), the minimal scaling factor satisfying the prior constraint yields an exact analytical closed-form:

\[r_{\min} = \frac{p_{\max} + \rho P_{\bar{S}}}{p_{\min} - \rho P_S}\]

Setting \(r^* = \max(1, r_{\min})\) and substituting it into \(q(r^*)\) produces the target distribution \(q^*\). This ensures that intra-group probability proportions remain strictly invariant while minimally expanding the inter-group margin.

2. Logits Output Distillation: Transferring rectified dark knowledge Once the compliant target \(q^*\) is obtained, logit-level distillation guides the student model's output distribution \(f^S\) to match \(q^*\). Unlike hard pseudo-label assignments or uniform candidate averaging, \(q^*\) retains the relative confidence ratios among non-candidate classes, transferring rich visual similarity and semantic relationship knowledge into the student.

The student is optimized end-to-end via cross-entropy loss against the rectified target:

\[\mathcal{L}_{kd} = - \sum_{c=1}^C q_c^* \log(f_c^S)\]

The teacher model parameters \(\theta^T\) are updated smoothly using an exponential moving average (EMA) of student parameters \(\theta^S\): \(\theta^T \leftarrow m \theta^T + (1-m) \theta^S\). This momentum mechanism stabilizes supervision signals across training steps and prevents error amplification from isolated batch noise.

3. Fine-Grained Semantic Distillation: Prototype-driven representation geometry alignment Because instance-dependent candidate confusion severely distorts the geometry of the embedding space, logit-level distillation alone cannot guarantee well-separated class clusters. To impose geometric constraints on the representation space, DKDP maintains a normalized class prototype \(\boldsymbol{\omega}_c\) for each category \(c \in \{1, \dots, C\}\) and updates it per mini-batch:

\[\boldsymbol{\omega}_c \leftarrow \mathrm{Normalize}(\gamma \boldsymbol{\omega}_c + (1 - \gamma) \bar{\boldsymbol{e}}_c)\]

where \(\bar{\boldsymbol{e}}_c\) denotes the average embedding of batch samples whose candidate-constrained prediction falls into class \(c\). The cosine similarity distribution \(\boldsymbol{\nu}_i\) between sample feature \(\boldsymbol{e}_i\) and all prototypes is computed with temperature \(\tau\): \(\nu_{ik} = \frac{\exp(\boldsymbol{e}_i \cdot \boldsymbol{\omega}_k / \tau)}{\sum_{c=1}^C \exp(\boldsymbol{e}_i \cdot \boldsymbol{\omega}_c / \tau)}\). This similarity distribution is aligned with a moving-average candidate confidence distribution \(\boldsymbol{\pi}_i\) via KL divergence:

\[\mathcal{L}_{fd} = \mathrm{KL}(\boldsymbol{\pi}_i \parallel \boldsymbol{\nu}_i)\]

This objective pulls student representations toward their correct class prototypes while repelling confusing candidate and non-candidate prototypes, establishing discriminative decision boundaries in the latent space.

Loss & Training

The overall training objective combines the logit output distillation loss and the fine-grained semantic distillation loss:

\[\mathcal{L} = \mathcal{L}_{kd} + \beta \cdot \mathcal{L}_{fd}\]

where \(\beta\) controls the relative trade-off between output and representation distillation. The student network parameters are trained using standard back-propagation, while the teacher parameters and class prototypes are updated momentum-wise without gradients. The default margin parameter is set to \(\rho = 0.05\) across ResNet-18 and ResNet-34 backbones.

Key Experimental Results

Main Results

DKDP is benchmarked against 12 deep PLL algorithms (comprising 7 IIPLL methods and 5 specialized IDPLL methods) across four standard benchmark datasets (Fashion-MNIST, Kuzushiji-MNIST, CIFAR-10, CIFAR-100) and three fine-grained vision datasets (Flower-102, Oxford-IIITPet, CUB-200). The classification accuracy results (from Table 1 and Table 2 in the paper) are summarized below:

Dataset Metric DKDP (Ours) Prev. SOTA (DIRK) Gain
Fashion-MNIST Top-1 Acc (%) 91.81 ± 0.10 91.48 ± 0.21 +0.33%
Kuzushiji-MNIST Top-1 Acc (%) 98.49 ± 0.05 96.80 ± 0.52 +1.69%
CIFAR-10 Top-1 Acc (%) 93.64 ± 0.17 90.87 ± 0.25 +2.77%
CIFAR-100 Top-1 Acc (%) 70.72 ± 0.20 68.77 ± 0.49 +1.95%
Flower-102 Top-1 Acc (%) 48.92 ± 0.35 44.03 ± 0.02 +4.89%
Oxford-IIITPet Top-1 Acc (%) 70.50 ± 0.49 64.95 ± 2.11 +5.55%
CUB-200 Top-1 Acc (%) 54.04 ± 0.23 49.29 ± 1.00 +4.75%

Ablation Study

The contribution of each distillation module is examined on five datasets with varying degrees of classification complexity (from Table 4 in the paper):

Configuration \(\mathcal{L}_{ce}\) \(\mathcal{L}_{kd}\) \(\mathcal{L}_{fd}\) CIFAR-10 CIFAR-100 Flower-102 Note
Full Model (DKDP) ✓ ✓ ✓ 93.64 70.72 48.92 Full dual distillation framework
w/o \(\mathcal{L}_{fd}\) ✓ ✓ ✗ 90.89 68.05 44.36 Removes prototype semantic distillation
w/o \(\mathcal{L}_{fd} \ \& \ \mathcal{L}_{kd}\) ✓ ✗ ✗ 72.95 62.97 37.72 Standard self-training baseline

Quantitative teacher distribution evaluations (Table 3 and Figure 4) reveal that DKDP incurs significantly lower KL divergence, total variation (TV) distance, and JS divergence compared to DIRK's heuristic scaling, successfully avoiding overcorrection on non-candidate classes. Training overhead analysis on CIFAR-10 (Table 5) shows that DKDP requires 96s per epoch, which is on par with ABLE (98s) and only slightly higher than DIRK (71s), demonstrating an efficient trade-off.

Key Findings

  • Fine-grained semantic distillation is essential for high-confusion datasets: Ablation results show that omitting \(\mathcal{L}_{fd}\) leads to modest drops on simpler datasets (+0.94% on KMNIST) but severe degradation on challenging datasets (dropping 2.67% on CIFAR-100 and 4.56% on Flower-102), confirming that feature-level prototype alignment is critical when inter-class semantic overlap is high.
  • Constrained projection prevents training divergence: Removing both \(\mathcal{L}_{fd}\) and \(\mathcal{L}_{kd}\) collapses CIFAR-10 accuracy from 93.64% to 72.95% (-20.69%), demonstrating that without prior-enforced teacher rectification, self-training reinforces wrong candidate associations.
  • Robustness across hyper-parameter sweeps: Model accuracy remains consistently high over a wide range of \(\beta\), and the margin constant \(\rho\) achieves optimal performance around 0.05 without distorting teacher probability structures.

Highlights & Insights

  • Analytically guaranteed minimal modification: Instead of ad-hoc clipping or heuristic probability scaling, formulating rectification within a candidate re-scaling family yields an exact closed-form solution with minimal KL divergence and preserved dark knowledge ratios.
  • Structural preservation of dark knowledge: DKDP demonstrates that enforcing partial label constraints does not require zeroing out or heavily suppressing non-candidate logits; maintaining their relative ratios provides vital negative transfer prevention.
  • Dual-level knowledge propagation: Coupling logit distillation with online class prototype matching bridges the gap between output probability confidence and latent metric geometry.

Limitations & Future Work

  • Single-scalar re-scaling restriction: The closed-form derivation assumes a uniform scaling factor \(r\) across the entire candidate set, which might restrict expressiveness when individual candidate labels exhibit highly non-uniform noise levels.
  • Cold-start sensitivity in prototype estimation: Moving average prototypes depend on early student predictions within the candidate set; in the initial training epochs with noisy representations, pseudo-label assignments might introduce transient prototype drift.
  • Future directions: Extending dual distillation to instance-dependent partial multi-label learning (ID-PML) and integrating adaptive temperature scheduling based on candidate set ambiguity.
  • vs DIRK (Wu et al., AAAI 2024): DIRK introduced distillation to IDPLL but employed heuristic scaling that disrupts relative non-candidate distributions; DKDP provides a closed-form minimal-perturbation projection that outperforms DIRK by 2.77% on CIFAR-10 and 5.55% on Oxford-IIITPet.
  • vs PICO (Wang et al., ICLR 2022): PICO combines prototype classifiers with contrastive learning under IIPLL; under IDPLL's severe feature correlation, its instance-level contrasts risk pushing semantically related true samples apart, whereas DKDP leverages smoothed teacher guidance to stabilize prototype alignment.
  • vs POP (Xu et al., ICML 2023): POP relies on progressive hard pruning of candidate sets where false eliminations cannot be recovered; DKDP maintains full soft probability distributions throughout training, offering superior noise tolerance.

Rating

  • Novelty: ⭐⭐⭐⭐ [Derives closed-form minimal-perturbation projection under PLL constraints and bridges output and prototype-embedding distillation]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated on 4 standard and 3 fine-grained benchmarks against 12 baselines with extensive metric divergence tests and ablation studies]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous mathematical formulation, clear theoretical explanations, and solid empirical visualizations]
  • Value: ⭐⭐⭐⭐⭐ [Offers an effective, lightweight paradigm for weakly supervised label noise that provides immediate practical utility for real-world annotation scenarios]