Skip to content

REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiency

Conference: ECCV 2026
Paper: ECCV 2026 Poster 3941
Cached Source: ../paper_cache/ECCV2026/eccv-3941.txt
Area: Interpretability / Model Compression (interpretability / model_compression)
Keywords: Knowledge Distillation, Robust Statistics, Power Divergence, Influence Function, Teacher Noise

TL;DR

Grounded in robust M-estimation, REDistill substitutes the conventional KL divergence in knowledge distillation with a theoretically derived power divergence (\(\lambda=2/3\)), adaptively downweighting unreliable teacher predictions and achieving state-of-the-art distillation across diverse architectures without hyper-parameter tuning.

Background & Motivation

Knowledge Distillation (KD) is a cornerstone technique for transferring knowledge from an overparameterized teacher network to an efficient, compact student. Conventional logit-based distillation approaches universally align student and teacher predictive distributions by minimizing the Kullback–Leibler (KL) divergence. However, the foundational assumption of this formulation is that the teacher provides accurate and trustworthy soft targets. In real-world scenarios, teacher networks are imperfect—often exhibiting miscalibration, overconfidence, or blatant misclassification. Because the KL divergence relies on the logarithm of the likelihood ratio, its gradient is extremely sensitive to large discrepancies between distributions. When the teacher is noisy, blindly forcing the student to match distorted probabilities leads to error compounding and severe student degradation.

To mitigate teacher noise, existing correction-based techniques typically resort to ad-hoc heuristics—such as swapping predicted top-class logits with ground truth, artificially augmenting ground-truth probabilities, or applying hard class masks. While these adjustments offer empirical patches, they distort natural inter-class correlations and dark knowledge. Crucially, their effectiveness hinges on fragile, model-specific hyper-parameter tuning that requires exhaustive grid searches across architectures and datasets, collapsing when evaluated in realistic, model-agnostic setups.

This paper tackles the problem from the principled foundation of robust statistics rather than empirical heuristics. Framing distillation as target distribution estimation under model misspecification, the authors argue that the loss should intrinsically bound the influence of corrupt likelihood ratios. Core idea: replace the standard KL objective with a power divergence derived from Box–Cox relaxed logarithms, utilizing statistical influence functions to theoretically fix the robustness parameter at \(\lambda=2/3\), thereby adaptively suppressing teacher noise while preserving dark knowledge and computational efficiency.

Method

Overall Architecture

REDistill operates strictly on output logits, avoiding intermediate feature projection heads or teacher structural alterations. Given an input sample \(x\), the teacher and student produce probability vectors \(p(k|x)\) and \(q_\theta(k|x)\). The optimization objective incorporates ground-truth supervision alongside decoupled target and non-target distillation components. In place of standard KL terms, REDistill adopts the power divergence of order \(\lambda=2/3\), paired with an explicit temperature scaling factor \(\tau^2\) to maintain balanced gradient magnitudes during optimization.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input image x and one-hot label y"] --> Net["Forward inference<br/>Teacher f_T(x) and Student f_S(x, theta)"]
    Net --> Logits["Temperature-scaled Softmax (tau)<br/>Distributions p^tau and q^theta_tau"]
    Logits --> CE["Supervised cross-entropy branch<br/>KL(y, q_theta)"]
    Logits --> Split["Decoupled logit partitioning<br/>Target vs. non-target distributions"]
    Split --> PD["Power divergence estimator (lambda=2/3)<br/>Adaptive downweighting of noisy likelihood ratios"]
    CE --> Total["Total loss optimization<br/>tau^2 scaling preserves gradient balance"]
    PD --> Total
    Total --> Out["Back-propagation updating student parameters theta"]

Key Designs

1. Power Divergence Formulation: Adaptive Downweighting of Outlier Logits The standard KL divergence computes the expected surprise using natural logarithms: \(\text{KL}(p, q_\theta) = \sum_{k=1}^K p_k \log \frac{p_k}{q_{\theta,k}}\). In the presence of teacher errors, extreme likelihood ratios \(p/q_\theta\) generate unbounded gradients. REDistill introduces the \((1-\lambda)\)-logarithm (Box–Cox transform of order \(1-\lambda\)): $$ \log_{1-\lambda}(u) = \begin{cases} \log(u), & \lambda = 0 \ \frac{u^\lambda - 1}{\lambda}, & \lambda \neq 0 \end{cases} $$ Substituting this relaxation into the divergence definition yields the power divergence of order \(\lambda\): $$ D_\lambda(p, q_\theta) = \frac{1}{\lambda} \sum_{k=1}^K p(k|x) \left[ \left(\frac{p(k|x)}{q_\theta(k|x)}\right)^\lambda - 1 \right] $$ For \(\lambda > 0\), the power transformation sub-linearly suppresses large deviations between \(p\) and \(q_\theta\), adaptively penalizing noisy teacher targets while smoothly recovering standard KL divergence as \(\lambda \to 0\).

2. Influence-Function-Derived Parameter Selection: Fixing Optimal \(\lambda=2/3\) In robust statistical estimation, the sensitivity of an M-estimator to contamination is quantified by its Influence Function (IF). Under the power divergence objective \(\mathcal{L}_\lambda\), the perturbation induced by an outlier \((x, y)\) evaluates to: $$ \text{IF}(\mathcal{L}\lambda, (x, y)) \propto \left( \frac{p(k|x)}{q\theta(k|x)} \right)^{-\lambda} $$ A larger \(\lambda > 0\) strictly bounds the supremum of the influence function, directly boosting resistance against corrupted teacher predictions. Conversely, smaller \(|\lambda|\) preserves Fisher information and asymptotic statistical efficiency. Aligning with classical results from goodness-of-fit literature (Cressie & Read 1984; Basu et al. 1998), \(\lambda=2/3\) establishes the near-optimal trade-off between statistical efficiency and outlier robustness. REDistill therefore fixes \(\lambda=2/3\) universally across all model architectures and datasets, eliminating manual hyper-parameter tuning.

3. Decoupled Structure and Temperature Scaling Invariance REDistill embeds the power divergence into a decoupled distillation framework, splitting alignment into target class and non-target class subspaces: $$ \mathcal{L}{\text{REDistill}}(\theta, x, y) = \text{KL}(y, q\theta(\cdot|x)) + \alpha D_{2/3}(p_{\text{target}}, q_{\theta,\text{target}}) + \beta D_{2/3}(p_{\text{nontarget}}, q_{\theta,\text{nontarget}}) $$ Furthermore, logit temperature scaling (\(\tau > 0\)) inherently attenuates the magnitude of backward gradients by a factor of \(1/\tau^2\). REDistill demonstrates that multiplying the power divergence objective by \(\tau^2\) (yielding \(\tau^2 D_{2/3}(p^\tau, q^\tau)\)) restores balanced gradient norms relative to the ground-truth cross-entropy term, ensuring rapid and stable convergence across heterogeneous backbone pairings.

Key Experimental Results

Main Results

The authors conduct comprehensive benchmarks on CIFAR-100 and ImageNet-1k across 14 distinct teacher–student pairs spanning homogeneous (e.g., ResNet-to-ResNet) and heterogeneous architectures (e.g., ResNet-to-ShuffleNet/MobileNet). Crucially, models are assessed under the model-agnostic protocol, where a single unified hyper-parameter set is applied across all architectures without validation tuning.

The table below highlights representative results on CIFAR-100 under model-agnostic settings (Table 1) and ImageNet-1k validation set (Table 3), averaged across multiple runs:

Teacher / Student Pair Teacher Top-1 Student Baseline KD (Hinton) DKD (CVPR'22) LSKD (CVPR'24) RLD (ICCV'25) REDistill (Ours)
ResNet32 / ShuffleNetV2 79.42% 71.82% 75.51% 77.01% 77.11% 76.74% 77.52% (+0.51%)
WRN-40-2 / ResNet8 75.61% 72.50% 75.03% 75.50% 76.55% 75.12% 77.21% (+1.71%)
WRN-40-2 / MobileNetV2 75.61% 64.60% 64.66% 69.27% 69.57% 68.91% 69.98% (+0.71%)
ResNet50 / MobileNetV2 79.34% 64.60% 64.40% 70.60% 70.61% 70.22% 70.92% (+0.32%)
ResNet110 / ResNet32 74.31% 71.14% 72.17% 73.53% 73.94% 73.09% 74.18% (+0.65%)
ImageNet: Res34 / Res18 73.31% 69.75% 71.03% 71.70% 71.88% 71.91% 72.00% (+0.30%)
ImageNet: Res50 / MN-V1 76.16% 68.87% 70.50% 72.05% 72.85% 72.75% 72.98% (+0.93%)

Across 14 evaluation pairs, REDistill achieves state-of-the-art accuracy on 13 setups (ranking second only on VGG13/VGG8). Compared to tuning-dependent approaches like RLD and LSKD, REDistill provides superior generalizability across unseen student architectures.

Ablation Study

The evaluation includes two critical ablations: sensitivity analysis on the order parameter \(\lambda\) (Table 6) and controlled teacher logit corruption experiments (Table 7) where true-label cross-entropy is zeroed out to strictly isolate teacher reliability.

Table 1: Ablation on Robustness Parameter \(\lambda\) (CIFAR-100 & ImageNet, Table 6 excerpts)

Parameter Choice \(\lambda\) ResNet32 / SHN-V2 WRN-40-2 / ResNet8 ResNet50 / MN-V2 ImageNet: Res50 / MN-V1 Note
\(\lambda = 0\) (DKD equivalent) 77.01% 75.50% 70.60% 72.05% Standard KL, vulnerable to outliers
\(\lambda = 1/3\) 77.12% 76.91% 70.55% 72.00% Under-penalized noise
\(\lambda = 2/3\) (REDistill theoretical) 77.52% 77.21% 70.92% 72.98% Optimal trade-off peak
\(\lambda = 1.0\) 77.50% 77.15% 70.51% 72.72% Minor over-suppression
\(\lambda = 3/2\) 77.01% 76.21% 70.42% 72.19% Noticeable accuracy drop
\(\lambda = 2.0\) 76.31% 75.80% 70.48% 71.33% Severe loss of statistical efficiency

Table 2: Resilience to Artificial Teacher Noise (ResNet50 / MobileNetV2, Table 7)

Shuffled Logit Ratio KD [Hinton] DKD [CVPR'22] LSKD [CVPR'24] RLD [ICCV'25] REDistill (Ours)
0% (clean teacher) 63.71% 70.25% 70.23% 70.12% 70.76%
15% random noise 63.15% (-0.56%) 68.00% (-2.25%) 67.50% (-2.73%) 70.28% (+0.16%) 70.50% (-0.26%)
30% random noise 62.74% (-0.97%) 61.88% (-8.37%) 53.06% (-17.17%) 69.35% (-0.77%) 70.12% (-0.64%)
45% extreme noise 59.44% (-4.27%) 54.45% (-15.80%) 41.91% (-28.32%) 68.53% (-1.59%) 69.24% (-1.52%)

Key Findings

  • Severe degradation of conventional methods under noise: At 45% corrupted teacher logits, DKD plummets by 15.80% and LSKD collapses by 28.32% (down to 41.91%). In contrast, REDistill drops by only 1.52% (holding at 69.24%), outperforming RLD (68.53%).
  • Theoretical validation of \(\lambda=2/3\): Across all evaluated configurations, \(\lambda=2/3\) achieves peak top-1 accuracy. Values exceeding \(\lambda > 1\) over-penalize discrepancies and impair useful dark knowledge transfer.
  • Seamless compatibility: Incorporating REDistill into KD, DKD, and multi-level logit distillation (MLKD) improves student accuracy across the board (e.g., MLKD+REDistill reaches 78.82% on ResNet32/SHN-V2), surpassing complex feature-based methods like ReviewKD and SimKD without architectural overhead.

Highlights & Insights

  • Statistical grounding replaces heuristic fixes: By replacing ad-hoc logit modifications (swapping/masking) with the power divergence family, REDistill achieves smooth, continuous noise attenuation directly grounded in M-estimation theory.
  • Zero architectural overhead: Operates purely on output probability distributions without requiring intermediate feature projection heads or auxiliary networks, adding less than 1ms per training step over vanilla KD.
  • Model-agnostic generalization: The core hyper-parameter \(\lambda=2/3\) is derived analytically from asymptotic efficiency considerations rather than empirical search, allowing seamless deployment to novel architectures.

Limitations & Future Work

  • Scope limited to classification: Empirical validation is restricted to image classification benchmarks (CIFAR-100, ImageNet-1k); behavior on dense structured outputs (object detection, segmentation) or autoregressive generation remains unexamined.
  • Fixed global \(\lambda\): Although \(\lambda=2/3\) performs robustly across diverse models, sample-adaptive dynamic divergence orders could further optimize handling under heterogeneous uncertainty profiles.
  • Discrete support formulation: The current mathematical formulation is developed for categorical distributions; extending robust divergence estimation to continuous feature representations or self-supervised contrastive distillation is a promising direction.
  • vs DKD [CVPR 2022]: DKD decoupled target and non-target probabilities under standard KL divergence, remaining sensitive to corrupted teacher signals; REDistill retains the decoupled efficiency while replacing KL with power divergence, outpacing DKD by 14.79% under 45% teacher noise.
  • vs RLD [ICCV 2025]: RLD applies hard logit masking to filter untrusted classes, requiring model-tailored threshold searches that degrade significantly in model-agnostic benchmarks; REDistill utilizes continuous smooth downweighting, achieving superior overall accuracy and stability.
  • vs ABKD [arXiv 2025]: ABKD also utilizes generalized power divergences but relies on 2D grid searches over \(\alpha\)-\(\beta\) parameters for each specific dataset; REDistill theoretically anchors \(\lambda=2/3\) via influence function analysis, establishing tuning-free transferability.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Principled integration of robust power divergence and influence function theory into knowledge distillation, replacing heuristic workarounds.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across 14 teacher-student pairs, model-agnostic protocols, explicit noise injection, and multi-method integration.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear mathematical motivation, transparent ablation analysis, and self-consistent narrative.
  • Value: ⭐⭐⭐⭐⭐ Highly practical, plug-and-play formulation with zero parameter overhead and strong cross-architecture generalization for model compression pipelines.