Don’t Teach Instability, Teach Robustness: Selective Sensitivity Gating for Adversarial Distillation¶

Conference: ECCV 2026
Paper: CVF Open Access
Code: https://github.com/JingqiJi03/AEGIS
Area: Model Compression
Keywords: Adversarial Robustness Distillation, Knowledge Distillation, Stability Punishment, Sensitivity Gating, Adversarial Defense

TL;DR¶

Addressing the "stability punishment" phenomenon where standard Adversarial Robustness Distillation blindly mimics the teacher's jagged decision boundaries, this paper proposes AEGIS to decouple supervision via semantic logit gating and sensitivity calibration gating, boosting ResNet-18 AutoAttack accuracy to 53.22% on CIFAR-10 (a +3.12% gain over ARD) while preserving ~50% accuracy under severe \(\epsilon=16/255\) perturbations.

Background & Motivation¶

Adversarial Robustness Distillation (ARD) serves as a primary avenue for transferring defensive capabilities from heavily over-parameterized models to deployment-friendly lightweight networks. Existing ARD paradigms—whether matching soft output probability distributions or aligning local input gradient landscapes—invariably rely on the fundamental premise that the teacher's decision boundary is an optimal geometric oracle. However, adversarially trained teacher models frequently suffer from severe robust overfitting, leading to jagged, unstable decision boundaries whose local probability shifts carry substantial sensitivity noise under perturbation.

Enforcing exact matching across all samples induces an unexpected "stability punishment" dilemma. Under adversarial attacks, a teacher model may experience drastic confidence drops (\(|\Delta p_t| \gg 0\)) around local boundary fissures. In contrast, smaller student models, whose constrained capacity acts as an implicit regularizer, often exhibit smoother local manifolds and higher intrinsic stability (\(|\Delta p_s| \approx 0\)). Standard gradient or logit alignment penalizes the student for resisting this drop, forcing it to enlarge its local gradient norm to replicate the teacher's volatility. Consequently, rather than inheriting defensive strength, the student is actively poisoned by the teacher's sensitivity noise.

To eliminate this fundamental flaw, distillation targets must evolve from indiscriminate alignment to fine-grained differential sensitivity filtering. An effective framework must not only correct categorical misclassifications but also selectively ignore the teacher whenever its local geometry is more fragile than that of the student. Core idea: propose AEGIS (Adversarial Error Gating and Instability Suppression), which decouples distillation into a Semantic Logit Gate (SLG) that reverts to clean teacher logits upon adversarial misclassification, and a Sensitivity Calibration Gate (SCG) that regulates target-class and non-target-class differential probability variations, enforcing gradient-level supervision only when the teacher exhibits genuinely superior geometric stability.

Method¶

Overall Architecture¶

AEGIS reorganizes the adversarial distillation pipeline into a dual-branch gated architecture. Given input pair \((x, x_{adv})\), the Semantic Logit Gate (SLG) first examines whether the teacher correctly predicts the ground-truth label on the perturbed input \(x_{adv}\). If the teacher is fooled, SLG shifts the distillation target back to the teacher's stable clean-image logits \(f_T(x)\), preventing error propagation. Concurrently, the Sensitivity Calibration Gate (SCG) operates in the differential probability variation space \(\Delta \mathbf{p} = \mathbf{p}(x_{adv}) - \mathbf{p}(x)\), branching into target-class sensitivity calibration and non-target-class structural filtering to ensure that local gradient alignment occurs strictly under favorable geometric conditions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input pair x and adversarial sample x_adv"] --> B["Forward pass through Teacher & Student"]
    B --> C["Semantic Logit Gate<br/>Verify teacher prediction against ground truth"]
    C -->|Teacher Correct| D1["Adversarial Logits Matching"]
    C -->|Teacher Incorrect| D2["Clean Logits Fallback Distillation"]
    B --> E["Sensitivity Calibration Gate<br/>Compute differential probability variation Δp"]
    E --> F1["Target-Class Sensitivity Calibration<br/>Constrain student drop against teacher weakest boundary"]
    E --> F2["Non-Target Structural Knowledge Filtering<br/>Mask misleading inverted non-target classes"]
    D1 & D2 --> G["Combined Objective Optimization"]
    F1 & F2 --> G

Key Designs¶

1. Semantic Logit Gate: Dynamic fallback to clean dark knowledge upon teacher failure

When an adversarial perturbation fools the teacher, forcing the student to emulate its adversarial output leads to catastrophic error inheritance. Rather than discarding such samples or substituting them with rigid one-hot ground-truth labels, SLG computes a dynamic reliability indicator \(G_{SLC} = \mathbb{I}(\operatorname*{argmax} f_T(x_{adv}) = y)\). When \(G_{SLC} = 1\), the student matches the teacher's adversarial soft logits; when \(G_{SLC} = 0\), the gate smoothly redirects the distillation target to the teacher's clean logits \(f_T(x)\):

\[\mathcal{L}_{SLG} = \mathbb{E}_x \left[ G_{SLC} \mathcal{L}_{KL}\big(f_S(x_{adv}), f_T(x_{adv})\big) + (1 - G_{SLC}) \mathcal{L}_{KL}\big(f_S(x_{adv}), f_T(x)\big) \right]\]

This mechanism halts the transmission of misclassified adversarial errors while preserving the rich dark knowledge—such as inter-class topological correlations—inherent in the teacher's clean distribution.

2. Target-Class Sensitivity Calibration: Bounding student degradation by the critical boundary

In adversarial geometry, attacks push inputs along gradients toward the closest decision boundary, making the non-target class with the maximal probability rise, \(k^* = \operatorname*{argmax}_{k \neq y} (\Delta \mathbf{p}_T)_k\), the direct indicator of the model's most vulnerable decision margin. If the student's ground-truth probability degradation \((\Delta \mathbf{p}_S)_y\) is faster than the teacher's expansion toward this closest error boundary \((\Delta \mathbf{p}_T)_{k^*}\), the student requires geometric stabilization. Conversely, if the student exhibits higher resistance, forcing alignment merely inflates the student's local gradient norm \(\|\nabla \mathbf{p}_S\|_2\). Thus, TRD establishes the adaptive gating condition \(G_{TC} = \mathbb{I}\big( (\Delta \mathbf{p}_S)_y < (\Delta \mathbf{p}_T)_{k^*} \big)\):

\[\mathcal{L}_{TRD} = G_{TC} \cdot \left| (\Delta \mathbf{p}_S)_y - (\Delta \mathbf{p}_T)_{k^*} \right|^2\]

When the student's local stability matches or surpasses the teacher's critical tolerance, the gate closes, protecting the student's smooth manifold from teacher noise.

3. Non-Target Structural Knowledge Filtering: Purging inverted classes to distill genuine manifold geometry

While non-target classes contain valuable relative confidence information, adversarial perturbation can abnormally elevate specific incorrect categories above the ground-truth confidence. Blindly aligning all non-target variations transfers distorted geometry. NTRD defines an element-wise gating mask \(G_{NTC} \in \{0, 1\}^K\), where \(G_{NTC}^{(k)} = \mathbb{I}(p_k(x_{adv}) < p_y(x_{adv})) \cdot \mathbb{I}(k \neq y)\). The structural variation loss is restricted strictly to legitimate non-target classes where the ground truth maintains superiority:

\[\mathcal{L}_{NTRD} = \frac{1}{\sum_{k \neq y} G_{NTC}^{(k)}} \sum_{k \neq y} G_{NTC}^{(k)} \left| (\Delta \mathbf{p}_S)_k - (\Delta \mathbf{p}_T)_k \right|^2\]

By filtering out spurious probability spikes, NTRD preserves faithful structural dark knowledge across benign classes.

Loss & Training¶

The overall training operates within an adversarial min-max optimization framework. The inner loop generates adversarial samples \(x_{adv} = x + \delta\) (\(\|\delta\|_\infty \le \epsilon\)) by maximizing the KL divergence between student and teacher outputs. The outer minimization simultaneously optimizes semantic correctness and geometric sensitivity calibration:

\[\min_{\theta_S} \mathbb{E}_{(x,y)} \left[ \mathcal{L}_{SLG} + \alpha \big( \mathcal{L}_{TRD} + \beta \mathcal{L}_{NTRD} \big) \right]\]

Optimal scaling parameters are established at \(\alpha = 2.0\) and \(\beta = 0.7\). Empirical tracking of gate activation ratios reveals that SCG closes much faster than SLG across training epochs, reflecting that lightweight student networks rapidly achieve superior local smoothness over the overfitted teacher, at which point the gate automatically detaches to insulate the student from late-stage teacher noise.

Key Experimental Results¶

Main Results¶

Evaluated on CIFAR-10 with a WideResNet-34-10 teacher against standard ResNet-18 and MobileNetV2 students under perturbation budget \(\epsilon = 8/255\) (averaged over 4 independent random seeds):

Student	Method	Clean (%)	FGSM (%)	PGD-10 (%)	PGD-20 (%)	CW2 (%)	AutoAttack (AA) (%)	Avg. (%)
ResNet-18	ARD [AAAI 2020]	85.10	61.40	55.29	53.91	79.74	50.10	60.09
ResNet-18	IAD [ICLR 2022]	83.26	61.83	57.73	56.76	79.38	52.20	61.58
ResNet-18	AdaAD [CVPR 2023]	84.16	59.87	56.26	55.43	80.23	51.41	60.64
ResNet-18	DGAD [ECCV 2024]	85.46	61.78	57.65	56.80	81.54	52.59	62.07
ResNet-18	IGDM [ICLR 2025]	82.21	60.51	57.15	56.57	78.91	51.80	60.99
ResNet-18	AEGIS (Ours)	84.69	62.66	58.68	57.89	81.16	53.22	62.72
MobileNetV2	ARD [AAAI 2020]	82.91	58.66	53.94	53.00	78.20	48.41	58.44
MobileNetV2	DGAD [ECCV 2024]	84.07	59.85	55.95	55.11	79.99	50.90	60.36
MobileNetV2	IGDM [ICLR 2025]	80.48	58.72	55.40	54.92	77.00	49.83	59.17
MobileNetV2	AEGIS (Ours)	84.43	60.98	56.74	56.08	81.20	51.34	61.26

Ablation Study¶

Ablation on CIFAR-10 using ResNet-18 student and WideResNet-34-10 teacher, incrementally adding each proposed component onto the AdaAD baseline:

Config	Clean (%)	FGSM (%)	PGD-10 (%)	PGD-20 (%)	CW2 (%)	AutoAttack (AA) (%)	Note
Baseline (AdaAD)	84.16	59.87	56.26	55.43	80.23	51.41	Ungated dynamic adversarial baseline
+ SLG	84.66	61.54	57.74	57.08	80.76	53.11	Clean logit fallback boosts AA by +1.70%
+ TRD	84.82	62.39	58.63	57.66	81.14	53.09	Target sensitivity gating suppresses jagged noise
+ NTRD	84.76	61.54	57.84	57.13	80.88	52.96	Purges misleading non-target inverted classes
+ SLG + TRD	84.57	62.40	58.38	57.55	81.13	53.35	Joint categorical & target geometric calibration
+ SLG + NTRD	84.97	61.60	57.80	57.12	80.96	53.13	Semantic gating combined with dark knowledge
Full Model (AEGIS)	84.69	62.66	58.68	57.89	81.16	53.22	Best overall robustness and highest average

Key Findings¶

Orthogonal contributions from dual gates: Adding SLG alone yields a notable +1.70% improvement under AutoAttack, proving that dynamic fallback to clean teacher logits effectively prevents error propagation; adding TRD improves PGD-20 accuracy by +2.23%, validating that cutting off unstable teacher gradients protects the student's manifold smoothness.
Resilience under severe perturbation: When perturbation strength is pushed to \(\epsilon = 16/255\), competitive baselines like ARD and IGDM collapse below 40% accuracy, whereas AEGIS maintains nearly 50% accuracy, demonstrating superior boundary robustness against large-magnitude shifts.
Superiority on capacity-constrained architectures: For MobileNetV2 with depthwise separable layers, where traditional methods force models to mimic unsupportable geometric complexities, AEGIS achieves 51.34% AutoAttack accuracy, surpassing several baselines on ResNet-18.

Highlights & Insights¶

Theoretical unmasking of stability punishment: The paper mathematically proves that standard distillation forces students to increase their local gradient norms when matching a more volatile teacher, explaining why high-capacity robust models can paradoxically degrade lightweight students.
Clean logits as an effective soft anchor: Rather than discarding failed teacher samples or using one-hot hard labels, falling back to clean logits preserves inter-class relationship priors without transferring adversarial misclassification errors.
Probability variation space for geometric alignment: Calibrating models within \(\Delta \mathbf{p}\) rather than unnormalized logit differences \(\Delta \mathbf{z}\) naturally focuses optimization on uncertain boundary samples due to the vanishing softmax Jacobian on saturated regions.

Limitations & Future Work¶

Author-identified limitations: While the outer minimization selectively filters harmful gradients, the inner maximization still relies on standard adversarial generation, without fully decoupling student-specific versus teacher-specific adversarial vulnerabilities.
Potential extensions: The scaling coefficients \(\alpha\) and \(\beta\) are currently fixed globally; future research could introduce confidence-aware dynamic scheduling. Furthermore, this selective sensitivity gating principle could be extended to diffusion models and multimodal large language model alignment.

vs ARD [Goldblum et al., AAAI 2020]: Vanilla ARD presumes the teacher is an unerring oracle across all inputs; AEGIS identifies stability punishment from teacher robust overfitting and applies selective gating to block noise propagation.
vs DGAD [Park & Min, ECCV 2024]: DGAD addresses label errors via error-corrective label swapping, succeeding on clean accuracy and weak CW attacks, but cannot resolve geometric gradient volatility; AEGIS simultaneously rectifies categorical labels and calibrates local differential probability variations.
vs IGDM [Lee et al., ICLR 2025]: IGDM matches input gradients indiscriminately, failing when student models naturally attain higher local smoothness; AEGIS gates gradient transfer based on differential stability parity.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formulates the counter-intuitive stability punishment phenomenon with rigorous theoretical and empirical backing.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks across CIFAR-10/100 and Tiny-ImageNet under multiple white-box and ensemble attacks, supplemented by extreme perturbation tests.
Writing Quality: ⭐⭐⭐⭐⭐ Clear progression from motivational scatter plots to formal proofs, architecture diagrams, and targeted ablations.
Value: ⭐⭐⭐⭐⭐ Highly practical, plug-and-play framework for deploying genuinely robust compact models on edge devices.