Improving Adversarial Robustness via Activation Amplification and Attenuation

Conference: ECCV 2026
arXiv: 2606.27784
Code: https://github.com/tgoncalv/A3
Area: AI Safety
Keywords: Adversarial Robustness, Activation Scaling, Plug-and-Play Defense, Contrastive Learning, Ranking Loss

TL;DR

This paper proposes A3 (Activation Amplification and Attenuation), a lightweight learnable activation scaling module that implements two modes—activation amplification and attenuation—using the same set of parameters. During training, the degraded predictions from the amplification mode are utilized as negative references to construct contrastive and ranking losses. During inference, only the attenuation mode is employed to enhance adversarial robustness without introducing significant computational overhead.

Background & Motivation

Adversarial attacks mislead the predictions of deep neural networks by adding imperceptible, tiny perturbations to inputs, presenting a core threat to AI safety. Existing defense strategies generally fall into two categories: improving training strategies (such as AT, TRADES, MART) to generate adversarial samples and optimize model parameters under a min-max framework; or inserting lightweight modules into the backbone network to directly modify intermediate feature representations to enhance intrinsic robustness. Most recent plug-and-play modules follow a common paradigm: computing a mask to identify and suppress non-robust features. For example, CAS and CIFS detect channels in the non-robust feature set and suppress them on a per-sample basis, while FPCM begins with the frequency domain to reconfigure high- and low-frequency components. However, recent works like FSR point out that completely discarding non-robust features is not optimal, as these features may still retain useful predictive signals. Consequently, they propose strategies to separate features into robust and non-robust parts and then reorganize the non-robust components.

This contradiction of "suppression vs. preservation" is essentially a balancing issue: one must weaken the influence of noise introduced by adversarial perturbations without over-correcting and losing discriminative information. The authors of A3 noticed an interesting phenomenon in recent out-of-distribution (OOD) detection studies (such as ASH and SCALE)—properly scaling activation values within the energy score function can significantly widen the score gap between in-distribution (ID) and OOD samples. Since adversarial samples are essentially a type of out-of-distribution data, can this idea be transferred to adversarial robustness? Unlike OOD detection, which only uses scaling to improve score separation during inference, adversarial training requires backpropagation-compatible end-to-end optimization. Furthermore, one can take it a step further: if scaling can separate "useful" and "useless" signals, can the model be trained not only to learn to attenuate useless signals but also to actively amplify useless signals to serve as "negative examples"?

Core Idea: Design a learnable activation scaling module, A3, where the same set of learnable parameters can switch between "amplification" and "attenuation" modes simply by reversing the sign of the scaling operation. During training, the attenuation branch (used for normal predictions) and the amplification branch (used as a negative reference) are output in parallel. A ranking loss ensures that the prediction loss of the attenuation branch is always lower than that of the amplification branch, while a contrastive logit loss pulls the logits of the attenuation branch on clean and adversarial samples closer and pushes them away from those of the amplification branch. Consequently, during inference, only the attenuation branch is used to obtain more robust representations.

Method

Overall Architecture

A3 is a plug-and-play module that can be inserted after a certain intermediate layer of the backbone network (typically after the final block). Its core processing workflow is as follows: given the activation map \(z\) of the current layer, channel-level features are obtained first through global average pooling and a learnable linear projection, and then a channel-level binary mask \(m\) is obtained via Gumbel-Softmax approximation sampling. Using this mask and the original activation amplitude distribution, two scaling factors \(f_m\) and \(f_{1m}\) are calculated and combined into the final scaling function \(Scale(z, m) \in [0, 1]\). By a simple sign flip, the attenuated version \(z_{att} = z \cdot [1 - Scale(z, m)]\) and the amplified version \(z_{amp} = z \cdot [1 + Scale(z, m)]\) are simultaneously obtained. During training, both branches run in parallel: the output of the attenuation branch is used for the main loss, while the output of the amplification branch serves as a negative reference for the ranking and contrastive losses. During inference, only the attenuation branch is retained.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Activation Map z<br/>(C×H×W)"] --> B["Global Average Pooling +<br/>Linear Projection Wm"]
    B --> C["Gumbel-Softmax<br/>→ Binary Mask m"]
    C --> D["Compute s1 / s2<br/>and Log Scaling Factors"]
    D --> E["Scaling Function<br/>Scale = fm·m + f1m·(1-m)"]
    E --> F["Attenuation Branch<br/>z_att = z·(1-Scale)"]
    E --> G["Amplification Branch<br/>z_amp = z·(1+Scale)"]
    F --> H["Prediction → Main Loss L_main"]
    G --> I["Prediction → Negative Reference"]
    H --> J["Inference: Only Attenuation Branch Used"]
    I --> J

Key Designs

1. Differentiable Channel-Level Mask Generation

A3 needs to identify which channels carry activation patterns that require differentiated treatment. It first applies global average pooling (GAP) to the activation map of the current layer to compress the spatial dimensions into a channel-level description vector, which is then linearly projected into the same dimension via a learnable weight matrix \(W_m \in \mathbb{R}^{C \times C}\). Finally, an approximate binary mask \(m\) is obtained via Gumbel-Softmax. By using the reparameterization trick, Gumbel-Softmax smooths discrete sampling into a continuously differentiable operation, avoiding the gradient obfuscation issues caused by traditional top-k hard thresholding—which was the fundamental reason why its predecessor, k-WTA, was broken by adaptive attacks. Ablation experiments show that after replacing Gumbel-Softmax with a hard threshold, A3 still achieves robustness gains (indicating that the scaling mechanism itself is effective), but the differentiable version achieves about 4 percentage points higher in AA accuracy.

2. Logarithmic Scaling Factor Based on Activation Amplitude

After the mask \(m\) is determined, a specific scaling amount needs to be calculated. A3 computes two scalars: \(s_1\) = the sum of the absolute values of all channel activations (measuring overall activation intensity), and \(s_2\) = the sum of the absolute values of the activations of the channels selected by the mask (measuring the activation intensity of the "channels to be processed"). Defining \(f_m = \ln(1+s_2)/\ln(1+s_1)\) and \(f_{1m} = 1 - f_m\), the final scaling function is \(Scale = f_m \cdot m + f_{1m} \cdot (1 - m)\), which means the channels selected by the mask are multiplied by \(f_m\) and the unselected ones are multiplied by \(f_{1m}\). Compared to the exponential formula used in OOD detection, this logarithmic formula offers two key advantages: first, its derivative smoothly decays as the input increases, which effectively mitigates gradient spike issues caused by perturbations in adversarial training; second, it is naturally bounded (\(s_2 \le s_1\) guarantees \(f_m \in [0, 1]\)), ensuring that the scaling operation does not excessively distort the original activations. Ablation experiments comparing linear, quadratic, exponential, and logarithmic variants show that the logarithmic version outperforms the linear one by approximately 7.5 percentage points on PGD-100.

3. Dual-Mode Mechanism via Sign Flipping and Contrastive Ranking Loss

The most ingenious design of A3 is that the same set of parameters simultaneously supports two modes: \(z_{att} = z \cdot (1 - Scale)\) and \(z_{amp} = z \cdot (1 + Scale)\). Since \(Scale\) is always between \([0, 1]\), the activation values are compressed to the \([0, z]\) range in the attenuation mode, and expanded to the \([z, 2z]\) range in the amplification mode. Grad-CAM visualizations clearly show that the attenuation branch suppresses responses in irrelevant regions (such as the background) in the activation map and focuses on class-related regions, while the amplification branch conversely reinforces irrelevant regions. During inference, only the attenuation branch is used, but the amplification branch plays an indispensable role as a "negative teacher"—it is precisely its degraded predictions that provide a clear optimization target for the ranking loss. Specifically, A3 defines two auxiliary losses: a ranking loss \(L_{rank} = \max(\text{CE}(p_{att}^{adv}, y) - \text{CE}(p_{amp}, y), 0)\) in a hinge form to ensure that the cross-entropy loss of the attenuation branch is strictly lower than that of the amplification branch; and a contrastive logit loss \(L_{cl} = -\log\left(\frac{\exp(\text{sim}(l_{att}^{adv}, l_{att})/\tau)}{\exp(\text{sim}(l_{att}^{adv}, l_{att})/\tau) + \exp(\text{sim}(l_{att}^{adv}, l_{amp})/\tau)}\right)\) to pull the logits of the attenuation branch on clean and adversarial samples closer while pushing them away from those of the amplification branch. Ablation experiments show that \(L_{rank}\) contributes more (removing it drops AA from \(47.28\%\) to \(45.41\%\)), and the hinge formulation outperforms directly using the difference of cross-entropy values.

Loss & Training

The total loss of A3 is \(L = L_{main} + \lambda_{rank} \cdot L_{rank} + \lambda_{cl} \cdot L_{cl}\), where \(L_{main}\) is the main loss of the chosen adversarial training method (which can be flexibly replaced with AT, TRADES, or MART). The default weights are \(\lambda_{rank}=1\) and \(\lambda_{cl}=5\). Training utilizes PGD-10 to generate adversarial samples (perturbation bound \(\varepsilon=8/255\), step size \(\varepsilon/4\)), handled by an SGD optimizer (momentum 0.9, weight decay 5e−4) with an initial learning rate of 0.1, decayed by 10 times at the 75th and 90th epochs, for a total of 100 epochs. Hyperparameters are \(\tau_m = 0.1\) (Gumbel-Softmax temperature) and \(\tau_{cl} = 10\) (contrastive loss temperature). A3 modules are inserted after block 4 in ResNet-18 and after block 3 in WideResNet-34-10.

Key Experimental Results

Main Results

Taking the AT training of ResNet-18 on CIFAR-10/100 as an example:

Dataset	Attack	Baseline (AT)	+A3	Gain
CIFAR-10	PGD-100	47.51	57.01	+9.50
CIFAR-10	C&W	48.19	52.33	+4.14
CIFAR-10	Ensemble	46.05	51.04	+4.99
CIFAR-10	AutoAttack	44.28	47.28	+3.00
CIFAR-100	PGD-100	24.16	31.78	+7.62
CIFAR-100	Ensemble	22.85	26.52	+3.67

The gain is even more significant on WideResNet-34-10: under AT training, CIFAR-10 AutoAttack increases from \(48.18\%\) to \(51.62\%\) (\(+3.44\%\)), and CIFAR-100 increases from \(23.70\%\) to \(27.68\%\) (\(+3.98\%\)). Consistent gains are also observed on Tiny ImageNet. A3 is compatible with multiple adversarial training strategies: it achieves continuous improvements under both TRADES and MART frameworks, demonstrating its independence from specific training paradigms.

Ablation Study

Configuration	Ensemble	AA	Description
A3 (Full)	51.04	47.28	Full Model
w/o \(L_{cl}\)	50.24	46.70	Without Contrastive Loss
w/o \(L_{rank}\)	48.11	45.41	Significant drop after removing ranking loss
Pure attenuation, no amplification training	48.73	46.72	Removing dual mode, serving only as a regular scaling module
Hinge \(\rightarrow\) CE Difference (Eq.13a)	48.74	46.84	Partial degradation after removing hinge
Logarithmic \(\rightarrow\) Linear Scaling	48.58	46.85	Swapped with linear scaling formula

Key Findings

• Ranking loss contributes the most: removing \(L_{rank}\) drops Ensemble from \(51.04\%\) to \(48.11\%\), showing that the hinge ranking mechanism using the amplification branch as a negative reference is the core driver of robustness enhancement.
• The logarithmic scaling formula performs best among six variants, verifying the dual advantages of the logarithmic form in gradient stability and boundedness.
• A3 is extremely lightweight: it adds only 0.27M parameters (a relative increase of \(2.4\%\)) and 0.0021G FLOPs to ResNet-18, which is far lower than FSR (+1.26M) and FTA2C (+10.71M); it adds 0.41M parameters to WideResNet-34-10.
• The attenuation mode also reduces the model's overconfidence in adversarial samples: the Expected Calibration Error (ECE) decreases from \(14.62\%\) (baseline) to \(11.12\%\) (A3 attenuation mode), while the ECE of the amplification mode rises to \(26.90\%\).
• The module is position-sensitive: inserting it in shallow layers (blocks 1-3) of ResNet-18 yields limited robustness improvements, while block 4 is optimal; inserting it simultaneously in blocks 3+4 actually causes a significant drop in accuracy (AA falls to \(36.87\%\)).

Highlights & Insights

• The most ingenious design of this work is "using degradation to promote upgrading": by training the model to learn both how to make predictions worse (amplification mode) and how to make them better (attenuation mode), the gap between the two is utilized as an optimization signal. This is more flexible than simply suppressing non-robust features, because the model must truly understand which signals are helpful or harmful to find the boundary between high and low-quality representations in high-dimensional space.
• The sign-flipping mechanism is extremely elegant. The same set of parameters can switch between amplification and attenuation by changing a single sign, acting as mutual negative/positive references during training without requiring extra parameters.
• It transfers the activation scaling concept from OOD detection to the field of adversarial training. By replacing the non-differentiable top-k operation with Gumbel-Softmax and the exponential scaling with logarithmic scaling, it successfully addresses two key adaptation challenges: gradient obfuscation and numerical instability.

Limitations & Future Work

• The accuracy of A3 on clean images drops slightly compared to the baseline (about 0.5-1%). The authors suggest this is because clean images lack harmful activations that require attenuation, thus gaining no benefit from the attenuation operation.
• The insertion position of the module is highly sensitive: inserting it at shallow layers severely interferes with low-level feature learning (AA is only \(42.55\%\) after WideResNet block 1 vs. \(51.62\%\) after block 3), requiring careful selection of the insertion point in practice.
• The experiments were only conducted on CIFAR-10/100 and Tiny ImageNet (64×64), failing to cover larger-scale datasets such as ImageNet-1K. The computational overhead and effectiveness in larger input sizes and high-resolution scenarios remain unclear.
• The amplification mode requires additional forward computation during training. Although the parameter size is minimal, its impact on training throughput for larger models deserves further evaluation.

Related Work & Insights

• vs FSR (CVPR 2023): FSR explicitly separates features into robust and non-robust components and reorganizes the latter via an MLP. A3 does not perform explicit separation but instead adjusts activation values in a continuous space through a learnable scaling factor, offering a simpler design with fewer parameters.
• vs ASH / SCALE (OOD Detection): The core scaling formula of A3 is inspired by the activation scaling concept in OOD detection, but replaces non-differentiable top-k with Gumbel-Softmax and exponential scaling with logarithmic scaling, resolving gradient obfuscation and numerical overflow issues in adversarial training environments.
• vs k-WTA: k-WTA also uses a top-k operation to suppress non-robust features, but was broken by adaptive attacks due to gradient obfuscation caused by non-differentiability. A3 is fully differentiable and passes three gradient obfuscation verifications under the AutoAttack framework.

Rating

• Novelty: ⭐⭐⭐⭐ Systematically transfers the concept of OOD activation scaling to the domain of adversarial robustness, and designs a dual-mode scaling + contrastive ranking loss framework, which is highly ingenious and well-conceived.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensively evaluated across two backbones, three datasets, and three adversarial training strategies; ablation studies cover five core design dimensions including loss components, scaling formulas, module positions, mask types, and scaling strengths.
• Writing Quality: ⭐⭐⭐⭐⭐ Natural motivation, step-by-step method derivation, and mutual validation between visualizations (Grad-CAM and activation distribution histograms) and textual explanations construct a comprehensive logical chain.
• Value: ⭐⭐⭐⭐ Introduces consistent robustness improvements with extremely low parameter and computational overheads, making it suitable to be integrated as a plug-and-play module into existing adversarial training pipelines.

Related Papers

• [NeurIPS 2025] Understanding and Improving Adversarial Robustness of Neural Probabilistic Circuits
• [CVPR 2026] Mitigating Error Amplification in Fast Adversarial Training
• [ICLR 2026] On the Interaction of Compressibility and Adversarial Robustness
• [CVPR 2026] Improving Adversarial Transferability with Local Perturbation Augmentation
• [ICLR 2026] When Flatness Does (Not) Guarantee Adversarial Robustness