Debiased Dual-Invariant Defense for Adversarially Robust Person Re-Identification

Conference: AAAI 2026
arXiv: 2511.09933
Code: Available
Area: Autonomous Driving / Person Re-Identification
Keywords: Adversarial Defense, Person Re-Identification, Meta-Learning, Data Balancing, Metric Learning

TL;DR

This work systematically identifies two unique challenges in person Re-ID adversarial defense (model bias and composite generalization needs) and proposes a Debiased Dual-Invariant Defense framework: resampling with a diffusion model in the data balancing phase alleviates bias, while dual generalization to unseen IDs and unseen attacks is achieved in the dual adversarial self-meta defense phase using metric adversarial training with farthest negative expansion softening and adversarially-enhanced self-meta learning.

Background & Motivation

Person Re-ID is a core capability in security surveillance, but deep learning-based Re-ID models are highly vulnerable to adversarial attacks. Existing defense methods are mostly designed for classification tasks, facing two unique challenges when transferred to the metric learning task of Re-ID:

Challenge 1: Model Bias - Inter-class imbalance: The number of samples per ID in Re-ID datasets varies dramatically (depending on how frequently they appear under the cameras), as clearly illustrated by the statistics from Market-1501 and DukeMTMC. - Insufficient intra-class diversity: Samples are typically extracted from video sequences, leading to highly redundant images for the same ID with limited visual diversity. - Bias effect: After adversarial training, the per-ID precision variance of the model increases from 18.78 to 23.40, exacerbating the bias.

Challenge 2: Composite Generalization - Robustness distributed on the classifier: Experimental verification shows that after adversarial training, fine-tuning only the classifier keeps the clean accuracy almost unchanged while robustness drops significantly (ResNet50 under PGD drops from 53.08 to 50.22), indicating that part of the robustness knowledge is allocated to the classifier \(H\), which is discarded during testing. - Two-dimensional generalization: Re-ID is an open-set task (unseen IDs appear during testing) while the types of attacks are too numerous to enumerate. Therefore, generalization to both unseen IDs and unseen attack types is simultaneously required.

Method

Overall Architecture

The method consists of two phases: Data Balancing Phase \(\rightarrow\) Dual Adversarial Self-Meta Defense Phase.

The model contains a feature encoder \(E\) (parameterized by \(\theta_E\)) and a classifier \(H\) (parameterized by \(\theta_H\)). During training, \(G = H(E(\cdot))\) is jointly optimized, while during testing, only \(E\) is utilized to extract features for retrieval.

Key Designs

(1) Diffusion-based Data Balancing

Addressing inter-class imbalance: For IDs with sample counts below a threshold \(\delta_1\), a conditional diffusion model is used to synthesize pseudo-samples for supplementation:

\[\mathcal{D} \leftarrow \mathcal{D} \cup \{x_i^{\text{pseudo},j} \mid i \in \mathcal{I}, n_i < \delta_1, j=1,\dots,\delta_1 - n_i\}\]

Addressing insufficient intra-class diversity: For IDs where a single camera accounts for more than \(\delta_2\) of the samples, complementary samples are synthesized from other camera perspectives:

\[\mathcal{D}_{i'} \leftarrow \mathcal{D}_{i'} \cup \{x_{i',c}^{\text{pseudo}} \mid c \in \mathcal{C} \setminus \{c_{i'}\}\}\]

This is implemented using the EDM framework, training the diffusion model under ID conditioning.

(2) Farthest Negative Expansion Softening (FNES) Metric Adversarial Training

Issues with existing metric PGD attacks: Fixed iteration directions lead to high similarity among adversarial samples, lacking diversity. FNES improves this through two steps:

Linear Scaling Perturbation: Linearly scaling the final adversarial perturbation to increase diversity:

\[x^{\text{temp}} = x + \gamma \cdot (\hat{x} - x), \quad x^{\text{adv}} = \omega x + (1-\omega) x^{\text{temp}}\]

where \(\gamma \geq 1\) is the scaling factor, and \(\omega \sim \mathcal{U}(a,b)\) is the mixup weight.

Farthest Negative Class Label Softening: Redistributing a portion of the label probability from the ground-truth class to the farthest negative class targeted by the metric attack:

\[y^{\text{adv}} = \omega \phi(y, \lambda_1) + (1-\omega) \tau(\phi(y, \lambda_2), \upsilon)\]

where \(\phi\) is the label smoothing function, and \(\tau\) transfers probability \(\upsilon\) from the ground-truth class to the farthest negative class. This allows the model to learn robustness knowledge regarding the farthest negative class and alleviates hard label overfitting.

(3) Adversarially-enhanced Learning

Introducing a feature discriminator \(D\) alongside the encoder \(E\) to form an adversarial learning framework, aiming to learn adversarial-invariant features (shared representations of clean and adversarial samples):

\[\min_E \max_D \mathcal{L}(E,D) = \mathbb{E}_x[\log D(E(x))] + \mathbb{E}_{x^{\text{adv}}}[\log(1-D(E(x^{\text{adv}})))]\]

When Nash equilibrium is reached, \(D\) cannot distinguish the source of the features \(\rightarrow\) the encoder extracts adversarial-invariant features.

(4) Self-Meta Learning

To learn generalization-invariant features shared between seen and unseen IDs: - Divide each batch of data into \(\mathcal{D}_{\text{meta-train}}\) and \(\mathcal{D}_{\text{meta-test}}\). - The model first undergoes one-step gradient descent on \(\mathcal{D}_{\text{meta-train}}\) to obtain a temporary model \(G_{\text{temp}}\): $\(\theta_G^{\text{temp}} = \theta_G - \alpha \nabla_{\theta_G} \mathcal{L}_{\text{meta-train}}\)$ - Then evaluate \(G_{\text{temp}}\) on \(\mathcal{D}_{\text{meta-test}}\). - The final loss is formulated as \(\mathcal{L}_{\text{self-meta}} = \mathcal{L}_{\text{meta-train}} + \mathcal{L}_{\text{meta-test}}\), performing gradient descent directly on \(\theta_G\).

The loss of each phase contains \(\ell = \mathcal{L}_{\text{cls}} + \mathcal{L}_{\text{tri}} + \mathcal{L}_E\) for both clean and adversarial samples.

Loss & Training

Overall optimization framework: - Inner loop (attack): Use metric PGD to maximize \(\mathcal{L}_{\text{metric}}\). - Outer loop (defense): Minimize classification loss + triplet loss, while performing adversarial training of the encoder-discriminator + self-meta learning.

Key Experimental Results

Main Results

Table 2: ResNet50 White-box Robustness (mAP/Rank-1)

Method	Market Clean	FNA 8/255	SMA 8/255	IFGSM 8/255
Origin	78.49/92.01	0.20/0.17	0.27/0.26	1.25/1.95
Adv_train	69.69/88.24	8.57/18.14	22.85/35.69	17.97/34.65
DAS	69.79/88.39	12.70/24.85	32.14/49.05	22.33/39.79
Ours	68.50/88.21	31.99/55.17	50.13/72.60	37.61/62.02

On Market under FNA attack, mAP increases from 12.70 \(\rightarrow\) 31.99, and under SMA attack from 32.14 \(\rightarrow\) 50.13, significantly leading the SOTA defense DAS.

Table 5: Cross-dataset Generalization (Market \(\rightarrow\) Duke, ResNet50)

Method	Clean	FNA 8/255	SMA 8/255	IFGSM 8/255
None	15.08/27.65	0.15/0.13	0.35/0.36	0.29/0.36
Metric AT	16.51/29.35	4.47/10.89	10.77/22.52	5.78/13.14
Ours	19.07/34.69	6.17/12.88	13.02/24.60	7.92/15.35

Ablation Study

Table 3: Ablation of Stepwise Module Addition (ResNet50, Market)

Configuration	Clean	FNA 8/255	SMA 8/255	IFGSM 8/255
Metric AT (baseline)	67.20/88.00	28.38/52.26	45.38/68.74	33.97/58.52
+Diffusion model	66.96/86.91	29.85/53.36	45.82/68.41	35.32/59.68
+Adversarial learning	67.81/88.21	30.34/53.77	48.10/70.72	35.70/60.27
+Self-meta learning	68.24/88.03	29.48/52.88	46.91/69.30	35.09/59.86
+FNES	68.29/88.07	30.98/54.45	49.25/71.11	37.16/61.49
All modules	68.50/88.21	31.99/55.17	50.13/72.60	37.61/62.02

Each module exhibits a positive contribution: diffusion model balances data \(\rightarrow\) adversarial learning extracts invariant features \(\rightarrow\) self-meta learning enhances generalization \(\rightarrow\) FNES boosts the diversity of adversarial training.

Key Findings

1. Validated the hypothesis that "robustness is distributed on the classifier" (Table 4): This method progressively enhances the robustness from AT_PGD \(\rightarrow\) Metric AT \(\rightarrow\) Ours, effectively mitigating the robustness loss caused by discarding the classifier during test time.
2. Effective cross-dataset generalization: Market training \(\rightarrow\) Duke testing shows a 32-40% improvement compared to Metric AT.
3. Grad-CAM visualization displays that the proposed method focuses on more reasonable body feature regions.
4. UMAP feature distribution demonstrates that this method maintains better inter-class separability under adversarial attacks.

Highlights & Insights

1. Deep problem insight: For the first time, two unique challenges of ReID adversarial defense are systematically identified, especially the convincing experimental validation of "classifier consuming robustness".
2. Ingenious FNES design: Linear scaling breaks the fixed iteration direction of metric PGD + farthest negative class label softening resolves both diversity and overfitting issues simultaneously.
3. Dual-invariant feature learning: Adversarial-invariant (clean \(\leftrightarrow\) adversarial) and generalization-invariant (seen \(\leftrightarrow\) unseen ID) representations complement each other.

Limitations & Future Work

1. Clean accuracy drops after defense (78.49 \(\rightarrow\) 68.50); the robustness-accuracy trade-off still exists.
2. The generation quality of the diffusion model affects the final performance, but the paper does not discuss quality control of generated samples in depth.
3. The sensitivity of the data division strategy (meta-train vs meta-test) in self-meta learning to the results is not fully analyzed.
4. Validated only on ResNet18/50; the applicability to Vision Transformer architectures remains unknown.

Related Work & Insights

• ReID Attacks: FNA (Bai 2020), SMA (Bouniot 2020) as white-box metric attacks, and black-box attacks (Liu 2023, Zhang 2020).
• ReID Defenses: Offline adversarial training (Bai 2020), virtual data augmentation (Bian 2025), and dynamic attack budget (Wei 2024).
• General Adversarial Training: PGD-AT (Madry 2018), TRADES (Zhang 2019).
• Insights: The concept of FNES can be generalized to the adversarial training of other metric learning tasks.

Rating

Dimension	Score	Description
Novelty	★★★★☆	Unique problem insights, innovative FNES design
Technical Depth	★★★★☆	Multi-layer design of diffusion model + adversarial learning + meta-learning
Experimental Thoroughness	★★★★★	Comprehensive white-box/black-box/cross-dataset/ablation/visualization evaluations
Writing Quality	★★★★☆	Clear problem analysis, substantial experimental evidence
Value	★★★★☆	Code is open-sourced, offering practical reference value for security ReID system defense

Related Papers

• [AAAI 2026] Hierarchical Prompt Learning for Image- and Text-Based Person Re-Identification
• [AAAI 2026] When Person Re-Identification Meets Event Camera: A Benchmark Dataset and An Attribute-guided Re-Identification Framework
• [CVPR 2026] FedBPrompt: Federated Domain Generalization Person Re-Identification via Body Distribution Aware Visual Prompts
• [NeurIPS 2025] GSAlign: Geometric and Semantic Alignment Network for Aerial-Ground Person Re-Identification
• [AAAI 2026] Minimum-Cost Network Flow with Dual Predictions