CMCC-ReID: Cross-Modality Clothing-Change Person Re-Identification¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/Xuan266/CMCC-ReID
Area: Human Understanding
Keywords: person re-identification, visible-infrared matching, clothing change, feature disentanglement, prototype learning

TL;DR¶

The paper introduces SYSU-CMCC, a benchmark combining modality discrepancy and clothing change, and a Progressive Identity Alignment Network (PIA) that disentangles identity from clothing before bidirectional prototype alignment, achieving 57.0% Rank-1 and 46.5% mAP for visible-to-infrared retrieval.

Background & Motivation¶

Person re-identification must associate the same person across cameras, but long-term scenarios contain more than one source of appearance change. Daytime visible images and nighttime infrared images differ substantially, while the same person may also change clothes between observations. Visible-infrared re-identification typically assumes unchanged clothing, whereas clothing-change re-identification primarily learns identity invariance under visible imaging. Both settings have advanced, yet neither fully answers what should be aligned when the two changes occur together. If a cross-modality model still relies on clothing, bringing its modality distributions closer may simply reinforce incorrect appearance associations.

Infrared imaging also makes clothing suppression itself harder: color and texture cues diminish, reducing contrast between clothing and other body regions. The authors use CAL attention maps to show that a model suppressing clothing responses in visible images can remain dominated by clothing regions in infrared images. Another observation in Figure 2 is that directly applying SAAI underperforms its ResNet-50 baseline, whereas adding its alignment module to CAL produces some benefit. These results support a specific interpretation: clothing interference is not residual noise to remove after modality alignment, but a condition determining whether alignment works. PIA therefore does not simply optimize existing clothing-change and modality-alignment losses together from the beginning; it first develops more reliable identity representations.

To test this argument, the paper builds SYSU-CMCC from PRCC and SYSU-MM01 rather than concatenating unrelated identities from the two datasets. Identical samples in PRCC camA/B and SYSU-MM01 cam1/2 provide cross-dataset identity anchors, followed by manual checks of PRCC camC and SYSU-MM01 cam3/6. Retained visible and infrared samples share identity but differ in clothing; the resulting benchmark contains 214 identities, 18,375 images, and 3 cameras. In the new benchmark, Cam1/2 provide infrared images with clothing A and Cam3 provides visible images with clothing B; these new camera indices must not be confused with source-dataset indices. Core Idea: first reduce the identity branch's dependence on clothing regions, then align identity representations using intra-modality and inter-modality prototypes, avoiding the propagation of clothing bias as identity knowledge.

Method¶

Overall Architecture¶

PIA processes visible or infrared person images with a shared convolutional backbone that produces spatial feature maps. Dual-Branch Disentanglement Learning (DBDL) converts these maps into identity and clothing vectors, with the identity vector serving as the retrieval representation. Bi-Directional Prototype Learning (BPL) is introduced only in the second stage, maintaining visible and infrared prototypes for each training identity. Its intra-modality constraints compact same-identity samples, while inter-modality constraints pull a sample toward the same identity's prototype in the other modality. The two stages differ in training objectives: the second stage neither discards DBDL nor completely freezes the first stage. Dashed arrows below denote training supervision; prototypes and classification labels are not required inputs attached to test queries.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Visible or infrared image"] --> Backbone["Shared ResNet-50"]
    Backbone --> DBDL["Dual-Branch Disentanglement Learning<br/>DBDL"]
    Labels["Training: identity and clothing labels"] -.-> DBDL
    DBDL --> Feature["Identity vector"]
    Feature -->|"Stage II training"| BPL["Bi-Directional Prototype Learning<br/>BPL"]
    BPL -.->|"Intra- and inter-modality supervision"| DBDL
    Feature -->|"Testing"| Retrieval["Cross-modality gallery ranking"]

Key Designs¶

1. Dual-Branch Disentanglement Learning: suppress clothing without discarding entire clothing regions

DBDL first learns spatial clothing attention from the shared feature map rather than invoking an external clothing segmentation model. It performs channel-wise maximum and average pooling, concatenates the two spatial statistics, and applies convolution and a sigmoid to obtain clothing weights. Maximum pooling retains strong responses, while average pooling captures broader response patterns; together they guide spatial selection in the clothing branch. Element-wise multiplication of clothing weights and backbone features, followed by global pooling, produces a clothing vector supervised by clothing classification labels. This explicit branch gives the network a recognizable and constrained clothing representation instead of merely expecting an identity classifier to ignore clothes automatically.

The identity branch uses the same clothing weights for smooth suppression, with its strength controlled by a learnable coefficient. The authors avoid hard exclusion that zeroes all high-clothing-response regions, since those locations can also contain useful body-structure information. For example, clothing-covered regions may still convey body shape, so deleting them can remove identity cues as well. Pooling the identity-attention-weighted features produces an identity vector supervised by identity classification. The two branches have distinct discriminative responsibilities, differing in spatial selection and classification objectives rather than their input images. Operators are missing from the extracted text of Equation (2), so this note describes smooth suppression without presenting a guessed expression as the authors' exact formula.

Separate classification heads alone do not prevent the identity vector from retaining clothing information, motivating an additional orthogonality constraint. The surrounding prose describes this as an average penalty on the absolute normalized dot product, reducing directional correlation between identity and clothing features. Its purpose is not to make the classifiers compete, but to reduce the opportunity for both representation spaces to share clothing cues. Orthogonality and spatial suppression are complementary: the former constrains vector relationships, while the latter controls where image information enters the identity branch. However, low correlation does not establish statistical independence or prove that all clothing information has been removed from the identity vector. This is a qualification of the regularizer's capabilities; the authors' use of disentanglement should not be read as a formal causal-identification guarantee.

2. Bi-Directional Prototype Learning: stabilize within-modality identity clusters and cross-modality correspondence

Starting from first-stage identity features, BPL stores two centers per training identity, representing visible and infrared observations respectively. Separate centers distinguish insufficient within-modality compactness from inconsistent cross-modality identity representations. Prematurely mixing all modalities into one center could combine clothing-induced and imaging-induced offsets in a single target. Prototypes are initialized from mean same-identity, same-modality features and updated using momentum and the current batch's identity-specific means. The default momentum is 0.9, preventing prototypes from following every batch-level fluctuation in pose and background. The initialization prose refers to all same-identity samples, but the normalization quantity near Equation (7) is described as a batch-level sample count; implementation should be checked rather than silently resolving this ambiguity.

Intra-modality learning pulls a visible sample toward its visible identity prototype and away from other visible identity prototypes; the infrared branch operates analogously. This reduces same-identity dispersion caused by pose, illumination, and other appearance factors, producing more stable identity clusters. Inter-modality learning exchanges the target prototype sets: visible samples contrast against infrared prototypes, and infrared samples against visible prototypes. The positive prototype is always selected by identity labels, not clothing similarity or unsupervised nearest neighbors. The following unifies the recognizable structure of Equations (10) through (13); it is a notational summary, not a verbatim reconstruction of corrupted equations:

\[ \ell_{m\to n}(\mathbf f,y)=-\log\frac{\exp(\mathbf f^{\mathsf T}\mathbf p_y^n/\tau)}{\sum_{k=1}^{K}\exp(\mathbf f^{\mathsf T}\mathbf p_k^n/\tau)},\qquad m,n\in\{V,I\}. \]

Here \(\mathbf f\) comes from modality \(m\), \(y\) is its identity label, \(K\) is the number of training identities, and \(\mathbf p_k^n\) is an identity prototype in modality \(n\). The case \(m=n\) gives intra-modality contrast, while \(m\ne n\) gives inter-modality contrast; each of the four directions is averaged over its corresponding samples. The temperature \(\tau=1/16\) controls the sharpness of similarity distributions, and the denominator includes every training identity prototype in the target modality. Thus, BPL does not merely make modalities closer: it brings matching identities together across modalities while keeping different identities distinguishable. It operates on DBDL identity vectors rather than pulling clothing vectors toward identity centers in the other modality. This choice, together with delayed activation, reduces the risk that clothing cues become persistent and reinforced prototype memories.

Loss & Training¶

Stage I jointly trains the shared backbone and DBDL using identity classification, clothing classification, and orthogonality regularization, without BPL. Stage II retains these objectives and adds visible and infrared intra-modality contrast plus bidirectional inter-modality prototype contrast. Equations (14) and (15) are incompletely extracted; in particular, the recognizable weight placement in Equation (15) conflicts with the following description of \(\lambda_2\) as weighting intra-modality terms. This note therefore omits an unverified full objective and records only the explicitly reported values \(\lambda_1=0.5\) and \(\lambda_2=1.5\). Training lasts 90 epochs, with the second-stage objective activated after epoch 55; this does not involve independently training two networks and then connecting them. The backbone is ImageNet-pretrained ResNet-50 with its final spatial downsampling removed to retain finer spatial responses. The implementation concatenates global average and maximum pooling outputs and applies Batch Normalization to the final features. Inputs have resolution \(384\times192\), with random horizontal flipping, random cropping, and random erasing. Adam starts at a learning rate of \(3.5\times10^{-4}\), reduced by a factor of 10 every 30 epochs; experiments use one NVIDIA A100. Each batch includes 8 identities with 4 visible and 4 infrared images per identity, totaling 64 images and ensuring paired identity supervision across modalities. At test time, identity features are extracted for queries and gallery images for retrieval; Figure 6 analyzes cosine distances between test features. The main text does not specify the complete testing implementation or training/test identity counts, so no additional reranking, gallery-sampling, or deployment-latency assumptions are made.

Key Experimental Results¶

Main Results¶

The following selection comes from Table 2 on page 12; every method is evaluated on SYSU-CMCC rather than its original dataset. V2I uses visible queries against an infrared gallery, and I2V uses infrared queries against a visible gallery; both directions include clothing change. Rank-1 is the fraction of queries whose first retrieved identity is correct, and mAP averages per-query average precision; all table values are percentages.

Method	Original task	V2I Rank-1	V2I mAP	I2V Rank-1	I2V mAP
SAAI	VI-ReID	20.4	21.2	19.1	19.2
DEEN	VI-ReID	28.3	24.8	24.7	28.5
CAL	CC-ReID	40.1	33.2	35.6	37.1
CSCI	CC-ReID	42.9	31.9	37.6	36.9
CaAug	CC-ReID	42.6	36.6	38.2	38.2
PIA	CMCC-ReID	57.0	46.5	50.4	50.3

Relative to CaAug, PIA improves V2I mAP by 9.9 percentage points and I2V mAP by 12.1 percentage points. The strongest prior method for V2I Rank-1 is CSCI, not CaAug; PIA improves over CSCI by 14.1 percentage points. These comparisons support an advantage on this joint-variation benchmark, not a claim that PIA leads on every conventional VI-ReID or CC-ReID dataset.

Ablation Study¶

The following reproduces the configurations in Table 3 on page 13, using a reproduced CAL baseline and percentage scores on SYSU-CMCC. Progressive training means delayed activation of prototype learning; row 6 explicitly removes this strategy and must not be treated as a progressive-training result.

Table 3 index	Config	V2I Rank-1	V2I mAP	I2V Rank-1	I2V mAP
1	CAL baseline	40.1	33.2	35.6	37.1
2	Baseline + DBDL, without orthogonality	43.2	35.7	38.6	38.7
3	DBDL + orthogonality	45.6	37.6	40.5	39.1
4	Config 3 + progressive + intra-modality BPL	46.9	39.5	42.7	41.6
5	Full PIA, adding inter-modality BPL	57.0	46.5	50.4	50.3
6	Full components without progressive training	42.8	35.4	42.6	40.7

Key Findings¶

Given disentangled features and progressive training, adding inter-modality contrast increases V2I Rank-1 from 46.9 to 57.0, a gain of 10.1 percentage points; this is not an independent module gain under arbitrary training conditions.
Keeping all components but removing progressive training lowers V2I Rank-1 from 57.0 to 42.8, a loss of 14.2 percentage points, showing that optimization order is itself a critical variable.
Non-progressive row 6 is worse than row 3 on V2I but slightly better on I2V; prototype constraints are not entirely ineffective on every metric, but fail to deliver consistent expected gains.
Figure 5a identifies 55 as the best second-stage starting point among tested settings; nearby regularization weights are relatively stable, whereas excessively small or large momentum harms performance.

Highlights & Insights¶

The most useful diagnostic asks whether the representation being aligned truly captures identity. It reframes alignment failure as contamination by confounding factors rather than simply insufficient capacity.
The clothing branch is not the final retrieval output, but supplies a learnable suppression target for the identity branch. Soft suppression also preserves potentially useful body-shape cues in clothing-covered regions.
Separate prototype sets distinguish intra-modality compactness from inter-modality consistency. A transferable hypothesis is to stabilize category representations within each domain before cross-domain interaction, although transfer benefits require new experiments.

Limitations & Future Work¶

Evaluation covers a single benchmark reconstructed from existing data, without independently collected scenarios or cross-dataset generalization tests; 214 identities cannot represent all long-term appearance changes.
The benchmark pairs different outfits with different modalities, creating structural coupling between the factors. A reader suggestion is to collect fully crossed clothing and modality conditions to isolate their individual effects.
The cached main text does not specify training/test identity counts or the complete gallery-sampling procedure, and reports no repeated-run variance; decimal precision in tables does not establish reproducibility.
Several equations are corrupted in extraction, and the prototype-initialization scope and total-loss weights retain the ambiguities described above; the feature analysis on page 15 also calls the method "FID" rather than PIA.
The authors claim the first benchmark, but their related work already cites a 2024 visible-infrared clothing-change dataset study; priority should be treated as their task-positioning claim, not an independently verified fact.
Manual cross-dataset verification improves identity consistency but does not replace privacy, consent, or fairness audits; these require separate evaluation before real-world use of identity-association benchmarks.

Compared with CAL: CAL uses clothing-adversarial learning to reduce clothing dependence; PIA explicitly learns dual branches with orthogonality and subsequently uses cleaner identity representations for cross-modality prototype training.
Compared with SAAI: SAAI emphasizes semantic alignment and affinity inference; this paper emphasizes clothing suppression before alignment, and the combination experiment in Figure 2 suggests the approaches are not simply mutually exclusive.
Compared with ProtoNCE: PIA adopts prototype contrast but maintains two modality-specific prototype sets using supervised identity labels and progressive training; it does not discover identity clusters without supervision.
Compared with Wei et al. (2024): Reference [44] already investigates visible-infrared clothing-change data in natural scenes; this paper emphasizes coupled-variation analysis and a two-stage method, warranting careful wording about priority.

Rating¶

Novelty: 4/5. The joint-variation analysis and progressive optimization are distinctive, but the first-benchmark claim should not be repeated without qualification.
Experimental Thoroughness: 3/5. Bidirectional comparisons and component ablations are useful, but independent datasets, variance, and protocol details remain limited.
Writing Quality: 3/5. The central logic is clear, but initialization descriptions, weight explanations, and method naming require checking.
Value: 4/5. The work provides a concrete benchmark and persuasive conditional evidence for removing confounding cues before alignment.