Granular Semantic Cognition for Visible-Infrared Person Re-Identification

Conference: ECCV 2026
Paper: ECCV Official
Area: Human Understanding
Keywords: visible-infrared person re-identification / granular semantic cognition / hierarchical prototype refinement / semantics-guided fusion / cross-identity disambiguation

TL;DR

To address the lack of color in infrared imagery that hinders cross-modal alignment, this paper proposes Granular Semantic Cognition (GSC), which progressively refines shared semantic prototypes across stages to guide channel-wise textual color compensation into corresponding infrared local regions, regularized by a fusion-weighted identity contrastive loss to suppress cross-identity ambiguity.

Background & Motivation

Visible-infrared person re-identification (VI-ReID) is critical for smart surveillance under low-light and nighttime environments. However, infrared sensors inherently capture only thermal radiation while losing rich color and fine texture cues, leading to a substantial cross-modality distribution discrepancy against daytime color imagery. Mainstream VI-ReID approaches typically attempt to embed both modalities into a shared feature space via distribution alignment or synthesize pseudo-visible images through generative adversarial networks. Yet, the former struggles due to the intrinsic loss of discriminative color cues in infrared data, while the latter frequently introduces generative artifacts and suffers from unstable optimization.

Recently, text-compensated VI-ReID paradigms (e.g., CSDN, TVI-LFM) have shown promise by leveraging natural language descriptions to replenish the missing color attributes in infrared images. Nevertheless, existing compensation schemes rely predominantly on coarse-grained, global textual fusion—directly concatenating or statically fusing global text embeddings with visual representations. The fundamental tension lies in the high spatial heterogeneity of human appearance (for instance, a pedestrian wearing a red top and black trousers): global textual guidance lacks spatial localization, readily diffusing color cues into irrelevant anatomical regions. Furthermore, conventional prototype-based semantic priors are too coarse to isolate fine-grained body parts, and shared semantic configurations across different pedestrians induce similar fusion weights, triggering severe cross-identity ambiguity.

To resolve these bottlenecks, this paper explores granular local semantic cognition and adaptive color compensation. Core idea: progressively refine learnable semantic prototypes into fine-grained local anatomical parts through multi-stage feature interactions, generate adaptive channel-wise fusion weights to selectively inject textual color cues into semantically aligned infrared regions, and impose a fusion-weighted identity contrastive loss to regularize fusion patterns and eliminate cross-identity semantic confusion.

Method

Overall Architecture

The overall pipeline of GSC aims to establish precise spatial correspondence between infrared local structures and textual color distributions. Built upon a dual-stream ResNet-50 visual backbone, the framework extracts visible features \(F_{vis}\) and infrared features \(F_{ir}\), while a text encoder produces descriptive textual embeddings \(F_t\). The architecture comprises two core functional modules and a specialized contrastive regularization objective: the Hierarchical Refinement Module (HRM) iteratively refines modality-shared semantic prototypes via multi-stage interactions; the Semantics-guided Adjustment Module (SAM) leverages the refined part semantics to drive bidirectional spatial-channel modulation and adaptive feature fusion; finally, the fusion-weighted identity contrastive loss (\(\mathcal{L}_{FWIC}\)) operates alongside standard cross-entropy and triplet losses to constrain fusion behaviors in an identity-aware manner.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: IR/VIS Images & Text Descriptions"] --> B["Hierarchical Prototype Refinement<br/>Multi-stage pixel-prototype interaction"]
    B --> C["Semantics-guided Channel Adjustment<br/>Bidirectional modulation & adaptive fusion"]
    C --> D["Fusion-weighted Identity Contrastive Constraint<br/>Identity-consistent fusion behavior regularizer"]
    D --> E["Cross-modal Identity Discriminative Embeddings"]

Key Designs

1. Hierarchical Prototype Refinement: progressively refining local anatomical part semantics

To overcome the coarseness of conventional prototype learning that fails to isolate fine-grained human body parts, HRM incorporates learnable semantic prototypes \(P^{(s)} = \{p_1^{(s)}, \dots, p_k^{(s)}\}\) together with learnable spatial positional encodings into a multi-stage refinement pipeline. At stage \(s\), the module computes the cosine similarity between pixel-wise visual features \(t_{m,j}^{(s)}\) and \(\ell_2\)-normalized prototypes, yielding a spatial assignment probability matrix:

\[S_m^{(s)}(i,j) = \frac{\exp\langle\bar{p}_i^{(s)}, \bar{t}_{m,j}^{(s)}\rangle}{\sum_{q=1}^k \exp\langle\bar{p}_q^{(s)}, \bar{t}_{m,j}^{(s)}\rangle}\]

Using \(S_m^{(s)}(i,j)\) as assignment weights, pixel features are aggregated into latent semantic part vectors \(g_{m,i}^{(s)}\). A lightweight two-layer MLP then enhances these part representations into refined vectors \(\tilde{g}_{m,i}^{(s)}\), which are broadcast back to update spatial pixel features for the subsequent stage: \(T_{m,j}^{(s+1)} = T_{m,j}^{(s)} + \sum_{i=1}^k S_m^{(s)}(i,j)\tilde{g}_{m,i}^{(s)}\). To prevent prototype collapse into redundant representations, HRM optimizes a prototype diversity loss \(\mathcal{L}_{pd}\) that penalizes pairwise correlations between prototype projections, encouraging the \(k\) prototypes to specialize into distinct, non-overlapping anatomical anchors.

2. Semantics-guided Channel Adjustment: adaptive color compensation steered by part priors

Once fine-grained part semantics are extracted, SAM seamlessly integrates textual color cues into infrared feature maps without spatial misplacement. SAM performs bidirectional feature modulation: first, global text features modulate infrared spatial maps via depthwise channel correlation, dynamically amplifying infrared regions consistent with verbal descriptions; second, global average pooling over the infrared map re-calibrates text channels via Sigmoid gating to inject visual-awareness into language embeddings. Subsequently, SAM passes the refined part semantics \(P^{(s)}\) from the final HRM stage through a gated MLP to predict channel-wise fusion weights:

\[\boldsymbol{\alpha} = \sigma\big(W_2 \delta(W_1 P^{(s)})\big)\]

where \(\delta\) is the ReLU function and \(\sigma\) is the Sigmoid activation. The fused infrared representation is obtained via element-wise blending: \(\tilde{F}_{ir} = \boldsymbol{\alpha} \odot F_{ir} + (1 - \boldsymbol{\alpha}) \odot F_t\). Consequently, channels strongly correlated with specific body parts absorb complementary color information from text, whereas unrelated channels retain intrinsic infrared structural geometry, precluding color bleeding across body parts.

3. Fusion-weighted Identity Contrastive Constraint: suppressing cross-identity fusion ambiguity

Even with granular part semantics, different pedestrians frequently exhibit similar body structures or clothing styles (e.g., dark tops), causing the model to generate nearly identical fusion weights for distinct individuals and resulting in feature confusion. To eliminate this ambiguity, the fusion-weighted identity contrastive loss (\(\mathcal{L}_{FWIC}\)) directly constrains the fusion weight space. To discard sample difficulty and scale variations while preserving relative channel preference patterns, the fusion vector \(\boldsymbol{\alpha}_i\) is \(\ell_2\)-normalized into \(\tilde{\boldsymbol{\alpha}}_i\). The pairwise fusion strategy similarity is formulated as \(s_{ij} = \tilde{\boldsymbol{\alpha}}_i^\top \tilde{\boldsymbol{\alpha}}_j / \tau\) with temperature \(\tau\), and optimized via:

\[\mathcal{L}_{FWIC} = -\frac{1}{B} \sum_{i=1}^B \frac{1}{|\mathcal{P}(i)|} \sum_{j \in \mathcal{P}(i)} \log \frac{\exp(s_{ij})}{\sum_{k \neq i} \exp(s_{ik})}\]

By maximizing fusion weight consistency among positive pairs sharing the same identity while penalizing similar fusion behaviors across negative pairs, \(\mathcal{L}_{FWIC}\) pushes identity-specific channel preference distributions apart, preventing misleading cross-identity semantic overlap.

Loss & Training

The entire network is optimized end-to-end using a joint objective function:

\[\mathcal{L}_{total} = \mathcal{L}_{id} + \mathcal{L}_{tri} + \mathcal{L}_{pd} + \mathcal{L}_{cs} + \lambda \mathcal{L}_{FWIC}\]

where \(\mathcal{L}_{id}\) is the identity cross-entropy loss applied to batch-normalized embeddings, \(\mathcal{L}_{tri}\) is the cross-modal hard-mining triplet loss, \(\mathcal{L}_{cs}\) is the center separation loss enforcing intra-class compactness and inter-class dispersion via margins \(\rho_1, \rho_2\), and \(\lambda\) balances the contribution of \(\mathcal{L}_{FWIC}\) (\(\lambda = 1.0\) on SYSU-MM01 and \(\lambda = 1.5\) on LLCM). Optimization is performed on a single NVIDIA RTX 4090 GPU using SGD with momentum 0.9. The learning rate warms up to 0.1 over the first 10 epochs and decays by a factor of 10 at epochs 20 and 50.

Key Experimental Results

Main Results

Comprehensive comparisons against state-of-the-art methods across three widely used VI-ReID benchmarks (SYSU-MM01, RegDB, LLCM) demonstrate the superiority of GSC.

Dataset	Protocol / Mode	Ours (GSC)	Prev. SOTA (TVI-LFM / ICRE / DSFAD)	Gain
SYSU-MM01	All search (Rank-1 / mAP)	89.6% / 90.8%	84.9% / 81.5% (TVI-LFM)	+4.7% / +9.3%
SYSU-MM01	Indoor search (Rank-1 / mAP)	90.1% / 91.4%	89.1% / 90.8% (TVI-LFM)	+1.0% / +0.6%
RegDB	IR → VIS (Rank-1 / mAP)	91.2% / 87.0%	90.2% / 84.3% (ICRE)	+1.0% / +2.7%
LLCM	IR → VIS (Rank-1 / mAP)	58.8% / 64.9%	58.4% / 64.7% (ICRE)	+0.4% / +0.2%

Ablation Study

Component-wise ablation experiments on SYSU-MM01 (All search) and LLCM (IR→VIS) using a standard ResNet-50 dual-stream baseline:

Config	SYSU-MM01 Rank-1	SYSU-MM01 mAP	LLCM Rank-1	LLCM mAP	Note
Baseline	79.7%	77.3%	56.1%	62.5%	Dual-stream ResNet-50 baseline
+ SAM	83.6%	80.6%	57.6%	63.5%	Coarse text adjustment (+3.9% R-1 / +3.3% mAP)
+ SAM + HRM	88.7%	88.9%	58.1%	64.2%	Hierarchical prototype refinement (+5.1% R-1 / +8.3% mAP)
+ SAM + HRM + \(\mathcal{L}_{FWIC}\) (Full GSC)	89.6%	90.8%	58.8%	64.9%	Fusion behavior contrastive regularization

Key Findings

• HRM is the primary driver of performance gains: Adding HRM on top of SAM yields an 8.3% mAP boost on SYSU-MM01, confirming that unrefined coarse prototypes fail to provide reliable spatial anchors for textual color assignment.
• Optimal window for refinement stages: On SYSU-MM01 all-search, increasing HRM stages from 2 to 6 steadily elevates Rank-1 from 83.1% to 89.6%, before over-refinement and gradient saturation slightly lower it to 85.1% at 7 stages; the optimal stage counts are 3 for indoor-search and 5 for LLCM.
• Favorable computational budget: GSC incurs only +67.31M parameters and +1173MB GPU memory during training over the baseline, an acceptable overhead given the +9.9% Rank-1 and +13.5% mAP overall gains.

Highlights & Insights

• Cyclic multi-stage prototype feedback: Instead of passive single-layer cross-attention, HRM iteratively clusters, non-linearly refines, and broadcasts part representations back to spatial pixels, naturally evolving prototypes into anatomical detectors.
• Contrastive learning on fusion strategies: Extending contrastive objectives from latent feature representations to gating weight vectors penalizes homogenized fusion patterns, effectively isolating spurious shared semantics across distinct identities.
• Transferable asymmetric compensation paradigm: Treating language as an asymmetric attribute supplier rather than a symmetric retrieval target provides a generalizable blueprint for thermal-to-visible, low-light enhancement, and depth completion tasks.

Limitations & Future Work

• Static prototype count: The number of semantic prototypes \(k\) is fixed globally during training, lacking elasticity to adjust to partially occluded or cropped pedestrian bounding boxes.
• Future directions: Dynamic routing mechanisms could be explored to prune or expand part prototypes adaptively based on visibility; additionally, lightweight on-device VLMs could replace pre-annotated static texts with online conversational descriptions.

Related Work & Insights

• vs TVI-LFM (NeurIPS 2024): TVI-LFM employs vision-language foundation models to generate rich texts and performs global concatenation; GSC reveals that global fusion causes local color misattribution and leverages granular part semantics to guide localized routing, outperforming TVI-LFM by 9.3% mAP on SYSU-MM01.
• vs CSDN (TMM 2024): CSDN aligns high-level descriptions statically; SAM introduces dynamic bidirectional spatial modulation and channel-wise adaptive gating, coupled with \(\mathcal{L}_{FWIC}\) to dramatically improve discriminability among identically dressed pedestrians.

Rating

• Novelty: ⭐⭐⭐⭐☆ The iterative prototype refinement loop and fusion-weight contrastive regularizer present elegant solutions to local color injection.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across three major benchmarks, exhaustive module ablations, hyperparameter sensitivity curves, and complexity audits.
• Writing Quality: ⭐⭐⭐⭐⭐ Clean formulation, clear mapping between architecture and mathematical mechanisms, and coherent experimental insights.
• Value: ⭐⭐⭐⭐☆ Offers actionable insights for asymmetric cross-modal attribute compensation and fine-grained feature representation.

Related Papers

• [CVPR 2026] Spatial-Frequency Collaborative Learning for Occluded Visible-Infrared Person Re-Identification
• [ECCV 2026] CMCC-ReID: Cross-Modality Clothing-Change Person Re-Identification
• [ECCV 2026] ANFI: Rethinking Neighbor Feature Interaction in Person Re-ID
• [AAAI 2026] Modality-Aware Bias Mitigation and Invariance Learning for Unsupervised Visible-Infrared Person Re-Identification
• [CVPR 2026] Towards Cross-Modal Preservation, Consistency and Alignment for Privacy-Preserving Visible-Infrared Person Re-Identification