STMI: Segmentation-Guided Token Modulation with Cross-Modal Hypergraph Interaction for Multi-Modal Object Re-Identification¶

Conference: AAAI 2026
arXiv: 2603.00695
Code: None
Area: 3D Vision / Multi-Modal
Keywords: Multi-Modal ReID, Cross-Modal Fusion, Hypergraph Interaction, Segmentation Guidance, Token Modulation

TL;DR¶

STMI proposes a three-component multi-modal object re-identification framework. It suppresses background noise through SAM segmentation-guided feature modulation (SFM), extracts compact representations via semantic token reallocation (STR), and captures high-order semantic relationships with cross-modal hypergraph interaction (CHI), achieving significant improvements on benchmarks such as RGBNT201.

Background & Motivation¶

Background: Multi-modal object re-identification (Multi-Modal ReID) leverages complementary information from different modalities (such as RGB, Near-Infrared NIR, and Thermal Infrared TIR) to retrieve specific targets. Existing methods typically extract features from each modality using Vision Transformers (ViT) before fusing them.

Limitations of Prior Work: (1) Hard token filtering may lose discriminative information, and simple fusion strategies fail to effectively utilize cross-modal complementarity; (2) background noise manifests differently across modalities, and there is a lack of an effective foreground-background separation mechanism.

Key Challenge: How to achieve compact representations while retaining all token information and effectively modeling high-order cross-modal semantic relationships.

Goal: To design a unified multi-modal learning framework that simultaneously addresses background suppression, information compression, and high-order cross-modal interaction.

Key Insight: Utilize segmentation masks generated by SAM for soft modulation rather than hard filtering; employ learnable query tokens for adaptive reallocation; and construct hypergraphs to model high-order relations across modalities.

Core Idea: Construct a complete information processing pipeline using three modules: SFM (foreground enhancement) + STR (representation compression) + CHI (multi-modal high-order interaction).

Method¶

Overall Architecture¶

Given multi-modal images, patch tokens are extracted for each modality using ViTs. The SFM module utilizes pre-generated segmentation masks from SAM to modulate token attention. The STR module compresses the modulated tokens into compact representations using learnable query tokens. Finally, the CHI module constructs a unified hypergraph over the compact representations of all modalities to capture high-order cross-modal semantic relationships.

Key Designs¶

Segmentation-Guided Feature Modulation (SFM):
- Function: Enhances foreground representation and suppresses background noise using SAM segmentation masks.
- Mechanism: Generates foreground masks using SAM in advance, and converts these masks into learnable attention weights to perform soft modulation on tokens across ViT layers. Unlike hard token filtering, SFM retains all tokens but reallocates their importance weights.
- Design Motivation: Hard filtering may erroneously discard discriminative foreground tokens, whereas soft modulation effectively reduces background interference while preserving information integrity.
Semantic Token Reallocation (STR):
- Function: Compresses variable-length patch tokens into a fixed number of compact semantic representations.
- Mechanism: Introduces learnable query tokens to interact with modulated patch tokens via cross-attention, achieving adaptive semantic reallocation. The number of query tokens is much smaller than that of patch tokens. Crucially, no tokens are discarded—all information is aggregated into the query tokens via attention.
- Design Motivation: Traditional top-k selection inevitably loses information, whereas STR reallocates all information into compact representations via attention.
Cross-Modal Hypergraph Interaction (CHI):
- Function: Models high-order semantic relationships across all modalities.
- Mechanism: Treats the compact representations of each modality as hypergraph nodes to construct a unified cross-modal hypergraph. Hyperedges connect multiple nodes to capture ternary and higher-order semantic correlations. Information is propagated through hypergraph convolutions.
- Design Motivation: Cross-modal information exhibits higher-order relations beyond pairwise associations, which hypergraphs can naturally model.

Loss & Training¶

An industry-standard ReID training strategy is adopted: ID Classification Loss (Cross-Entropy) + Metric Learning Loss (Triplet Loss).

Key Experimental Results¶

Main Results¶

Dataset	Metric	STMI	Prev. SOTA	Gain
RGBNT201	mAP	Best	-	Significantly outperforms all baselines
RGBNT100	mAP	Best	-	Multi-modal fusion shows clear advantages
MSVR310	mAP	Best	-	Hypergraph interaction is effective

Ablation Study¶

Configuration	Effect	Description
Full STMI	Best	Synergy of three modules
w/o SFM	Significant drop	Increased background interference
w/o STR	Drop	Token redundancy reduces discriminability
w/o CHI	Drop	Lack of cross-modal high-order interaction

Key Findings¶

The SFM module contributes the most, validating that background noise is the primary bottleneck in multi-modal ReID.
CHI hypergraph interaction shows distinct advantages over general graph convolutions, demonstrating the necessity of high-order relationship modeling.
STR retains more discriminative information compared to hard top-k selection.

Highlights & Insights¶

SAM as a General Foreground Extractor: Utilizing the zero-shot segmentation capability of SAM to provide foreground masks for ReID, which can be transferred to tasks such as pedestrian attribute recognition and vehicle ReID.
Hypergraphs vs. Ordinary Graphs: Hypergraphs can model high-order semantics beyond pairwise relations, rendering them more expressive in multi-modal scenarios.
Token-Preserving Compression: The "reallocate rather than drop" philosophy of STR balances efficiency with information integrity.

Limitations & Future Work¶

Heavy reliance on pre-generating masks using SAM increases pre-processing overhead.
The construction and convolution of hypergraphs increase computational complexity.
Validation is restricted to the RGB-NIR-TIR tri-mode, leaving the performance on other modality combinations unknown.
SAM may fail in low-quality or extreme scenarios, potentially degrading the performance of SFM.

vs. TOP-ReID: TOP-ReID uses token pruning, which may lose information, whereas STMI's STR avoids information loss through reallocation.
vs. Traditional Multi-Modal Fusion: Simple concatenation or addition fails to capture high-order cross-modal relationships, which CHI handles more effectively.
vs. TransReID: While multi-modal settings provide complementary information, STMI demonstrates the advantage of structured fusion.

Rating¶

Novelty: ⭐⭐⭐⭐ The integration of SAM and hypergraphs is novel, with all three modules showcasing innovation.
Experimental Thoroughness: ⭐⭐⭐⭐ Comprehensively evaluated across three benchmarks.
Writing Quality: ⭐⭐⭐⭐ The framework description is exceptionally clear.
Value: ⭐⭐⭐⭐ Highly beneficial for pushing the boundaries of the multi-modal ReID field.