Progressively Spiral Mamba Fusion for Multimodal Tracking

Conference: ECCV 2026
Paper: ECCV Official
Area: Video Understanding
Keywords: Multimodal Tracking / State Space Model / Mamba / Spiral Scanning / Cross-Layer Feature Interaction

TL;DR

To overcome quadratic computational complexity and the neglect of intra-modal cross-layer complementarity in multimodal tracking, PSMTrack presents a progressive SSM tracker that incorporates difference-guided bidirectional Mamba enhancement, sparse-routed cross-layer interaction, and hybrid local-global spiral Mamba fusion, achieving state-of-the-art performance across RGB-D/T/E benchmarks at 43.6 FPS.

Background & Motivation

Single Object Tracking (SOT) aims to continuously estimate the state and position of a target throughout an evolving video sequence given only the initial bounding box in the first frame. Although modern RGB-based trackers have achieved impressive accuracy under conventional conditions, they inherently degrade when confronted with severe real-world challenges, such as adverse illumination, heavy occlusion, abrupt motion blur, and distracting background clutter. To reinforce robustness, integrating complementary sensing modalities (such as thermal infrared TIR, depth D, or event streams E, collectively denoted as RGB-X) provides essential geometric, thermal, or microsecond temporal edge cues that compensate for visual degradation in the visible spectrum.

Nonetheless, existing multimodal tracking frameworks face two fundamental architecture bottlenecks. First, conventional methods predominantly rely on cross-modal attention mechanisms or lightweight adapter layers. Because cross-attention incurs quadratic computational complexity relative to sequence length, it is typically restricted to a few isolated deep Transformer blocks, leaving multi-stage complementary cues largely unexploited; conversely, parameter-efficient adapters exhibit weak cross-modal interaction capacity, requiring exhaustive multi-layer stacking and longer token sequences that impose unfavorable latency trade-offs. Second, current paradigms routinely overlook cross-layer feature complementarity within each modality: lower-level layers preserve fine spatial textures and precise target boundaries, whereas deeper layers encapsulate invariant high-level semantics. Relying exclusively on the final layer for downstream cross-modal fusion causes noticeable information forgetting and impairs tracking stability.

Addressing the core tension between dense multi-layer fusion demands and quadratic computational overhead, this paper explores the linear-complexity sequence modeling power of selective State Space Models (Mamba). Core idea: decompose multimodal tracking into a three-stage progressive pipeline consisting of layer-wise difference-guided bidirectional enhancement, dynamic sparse cross-layer aggregation, and hybrid local-to-global spatial spiral Mamba fusion, enabling holistic cross-modal alignment with near-linear complexity.

Method

Overall Architecture

The overall architecture of PSMTrack processes paired RGB and auxiliary \(X\) inputs through a dual-branch ViT backbone. After patch projection, image tokens progress through 12 Transformer blocks. Following each Transformer block, the Difference-guided Bidirectional State-space Mamba Enhancement (DBME) module operates symmetrically on both branches to inject differential cues without altering sequence dimensions. At the backbone output stage, the Cross-Layer Interaction (CLI) module dynamically selects an informative subset of intermediate layers via lightweight routing to reconcile low-level textures with high-level semantics. Finally, the Hybrid Spiral Mamba Fusion (HSMF) module performs consecutive local-window and full-map spiral scanning on concatenated multimodal features before feeding the unified representation into a frozen FCN prediction head for classification and bounding-box regression.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["RGB & Auxiliary X Modality Inputs<br/>Template and Search Patch Embeddings"] --> B["Dual-Stream Transformer Backbone"]
    B --> C["Difference-guided Bidirectional Mamba Enhancement (DBME)<br/>Modal Discrepancy Mining + Bidirectional SSM Residual"]
    C --> D["Dynamic Routing & Cross-Layer Interaction (CLI)<br/>Top-k Sparse Layer Selection + SCSA Denoising + AConv/SCConv"]
    D --> E["Hybrid Spiral Mamba Fusion (HSMF)<br/>k×k Local Bidirectional Spiral + Full-Map Global Spiral Scanning"]
    E --> F["Tracking Head<br/>Target Classification & Bounding Box Regression"]

Key Designs

1. Difference-guided Bidirectional State-space Mamba Enhancement: Layer-Wise Fine-Grained Cue Injection

Existing multimodal trackers typically interact features solely via generic cross-attention or simple feature summing, which lacks explicit discrepancy awareness. DBME addresses this by deploying after each of the 12 Transformer layers in a residual formulation without altering channel or token dimensions. To maintain spatial boundary consistency, DBME performs bidirectional sequence scanning using forward and backward SSMs on each modality's token sequence: $\(F_i^{r\text{-}f} = \Phi_{\mathrm{SSM}}(F_i^r), \quad F_i^{r\text{-}b} = \mathrm{flip}\left(\Phi_{\mathrm{SSM}}\left(\mathrm{flip}(F_i^r)\right)\right)\)$ Concurrently, the module computes the element-wise absolute difference \(F_i^d = |F_i^r - F_i^x|\) to explicitly highlight cross-modal discrepancies and complementary regions. This difference map is processed by a selective Mamba block and normalized with a \(\tanh\) activation function to suppress common-mode background noise. The resulting gated discrepancy response modulates the bidirectional feature sum, which is projected by a linear layer \(\Phi_L\) and added back residually: $\({F'_i}^r = \Phi_L\left((F_i^{r\text{-}f} + F_i^{r\text{-}b}) \odot \tanh(\mathrm{Mamba}(F_i^d))\right) + F_i^r\)$ The auxiliary modality \(X\) is updated symmetrically. This progressive design ensures that clean complementary cues are injected layer by layer while preventing background distractors from propagating.

2. Dynamic Routing Cross-Layer Interaction: Adaptive Multi-Scale Semantic Aggregation

Relying exclusively on the final backbone layer loses fine geometric details vital for precise localization, whereas static multi-layer concatenation cannot adapt to varying scene dynamics. CLI incorporates a modality-shared routing mechanism \(R\) that takes the channel-concatenated representations of all \(L\) Transformer layers, applies spatial pooling \(P(\cdot)\), and generates layer selection probabilities via an MLP: $\(s^n = \mathrm{MLP}(P([F_1; F_2; \dots; F_L]))\)$ The top-\(k\) layers with highest scores are dynamically selected, with the final layer permanently retained as a shared expert. To counteract spatial misalignment and channel redundancy from direct concatenation, CLI applies Spatial-Channel Synergistic Attention (SCSA) along both paths for explicit calibration and denoising. Furthermore, it substitutes the standard linear projection in Non-Causal State-Space Duality (NC-SSD) with an adaptive \(1\times 1\) convolution (AConv) and replaces depthwise convolution with Spatial and Channel reconstruction Convolution (SCConv). This maintains a \(3\times 3\) receptive field while stabilizing feature reconstruction against spurious activations under blur and occlusion. The final cross-layer representation is formed via an equal-weighted sum over selected layers: \(F = \sum_{l \in S} F_{\text{interaction}}^l\).

3. Hybrid Spiral Mamba Fusion: Local Disparity Alignment and Global Coherent Modeling

Directly concatenating multimodal features \(F = \mathrm{Concat}(F^{\mathrm{RGB}}, F^X)\) prior to regression often introduces semantic fragmentation and boundary instability due to heterogeneous sensor responses. HSMF structures spatial interaction into a dual-stage spiral Mamba framework operating under linear computational complexity. - Local Mamba Stage: The feature map \(F\) is divided into non-overlapping \(k \times k\) local windows. Within each window, tokens are sequenced along both inward and outward spiral paths and modeled using Mamba. Because the continuous spiral trajectory preserves spatial contiguity, local Mamba aligns modal disparities within each neighborhood, suppressing single-modality false boundaries while amplifying co-supported target structures. - Global Mamba Stage: Taking the refined output of the local stage, global Mamba treats the entire \(H \times W\) feature map as an integrated region and traverses all spatial locations along a macro spiral trajectory. Unlike conventional raster-scan unrolling that ruptures vertical 2D adjacency across row transitions, spiral scanning maintains spatial proximity between neighboring coordinates throughout the sequence. This enables smooth long-range contextual propagation, allowing target features corrupted by occlusion or lighting failure to be reconstructed from uncorrupted spatial evidence, outputting a globally consistent feature map \(F_{\mathrm{fusion}}\).

Loss & Training

Following established visual tracking pipelines, the tracker employs a frozen Fully Convolutional Network (FCN) prediction head. The total objective balances target classification weighted Focal Loss \(\mathcal{L}_{\mathrm{cls}}\), Generalized IoU loss \(\mathcal{L}_{\mathrm{giou}}\), and smooth \(\mathcal{L}_1\) bounding-box regression loss: $\(\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{cls}} + \lambda_1 \mathcal{L}_{\mathrm{giou}} + \lambda_2 \mathcal{L}_1\)$ with balancing hyper-parameters set to \(\lambda_1 = 2\) and \(\lambda_2 = 5\). Dual-branch backbones are initialized with pretrained weights from the OSTrack foundation tracker. Optimization uses AdamW with an initial learning rate of \(10^{-5}\) and weight decay of \(10^{-4}\). Fine-tuning runs for 50 epochs on LasHeR (RGB-T), 30 epochs on DepthTrack (RGB-D), and 70 epochs on VisEvent (RGB-E) with batch size of 60,000 image pairs per epoch on a single NVIDIA RTX 4090 GPU.

Key Experimental Results

Main Results

PSMTrack is comprehensively evaluated against recent state-of-the-art multimodal trackers across five major benchmarks spanning RGB-T, RGB-D, and RGB-E tracking. Running on a single NVIDIA RTX 4090 GPU, it achieves an inference throughput of 43.6 FPS.

Quantitative results from Table 1, Table 2, and Table 3 of the paper are summarized below:

Dataset	Modality	Metrics	Ours (PSMTrack)	Prev. SOTA (STTrack)	Gain
LasHeR	RGB-T	PR (%) / SR (%)	76.5 / 61.2	76.0 / 60.3	+0.5 / +0.9
RGBT234	RGB-T	MPR (%) / MSR (%)	91.2 / 67.3	89.8 / 66.7	+1.4 / +0.6
DepthTrack	RGB-D	Re (%) / Pr (%) / F-score (%)	64.2 / 64.3 / 64.2	63.4 / 63.2 / 63.3	+0.8 / +1.1 / +0.9
VOT-RGBD2022	RGB-D	EAO (%) / Accuracy (%) / Robustness (%)	77.8 / 82.9 / 93.5	77.6 / 82.5 / 93.7	+0.2 / +0.4 / -0.2
VisEvent	RGB-E	AUC (%) / Pr (%)	62.7 / 79.9	61.9 / 78.6	+0.8 / +1.3

Ablation Study

To validate the contributions of the three core architectural modules, systematic ablation studies were conducted on LasHeR, DepthTrack, and VisEvent (Table 4 in the original paper):

Variant	DBME	CLI	HSMF	LasHeR (PR / SR)	DepthTrack (F-score)	VisEvent (AUC / Pr)	Note
Ours	✓	✓	✓	0.765 / 0.612	0.642	0.627 / 0.799	Full model
Var. 1	×	✓	✓	0.752 / 0.605	0.636	0.623 / 0.789	w/o DBME module
Var. 2	✓	×	✓	0.753 / 0.607	0.636	0.624 / 0.793	w/o CLI module
Var. 3	✓	✓	×	0.748 / 0.603	0.632	0.616 / 0.775	w/o HSMF (replaced by channel concat)
Var. 4	×	×	✓	0.733 / 0.596	0.624	0.614 / 0.772	only HSMF retained

In addition, evaluating the internal design components of CLI (Table 5 in the paper) on LasHeR SR reveals that vanilla NC-SSD achieves 60.5%, adding AConv and SCConv improves to 60.9%, and incorporating SCSA reaches the best score of 61.2%. Ablating the layer selection count \(k\) in CLI (Table 6) shows that \(k=3\) delivers optimal balance across all three benchmarks (LasHeR: 61.2%, DepthTrack: 64.3%, VisEvent: 62.7%), whereas \(k=1\) (60.7%) yields insufficient cross-layer context and \(k=4\) (60.8%) introduces distracting feature redundancy.

Key Findings

• Spatial spiral fusion provides the primary performance anchor: Replacing HSMF with naive concatenation (Variant 3) produces the most severe performance degradation, causing LasHeR PR to drop by 1.7% (from 76.5% to 74.8%) and VisEvent Pr to plummet by 2.4% (from 79.9% to 77.5%). This demonstrates that local noise suppression and global continuity modeling are indispensable for resolving inter-modal spatial disparities.
• Discrepancy guidance prevents error propagation: The DBME module explicitly isolates modality differences; removing it (Variant 1) drops LasHeR PR by 1.3%, showing that bidirectional SSM scanning paired with discrepancy gating effectively refines target representations throughout the 12 backbone layers.
• Optimal depth for multi-scale feature routing: Selecting \(k=3\) intermediate layers in CLI yields the best tracking stability. Incorporating earlier layers supplies essential boundary texture, but selecting too many layers (\(k \ge 4\)) injects high-frequency noise from cluttered backgrounds that degrades target localization.

Highlights & Insights

• Synergy of Discrepancy Guidance and State-Space Scanning: Explicitly computing modality differences and modeling them via selective Mamba provides a zero-overhead gating mechanism that enhances target saliency while suppressing common-mode noise.
• Topology-Preserving 2D Spiral Scanning: Rather than unrolling 2D image grids via raster scans that disrupt spatial adjacency at line breaks, the dual-scale inward/outward spiral scanning preserves 2D neighborhood relationships under 1D state-space recurrences.
• Unified and Real-Time RGB-X Tracking: By freezing the foundation tracking head and inserting linear-complexity SSM modules, PSMTrack achieves real-time execution (43.6 FPS) across thermal, depth, and event modalities within a single unified architecture.

Limitations & Future Work

• Author-Acknowledged Limitations: The framework assumes synchronized, spatially calibrated multimodal inputs and does not explicitly formulate recovery strategies for complete modal sensor dropouts or severe optical distortions. Additionally, the spiral trajectory uses a fixed geometric center, which may not align optimally with non-square or elongated target aspect ratios.
• Potential Improvement Directions: Investigating dynamic, content-adaptive scanning trajectories where the spiral focal point shifts toward target saliency centers could improve feature extraction efficiency. Furthermore, integrating cross-frame memory state recurrence with spatial spiral Mamba could provide stronger spatio-temporal tracking continuity under extended target occlusions.

Related Work & Insights

• vs STTrack: While STTrack incorporates cross-modal Mamba to generate temporal state tokens across frames, its spatial fusion within individual frames remains tied to discrete layers. PSMTrack focuses on intra-frame progressive cross-layer interaction and local-global spiral spatial modeling, outperforming STTrack across LasHeR (PR 76.5% vs 76.0%) and VisEvent (AUC 62.7% vs 61.9%).
• vs SDSTrack / OneTracker: SDSTrack and OneTracker rely on Transformer adapter tuning or prompt tuning, which encounter high latency when scaled across multiple layers. PSMTrack achieves comprehensive multi-layer interaction and adaptive routing with near-linear complexity, demonstrating superior accuracy and operational efficiency.

Rating

• Novelty: ⭐⭐⭐⭐☆ (Creative adaptation of 2D spiral scanning trajectories within selective state-space models for multimodal tracking)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluated on 5 benchmark datasets across RGB-T, RGB-D, and RGB-E with thorough component and hyperparameter ablations)
• Writing Quality: ⭐⭐⭐⭐☆ (Coherent narrative, clean mathematical descriptions, and informative module illustrations)
• Value: ⭐⭐⭐⭐☆ (Offers an efficient, linear-complexity architectural blueprint for spatial multimodal fusion in edge-oriented computer vision)

Related Papers

• [CVPR 2026] HieraMamba: Video Temporal Grounding via Hierarchical Anchor-Mamba Pooling
• [CVPR 2026] DarkAct: A RGB-Thermal Dataset and Fusion Framework for Multimodal Low-Light Action Recognition
• [CVPR 2026] TAPFormer: Robust Arbitrary Point Tracking via Transient Asynchronous Fusion of Frames and Events
• [AAAI 2026] PlugTrack: Multi-Perceptive Motion Analysis for Adaptive Fusion in Multi-Object Tracking
• [ICML 2026] Unified Multimodal Visual Tracking with Dual Mixture-of-Experts