---

title: >- [Paper Note] Local-to-global Cross-modal Coordination for Self-supervised RGB-T Tracking description: >- [ECCV 2026][self_supervised][RGB-T Tracking] A self-supervised RGB-T tracker with Hierarchical Local Coordination and Turning-phase Global Coordination within a closed-loop verification framework. tags: - ECCV 2026 - self_supervised - RGB-T Tracking - Cross-modal Coordination - Closed-loop Verification date: 2026-09-19 content_hash: c054ce4c128e00fe

Local-to-global Cross-modal Coordination for Self-supervised RGB-T Tracking

Conference: ECCV 2026
Paper: ECCV Original
Code: https://github.com/zyyyya/LGCTrack
Area: Self-Supervised Learning
Keywords: RGB-T tracking, self-supervised learning, single object tracking, cross-modal coordination, closed-loop verification

TL;DR

To tackle error accumulation and modality drift caused by dynamic reliability discrepancies between visible and thermal modalities in forward-backward self-supervised loops, LGCTrack introduces hierarchical local coordination (HLC) and turning-phase global coordination (TGC) to achieve state-of-the-art self-supervised RGB-T tracking while rivaling fully supervised methods.

Background & Motivation

Single object tracking is a foundational computer vision task vital for video surveillance, robotics, and autonomous driving. Recent trackers built on Transformer backbones and large-scale visible datasets have advanced significantly. However, under adverse conditions like low illumination, severe shadows, and intense appearance variations, visible cues alone often degrade. Complementing RGB with thermal infrared (TIR) images provides invariant cross-spectral information across day and night. Nevertheless, top-performing RGB-T trackers rely heavily on dense frame-level manual bounding-box annotations. Annotating multi-modal video sequences is prohibitively labor-intensive and costly, severely curtailing dataset scalability.

This bottleneck has spurred interest in self-supervised RGB-T tracking, where tracking representations are learned from temporal continuity via forward-backward cycle consistency. Under this paradigm, an object is tracked forward from an initial frame to a subsequent frame and then back to the start, using the initial annotation as a verification signal. However, visible and thermal cues exhibit dynamic reliability discrepancies over time: when a modality undergoes thermal crossover or severe over-exposure, generated pseudo-labels become noisy and unreliable. Uncontrolled cross-modal interaction allows noise to cross-contaminate both streams, distorting intermediate representations and compounding drift during forward tracking. When transitioning to backward tracking, degraded target representations cause catastrophic verification failure.

The core tension in loop-based self-supervised RGB-T tracking lies in regulating cross-modal interactions: the network must exploit complementary cross-modal information during feature extraction without transferring unreliable cues, and it must anchor the target semantics at the critical forward-to-backward transition. Core idea: coordinate cross-modal interactions across two granularities by integrating layer-wise reliability-gated local coordination (HLC) inside the dual-stream encoder and turning-phase global coordination (TGC) at the forward-backward transition to filter pseudo-label noise and prevent modality drift.

Method

Overall Architecture

LGCTrack is built upon a dual-stream ViT backbone and a temporal forward-backward closed-loop verification paradigm. The overall pipeline comprises forward tracking, turning-phase global token refinement, backward tracking, and closed-loop loss optimization. For each modality stream (RGB and TIR), a compact global state token \(S = (s^{\mathrm{rgb}}, s^{\mathrm{tir}})\) is prepended to the image patch tokens to summarize target states. Given the initial template \(Z\) at frame \(t_0\), the forward tracker sequentially predicts target bounding boxes. In each encoder layer, Hierarchical Local Coordination (HLC) enables reliability-aware cue exchange. Upon reaching the sequence end at frame \(N\), Turning-phase Global Coordination (TGC) refines the global target tokens via cross-modal attention and cross-initializes the reverse tracking branches. The backward tracker traverses the sequence in reverse temporal order to reconstruct the target state at frame \(t_0\), computing cycle-consistency supervision against the initial ground truth.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Dual-modal Video Input<br/>Synchronized RGB and TIR Streams"] --> B["Forward Tracking with HLC<br/>Layer-wise Gated Local Coordination"]
    B --> C["Turning-phase Global Coordination<br/>Cross-modal Context Aggregation"]
    C --> D["Cross-modal Token Initialization<br/>Coupled Backward Guidance"]
    D --> E["Backward Tracking Loop<br/>Reverse Temporal Verification"]
    E --> F["Closed-loop Supervision<br/>Focal + GIoU + L1 Loss Optimization"]

Key Designs

1. Hierarchical Local Coordination (HLC): Layer-wise Reliability-Gated Feature Fusion

In self-supervised tracking loops, intermediate feature interactions lack per-frame supervision to rectify corrupted features. If cross-modal features are injected indiscriminately, sensory degradation in one modality directly impairs the other. HLC is incorporated as a bypass branch inside each block of the dual-stream encoder. For the \(l\)-th layer intermediate representations \(F_m^{(l)}\) (\(m \in \{\mathrm{rgb}, \mathrm{tir}\}\) and \(\bar{m}\) denoting its counterpart), linear projections generate queries, keys, and values. Queries of modality \(m\) attend to the keys and values of \(\bar{m}\) to extract complementary context \(C_m\):

\[C_m = \mathcal{M}\left(\mathrm{Softmax}\left(\frac{Q_m K_{\bar{m}}^T}{\sqrt{d}}\right) V_{\bar{m}}\right)\]

where \(\mathcal{M}(\cdot)\) is a lightweight Linear-GELU layer and \(d\) is the projection dimension. To prevent unreliable cues from polluting the stream, HLC concatenates the original representation with \(C_m\) and feeds them into a two-layer MLP to generate a token-wise scalar gate \(g_m \in [0, 1]\):

\[g_m = \sigma\left(W_{\mathrm{gate}}^m [F_m^{(l)}; C_m]\right)\]

The gated cross-modal features are then merged into the main stream with a learnable scaling parameter \(\alpha_m\) (initialized to zero): \(F_m' = F_m^{(l)} + \mathrm{MSA}(\mathrm{LN}(F_m^{(l)})) + \alpha_m \, g_m \odot C_m\), followed by FFN transformation. Across all 12 encoder layers, HLC progressively fuses complementary cues while dynamically blocking corrupted signals.

2. Turning-phase Global Coordination (TGC): Cross-modal Refinement and Cross-write Initialization

The turning phase between forward and backward tracking represents the most vulnerable bottleneck in closed-loop verification. Since backward tracking is initialized by the final forward prediction, unimodal tracking errors accumulated at the turnaround point will inevitably propagate throughout the reverse trajectory. Rather than propagating unimodal global tokens \(s_{\mathrm{turn}}^m\), TGC aggregates spatial tokens \(F_N^{\bar{m}}\) from the opposite modality to refine the target semantic representation:

\[\tilde{s}_{\mathrm{turn}}^m = \mathrm{Proj}\left(\mathrm{Softmax}\left(\frac{Q(s_{\mathrm{turn}}^m) K(F_N^{\bar{m}})^T}{\sqrt{d}}\right) V(F_N^{\bar{m}})\right)\]

Crucially, instead of writing back to its source modality, LGCTrack employs a cross-write initialization strategy: the refined token \(\tilde{s}_{\mathrm{turn}}^{\mathrm{rgb}}\) is injected into the TIR backward stream, while \(\tilde{s}_{\mathrm{turn}}^{\mathrm{tir}}\) is assigned to the RGB backward stream. This cross-write coupling binds both streams within the verification loop, preventing either branch from over-relying on unimodal momentum and enforcing cross-modal semantic consensus. During inference, TGC serves to update the global token based on preceding tracking states.

Loss & Training

Given an unlabeled clip of \(N\) frames with initial ground-truth box \(b_{t_0}^{gt}\), forward tracking recursively predicts boxes until frame \(N\). After TGC calibration, reverse tracking returns to frame \(t_0\), yielding backward prediction \(\tilde{b}_{t_0}\). The network is optimized end-to-end using the closed-loop verification objective at frame \(t_0\):

\[\mathcal{L} = \mathcal{L}_{\mathrm{cls}}(\tilde{b}_{t_0}, b_{t_0}^{gt}) + \lambda_{\mathrm{giou}}\mathcal{L}_{\mathrm{giou}}(\tilde{b}_{t_0}, b_{t_0}^{gt}) + \lambda_{L_1}\mathcal{L}_1(\tilde{b}_{t_0}, b_{t_0}^{gt})\]

where \(\mathcal{L}_{\mathrm{cls}}\) denotes the focal classification loss, and \(\mathcal{L}_{\mathrm{giou}}\) and \(\mathcal{L}_1\) represent GIoU and L1 bounding-box regression losses with loss weights \(\lambda_{\mathrm{giou}}=2\) and \(\lambda_{L_1}=5\). The tracker initializes with SSTrack weights and is optimized using AdamW (learning rate \(10^{-4}\), weight decay \(10^{-4}\)) for 10 epochs with batch size 4 on NVIDIA RTX 4090 GPUs (and verified on Atlas 800T A3).

Key Experimental Results

Main Results

LGCTrack was extensively evaluated on three standard RGB-T tracking benchmarks: LasHeR, RGBT234, and GTOT. It establishes new state-of-the-art performance among self-supervised trackers across all metrics and substantially closes the performance gap to fully supervised models.

Table 1: State-of-the-art comparison on LasHeR, RGBT234, and GTOT (selected from Table 1 of the paper)

Method	Source	Sup	LasHeR PR (%)	LasHeR NPR (%)	LasHeR SR (%)	RGBT234 PR (%)	RGBT234 SR (%)	GTOT PR (%)	GTOT SR (%)
APFNet	AAAI 22	Full	50.0	-	36.2	82.7	57.9	90.5	73.7
CMD	CVPR 23	Full	59.0	54.6	46.4	82.4	58.4	89.2	73.4
CAFormer	AAAI 25	Full	70.0	66.1	55.6	88.3	66.4	91.8	76.9
S2OTFormer	TOMM 24	Self	39.8	-	29.5	68.4	47.7	83.1	70.2
GDSTrack	IJCAI 25	Self	45.9	39.3	35.4	70.9	48.5	73.9	59.8
SSTrack	AAAI 25	Self	54.8	49.8	38.4	78.5	55.8	79.4	65.3
LGCTrack (Ours)	ECCV 2026	Self	59.5	55.3	45.7	84.7	62.0	91.9	77.6

Note: On LasHeR, LGCTrack outperforms the prior best self-supervised method SSTrack by +4.7% PR, +5.5% NPR, and +7.3% SR. On GTOT, LGCTrack even marginally surpasses the fully supervised CAFormer (91.9% vs. 91.8% PR).

Ablation Study

Table 2: Ablation of core components and interaction designs (summarized from Tables 3 & 4 of the paper)

Experiment	Variant	LasHeR PR (%)	LasHeR NPR (%)	LasHeR SR (%)	RGBT234 PR (%)	RGBT234 SR (%)	Note
Components (Tab. 3)	Baseline (Unimodal loops)	54.8	49.8	38.4	78.5	55.8	No cross-modal coordination
Components (Tab. 3)	+ HLC only	58.3	54.1	43.9	81.4	59.7	Layer-wise feature fusion
Components (Tab. 3)	+ TGC only	55.7	51.0	42.6	81.3	57.4	Turning-phase token refinement
Components (Tab. 3)	Full LGCTrack (HLC + TGC)	59.5	55.3	45.7	84.7	62.0	Combined local-to-global design
Interaction (Tab. 4)	Vanilla direct addition	58.3	53.3	41.7	80.8	57.3	Lacks gating mechanism
Interaction (Tab. 4)	HLC w/o dynamic gate	58.2	54.2	44.4	83.0	60.4	Degrades without noise control

Table 3: Ablation of TGC update strategies and inference speed (summarized from Tables 6 & 8 of the paper)

Evaluation Aspect	Variant / Method	LasHeR PR (%)	RGBT234 SR (%)	Params (M)	FLOPs (G)	FPS
TGC Update Strategy (Tab. 6)	Direct Token Swap (Dir.)	58.9	60.8	-	-	-
TGC Update Strategy (Tab. 6)	Refine and Self-write (Self)	58.4	60.2	-	-	-
TGC Update Strategy (Tab. 6)	Refine and Cross-write (Cross)	59.5	62.0	-	-	-
Efficiency Comparison (Tab. 8)	GDSTrack (IJCAI 25, Self)	45.9	48.5	202.1	129.4	15.0
Efficiency Comparison (Tab. 8)	STTrack (AAAI 25, Full)	-	-	128.3	181.2	42.4
Efficiency Comparison (Tab. 8)	LGCTrack (Ours)	59.5	62.0	167.1	178.4	52.0

Key Findings

• Dual-granularity Synergy: Adding HLC alone lifts LasHeR SR from 38.4% to 43.9% (+5.5%), and TGC alone increases it to 42.6% (+4.2%). Combining both achieves 45.7% (+7.3%), validating that progressive local interaction and terminal global semantic anchoring are mutually reinforcing.
• Critical Role of Reliability Gating: Removing the dynamic gate from HLC drops LasHeR PR from 59.5% to 58.2% and RGBT234 SR from 62.0% to 60.4%, demonstrating that ungated fusion leaks modality-specific noise during unannotated self-supervised tracking.
• Superiority of Cross-modal Write-back: Writing the refined token back to its own stream yields 58.4% PR on LasHeR, whereas cross-writing to the dual stream achieves 59.5% PR, confirming that forcing cross-modal dependency breaks unimodal error drift.
• Real-Time Efficiency: LGCTrack achieves 52.0 FPS on an RTX 4090, vastly outperforming prior self-supervised tracker GDSTrack (15.0 FPS) while maintaining a balanced parameter footprint (167.1M).

Highlights & Insights

• Dual-Defense Architecture for Cycle Verification: By dissecting tracking loops into intermediate extraction and terminal transition, LGCTrack effectively resolves error accumulation with HLC and prevents semantic drift with TGC.
• Cross-write Initialization Trick: The turning-phase cross-injection forces the reverse tracker of one modality to condition on features derived from the other, establishing a natural self-checking constraint that prevents shortcut learning.
• High Transferability: The local-gated feature interaction and turning-phase global cross-calibration principle can readily transfer to other self-supervised multimodal tracking modalities, such as RGB-Depth (RGB-D) and RGB-Event (RGB-E).

Limitations & Future Work

• Concurrent Dual-modality Degradation: When a target encounters pitch-black environments alongside severe thermal clutter simultaneously, both streams fail, causing unrecoverable loop failure.
• Fixed-length Clip Constraints: The current loop optimization operates on fixed short clips, lacking dynamic memory retrieval mechanisms to re-identify targets after extended occlusions in long-duration tracking.
• Potential Extension: Incorporating a temporal memory buffer to retrieve reliable historical anchors across the cycle would further stabilize long-term tracking.

Related Work & Insights

• vs SSTrack (AAAI 2025): SSTrack leverages instance-level contrastive cycle consistency but lacks fine-grained modality reliability filtering. LGCTrack incorporates HLC and TGC, outperforming SSTrack on LasHeR by +7.3% SR.
• vs GDSTrack (IJCAI 2025): GDSTrack pioneers self-supervised RGB-T tracking but suffers from heavy computation (15 FPS) and error accumulation. LGCTrack delivers +13.6% PR gains on LasHeR and operates more than 3x faster (52.0 FPS).
• vs Fully Supervised Trackers (ViPT / CAFormer): While supervised trackers rely on extensive per-frame bounding boxes, LGCTrack demonstrates that properly regulated cycle consistency can match or exceed supervised baselines on benchmarks like GTOT.

Rating

• Novelty: ⭐⭐⭐⭐☆ Elegant multi-granularity coordination resolving fundamental cycle-consistency drift in multimodal tracking.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across three benchmarks, five attribute evaluations, full ablations, and hardware validation.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, self-consistent mathematical formulations, and thorough qualitative analysis.
• Value: ⭐⭐⭐⭐☆ Bridges the gap between unsupervised and supervised RGB-T tracking with an open-source real-time tracker.

Related Papers

• [ECCV 2024] SCPNet: Unsupervised Cross-modal Homography Estimation via Intra-modal Self-supervised Learning
• [ICML 2026] Can Local Learning Match Self-Supervised Backpropagation?
• [CVPR 2026] Semantic-Guided Global-Local Collaborative Prompt Learning for Few-Shot Class Incremental Learning
• [CVPR 2026] Dual-Estimator: Decoupling Global and Local Semantic Shift for Drift Compensation in Class-Incremental Learning
• [ICLR 2026] Rethinking Unsupervised Cross-Modal Flow Estimation: Learning from Decoupled Optimization and Consistency Constraint