SFDATrack: Generalized Source-Free Domain Adaptive Tracking Under Adverse Weather Conditions¶

Conference: ECCV2026
arXiv: 2607.00369
Code: https://github.com/watcherBR0/sfdatrack
Area: Video Understanding
Keywords: Object Tracking, Source-Free Domain Adaptation, Adverse Weather, Mamba, Prototype Projection

TL;DR¶

SFDATrack proposes the first source-free domain adaptive framework for visual tracking under adverse weather conditions. Through a bidirectional interactive Mamba module and a hyperspherical prototype projection module within a Mean-Teacher architecture, it robustly adapts to multiple weather conditions without accessing any source domain data, achieving state-of-the-art performance on both synthetic and real-world adverse weather tracking benchmarks.

Background & Motivation¶

Visual object tracking plays a foundational role in applications such as autonomous driving, video surveillance, and embodied AI. Modern deep-learning-based trackers perform exceptionally well in standard scenarios (e.g., daytime, clear weather), but their performance drops drastically when they are deployed in unseen adverse weather environments (e.g., night, heavy fog, rainstorms)—where non-uniform perturbations like low contrast and atmospheric noise severely degrade the model's target recognition capabilities. To address this, Domain Adaptive Video Object Tracking (DAVOT) methods have emerged recently, attempting to learn cross-weather invariant robust features through joint training on labeled source domain data (e.g., daytime) and unlabeled target domain data (e.g., night, foggy days).

However, these existing methods rely on an assumption that is often impractical in real-world scenarios: source domain data must always be accessible during the adaptation stage. In practice, storage constraints, computational costs, or data privacy regulations often make this assumption invalid—for instance, edge devices with limited memory cannot store large-scale source domain videos, and bandwidth-constrained networks struggle to transmit source data. This raises a crucial question: Can a pre-trained source model be successfully adapted to an arbitrary adverse weather target domain without access to any original source data?

This paper aims to answer this question. Core Idea: Utilize a Mean-Teacher framework to align target tokens under weak/strong augmentations through bidirectional Mamba interactions, then project them onto a learnable hyperspherical prototype space. Use dual domain-specific and domain-invariant alignment constraints to generalize the model to various weather conditions without seeing any source data.

Method¶

Overall Architecture¶

SFDATrack implements source-free multi-domain knowledge distillation based on a Mean-Teacher pipeline. The input video frames undergo weak augmentation (Teacher branch) and strong augmentation (Student branch) before being fed into encoders with shared structures. The teacher branch generates pseudo-labels for weakly augmented frames, while the student branch learns from strongly augmented frames. The teacher parameters are updated from the student via Exponential Moving Average (EMA). The encoded features are sequentially processed by the Dual Interactive Mamba (DIM) module and the Hyperspherical Prototype Projection (HPP) module, and finally optimized with a joint loss to achieve robust tracking across weather domains.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Target Domain Video"] --> B["Weak Augmentation (Teacher)"]
    A --> C["Strong Augmentation (Student)"]
    B --> D["ViT Encoder θᵀ"]
    C --> E["ViT Encoder θˢ"]
    D --> F["Dual Interactive Mamba<br/>Bidirectional Sequence Interaction"]
    E --> F
    F --> G["Hyperspherical Prototype<br/>Projection (HPP)"]
    G --> H["Domain-Specific Alignment (DSA)"]
    G --> I["Domain-Invariant Alignment (DIA)"]
    H --> J["Joint Loss L"]
    I --> J
    J --> K["Target State Estimation"]
    D -.->|EMA Update| E

Key Designs¶

1. Dual Interactive Mamba (DIM): Bidirectional Sequence Interaction to Bridge the Domain Gap

Although the Mean-Teacher framework provides a basic knowledge distillation mechanism, its generalization capability across multiple weather conditions is limited by the feature inconsistency between the weak and strong augmentation branches. The core idea of the DIM module is to enable bidirectional information exchange between the search region tokens of both branches, thereby distilling candidate target tokens that are robust to weather changes. Specifically, for the encoded search tokens \(X'\) and \(X''\) of the teacher and student branches, a set of learnable interaction prompt tokens \(P'\) and \(P''\) are introduced. The search tokens of both branches are flipped and concatenated with the opponent's prompt tokens: the teacher branch uses \(P''\) + flipped \(X'\), and the student branch uses \(P'\) + flipped \(X''\). These concatenated tokens are normalized and linearly projected to generate gating vectors, which are then fed into a shared Mamba SSM block for sequence modeling. Finally, the enhanced features from both paths are aggregated back into the search tokens of each branch through a gating fusion mechanism, where the gating mechanism controls the amount of information flow from the opponent branch. By stacking multiple DIM blocks, the model progressively enhances cross-view semantic alignment, driving the target representations under weak and strong augmentations to converge in the embedding space.

2. Hyperspherical Prototype Projection (HPP): Multi-Domain Generalization in Hyperspherical Prototype Space

When source domain data is inaccessible, directly adapting a pre-trained model to different target domains HTML-constructs inevitably leads to performance degradation. The HPP module projects the search features output by DIM onto a set of learnable prototypes, organizing multi-domain knowledge in a latent hyperspherical space. Specifically, after applying \(L_2\) normalization to the pooled search features, the similarity between each feature and \(K\) learnable prototypes is calculated to obtain a soft assignment matrix \(S\). An optimal transmission problem with an entropy regularization term is then solved using the Sinkhorn-Knopp algorithm to obtain a hardened soft assignment target \(Q\), ensuring that each prototype is equally allocated within a mini-batch. Based on this, HPP designs two complementary alignment strategies:

Domain-Specific Alignment (DSA): It is posited that augmented samples associated with the same weather should maintain consistent localization capabilities in both teacher and student networks. For \(M\) weather domains, the student's domain-specific similarity matrices \(S_m\) are constructed respectively, using the soft assignment \(Q_m\) provided by the teacher for the corresponding domain as pseudo-labels. A cross-entropy loss is employed to bring the teacher and student predictions closer within each domain.
Domain-Invariant Alignment (DIA): It further forces the representation distributions of multiple domains to be consistent in the hyperspherical space. First, k-means clustering is performed on the features output by DIM to obtain \(M\) cluster centers. Distance-based soft weights are then used to fuse the cluster-weighted features, yielding a domain-invariant representation \(X^*\), which is subsequently projected onto the same prototype space and aligned with the soft assignment target provided by the teacher using KL divergence. This cluster-weighted fusion enables the model to extract shared visual semantics from various weather domains, alleviating preference for any specific domain.

The combined effect of both alignments is: DSA ensures prediction consistency within each weather domain, while DIA ensures distribution consistency across domains, co-operating to enable stable generalization of the model under zero-source data constraints.

Loss & Training¶

The overall loss consists of three parts: the target supervision loss \(L_S\) of the tracking baseline (classification + \(L1\) localization + GIoU), the domain-specific alignment loss \(L_{DSA}\), and the domain-invariant alignment loss \(L_{DIA}\), with weighting coefficients fixed at 0.5 and 0.5 respectively.

Key Experimental Results¶

Main Results¶

Dataset	Metric	SFDATrack	Prev. SOTA (UMDATrack)	Gain
GOT-10k-Foggy	AO / SR₀.₇₅	70.1 / 67.1	66.6 / 62.2	+3.5 / +4.9
GOT-10k-Dark	AO / SR₀.₇₅	67.3 / 62.2	65.4 / 57.3	+1.9 / +4.9
GOT-10k-Rainy	AO / SR₀.₇₅	72.1 / 70.1	68.5 / 63.2	+3.6 / +6.9
NAT2021	AUC / Precision	56.78 / 73.60	54.58 / 70.78	+2.20 / +2.82
UAVDark70	AUC / Precision	61.22 / 76.51	60.05 / 73.35	+1.17 / +3.16
AVisT	AUC / Precision	60.87 / 59.70	60.50 / 59.01	+0.37 / +0.69

On the three adverse weather variants of the synthetic dataset GOT-10k, SFDATrack outperforms the previous best domain adaptation method across all metrics, with a particularly significant advantage in the high-precision localization metric \(SR_{0.75}\) (up to 6.9 percentage points). It also takes an all-around lead in real-world nighttime scenarios (NAT2021, UAVDark70) and diverse natural weather scenarios (AVisT).

Ablation Study¶

Configuration	AUC (%)	Precision (%)	Description
Baseline (w/o DIM / w/o HPP)	51.22	67.62	Basic Mean-Teacher
+ DIM	53.56	70.88	+2.34 AUC
+ HPP	53.24	70.37	+2.02 AUC
+ DIM + HPP (Full Model)	56.78	73.60	Best, synergy between both
HPP DSA Only	59.65	74.07	Domain-specific alignment only
HPP DIA Only	59.55	73.91	Domain-invariant alignment only
HPP DSA + DIA (Full)	61.22	76.51	Dual alignment complementarity
DIM with Transformer	56.77	70.45	Reference baseline
DIM with Bidirectional Mamba	60.33	75.60	Bidirectional > Unidirectional
DIM Full (Ours)	61.22	76.51	W/ prompt exchange + gated fusion

Key Findings¶

DIM is the more contributing module: adding DIM alone improves AUC by 2.34%, adding HPP alone improves AUC by 2.02%, and stacking both boosts performance by 5.56%—demonstrating a significant synergistic effect.
Both DSA and DIA are indispensable: using either strategy alone brings an improvement of about 1.3-1.7% AUC, but using them together reaches 61.22% AUC—showing that maintaining intra-domain prediction consistency and cross-domain distribution consistency are orthogonal and complementary dimensions.
In DIM, shifting from Transformer to unidirectional Mamba, then to bidirectional Mamba, and finally to the full DIM (including prompt exchange and gated fusion) brings stable improvements step by step.
The inference speed is 91 FPS, leading the efficiency among cross-domain methods that achieve the best accuracy, and is comparable to the lightweight tracker ARTrackV2 (95 FPS).

Highlights & Insights¶

First source-free domain adaptive framework for tracking: Introducing the SFDA setting to tracking, a task with strong temporal dependency, solves the unique challenge caused by accumulated pseudo-label drift between frames. It serves as a solid baseline for the intersection of VOT and SFDA.
Elegant and effective design of DIM: Interchanging search tokens from weak/strong augmentation branches and conducting bidirectional sequence interaction through a shared Mamba block, combined with gated fusion to control information flow—this approach of "gated viewpoint information exchange instead of simple concatenation" can be transferred to any dual-branch teacher-student framework.
Paradigm of hyperspherical prototype space + dual alignment: Using Sinkhorn-Knopp for uniform allocation + k-means clustering weight to construct domain-invariant representations avoids the hassle of manually designing domain fusion rules, achieving elegant multi-domain generalization under the constraint of zero source data.
Strong weather transfer capability: Not only does it lead by a wide margin on synthetic datasets (\(72.1\%\) AO and \(70.1\%\) \(SR_{0.75}\) on GOT-10k-Rainy), but it also comprehensively outperforms previous methods that require source data in real-world night (NAT2021/UAVDark70) and mixed weather (AVisT) scenarios, proving that source data is not indispensable in domain adaptive tracking.

Limitations & Future Work¶

The model was trained only on synthetic data from three weather domains (night/fog/rain); its generalization capability to more diverse real-world adverse weather conditions (e.g., sandstorms, blizzards, underwater) has not yet been verified.
Weighting coefficients \(\omega\) and \(\lambda\) were fixed at 0.5/0.5 across all experiments, lacking an adaptive adjustment mechanism—optimal weights for different weather domains might differ during real-world deployment.
No sensitivity analysis was conducted for the setting of prototype number \(K\). It remains an open question whether the prototypes can still allocate tasks effectively when the number of domains far exceeds \(K\).

vs UMDATrack (previous work by the same team): UMDATrack is a classic Unsupervised DA paradigm requiring source domain data for training; SFDATrack achieves better or comparable performance without looking at the source data at all, demonstrating that the SFDA setting is feasible and even superior in tracking.
vs General SFDA methods (such as SHOT): General SFDA focuses on static tasks like image classification or object detection. The design of SFDATrack specifically considers the spatial-temporal continuity constraints of tracking; the cross-frame feature exchange in the DIM module and the equivalent allocation constraint of Sinkhorn-Knopp are customized designs for the tracking task.
vs Transformer-based trackers: SFDATrack replaces Transformer with Mamba as the interaction module, benefiting from the linear complexity of SSM to achieve a real-time inference speed of 91 FPS with 93M parameters.

Rating¶

Novelty: ⭐⭐⭐⭐ The first work to introduce SFDA into VOT. Although the methodology design has a feeling of assembling modular components, the problem setting itself possesses significant practical value.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated against 10+ methods on 2 synthetic datasets (6 subsets) and 3 real-world datasets. Ablations cover every module and strategy branch, and analyses are provided for EMA frequency and loss weights.
Writing Quality: ⭐⭐⭐⭐ Clear structure and motivation, with adequate description of the method. However, some formula derivations (e.g., the notations in Eq. 3-4) are slightly convoluted upon first reading.
Value: ⭐⭐⭐⭐⭐ Scenarios where source data is unavailable are extremely common in deployment. This work provides a feasible and highly efficient baseline for source-free domain adaptive tracking, holding strong practical reference value.