TR-MoE: Temporal Reliability-Aware Mixture-of-Experts for Robust Tracking¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/Avant1v/TR-MoE
Area: Video Understanding
Keywords: Visual Object Tracking, Mixture-of-Experts, State Space Models, Temporal Reliability, Kolmogorov–Arnold Networks
TL;DR¶
Addressing the complementary nature of SAM2's fine-grained segmentation and discriminative trackers' global semantic robustness under occlusion, TR-MoE reformulates multi-tracker fusion as a temporal sequence modeling problem, employing a Mamba-based trajectory refiner and a KAN-enhanced disentangled head to achieve state-of-the-art results on LaSOT and GOT-10k.
Background & Motivation¶
Visual object tracking has long been anchored by discriminative paradigms, progressing from Siamese correlation networks to modern one-stream Vision Transformers such as OSTrack and LoRAT. Leveraging global semantic representations, these discriminative trackers excel at target re-identification amid severe occlusion, out-of-view disappearance, and extreme motion blur. Concurrently, the advent of vision foundation models, notably Segment Anything Model 2 (SAM2), has revolutionized video perception by delivering precise zero-shot mask predictions and robust distractor suppression via spatio-temporal memory. However, SAM2 remains prone to catastrophic tracking loss when targets experience extended occlusion. The distinct, complementary failure modes and inductive biases of these two paradigms make a Mixture-of-Experts (MoE) fusion framework an appealing direction.
Nevertheless, orchestrating heterogeneous tracking experts presents a core dilemma. Existing multi-tracker combination methods rely primarily on Kalman filtering, hand-crafted geometric heuristics, or static per-frame confidence scores. These approaches suffer from fundamental short-sightedness: by relying solely on current-frame instantaneous responses, they remain blind to the temporal dynamics of tracking error accumulation. In practice, tracker drift is rarely an isolated single-frame anomaly; it manifests as a progressive temporal deviation. Lacking a continuous modeling of error histories, inter-expert consistency trends, and trajectory inertia, conventional fusion mechanisms fail to perform timely arbitration during sudden target loss, frequently causing failure modes to propagate across subsequent frames.
The key insight of this paper is to reframe multi-expert tracker fusion as a time-series state trajectory refinement problem. Core idea: TR-MoE explicitly models the temporal evolution of expert error regimes through a linear-complexity Mamba selective state space model, coupled with a Kolmogorov–Arnold Network (KAN) decoupled head that factorizes trajectory refinement into directional offsets, dynamic scales, and adaptive reliability weights.
Method¶
Overall Architecture¶
For each frame, TR-MoE ingests the candidate bounding boxes from both SAM2 and a base discriminative tracker, assembling an 11-dimensional state vector that captures relative coordinate offsets, inter-expert IoU consensus, and normalized target dimensions across a sliding temporal window of \(N=32\) frames. The end-to-end refinement workflow operates in three distinct phases: first, an input KAN projection layer nonlinearly maps the low-dimensional state sequence into a latent representation manifold; second, stacked Mamba-based Trajectory Refiner (MTR) blocks continuously update hidden state dynamics to model long-range error accumulation with linear complexity; third, a KAN-enhanced disentangled head independently predicts directional offsets, dynamic scale suppression factors, and confidence scores for each expert branch, deriving the final fused bounding box through reliability-weighted aggregation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Historical Sequence<br/>SAM2 & Tracker relative offsets + IoU + scale"] --> B["KAN Feature Embedding<br/>Edge-based learnable splines mapping to latent manifold"]
B --> C["Mamba Trajectory Refiner<br/>Selective state space modeling long-term error dynamics"]
C --> D["KAN Disentangled Refinement Head<br/>Factorized direction / dynamic scale / reliability scores"]
D --> E["Reliability-Aware Mixture Fusion<br/>Adaptive gated routing producing refined bounding box"]
Key Designs¶
1. KAN Feature Embedding: Mapping Nonlinear Error Manifolds Conventional tracker fusion typically applies standard linear layers for input feature projection. However, in low-dimensional geometric state sequences, sudden drops in inter-expert IoU correlate with subtle spatial bounding box jitters in a highly nonlinear manner that rigid weight matrices struggle to approximate efficiently. TR-MoE introduces a KANLinear layer coupled with LayerNorm as the input interface. Grounded in the Kolmogorov–Arnold representation theorem, this module places learnable univariate B-spline activations directly on network edges. It projects the 11-dimensional state vector \(X_t = [\Delta B_{sam}, \Delta B_{track}, \mathcal{C}_t, S_{t-1}]\) into a 128-dimensional latent space \(E_t\), where \(\Delta B\) captures the spatial displacement relative to the previous frame's fused result, \(\mathcal{C}_t = \text{IoU}(B_{sam}, B_{track})\) serves as an instantaneous consensus gate, and \(S_{t-1}\) provides scale context. This edge-based parameterization captures complex geometric-confidence interactions with minimal parameter overhead.
2. Mamba Trajectory Refiner: Long-Horizon Error Evolution Modeling Static fusion mechanisms are easily fooled by transient detection noise. The MTR module stacks \(L=4\) residual Mamba blocks to model temporal dependencies via discretized selective state space equations: $\(h_t = \overline{\mathbf{A}}_t h_{t-1} + \overline{\mathbf{B}}_t x_t, \quad y_t = \mathbf{C}_t h_t\)$ Unlike linear time-invariant systems, Mamba's discretization parameters \(\Delta_t, \mathbf{B}_t, \mathbf{C}_t\) are input-dependent functions of the current state \(x_t\). When the tracking trajectory is stable and inter-expert IoU remains high, the system automatically shortens the timescale parameter \(\Delta_t\), resetting historical hidden states to prioritize immediate local motion cues. Conversely, when severe trajectory discrepancies or abrupt drops in consensus occur, the model dynamically expands \(\Delta_t\) to assimilate broader historical contexts across up to 32 frames. This enables MTR to reliably differentiate between benign short-term occlusions and irreversible systematic drift while retaining \(O(N)\) linear inference speed.
3. KAN Disentangled Refinement Head: Orthogonal Decomposition of Corrections Directly predicting refined four-coordinate bounding box values using unified dense layers leads to severe coupling among positional shifts, scale variations, and expert credibility. TR-MoE introduces a KAN-enhanced disentangled head that factorizes the geometric update into three independent components for each expert \(k \in \{\text{SAM2}, \text{Tracker}\}\). Specifically, directional vectors are regressed via B-spline KAN branches \(\mathbf{d}_k = \mathcal{K}_{dir}^{(k)}(H_t^{(L)})\), dynamic magnitude factors are computed via scalar gates \(m_k = \sigma(\text{Linear}(H_t^{(L)}))\), and the refined displacement is modulated as \(\Delta B_k = m_k \mathbf{d}_k\). Concurrently, a separate KAN scoring head predicts reliability confidence \(s_k = \mathcal{K}_{score}^{(k)}(H_t^{(L)})\), normalized through Softmax to yield dynamic fusion weights \([w_{sam}, w_{track}]\). This orthogonal separation allows the system to instantly suppress a drifting tracker by collapsing its magnitude \(m_k\) and weight \(w_k\) toward zero.
A Worked Example¶
Consider a challenging sequence where a target is partially occluded by dense foliage at frame 980. SAM2 experiences subtle boundary slippage due to local edge confusion, while the discriminative LoRAT tracker remains locked onto the global semantic target; the consensus score \(\mathcal{C}_t\) plunges from 0.85 to 0.32. MTR observes this persistent conflict across consecutive frames, expanding its selective timescale \(\Delta_t\) to inspect past trajectory continuity. By frame 982, LoRAT severely drifts onto a visually similar distractor in the background. Recognizing this abrupt structural departure against the 32-frame historical motion baseline, MTR identifies LoRAT's systemic failure mode. The KAN scoring head slashes LoRAT's confidence \(s_{track}\) from 0.80 down to 0.15 while boosting \(s_{sam}\) to 0.85, seamlessly shifting the fusion weight \(w_{sam}\) from 0.30 to 0.88. Simultaneously, the dynamic scale factor \(m_{track}\) is dampened, preventing distractor contamination and keeping tracking performance smooth and robust.
Loss & Training¶
The overall learning objective balances four distinct terms: $\(\mathcal{L}_{total} = \mathcal{L}_{main} + \lambda_{aux}\mathcal{L}_{aux} + \lambda_{score}\mathcal{L}_{score} + \lambda_{reg}\mathcal{L}_{reg}\)$ with balancing hyperparameters empirically tuned to \(\lambda_{aux} = 0.5, \lambda_{score} = 2.0, \lambda_{reg} = 5.0\). - Main Regression Loss \(\mathcal{L}_{main}\): Evaluated solely on the fused output \(\hat{B}_{fused}\) using a combined L1 loss (weight 5.0) and GIoU loss (weight 2.0). A global validity mask \(\mathcal{M}_{valid} = [\max(\text{IoU}_{sam}, \text{IoU}_{track}) > \tau_{valid}]\) gates out samples where both experts have lost the target, guarding gradients against unrecoverable tracking noise. - Selective Auxiliary Supervision \(\mathcal{L}_{aux}\): Incorporates a "teachability" threshold \(\tau_{teach}\), applying bounding box loss to individual expert heads only when their raw predictions retain sufficient overlap with the ground truth, thus preventing heads from hallucinating corrections on lost targets. - Reliability Alignment Loss \(\mathcal{L}_{score}\): Supervised via mean squared error to force predicted confidence scores \(\hat{s}_k\) to match the true IoU between raw tracker candidates and ground truth. - Scale Regularization \(\mathcal{L}_{reg}\): An L2 penalty \(||\Delta S||_2^2\) imposed on scale factors to enforce minimal intervention.
Key Experimental Results¶
Main Results¶
Evaluation is conducted across three benchmarks: LaSOT (long-term), LaSOText (complex attributes), and GOT-10k (zero-overlap short-term). The primary TR-MoE baseline pairs SAM2-T with LoRAT-L378.
| Dataset | Metric | TR-MoE (Ours) | LoRAT-L378 | SAM2-T | MCITrack-L384 | SPMTrack-L378 | Relative Gain (vs Best Base) |
|---|---|---|---|---|---|---|---|
| LaSOT | AUC (%) | 77.3 | 75.1 | 67.3 | 76.6 | 76.8 | +0.5 (+2.2 vs LoRAT) |
| LaSOT | PNorm (%) | 86.5 | 82.0 | 74.2 | 86.1 | 85.9 | +0.4 (+4.5 vs LoRAT) |
| LaSOT | P (%) | 85.3 | 84.1 | 71.3 | 85.0 | 84.0 | +0.3 (+1.2 vs LoRAT) |
| LaSOText | AUC (%) | 58.4 | 56.6 | 52.4 | 55.7 | - | +1.8 vs LoRAT |
| LaSOText | PNorm (%) | 70.8 | 69.0 | 61.8 | 66.5 | - | +1.8 vs LoRAT |
| LaSOText | P (%) | 68.3 | 65.1 | 59.7 | 62.9 | - | +3.2 vs LoRAT |
| GOT-10k | AO (%) | 81.4 | 77.5 | 78.5 | 80.0 | 80.0 | +1.4 (+2.9 vs SAM2) |
| GOT-10k | SR0.5 (%) | 91.9 | 86.2 | 88.9 | 88.5 | 89.4 | +2.5 (+3.0 vs SAM2) |
| GOT-10k | SR0.75 (%) | 78.8 | 78.1 | 72.4 | 80.2 | 79.9 | -1.4 vs MCITrack |
Ablation Study¶
All ablations are performed on LaSOT utilizing the combination of SAM2-T and LoRAT-B224.
Table 1: Architectural Component Ablation
| Config # | Embedding | Backbone | Head | AUC (%) | PNorm (%) | P (%) |
|---|---|---|---|---|---|---|
| A | Linear | MLP | Linear | 72.54 | 81.49 | 78.76 |
| B | Linear | Transformer | Linear | 74.34 | 83.85 | 79.82 |
| C | Linear | Mamba | Linear | 74.53 | 83.94 | 80.84 |
| Ours | KAN | Mamba | KAN | 75.22 | 84.50 | 81.70 |
Table 2: Decoupled Head, Dynamic Scale, and Adaptive Fusion Ablation
| Config # | Head Design | Scale Modeling | Fusion Mechanism | AUC (%) | PNorm (%) | P (%) | Note |
|---|---|---|---|---|---|---|---|
| A | Coupled | Implicit | Average | 71.01 | 79.57 | 75.83 | Naive static baseline |
| B | Decoupled | Implicit | Average | 72.48 | 81.35 | 78.55 | Direction/magnitude separation (+1.47% AUC) |
| C | Decoupled | Implicit | Adaptive | 74.82 | 83.75 | 81.04 | Dynamic routing (+2.34% AUC) |
| D | Decoupled | Dynamic | Average | 72.62 | 82.16 | 79.43 | Explicit dynamic scale adjustment |
| E | Coupled | Implicit | Adaptive | 74.27 | 83.13 | 80.32 | Adaptive routing without head decoupling |
| Ours | Decoupled | Dynamic | Adaptive | 75.22 | 84.50 | 81.70 | Full proposed architecture |
Key Findings¶
- Replacing an MLP baseline with the selective state space Mamba backbone boosts AUC from 72.54% to 74.53%, surpassing a comparable Transformer encoder (74.34%) and verifying that continuous state space models offer superior inductive bias for low-dimensional error tracking.
- Employing KAN parameterization across both input embedding and prediction heads contributes a notable +0.69% AUC improvement over standard linear layers, highlighting the efficacy of edge-based B-splines in mapping nonlinear geometric residuals.
- Generalizability trials across diverse trackers show that even when fused with lightweight SGLATrack (AUC 63.0%), TR-MoE reaches 70.5% AUC—outperforming standalone SAM2 by 3.2% without degrading into the weaker expert's inaccuracies.
Highlights & Insights¶
- Reframing Tracker Fusion as Temporal Dynamics Modeling: Moves beyond isolated per-frame confidence metrics by tracking historical inter-expert consistency trends, endowing the framework with proactive error mitigation capabilities.
- Synergy Between SSM and KAN in Geometric Time-Series: Bridges the gap between linear-time long-context sequence modeling (Mamba) and edge-parameterized nonlinear function decomposition (KAN), achieving high performance with minimal inference overhead.
- Teachability-Gated Multi-Task Objective: Incorporates teachability thresholds and validity masks into the regression loss, preventing gradient destabilization and hallucinated bounding box corrections during total tracking failure.
Limitations & Future Work¶
- Throughput Bottleneck of Upstream Trackers: While the MTR module operates with negligible latency, overall runtime remains bounded by the heaviest expert (notably SAM2's mask decoder). Future work could integrate compressed or distilled expert models.
- Dual-Expert Paradigm Constraints: Current formulations explicitly focus on two complementary tracker streams; extending this formulation to arbitrary N-expert pools (including optical flow and depth estimators) remains an open challenge.
- Two-Stage Decoupled Training: TR-MoE operates as a trajectory refinement framework atop frozen experts; end-to-end joint fine-tuning of base tracking backbones with MTR could unlock further gains.
Related Work & Insights¶
- vs SAMURAI / HiM2SAM: While SAMURAI and HiM2SAM introduce heuristic motion priors into SAM2's internal memory bank, they remain restricted to a single model's representational ceiling. TR-MoE leverages external discriminative experts to resolve fundamental tracking breakdown.
- vs SPMTrack / LoRATv2: SPMTrack embeds MoE layers directly into the Transformer backbone for parameter-efficient fine-tuning (PEFT), representing an intra-model representation mixture. TR-MoE functions at the task decision level as an inter-model temporal MoE, presenting a complementary and modular architecture.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ [Pioneers temporal error evolution modeling for tracker fusion using Mamba and KAN]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across LaSOT, LaSOText, and GOT-10k with thorough ablations and 5-tracker generalizability analysis]
- Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous formulation, clear narrative, and detailed mathematical exposition]
- Value: ⭐⭐⭐⭐☆ [Offers an actionable, generalizable blueprint for fusing vision foundation models with classical discriminative algorithms]