FusionTrack: Collaborative Multi-Object Tracking with Arbitrary Multi-UAVs¶

Conference: ECCV 2026
Paper: CVF Open Access
Code: https://github.com/aircas501/FusionTrack
Area: Video Understanding
Keywords: Multi-View Multi-Object Tracking, Multi-UAV Tracking, Collaborative Tracking, Cross-View Association

TL;DR¶

Addressing arbitrary UAV fleet dynamics, frequent target entries/exits, and severe cross-view appearance variations, FusionTrack discards the conventional decoupled track-then-associate paradigm in favor of an end-to-end bidirectional fusion framework between Tracklet Memory and Trajectory Identity Pools, paired with View-aware Hierarchical Clustering to achieve state-of-the-art collaborative tracking.

Background & Motivation¶

Multi-object tracking (MOT) across coordinated UAV swarms is a core capability for low-altitude urban governance, traffic monitoring, and wide-area security surveillance. However, a single drone is severely bottlenecked by a narrow field-of-view (FoV) and high susceptibility to flight motion perturbations, rendering monocular observation prone to identity switches and tracking failures during severe occlusions. While multi-view multi-object tracking (MVMOT) expands spatial coverage via collaborative camera networks, deploying dynamic UAV swarms introduces open-world complexities: flight paths are unconstrained, view overlap dynamically shifts across frames, targets enter and exit camera boundaries unpredictably, and viewpoint elevation changes induce severe geometric and appearance distortions across views.

Existing MVMOT benchmarks (e.g., MvMHAT, CityFlow, DIVOTrack) predominantly assume static camera positions with fixed spatial overlap topologies, failing to reflect the unconstrained dynamic trajectories of airborne drone fleets. Methodologically, mainstream solutions follow a decoupled two-stage "track-then-associate" pipeline: each drone first executes single-view tracking (e.g., ByteTrack), after which cross-view tracklets are linked using ReID feature matching, homography estimation via image registration, or matrix factorization. These decoupled systems suffer from severe cascaded error propagation: single-view identity switches or false detections irreversibly corrupt global matching, while cross-view association cues cannot be propagated back to rectify upstream tracking drift.

To overcome these structural limitations, this paper leverages Transformer-based query representations to bridge single-view tracking and cross-view association into an end-to-end closed loop. Core idea: construct an end-to-end framework featuring bidirectional fusion between a Tracklet Memory Pool and a Trajectory Identity Pool (OFM) to let global multi-view identity representations backpropagate into tracking queries, complemented by View-aware Hierarchical Clustering (VHC) with spatial neighborhood filtering for robust association under arbitrary camera configurations.

Method¶

Overall Architecture¶

FusionTrack integrates monocular feature extraction, single-view detection and tracking decoders, a Tracklet Memory Pool (TMP), a Trajectory Identity Pool (TIP), an Object Fusion Module (OFM), and inference-stage View-aware Hierarchical Clustering (VHC).

Given synchronized frames \(I = \{I_t^c\}_{c=1,t=1}^{C,T}\) from \(C\) unconstrained UAV cameras, each view extracts multi-scale feature maps via a ResNet-50 backbone and Transformer encoder. Single-view tracking generates detection embeddings \(E_D^t\) via learnable queries \(Q_D^t\), concatenates them with prior track queries \(Q_T^{t-1}\), and processes them through the tracking decoder to obtain updated track queries \(Q_T^t\). Then, bidirectional fusion takes place: recent track queries buffered in TMP are aggregated through time-decayed attention and projected into ReID space to momentum-update the global TIP; conversely, global multi-view identity embeddings cached in TIP are projected back into tracking space to refine the single-view tracking queries, establishing a mutually reinforcing closed loop.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Synchronized Multi-UAV Frames<br/>C unconstrained dynamic views"] --> B["ResNet-50 + Encoder<br/>Multi-scale feature extraction"]
    B --> C["Single-View Detection & Tracking<br/>Iterative query decoding"]
    C --> D["Bidirectional Memory & Identity Fusion<br/>TMP temporal buffer & TIP momentum update"]
    D --> E["Cross-Frame & Cross-View Adaptation<br/>OFM bidirectional projection & feedback"]
    E --> F["View-aware Hierarchical Clustering<br/>VHC mask constraints + NFM + SWV"]
    F --> G["Collaborative Tracking Output<br/>Single-view tracklets & global identities"]

Key Designs¶

1. Bidirectional Memory and Identity Fusion: Unifying Short-term Tracking and Long-term Cross-View ReID

Single-view tracking queries specialize in smooth short-term spatio-temporal continuity, whereas cross-camera association requires view-invariant, identity-discriminative representations. To bridge this gap, FusionTrack introduces the Tracklet Memory Pool (TMP) to cache track queries over a temporal sliding window \(\tau\): \(\{Q_T^{t-\tau}, \ldots, Q_T^t\}\). A lightweight linear projection \(\phi_1\) decouples tracking space from identity space, after which a multi-layer FFN \(f\) aggregates the temporal sequence to extract the ReID feature \(F_{id}^t = f(\mathrm{Concat}(\{\phi_1(Q_T^{t-\tau}), \ldots, \phi_1(Q_T^t)\})) \in \mathbb{R}^d\). Concurrently, a Trajectory Identity Pool (TIP) caches global cross-view identity features using a momentum update rule: \(F_R^t = \mu F_R^{t-1} + (1-\mu) F_{id}^t\). This coordinated memory pool architecture decouples single-view and cross-view objectives while maintaining unified representations.

2. Cross-Frame and Cross-View Adaptation: Closed-Loop Contextual and Global Feedback

Within the Object Fusion Module (OFM), track queries first undergo cross-frame temporal aggregation. Current track queries \(Q_T^t\) attend to historical queries \(\{Q_T^{t-\tau}, \ldots, Q_T^{t-1}\}\) from the TMP using cross-attention modulated by a time-decay factor \(W = \exp(-\alpha \cdot \Delta t)\), where \(\Delta t \in [0, \tau]\) prioritizes fresh observations over stale temporal history. Subsequently, cross-view reverse fusion is performed: multi-view identity features cached in TIP are averaged across available views \(\mathrm{Avg}(F_R^t|_1^C)\), concatenated with the local track query, and projected back into tracking space via an inverse projection \(\phi_2\):

\[Q_T^{t''} = Q_T^t + \phi_2(\mathrm{Concat}(Q_T^t, \mathrm{Avg}(F_R^t|_1^C)))\]

This allows single-view tracking decoders to leverage global identity evidence from other unoccluded drones, directly rectifying tracking drift caused by severe monocular viewpoint changes or temporary occlusions.

3. View-Aware Hierarchical Clustering: Geometric Neighborhood Filtering and Temporal Voting

During inference, direct cosine distance clustering often suffers from error accumulation and many-to-one identity conflicts. View-aware Hierarchical Clustering (VHC) enforces two structural constraints: Rule 1 forbids intra-view matching by setting same-camera pairwise distances to \(+\infty\) (mask \(M_1\)), and Rule 2 guarantees that matched target pairs cannot re-match with other candidates from each other's views. A mutual top-\(k\) (\(k=5\)) neighbor candidate filter constructs a sparse graph \(M_2\), drastically cutting computation to \(O(N_f^2 \log N_f)\) via priority queue-accelerated agglomeration.

To eliminate cross-camera matching ambiguity among visually similar distractors (e.g., identical white sedans), a Neighborhood Filtering Mechanism (NFM) exploits local spatial neighborhood stability: a candidate pair \((i, j)\) is preserved only if the proportion of matched spatial neighbors between \(\mathcal{N}(i)\) and \(\mathcal{N}(j)\) exceeds threshold \(\delta > 50\%\). Finally, a Sliding-Window Voting (SWV) mechanism (window size 5) requires an identity switch to be endorsed by more than half of the recent temporal frames before approval, effectively suppressing transient cross-view identity flickering.

Loss & Training¶

The framework is optimized in a progressive three-stage curriculum on 4 NVIDIA A800 GPUs: stage 1 trains single-view tracking only for 20 epochs (\(\mathcal{L}_T = \mathcal{L}_{cls} + \mathcal{L}_{reg} + \mathcal{L}_{giou}\)); stage 2 activates the ReID objective for 20 epochs; stage 3 conducts full end-to-end joint fine-tuning for 60 epochs. The overall objective applies uncertainty-aware homoscedastic weighting:

\[\mathcal{L}_{total} = e^{-\omega_1}\mathcal{L}_T + e^{-\omega_2}\mathcal{L}_R + \omega_1 + \omega_2\]

where \(\mathcal{L}_R = \mathcal{L}_{id} + \mathcal{L}_{trip} + \mathcal{L}_{rc}\). To prevent dimensional collapse and feature distortion across the bidirectional projection mappings \(\phi_1\) and \(\phi_2\), a bidirectional cycle-reconstruction regularization loss is introduced:

\[\mathcal{L}_{rc} = \frac{1}{N} \left( \| Q_T^t' - \phi_2(\phi_1(Q_T^t')) \|_2^2 + \| R_T^t - \phi_1(\phi_2(R_T^t)) \|_2^2 \right)\]

ensuring consistency and mutual invertibility between tracking and ReID embedding manifolds.

Key Experimental Results¶

Main Results¶

Evaluation on the newly proposed unconstrained drone benchmark MDMOT and four established multi-view benchmarks (CAMPUS, WILDTRACK, MvMHAT, DIVOTrack) demonstrates that FusionTrack sets a new state of the art across both overlapping and non-overlapping camera settings.

Dataset	Method	CVMA (↑)	CVIDF1 (↑)	Note
MDMOT	OSNet	50.1	45.5	Standard appearance ReID baseline
MDMOT	Strong	52.3	51.4	Deep person ReID bag-of-tricks
MDMOT	Citytrack	57.3	56.1	Multi-camera vehicle tracker
MDMOT	MvMHAT	78.8	63.7	Self-supervised multi-view tracker
MDMOT	CrossMOT	78.5	72.9	Collaborative tracking baseline
MDMOT	GMT	79.9	73.8	Previous SOTA multi-view tracker
MDMOT	FusionTrack (Ours)	81.8	75.5	Outperforms GMT by +1.9 / +1.7
CAMPUS	GMT / Ours	66.4 / 68.6	66.8 / 67.5	Gain +2.2 / +0.7
WILDTRACK	GMT / Ours	61.7 / 62.5	72.0 / 72.3	Gain +0.8 / +0.3
MvMHAT	GMT / Ours	94.1 / 94.8	95.6 / 95.9	Gain +0.7 / +0.3
DIVOTrack	GMT / Ours	74.5 / 74.8	73.2 / 77.5	Substantial CVIDF1 gain +4.3

Ablation Study¶

Ablation analysis on MDMOT isolates the contribution of each training component (cross-frame fusion, cross-view fusion, cycle-reconstruction loss) and inference design (VHC clustering, SWV voting, NFM filtering), corresponding to Table 4 in the original paper:

Stage / Configuration	Training Components	Inference Strategy	CVMA (↑)	CVIDF1 (↑)	Note
Baseline Model	No cross-frame / cross-view / \(\mathcal{L}_{rc}\)	VHC + SWV + NFM	76.5	66.8	Substantial drop without fusion (-5.3 / -8.7)
+ Cross-Frame Fusion	Time-decayed cross-attention	VHC + SWV + NFM	78.8	72.2	Temporal cues boost tracking continuity
+ Cross-View Fusion	Cross-frame + cross-view feedback	VHC + SWV + NFM	79.5	74.3	Multi-view complementarity improves queries
Full Training (w/o Post-processing)	Full training pipeline	Raw VHC only	77.1	72.3	Ambiguous associations degrade performance
+ SWV Voting	Full training pipeline	VHC + SWV	79.2	74.1	Temporal voting suppresses spurious switches
Baseline Bipartite Matching	Full training pipeline	Hungarian + SWV + NFM	78.5	73.3	Bipartite matching falls short of VHC
Full Model	Cross-frame + Cross-view + \(\mathcal{L}_{rc}\)	VHC + SWV + NFM	81.8	75.5	Optimal performance across all metrics

Key Findings¶

In monocular single-view tracking (SVT) on MDMOT (Table 3), FusionTrack ranks first among Transformer trackers with 88.75 MOTA, 89.22 MOTP, 93.06 IDF1, 84.88 HOTA, and achieves the lowest ID switches (81 vs. 202 for TransTrack and 101 for COMOT), verifying that cross-view feedback directly enhances single-camera trajectory stability.
The bidirectional reconstruction loss \(\mathcal{L}_{rc}\) plays a crucial role in preventing manifold degradation between tracking and ReID embeddings, boosting CVMA from 79.5 to 81.8 (+2.3) and CVIDF1 from 74.3 to 75.5 (+1.2).
Combining geometric Neighborhood Filtering (NFM) and Sliding-Window Voting (SWV) resolves ambiguous pairings among visually identical targets, yielding an aggregate improvement of +4.7 CVMA and +3.2 CVIDF1 over unconstrained VHC clustering.

Highlights & Insights¶

Closed-Loop Bidirectional Feature Propagation: Unlike traditional one-way tracking-to-ReID pipelines, FusionTrack projects multi-camera global identity embeddings back into single-view track queries, allowing cross-view consensus to dynamically rescue occluded single-camera tracklets.
Topological Spatial Neighbor Consistency (NFM): Leveraging the spatial adjacency structure of adjacent vehicles or pedestrians as a geometric verification constraint provides an elegant inductive bias to break visual ambiguity among identical vehicle makes and colors.
First Systematic Arbitrary-View Fleet Benchmark: Introducing MDMOT (122k frames, 24.81M boxes, 20.34 average targets per frame across 3–5 drones) provides the research community with the first realistic testbed for unconstrained, dynamically shifting multi-UAV tracking.

Limitations & Future Work¶

Throughput and Latency Constraints: Operating at 3.9 FPS on a single NVIDIA A800 GPU, the computational latency is dominated by multiple single-view Transformer decoders (>80% of runtime), presenting a challenge for real-time edge deployment on embedded UAV hardware.
Sensitivity to Extreme Drone Manuevers: Rapid pitch and roll maneuvers or heavy sensor noise during night flights can distort local spatial geometry, potentially degrading the neighbor set consistency required by NFM.
Future Directions: Exploring parallel multi-view query scheduling, knowledge distillation for real-time inference, and fusing UAV onboard GPS/IMU flight telemetry to provide coarse 3D spatial priors.

vs. Monocular Trackers (ByteTrack, FairMOT): Monocular approaches inevitably fail when targets exit single-drone FoV or endure prolonged tree cover; FusionTrack bridges multi-drone boundaries through the Trajectory Identity Pool, achieving instant target re-acquisition upon re-entry.
vs. Two-Stage Multi-Camera Methods (GMT, CrossMOT): Conventional multi-view methods rely heavily on strict homography approximations or discrete graph matching; FusionTrack eliminates rigid homography assumptions through end-to-end query interaction and view-aware hierarchical clustering.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Pioneers arbitrary multi-UAV collaborative tracking with an end-to-end bidirectional query-identity fusion framework.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks across MDMOT and 4 external multi-view datasets with rigorous SVT and MVMOT ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Cohesive narrative structure, clear mathematical formulations, and well-designed visualizations.
Value: ⭐⭐⭐⭐☆ Highly impactful for low-altitude drone swarm perception, open-source codebase, and urban intelligence systems.