title: >- [Paper Note] Instance Segmentation as Tracking: A New Paradigm for Multi-Small-Object Tracking with Event Cameras description: >- [ECCV 2026][3D Vision][Event Camera Tracking] Reformulating event-based multi-small-object tracking as 3D spatio-temporal instance segmentation via sparse convolutions and geometric graph clustering. tags: - ECCV 2026 - 3D Vision - Event Camera - Multi-Object Tracking - Instance Segmentation date: 2026-09-19 content_hash: 48533857f907ca90

Instance Segmentation as Tracking: A New Paradigm for Multi-Small-Object Tracking with Event Cameras¶

Conference: ECCV 2026
Paper: ECCV 2026
Area: 3D Vision
Keywords: Event Camera / Multiple Small Object Tracking / Spatio-Temporal Point Cloud / Instance Segmentation / 3D Sparse Convolution

TL;DR¶

To overcome temporal discretization and heavy background redundancy in traditional event-based "convert-then-detect-and-track" paradigms, this paper reformulates multi-small-object tracking as an instance segmentation task on 3D spatio-temporal event point clouds, introducing Ev-ISNet with fine-grained 3D sparse convolutions and geometry-consistent graph clustering for high-accuracy, streaming tracking.

Background & Motivation¶

Multiple small object tracking (MSOT) is of paramount importance across critical applications, including anti-UAV defense systems, perimeter intelligent surveillance, and low-altitude airspace monitoring. However, conventional tracking frameworks relying on visible RGB or infrared sensors remain constrained by limited dynamic ranges (typically ~60 dB) and low exposure frame rates (30–60 Hz). When tracking fast-maneuvering small objects under extreme dynamic illumination, intense backlighting, or heavily cluttered backgrounds, these systems suffer from severe motion blur and overexposure, causing fragile object signals to vanish and trajectories to fragment. Neuromorphic event cameras overcome these physical bottlenecks by asynchronously outputting brightness change events with over 120 dB dynamic range and microsecond temporal resolution (up to 1 MHz), providing continuous trajectories of fast-moving targets.

Nevertheless, the vast majority of existing event-based trackers (such as RVT or SpikeYOLO combined with SORT, DeepSort, or TrackTrack) strictly adhere to the conventional "convert-then-detect-and-track" paradigm inherited from video vision. They collapse continuous event streams into dense voxel grids or count frames over fixed temporal windows, subsequently executing standard 2D detectors and association filters. This paradigm encounters three severe dilemmas when dealing with high-speed micro-targets: first, projecting asynchronous events onto frame-like grids causes severe temporal downsampling and breaks the microsecond continuity of trajectories; second, tiny targets typically occupy only a few pixels and lack distinguishable appearance cues such as texture, color, or shape, rendering re-identification networks powerless; third, framing highly sparse events injects immense background redundancy, dramatically inflating computational overhead and preventing low-latency streaming deployment on edge devices.

Conversely, directly borrowing off-the-shelf 3D point cloud instance segmentation models (e.g., PointGroup, HAIS) fails due to two fundamental architectural mismatches: conventional 3D instance segmentation heavily presumes compact, centroid-centered object geometries, whereas spatio-temporal event streams generated by fast targets manifest as slender, elongated 3D curves whose geometric centroids fall outside the true trajectory; furthermore, traditional 3D segmentation architectures are designed for static, offline batch processing, fundamentally clashing with the causal, streaming requirements of real-time object tracking. Core Idea: Reformulate event-based MSOT as an instance segmentation task on 3D spatio-temporal event point clouds, proposing Ev-ISNet to extract voxel features and predict local motion directions via fine-grained 3D sparse convolutions, followed by geometric-consistency-driven graph association and union-find clustering for efficient online streaming tracking.

Method¶

Overall Architecture¶

Ev-ISNet circumvents the detached "bounding box detection followed by frame-to-frame matching" pipeline by partitioning incoming continuous event streams into sequential packets of fixed temporal duration ($\Delta T = 800\text{ ms}$). For each packet, the network discretizes raw events into an ultra-fine non-overlapping 3D sparse voxel grid of $1\text{ pixel} \times 1\text{ pixel} \times 1\text{ ms}$, preserving complete temporal resolution. A 3D sparse U-Net with submanifold sparse convolutions extracts point-wise voxel features and concurrently predicts foreground semantic confidence and local motion direction vectors. On the filtered foreground events, an intra-packet association graph is constructed to assess edge-wise instance affinities via a lightweight Geometric Consistency (GC) network, followed by union-find clustering into trajectory subgraphs. Finally, an inter-packet association module links adjacent packets along temporal interfaces using candidate edges and the shared GC network to produce continuous, causal streaming tracking outputs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Continuous Event Stream Packet<br/>(x, y, t, p)"] --> B["3D Sparse Voxel & Dual-Branch Prediction<br/>1×1 pixel×1ms grid + semantic/direction branches"]
    B --> C["Intra-Packet Geometry-Consistent Graph Association<br/>k-NN graph + relative geometry features + Union-Find"]
    C --> D["Inter-Packet Graph Association & Streaming Aggregation<br/>adjacent packet candidate edges + weight-shared GC network"]
    D --> E["Streaming Spatio-Temporal Trajectory Segmentation<br/>event-wise tracking labels"]

Key Designs¶

1. Ultra-Fine 3D Sparse Voxel Representation with Dual-Branch Prediction: Preserving Temporal Fidelity and Motion Continuity
To circumvent the information loss and computational bloat of dense 50 ms framing, Ev-ISNet maps asynchronous events $E = \{(x_k, y_k, t_k, p_k)\}$ into a non-overlapping sparse voxel grid at an ultra-fine resolution of $1\text{ pixel} \times 1\text{ pixel} \times 1\text{ ms}$. Utilizing submanifold sparse convolutions, the 3D sparse U-Net backbone processes active voxels exclusively, achieving high efficiency while retaining microsecond temporal fidelity. The extracted event features $F_{\text{event}}$ feed two lightweight, two-layer MLP branches: the semantic branch outputs foreground target probabilities to filter out background noise at threshold $\tau_{\text{object}} = 0.5$, while the direction branch estimates local 3D motion tangent vectors. Ground-truth motion directions are derived from the normalized principal eigenvector of the local $k$-NN covariance matrix: $$C = \sum_{e_j \in \text{kNN}(e)} (e_j - \bar{e})(e_j - \bar{e})^T$$ By discretizing 3D directions into azimuth and elevation classification targets, the network explicitly provides geometric momentum cues to reliably extrapolate trajectories across occlusions, trajectory intersections, and sparse detection gaps.

2. Intra-Packet Geometry-Consistent Graph Association: Overcoming Centroid Assumptions for Slender Trajectories
Conventional 3D segmentation approaches rely on spatial Euclidean proximity and center regression, which inevitably cause fragmentation when applied to slender, long-range event trajectories stretching across hundreds of milliseconds. To overcome this, Ev-ISNet establishes an undirected $k$-NN graph ($k=15$) among foreground events, generating $k \cdot N$ candidate edges. A lightweight Geometric Consistency (GC) network, instantiated as a two-layer MLP, evaluates edge-wise instance affinity based on feature differences $\Delta F_{ij} = F_i - F_j$, which encode relative spatial displacement $\Delta L$ and relative motion direction deviation $\Delta D$: $$p_{ij} = \sigma(\text{MLP}(\Delta F_{ij}))$$ Ablation experiments verify that relative geometric features offer robust translation invariance and noise immunity compared to absolute coordinates. Edges satisfying $p_{ij} \ge \tau_{\text{edge}} = 0.5$ are retained, and connected subgraphs are extracted using an efficient Union-Find data structure, allowing instance identities to diffuse naturally along curved trajectories.

3. Inter-Packet Graph Association and Streaming Union-Find Aggregation: Enabling Online Causal Tracking
To transition from batch-wise static processing to online streaming tracking without heavy recurrent state-space models, Ev-ISNet incorporates an inter-packet association mechanism. Between consecutive packets $G_t = (V_t, E_t)$ and $G_{t+1} = (V_{t+1}, E_{t+1})$, candidate cross-packet edges $E_{t, t+1} \subseteq V_t \times V_{t+1}$ are identified via localized boundary $k$-NN search. The GC network, fully sharing weights with the intra-packet module, predicts edge affinities across temporal packet interfaces, followed by Union-Find operations to merge identical trajectory IDs incrementally. This design guarantees deterministic, high-throughput causal tracking with near-zero latency degradation.

Loss & Training¶

The entire network is trained end-to-end using a multi-task joint objective: $$\mathcal{L} = \mathcal{L}_{\text{seg}} + \alpha \mathcal{L}_{\text{direction}} + \beta \mathcal{L}_{\text{edge}}$$ where $\mathcal{L}_{\text{seg}}$ is the binary cross-entropy (BCE) loss on foreground semantic labels, $\mathcal{L}_{\text{direction}}$ is the cross-entropy loss over discretized motion direction classes evaluated exclusively on target events masked by $M_{\text{obj}}$, and $\mathcal{L}_{\text{edge}}$ is the BCE loss on candidate edge affinities. Hyperparameters are set to $\alpha = \beta = 1$. Optimization uses Adam with a batch size of 1 for 50 epochs, featuring a 5-epoch linear warmup from initial rates to $1 \times 10^{-2}$, followed by cosine annealing decay down to $1 \times 10^{-3}$.

Key Experimental Results¶

Main Results¶

On the comprehensive EV-UAV-Track benchmark (147 event sequences, 2.3M+ annotated events, average object size $6.8 \times 5.4$ pixels), Ev-ISNet was evaluated against 16 state-of-the-art methods spanning frame-based trackers, event detectors paired with tracking filters, spiking neural network (SNN) trackers, and 3D point cloud instance segmenters. For processing an 8-second sequence, results are as follows:

Method	Event Rep.	MOTA (%) ↑	MOTP (px) ↓	IDF1 (%) ↑	mCov (%) ↑	mIoU (%) ↑	#Params (MB) ↓	Runtime (s) ↓
CenterTrack	Dense + Short	26.80	8.92	16.61	12.22	13.23	114.1	4.13
FairMOT	Dense + Short	31.72	7.35	12.58	11.02	12.18	247.3	3.53
ByteTrack	Dense + Short	29.83	7.91	18.67	8.89	9.52	68.5	3.12
MOTIP	Dense + Short	33.89	8.01	16.28	13.35	12.36	465.8	6.87
RVT + SORT	Dense + Short	37.08	6.23	21.71	22.17	17.68	9.9	1.92
RVT + TrackTrack	Dense + Short	36.91	6.99	23.16	22.20	17.01	319.2	2.24
SpikeYOLO + DeepSort	SNN + Short	34.88	6.87	23.87	24.12	17.02	72.1	2.56
SNNTracker	SNN + Long	35.20	5.53	26.11	24.30	17.19	24.8	3.18
PointGroup	Sparse + Long	51.19	2.38	29.86	30.72	20.18	29.6	1.98
HAIS	Sparse + Long	52.71	1.95	30.19	32.35	22.29	353.2	1.57
OneFormer3D	Sparse + Long	55.09	1.81	33.21	31.21	30.18	220.4	1.87
Ev-ISNet (Ours)	Sparse + Long	64.48	1.29	48.73	37.12	36.49	7.73	1.48

On the challenging real-world Bee Swarm benchmark (an active swarm averaging 80 bees per moment against wind-blown tree clutter), Ev-ISNet achieves 31.5% MOTA and 28.9% IDF1, decisively outperforming HAIS (23.1% MOTA) and SNNTracker (22.1% MOTA).

Ablation Study¶

1. Component Contribution to Tracking Performance

Config	Dir.	Intra.	Inter.	Online	MOTA (%) ↑	MOTP (px) ↓	IDF1 (%) ↑	mIoU (%) ↑
Baseline (HAIS style)	-	-	-	No	52.71	1.95	30.19	22.29
+ Dir.	✓	-	-	No	53.89	1.76	32.23	24.16
+ Intra.	-	✓	-	No	60.84	1.30	46.81	34.98
+ Intra. + Inter.	-	✓	✓	Yes	60.85	1.31	46.82	34.99
Full Model (Ev-ISNet)	✓	✓	✓	Yes	64.48	1.29	48.73	36.49

2. Impact of Geometric Features on Edge Prediction

Config	Loc $L$	Rel Loc $\Delta L$	Dir $D$	Rel Dir $\Delta D$	MOTA (%) ↑	MOTP (px) ↓	IDF1 (%) ↑	mIoU (%) ↑
Abs Loc	✓	-	-	-	55.73	1.98	42.04	26.37
Rel Loc	-	✓	-	-	60.85	1.31	46.82	34.99
Abs Dir	-	-	✓	-	42.35	2.56	38.86	14.23
Rel Dir	-	-	-	✓	45.23	2.13	40.75	17.32
Relative Pair ($\Delta L + \Delta D$)	-	✓	-	✓	64.48	1.29	48.73	36.49
All Features	✓	✓	✓	✓	58.22	1.43	40.23	25.24

Key Findings¶

Intra-packet association is the primary driver: Incorporating intra-packet graph clustering yields an immediate leap of +8.13% in MOTA (from 52.71% to 60.84%) and +16.62% in IDF1, highlighting the power of geometric diffusion over spatial centroid grouping.
Relative geometric features outperform absolute coordinates: Relying exclusively on absolute positions or combining all features degrades MOTA to 58.22%. Pairing relative displacement with relative motion directions ($\Delta L + \Delta D$) achieves peak performance (64.48% MOTA), providing translation-invariant relational encoding that filters global scene drift.
Temporal resolution is critical: Expanding voxel temporal resolution from 1 ms to 50 ms collapses 10.4% of events into identical spatio-temporal bins, slashing MOTA from 64.4% to 45.7%. Packet duration peaks at $\Delta T = 800\text{ ms}$; shorter windows (100 ms, 55.32% MOTA) lack long-range context, while longer windows (1200 ms, 63.89% MOTA) accumulate obsolete motion patterns.
Superior efficiency and real-time streaming: Ev-ISNet requires only 7.73 MB in model parameters (under 2% of MOTIP's 465.8 MB). With a 20 ms sliding-window stride, update latency is merely 18.2 ms, comfortably surpassing real-time throughput demands for high-frequency event streams.

Highlights & Insights¶

Paradigm Shift (Instance Segmentation as Tracking): Demonstrates that tracking tiny targets in neuromorphic streams is mathematically and architecturally equivalent to 3D point cloud instance segmentation, breaking free from the frame-conversion trap.
Lightweight Geometry-Consistent Clustering: Replaces complex spatial attention networks with a lean relative-feature GC network and Union-Find clustering, achieving high trajectory fidelity at marginal computational overhead.
EV-UAV-Track Benchmark: Establishes the first large-scale MSOT dataset featuring event-point-level 3D tracking identity annotations, resolving the severe ID switch issues of prior frame-level datasets.

Limitations & Future Work¶

Author-Admitted Limitations: In dense scenarios with intersecting trajectories or tight swarming maneuvers, local $k$-NN edge predictions can occasionally introduce erroneous bridge connections, causing identity merges.
Identified Limitations: The neighborhood size is fixed at $K=15$; under drastic dynamic shifts in target density (e.g., transitioning from isolated drones to dense swarms), the graph topology cannot adaptively scale.
Future Directions: Exploring density-adaptive graph edge pruning, incorporating temporal curvature constraints, and integrating spike neural network memory to further minimize power consumption on edge platforms.

vs Frame-based Trackers (CenterTrack, ByteTrack, MOTIP): Conventional trackers rely on dense appearance features and bounding box regression, failing completely on textureless small targets (MOTA < 34%); Ev-ISNet improves MOTA by over 30 percentage points via temporal continuity.
vs Event Detect-then-Track (RVT + SORT / TrackTrack): RVT aggregates events into coarse 50 ms grids, sacrificing fine temporal resolution while carrying large model sizes (9.9–319 MB); Ev-ISNet achieves superior accuracy with a compact 7.73 MB model.
vs 3D Point Cloud Segmenters (PointGroup, HAIS, OneFormer3D): Conventional 3D methods assume static, centroid-clustered objects; Ev-ISNet replaces centroid priors with geometric consistency along open-ended trajectories and adds streaming causal cross-packet association.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Pioneers the paradigm of reformulating neuromorphic multi-small-object tracking into 3D spatio-temporal instance segmentation]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive benchmarking against 16 SOTA methods, comprehensive ablations, and dense swarm evaluation]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural flow, precise mathematical formulations, and compelling motivation]
Value: ⭐⭐⭐⭐⭐ [Delivers both an impactful new paradigm and a high-quality benchmark dataset for event-based tracking]