Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/haohq19/ssla
Area: Object Detection
Keywords: event cameras, spatial sparsity, linear attention, asynchronous detection, parallel training

TL;DR¶

SSLA decomposes the global state of linear attention into geometrically routed local substates, enabling parallel training and event-by-event recurrent inference for low-latency detection; SSLA-L achieves 0.375 mAP on Gen1 with 0.724 MFLOPS per event and a measured single-core processing latency of 7.20 microseconds.

Background & Motivation¶

Event cameras produce streams of events carrying positions, timestamps, and brightness-change polarities rather than fixed-rate images. Accumulating events into images allows detectors to reuse established visual backbones, but requires waiting for data to accumulate; updating predictions event by event preserves the sensor's low-latency advantage. Existing asynchronous graph networks, sparse convolutions, and recurrent models try to compute only what a new event affects. However, deeper networks and larger receptive fields can make local inputs trigger widespread updates, while recurrent training on long event sequences is difficult to parallelize efficiently.

Linear attention appears well suited to this problem: it supports sequence-parallel training and stateful recurrent updates at deployment. Yet standard implementations update the entire state for every event. Classification can compress history into a global vector, whereas detection must remember where objects are and therefore needs finer spatial states. Simply enlarging the state also increases computation for every event. The challenge is not merely sparse input, but a sufficiently large spatial memory that is only locally activated and remains parallel-trainable.

The paper organizes independent substates in overlapping spatial windows, routes each event only to windows covering its position, and gathers their outputs back to that event. Core Idea: decouple spatial memory capacity from per-event activation, and use order-preserving sequence reorganization so that the same sparse-state model supports both parallel training and event-by-event recurrent inference.

Method¶

Overall Architecture¶

SSLA-Det takes temporally ordered raw events, uses polarity and time difference as input features, and uses coordinates to select spatial substates. The backbone contains 4 stages with 2 SSLA layers per stage. Sparse pooling and temporal dropout follow the first three stages. Output features are then written back to their spatial positions, where an asynchronous YOLOX detection head updates predictions locally.

A single SSLA layer processes events through mixture-of-spaces states, position-aware projection, and dual-mode state computation. These modes are not separate models: training reorganizes a complete sequence into window-specific subsequences for parallel computation, whereas inference updates only the states reached by the current event. Both implement the same causal computation. The asynchronous detection head prevents the final prediction stage from reverting to full-feature-map computation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Event stream<br/>Coordinates, polarity, time difference"] --> B["Mixture-of-spaces states<br/>Geometric routing to overlapping windows"]
    B --> C["Position-aware projection<br/>Window-relative position encoding"]
    C --> D["Dual-mode state computation<br/>Training: scatter, parallel compute, gather<br/>Inference: local recurrence, gather"]
    D --> E["Four-stage backbone output<br/>Inter-stage sparse pooling and temporal dropout"]
    E --> F["Asynchronous detection head<br/>Update only the output event position"]
    F --> G["Classes and bounding boxes"]

The sequence from mixture-of-spaces states to dual-mode state computation forms each SSLA layer and repeats throughout the backbone; it is not executed only once for the whole model. Layers also use residual connections and layer normalization. Pooling and temporal dropout are reused supporting components, not additional attention mechanisms.

Key Designs¶

1. Mixture-of-spaces states: retain fine-grained memory while accessing only a local subset

Mixture-of-spaces (MOS) partitions the current spatial domain into overlapping windows with stride 1. Each window maintains an independent state, while all windows share linear attention parameters. A new event is routed by its coordinates to windows covering its position, rather than to arbitrary memories selected by a learned router. Padding ensures that every event accesses the same number of windows: with window side length \(P\), the active count is \(A=P^2\), independent of the total number of windows across the image.

This separates how much spatial information is stored from how much information is updated at a time. Each selected window produces an interim output, but these outputs are aggregated into one event feature at the original position rather than propagated as additional spatial events. A single SSLA layer therefore preserves sequence length. Deeper layers can accumulate information from larger receptive fields without multiplying event counts merely because windows overlap. Total state storage still grows with the spatial domain; local computational savings do not make storage resolution-independent.

2. Position-aware projection: the same event plays different roles in different windows

An event may lie near one window's upper-left corner and another window's center. Sending the same vector to all windows would omit this distinction in local spatial layout. Position-aware projection (PAP) uses the event's two-dimensional offset from a window's upper-left corner to index learnable linear transformations. It projects the input feature before passing it to that window's linear attention. This does not merely append absolute coordinates to a feature; it explicitly encodes the event's position relative to the particular local memory.

The output side has a corresponding relative-position projection. After state computation, each window's interim output is transformed before summation back to the original event. Together, input and output projections preserve spatial roles, allowing parameter-shared windows to distinguish local object structure. In Table 7, removing both PAP sides reduces validation mAP from 0.335 to 0.014. Having many local memories is therefore insufficient without spatially informed reading and writing.

3. Dual-mode state computation: reorganize complete sequences for training, process only new events at inference

Training starts with an ordered event sequence of length \(L\). A precomputed lookup table provides the window indices and relative positions associated with each coordinate. Every event expands into \(A\) copies projected by input PAP, producing a sequence of length \(AL\). A stable sort by window index groups copies belonging to the same window while preserving their original temporal order. Caching this permutation yields independent window-specific subsequences; this is the scatter step.

The compute step processes different windows simultaneously. Within each window, parallel scans or chunk-wise algorithms for linear recurrence can also process long sequences. Windows share parameters but not states. Each subsequence remains causal: access to the complete input during training does not permit future events to affect past outputs. The implementation uses a real-valued Linear Recurrent Unit (LRU) with Triton rather than requiring an additional softmax attention matrix.

The gather step applies the cached inverse permutation to recover event groupings, applies output PAP to the window-specific interim outputs, and sums the results belonging to each event. An event thus draws on several local states but passes only one feature to the next layer. Training-time copies organize computation; they are not a complete segment of future input that deployment must buffer.

Inference does not wait for complete subsequences or sort future events. When an event arrives, the lookup table identifies its covering windows, whose historical states are read and updated. Output PAP and aggregation produce the current feature, while unselected states remain unchanged. Agreement between parallel training and recurrent inference relies on preserving each window's event order, using the same state-update rules, and maintaining independent window states.

The paper gives the SSLA module's per-event inference complexity as:

\[ \mathcal{O}(P^2 D_{out}^2). \]

With fixed window size and channel dimension, this does not grow directly with image resolution. Memory access, vectorization, and cache behavior can nevertheless affect measured latency. The cached text has missing operators in the recurrence and several projection equations; this note explains the mechanism from Algorithm 1 and the prose without reconstructing the authors' exact recurrence.

4. Asynchronous detection head: preserve local updates through prediction

The backbone maintains a spatial feature representation, and each output event overwrites only its corresponding location. All convolutions in the YOLOX head are changed to \(1\times1\), so a location update requires refreshing only that position's predictions rather than recomputing the complete feature map. Earlier SSLA state interactions provide spatial context, removing the need for cross-position convolutions in the detection head.

Sparse pooling in the first three stages rescales event coordinates to organize space at larger scales but does not reduce event count. Temporal dropout compresses the sequence and introduces an accuracy-computation trade-off. Channel dimensions double at each stage, while residual connections and layer normalization stabilize training. Event-by-event processing does not mean every input refreshes every deep-layer position: events removed by temporal dropout do not trigger subsequent computation, and retained backbone outputs update the corresponding predictions.

A Worked Example¶

With the default \(P=3\), a newly arriving event on a car edge reaches 9 windows in a layer. Each window retains its own state from previous events. Because the event has different relative positions in these windows, it produces 9 input PAP results. After local state updates, output PAP transforms the interim results, which are aggregated into a single feature at the event's position.

For a training sequence containing \(L\) events, the layer first expands the sequence to \(9L\) copies, groups them by window for parallel computation, and then gathers them back into \(L\) outputs, excluding inter-layer temporal dropout here. Online inference performs these 9 local accesses only when the new event arrives, without waiting for the next event. This example illustrates data flow rather than an additional experiment or a fixed whole-network access count, since pooling and temporal dropout change coordinates and event counts in subsequent layers.

Loss & Training¶

SSLA-S, SSLA-B, SSLA-M, and SSLA-L use first-stage channel dimensions of 12, 16, 24, and 32, respectively, with a default window side length of 3. Gen1 training uses AdamW for 40 epochs, batch size 32, and a base learning rate of \(1\times10^{-3}\) with cosine decay. Augmentation includes random flipping with probability 0.5 and input-event dropout with a retention ratio sampled between 0.8 and 1.0.

N-Caltech101 training runs for 200 epochs with batch size 64. Besides event dropout, it uses random cropping to 75% of the original resolution with probability 0.2 and translation by up to 10% of the original resolution. Training uses exponential moving averaging. Input-event dropout for augmentation should be distinguished from temporal dropout inside the backbone. The supplied full text does not explicitly enumerate detection loss terms and weights, so this note does not insert assumed default YOLOX loss equations.

Key Experimental Results¶

Main Results¶

Gen1 covers automotive scenes at \(304\times240\) resolution with car and pedestrian detection. Evaluation removes boxes with a diagonal below 30 pixels or width below 20 pixels. N-Caltech101 contains 101 classes at \(240\times180\) resolution, recorded through saccadic motion over displayed images, and should not be equated with natural driving video. The following results come from Tables 1 and 2. mAP and AP50 at a fixed IoU threshold of 0.50 are listed separately; missing results are not inferred.

Dataset	Method	mAP	AP50	MFLOPS/event
Gen1	DAGr-L	0.321	Not reported	17.4
Gen1	SSLA-S	0.334	0.629	0.102
Gen1	SSLA-B	0.351	0.655	0.182
Gen1	SSLA-L	0.375	0.675	0.724
N-Caltech101	DAGr-L	Not reported	0.732	18.9
N-Caltech101	SSLA-S	0.444	0.681	0.131
N-Caltech101	SSLA-L	0.515	0.743	0.926

On Gen1, SSLA-L exceeds DAGr-L by 0.054 mAP with approximately 24-fold lower computation; SSLA-S exceeds it by 0.013 mAP with approximately 171-fold lower computation. Cross-method comparison on N-Caltech101 should use AP50: 0.743 versus 0.732 is a gain of 1.1 percentage points, with approximately 20.4-fold lower computation. The synchronous EventPillars method achieves 0.531 mAP on Gen1, so the advantage concerns accuracy and efficiency among asynchronous methods, not the highest accuracy across all detectors.

Ablation Study¶

The following table summarizes Gen1 validation results from Tables 5 and 7. The full model scores 0.335 mAP on validation, which should not be mixed with the main result of 0.334. TD denotes temporal dropout and SP denotes sparse pooling.

Config	mAP	MFLOPS/event	Note
SSLA-S full model	0.335	0.102	Default spatial sparsity and both PAP sides
Without TD	0.370	1.02	Computation increases 10-fold, with higher accuracy
Without TD, dense state activation	0.370	388	Retains MOS and PAP but activates every window per event
Without SP	0.014	0.102	Unchanged computation, substantially lower detection accuracy
Input PAP only	0.306	0.085	Removes output-side position dependence
Output PAP only	0.224	0.089	Removes input-side position dependence
Neither PAP side	0.014	0.072	Uses position-independent learnable projections

With TD absent in both configurations, local activation reduces computation from 388 to 1.02 MFLOPS/event, approximately 380-fold, while both reach 0.370 mAP. This comparison isolates spatial sparsity; the complete difference between the full model and the dense variant should not all be attributed to SSLA.

Key Findings¶

Parallel training benefits are measured: Table 3 reports 1.05 hours per epoch for LSTM and 0.25 hours for SSLA-S on 4 A800 GPUs. SSLA-B takes 0.28 hours and achieves 0.351 mAP versus LSTM's 0.353, showing similar accuracy at comparable computation with faster training.
Computation is not a substitute for latency: the C++ recurrent implementation in Table 4 runs on one AMD Ryzen 9 9950X3D core. Gen1 latencies for S/B/M/L are 3.43/2.44/6.02/7.20 microseconds, with B faster than S. These are per-event model processing times, not end-to-end latency including the sensor, queuing, and system communication.
Simply widening a model cannot replace fine-grained states: in Table 6, standard linear attention with first-stage width 36 requires 0.093 MFLOPS/event, close to SSLA-S at 0.102, but achieves only 0.001 validation mAP. Their final-layer state sizes are 0.281K and 106.9K, respectively.
Window size controls local interaction range: Table 8 reports validation mAP of 0.200/0.335/0.371 for \(P=2/3/4\), with 0.047/0.102/0.179 MFLOPS/event. Larger windows also increase training memory and time; the default of 3 is a trade-off, not the accuracy-maximizing setting.

Highlights & Insights¶

Separating memory capacity from activation is better matched to localization than merely reducing channel dimensions. The transferable principle is to retain many geometrically organized local memories while paying only for local updates, rather than compressing all spatial detail into one global state.
Gather does more than restore ordering: it prevents sparse events from expanding into additional output events in deeper layers. Spatial context growth and event-activation growth thus become separately controllable properties.
Training-time sequence reorganization and deployment-time event scheduling can differ while preserving the same causal state computation. This suggests extensions to other streaming tasks with explicit local routing, provided state independence and positional encoding are revalidated.

Limitations & Future Work¶

The authors acknowledge an accuracy gap to synchronous methods: 0.375 mAP on Gen1 remains below synchronous results exceeding 0.5. Low latency is valuable, but does not eliminate the accuracy gap.
Insufficient relative motion between camera and object can produce too few events and cause missed detections. Figure 4 also includes apparent false positives caused by missing annotations. Event-image fusion is proposed as future work, with additional sensor requirements, alignment challenges, and system complexity.
For deployment, single-core latency is not equivalent to sustained throughput, tail latency, or energy consumption at high event rates. Total state memory still grows with the spatial domain, making bandwidth and queuing behavior under changing resolution and event density important additional measurements.
PAP, pooling, and local states are all important components. The results do not establish that arbitrary sparsification of linear attention yields the same benefits. FPGAs and specialized accelerators are discussed as future directions, not demonstrated deployments in this work.

vs DAGr: DAGr uses event graphs and local updates for asynchronous detection. This work pursues the same low-latency objective with spatially organized linear recurrent states and explicitly designs for sequence-parallel training. Cross-method comparisons must fix the dataset and metric, especially avoiding comparisons between AP50 and mAP.
vs EventSSM / S7 / EVA: The first two demonstrate the value of state space models for event classification, where global tasks do not require fine-grained localization. EVA explores linear attention for event detection but still requires a dense backbone. SSLA-Det connects local states to a local detection head to maintain asynchronous processing throughout the network.
vs Mixture-of-Memories / Col2a: The former selects memories through learned routing, whereas MOS routes by known geometry. The latter also investigates local states but relies on spatial contraction at discrete timestamps. This work retains the event sequence and uses scatter-compute-gather to parallelize training of an event-by-event causal model.

Rating¶

Novelty: 4/5. Geometrically routed states, relative-position reading and writing, and parallelizable sparse computation form an integrated response to asynchronous detection bottlenecks.
Experimental Thoroughness: 4/5. Two datasets, training time, single-core latency, and component ablations are covered, but broader hardware and sustained event-load evaluation are missing.
Writing Quality: 4/5. The mechanism and algorithm are clearly presented and metrics are carefully distinguished; damaged equations in the current full-text cache limit exact implementation verification.
Value: 4/5. The work provides measured support for decoupling state capacity from computation in fine-grained streaming vision, while deployment benefits still require hardware-specific evaluation.