Match-Any-Events: Zero-Shot Motion-Robust Feature Matching Across Wide Baselines for Event Cameras

Conference: ECCV2026
arXiv: 2604.18744
Code: https://github.com/spikelab-jhu/Match-Any-Events
Area: Event Camera Feature Matching
Keywords: Event Camera, Wide-Baseline Matching, Zero-Shot Generalization, Spatio-Temporal Transformer, Token Pruning

TL;DR

This paper proposes the first event camera wide-baseline feature matching model that achieves zero-shot generalization across datasets. It learns highly efficient multi-temporal-scale event features using a separable spatial-temporal attention backbone (TAg) and sparse-aware token adaptive pruning (SETS). Trained on a large-scale synthetic (E-MegaDepth) and real-world (ECM) wide-baseline dataset, it outperforms the previous state-of-the-art by 37.7% on multiple benchmarks.

Background & Motivation

Due to their high temporal resolution and extreme dynamic range, event cameras exhibit extraordinary transient motion estimation capabilities in fast-moving and low-light scenarios. However, a long-ignored issue is that when establishing wide-baseline feature correspondences between two arbitrary perspectives—such as observing the same scene from entirely different positions at different points in time—event cameras perform significantly worse than conventional frame-based cameras. This gap stems from three aspects. At the data level, existing event datasets rely almost entirely on adjacent-frame optical flow annotations inferred from SLAM poses; thus, networks never encounter true large viewpoint changes during training. At the model level, event data is intrinsically a 3D spatio-temporal representation (time \(\times\) height \(\times\) width); performing simple self-attention over the full spatio-temporal volume incurs a prohibitive computational complexity of \(O((THW)^2)\), rendering large-scale network training intractable. At the representation level, the appearance of events varies drastically with motion speed: the same scene yields dense event distributions under slow motion, but sparse, edge-dominated signals under fast motion. Traditional hand-crafted representations (such as fixed time-window event frames) cannot simultaneously accommodate varying motion speeds. These three challenges are highly coupled—without a sufficiently large wide-baseline dataset, training large networks is impossible, yet without an efficient architecture, even processing limited data is computationally unaffordable.

This paper addresses these challenges from both architectural design and training data. Architecturally, it designs a separable spatial-temporal attention mechanism that decomposes 3D volume attention into alternating 2D spatial and 1D temporal attention calculations, combined with adaptive sparse token pruning to further reduce the active computation to a level feasible for million-scale pairwise training. Data-wise, it synthesizes 3 million pairs of wide-baseline event streams based on MegaDepth and establishes a real-world heterogeneous stereo system to capture genuine event-image pairs, providing the wide-baseline supervision signals the model genuinely needs. Core Idea: A separable spatial-temporal attention temporal aggregator (TAg) is proposed, which decouples 30 event voxel features into alternating attention computations across spatial and temporal dimensions, reducing the complexity from \(O((THW)^2)\) to \(O(T(HW)^2 + HW \cdot T^2)\); it is coupled with Sparse-Aware Event Token Selection (SETS) to adaptively prune redundant computations in event-free regions in a differentiable manner; building on this, end-to-end training is performed using large-scale synthetic and real-world wide-baseline event pairs, achieving zero-shot cross-dataset wide-baseline feature matching on event cameras for the first time.

Method

Overall Architecture

The overall pipeline of Match-Any-Events is divided into four stages. First, the raw event stream is partitioned into multiple temporal bins using a logarithmic time window to generate multi-temporal-scale event voxel representations. The voxels are tokenized and fed into the Temporal Aggregator Transformer (TAg). TAg first performs feature interaction along the spatial and temporal dimensions alternately via separable spatial-temporal attention, and then fuses the multi-scale features of multiple temporal bins into a single motion-invariant feature map through a temporal aggregation refinement module. Next, the Sparse-Aware Event Token Selection (SETS) module adaptively prunes event-free regions (such as large stationary backgrounds) based on the accumulated halting score of each spatial location at each time step, reducing downstream computational overhead. Finally, in the coarse-to-fine matching stage, the coarse-grained feature maps after feature interaction are used to select initial matching pairs via dual-softmax and Mutual Nearest Neighbor (MNN) search, which are then refined to sub-pixel accuracy within fine-grained local windows.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Event Stream Input"] --> B["Multi-temporal Scale<br/>Voxel Encoding"]
    B --> C["TAg: Separable Spatial-Temporal Attention + Temporal Aggregation Refinement"]
    C --> D["SETS: Sparse-aware<br/>Token Adaptive Pruning"]
    D --> E["Coarse Matching Layer: Cross/Self-Attention<br/>+ Dual-softmax + MNN"]
    E --> F["Fine Matching Layer: Local Window<br/>Refinement + Sub-pixel Estimation"]
    F --> G["Output: Semi-dense Correspondence Field"]

Key Designs

1. TAg Temporal Aggregator Transformer: Decoupling Spatio-Temporal Dimensions for Multi-Scale Event Feature Fusion

The core computational bottleneck of event Transformers lies in the \(O((THW)^2)\) complexity of 3D spatio-temporal attention. The key insight of TAg is that spatial neighborhood relationships and feature variations along the temporal direction can inherently be processed separately, eliminating the need for joint attention over the global spatio-temporal volume all at once. TAg splits the computation into two stages.

The first stage is Separable Spatial-Temporal Attention. It first performs self-attention over the \(H \times W\) spatial dimensions individually within each temporal bin, facilitating feature interaction in the spatial domain. Then, for each spatial location, it conducts self-attention along the temporal dimension \(T\), enabling cross-temporal bin motion feature fusion. These two attention layers are stacked alternatively \(N_l\) times. The complexity of this design is only \(O(T(HW)^2 + HW \cdot T^2)\)—the first term represents the overhead of spatial attention for each of the \(T\) temporal bins, and the second term is the overhead of temporal attention for each of the \(HW\) spatial locations. When \(T\) is large, this is significantly lower than the tightly coupled \(O((THW)^2)\).

The second stage is Temporal Aggregation Refinement. Information regarding different motion speeds is distributed across different temporal scales: edge signals of fast motion are concentrated in short temporal bins (with high resolution and sharp details), whereas slow-motion textures rely on long temporal bins to accumulate more information. TAg uses the token from the first temporal bin (which has the highest resolution) as the query, and tokens from the remaining temporal bins as keys/values, integrating multi-scale information into a single feature map via cross-temporal attention. During aggregation, a learnable bias vector \(b \in \mathbb{R}^D\) is introduced to compensate for the degradation of feature quality when the smallest temporal bin contains too little information (e.g., when there is absolutely no fast motion in that region). Visualization results demonstrate that the model adaptively focuses on fine-grained temporal bins in fast-moving regions and switches to coarse-grained temporal bins in slow-moving areas.

2. SETS Sparse-Aware Token Selection: Adaptively Pruning Redundant Event Tokens in a Differentiable Manner

Although TAg has decoupled the temporal dimension, the spatial attention within each temporal bin still needs to process all \(H \times W\) tokens—many of which correspond to stationary, event-free regions. SETS predicts a halting score \(h_n^\tau \in [0,1]\) (outputted via MLP + Sigmoid) for each spatial location \(n\) at each time step \(\tau\). The scores accumulate along the time steps. When the cumulative sum reaches a threshold \(1-\varepsilon\), the token at that location stops being processed, and no further computation is allocated to it in subsequent time steps. To ensure differentiability, the halting score of the final time step is replaced by a residual term \(R_n = 1 - \sum_{i=1}^{N_n-1} h_n^i\).

Unlike earlier works that use halting scores as a weighted average for model outputs (ponder cost), SETS directly injects the accumulated halting scores into spatial attention weights as a bias term:

\[ \text{bias}_n^\tau = \begin{cases} \log(1 - \sum_{i=1}^{\tau-1} h_n^i), & \tau \leq N_n \\ -\infty, & N_n < \tau \leq T \end{cases} \]

Tokens with high halting scores decay exponentially in subsequent attention computation, thereby achieving actual "halting." The ponder loss \(L_{\text{ponder}} = \frac{1}{HW} \sum_n (N_n + R_n)\) forms a push-and-pull mechanism with the matching loss: the matching loss requires retaining all features useful for matching, while the ponder loss drives early halting, allowing the model to automatically balance the two. On ECM, SETS reduces spatial attention FLOPs by 21.5% (from 62.22G to 48.87G) with minimal impact on overall matching performance.

3. Coarse-to-Fine Progressive Matching: Dense Correspondence Estimation via Dual-Softmax + Mutual Nearest Neighbors

After TAg and SETS extract motion-robust multi-scale features, the matching stage adopts a coarse-to-fine strategy. At the coarse-grained level (stride=14), the feature maps of the two views undergo alternating cross-attention and self-attention for feature interaction to construct a correlation matrix \(S\). Softmax is applied to the rows and columns of the matrix respectively to obtain the matching probability \(P\). Then, the most confident matching pairs are selected via Mutual Nearest Neighbors (MNN):

\[ (x_i, y_j) = (\arg\max_x S(x, y_j), \arg\max_y S(x_i, y)) \]

The indices obtained from the coarse matching are used to crop local windows around corresponding positions in the fine-grained feature maps (i.e., the neighborhood of the candidate match in the fine-grained features), and the MNN selection is repeated within these windows. Finally, the expected coordinates are computed within a \(3 \times 3\) local region to achieve sub-pixel precision refinement.

Loss & Training

The training loss consists of four components: the cross-entropy loss \(L_c\) of the coarse matching layer (based on the ground truth matching matrix \(M_c^{\text{gt}}\)), the cross-entropy loss \(L_f\) of the fine matching layer (based on \(M_f^{\text{gt}}\)), the \(L_2\) coordinate regression loss \(L_l\) for sub-pixel refinement, and the ponder loss \(L_{\text{ponder}}\) of SETS. The final loss is formulated as \(L = L_c + \alpha L_f + \beta L_l + \gamma L_{\text{ponder}}\). The model is trained on a single NVIDIA H100 (96G) for 3 days with a batch size of 16 and an input resolution of \(560 \times 336\). The ViT backbone is initialized with DINO pre-trained weights to accelerate convergence, while the remaining modules are trained from scratch. Processing a pair of event streams with a resolution of \(350 \times 630\) on an RTX 4080 takes 49ms with the full model.

Key Experimental Results

Main Results

Dataset	Task	Metric	Ours	Prev. SOTA	Gain
ECM	Event \(\to\) Event	AUC@5°	54.61	11.40 (SuperEvent)	+379%
ECM	Event \(\to\) Event	AUC@10°	72.24	22.12 (SuperEvent)	+226%
ECM	Event \(\to\) Image	AUC@5°	48.58	23.60 (MatchAnything)	+106%
M3ED	Event \(\to\) Event	AUC@5°	52.99	24.89 (MatchAnything)	+113%
M3ED	Event \(\to\) Image	AUC@5°	54.97	26.35 (MatchAnything)	+109%
EDS	Event \(\to\) Event	AUC@5°	40.4	25.4 (SuperEvent)	+59%

Ablation Study

Configuration	AUC@5°	Accuracy (%)	Explanation
Full model	54.61	68.90	Full model
w/o TAg	50.12	67.18	Remove TAg, squeeze temporal dimension directly into channels
w/o TAg + ConcentrateNet	48.37	64.79	Use traditional CNN to aggregate the temporal dimension instead of TAg
w/o Multi-scale Input	39.86	66.87	Single temporal scale input
Synthetic data only (E-MegaDepth)	52.63	68.04	Trained only on E-MegaDepth
Real-world data only (M3ED)	11.33	20.25	Trained only on M3ED

Key Findings

• The TAg module contributes significantly: removing it results in an AUC@5° drop of approximately 4.5 points; replacing TAg with the traditional ConcentrateNet causes a larger performance drop, demonstrating that the proposed separable spatial-temporal attention is significantly better than simple CNN temporal squeezing.
• Multi-temporal scale input is the most critical single factor: removing multi-scale input causes the AUC@5° to drop sharply to 39.86 (a drop of nearly 15 points), indicating that effective information under different motion speeds is distributed across different time windows, and a single window cannot simultaneously cover both fast and slow motions.
• Synthetic data (E-MegaDepth) provides indispensable wide-baseline diversity: models trained solely on real-world driving scene data (M3ED) exhibit extremely poor generalization (AUC@5° of only 11.33), whereas the model trained solely on synthetic data already approaches the full performance (52.63 vs 54.61), with the mixture of both yielding the best results.
• Motion robustness experiments show that the performance of the model barely degrades across a wide input window range from 20ms to 500ms, whereas baseline methods experience significant decay as the window size increases.

Highlights & Insights

• First realization of zero-shot wide-baseline matching for event cameras: Prior event matching methods either required test-time fine-tuning or were only effective under narrow-baseline settings. In contrast, the proposed method can directly generalize to any unseen dataset in a single forward pass without any domain adaptation or fine-tuning.
• Clear and transferable complexity analysis of TAg: The separation strategy that reduces the complexity from \(O((THW)^2)\) to \(O(T(HW)^2 + HW \cdot T^2)\) is particularly suited for 3D sparse data like event streams, and can be naturally extended to feature learning in LiDAR point clouds or other 3D sensors.
• SETS replaces hard pruning with differentiable bias: Compared with fixed-ratio pruning schemes like DynamicViT, SETS automatically learns the halting step for each location via a ponder loss push-pull mechanism, without requiring task-specific predefined pruning ratios. Injecting halting via biases rather than weighted averages is also more direct.
• Practical value of the datasets: E-MegaDepth (3 million synthetic event stream pairs) and ECM (real heterogeneous stereo event-image pairs, including COLMAP poses and VGGT depth) fill the gap of large-scale wide-baseline annotated data in the event camera domain.
• Simple and practical architecture: The model only requires 3 days of training on a single H100, and takes only 49ms per pair for inference on a single RTX 4080, showing great promise for practical deployment.

Limitations & Future Work

• Performance still degrades under extreme motion (<20ms window) because highly sparse events cause insufficient texture information—this boundary might be an intrinsic physical limitation of event cameras themselves.
• The output consists of semi-dense feature correspondences instead of pixel-wise dense matching, which may have insufficient accuracy for downstream tasks requiring global dense alignment (such as dense SfM or panoramic alignment).
• Currently, only event-to-event and event-to-image modalities are supported, and the model has not been extended to pure image-to-image matching (though not the target for event-based systems, incorporating mixed-modality training could further boost generalization capabilities).

Related Work & Insights

• vs SuperEvent: SuperEvent adopts a detector-first architecture (detecting keypoints before describing them), which is constrained by the number of sparse keypoints and suffers from poor keypoint repeatability in wide-baseline scenarios. In contrast, the proposed detector-free method produces semi-dense correspondences and is highly robust to wide baselines, outperforming SuperEvent by approximately 43 points in AUC@5° on ECM.
• vs MatchAnything: MatchAnything is a general cross-modal matching framework with decent performance on event-to-image tasks, but exhibits a significant gap on event-to-event tasks (where the proposed method is higher by approximately 30 points in AUC@5°). The proposed method also naturally supports event-to-event matching tasks.
• vs VGGT: As a 1.2-billion-parameter foundation model, VGGT is powerful in 3D reasoning but is not "natively" compatible with event streams—requiring E2VID to reconstruct images from events before feeding them into VGGT. In contrast, the proposed method operates directly on event streams and features a significantly smaller parameter footprint.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First to achieve zero-shot cross-dataset wide-baseline matching on event cameras; the designs of TAg and SETS are highly concise, efficient, and transferable.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across three datasets (ECM, M3ED, EDS), covering both event-to-event and event-to-image tasks, complete ablation studies, motion robustness analysis, and downstream SfM applications.
• Writing Quality: ⭐⭐⭐⭐ Clear motivation, complete methodology, and logically sound ablation designs, though the biological vision background in the introduction is slightly verbose.
• Value: ⭐⭐⭐⭐⭐ Resolves a long-ignored key challenge in the event camera field, provides open-source code and data, and establishes a new baseline for zero-shot event matching.

Related Papers

• [CVPR 2026] NAF: Zero-Shot Feature Upsampling via Neighborhood Attention Filtering
• [CVPR 2025] Full-DoF Egomotion Estimation for Event Cameras Using Geometric Solvers
• [ACL 2025] Are Any-to-Any Models More Consistent Across Modality Transfers Than Specialists?
• [ECCV 2026] Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation
• [AAAI 2026] Spike Imaging Velocimetry: Dense Motion Estimation of Fluids Using Spike Cameras