EVEE: Event-Based Online Adaptation for Matching on Unknown Targets

Conference: ECCV 2026
Paper: ECCV Official Link
Code: https://github.com/ZeJZhao/EVEE
Area: Others
Keywords: Event-camera, Online adaptation, Local feature matching, Proxy supervision, 6-DoF pose estimation

TL;DR

Addressing the severe degradation of offline-trained matchers on previously unseen targets and their frozen inference weights, EVEE turns short-window event activity into label-free online proxy supervision, pairing a multi-threaded Training Warehouse (TraW), a decayed Event Voting Bank (EVB), and a negative-only ReWeight gate to deliver 51.3 FPS unknown-target matching with a 1.9% pose ACC/AUC gain over state-of-the-art baselines.

Background & Motivation

Local feature detection and matching serve as foundational primitives across 3D reconstruction, SLAM, and robotic 6-DoF pose tracking. Classical learned keypoint extractors such as SuperPoint and ALIKED, paired with attention-based matchers including SuperGlue, LightGlue, and OmniGlue, demonstrate impressive performance on standard closed-world benchmarks. However, when deployed in dynamic robotics scenarios involving novel objects, these offline-trained models degrade significantly because offline pretraining cannot anticipate the structural, geometric, and radiometric variations of unknown targets. Contemporary foundation-model-guided approaches such as Zero-1-to-3 and OmniGlue demand massive multi-GPU offline training clusters and remain completely frozen at inference, offering no mechanism for rapid test-time adaptation.

Event cameras, characterized by microsecond temporal resolution and high dynamic range, provide physical immunity to high-speed motion blur and drastic illumination shifts, making them ideally suited for capturing sharp contours of fast-moving targets. Nevertheless, existing event-based vision methods typically follow two flawed extremes: they either treat events as an auxiliary input modality via naive feature concatenation or cross-modal attention (which empirically injects sparse sensor noise into calibrated RGB representations and harms accuracy), or they construct fully asynchronous, complex event-matching pipelines. More critically, directly accumulated event frames disperse activations along arbitrary edges and sensor noise, violating the spatial sparsity and corner repeatability required for reliable geometric correspondence; conversely, discarding real physical events in favor of RGB-predicted edge maps sacrifices physical robustness against blur and lighting swings.

The key insight of EVEE is to shift the role of event streams from passive "input features" to active "test-time proxy supervision" that directly drives the online parameter evolution of a lightweight RGB keypoint detection head. Core idea: by bootstrapping from an offline RGB-to-event corner mapping, EVEE leverages short-window physical event evidence as test-time proxy supervision, utilizing multi-threaded parallel trial-and-error with weight broadcasting (TraW) alongside a temporally decayed bidirectional Event Voting Bank (EVB) and a purely attenuating ReWeight gate to achieve label-free, target-specific feature matching on unseen objects in real time.

Method

Overall Architecture

EVEE operates on synchronized RGB and event-camera streams from a calibrated sensor rig. The end-to-end framework comprises three sequential yet coordinated phases: offline event-corner pre-training, online adaptation, and online matching. In the offline phase, the network learns to predict event-consistent corner responses from RGB images alone. In the online adaptation phase, upon encountering an unknown target, the system conducts lightweight multi-threaded parameter updates on top of a frozen backbone within a short observation window. In the online matching phase, the TraW-selected optimal head is frozen, and candidate correspondences generated from shared features and EVB priors are filtered through ReWeight and dual-softmax competition to produce robust geometric matches.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    subgraph S1["Offline Pre-training: Cross-Modal Corner Prior Mapping"]
        direction TB
        A["RGB Frames + Real Event Streams"] --> B["Haar Wavelet Contours + Conv3D Temporal Fusion<br/>Aligned with MagicPoint Event Corners"]
    end
    subgraph S2["Online Adaptation: Rapid Test-Time Optimization"]
        direction TB
        B --> C["Training Warehouse TraW: Parallel Trial-and-Error<br/>23 Worker Threads + Best-Weight Broadcast + Loss Thresholding"]
        A --> D["Event Voting Bank EVB: Bidirectional Prototypes<br/>Confidence Weighting + Temporal Decay ρ=0.6 + Morphology"]
    end
    subgraph S3["Online Matching: Robust Correspondence & Pose Solving"]
        direction TB
        C --> E["Optimal Adaptive Keypoint Head θ*H<br/>Target-Aware Keypoint Heatmaps"]
        D --> F["EVB Foreground & Background Priors"]
        E & F --> G["ReWeight Gate: Candidate Attenuation<br/>Base Logits + Soft Gaussian Cues + Negative Log-Penalty"]
        G --> H["Dual-Softmax Mutual Selection & Dustbin<br/>Final 6-DoF Pose Solution"]
    end

Key Designs

1. Offline Event-Corner Pre-training & Temporal Feature Aggregation: Anchoring Geometry Priors

Because the exact geometric structure of an unknown target cannot be known in advance, sampling dense points along long straight edges induces the aperture problem and degrades matching repeatability. EVEE pre-trains on the shapes_6dof sequence of the DAVIS dataset by extracting stable corner points via MagicPoint on RGB frames and aligning them with synchronized event activations to establish "event corners". To capture high-frequency contour responses at negligible computational overhead while suppressing low-frequency textures, the feature finding module integrates a deterministic Haar-wavelet contour extractor. These features are then fed into a lightweight \(3\times 3\times 3\) Conv3D block across neighboring frames. This short-term temporal aggregation stabilizes keypoint activations under rapid motion and suppresses transient sensor noise without requiring heavy Transformer or recurrent backbones, followed by a lightweight convolution head yielding keypoint probabilities in \([0, 1]\).

2. Training Warehouse (TraW): Multi-Threaded Parallel Exploration and Weight Broadcasting

Deploying online adaptation on robotic platforms requires convergence within a narrow observation window of seconds. TraW addresses this by maintaining a parallel pool of 23 lightweight training threads. Each thread sequentially consumes short video segments (up to 64 frames) containing RGB frames and their corresponding fused event-derived proxy supervision. To prevent catastrophic overfitting to near-converged or noisy samples, TraW introduces a loss threshold \(\tau = 0.0001\), skipping frames whose supervised loss falls below \(\tau\). Periodically, TraW evaluates the mean loss across all worker threads and selects the checkpoint with the lowest loss as the current best model \(\theta^*_H\), broadcasting its weights across all threads for subsequent iterations. This architecture completes approximately 250 lightweight updates within 20 seconds (compared to only 50 updates in a single-threaded regime), enabling the RGB detector to swiftly internalize target-specific geometries at test time.

3. Event Voting Bank (EVB): Temporally Decayed Bidirectional Prototype Memory

Although physical event streams preserve sharp object boundaries under motion, they simultaneously introduce severe background noise and camera ego-motion artifacts. EVB operates at the feature resolution \((H_f, W_f)\) to separate target structures from background clutter. Pixel-level event counts \(E(i, j)\) across the receptive field \(\Omega_{pq}\) are pooled and normalized into a continuous spatial event-confidence map \(w(p, q) \in [0, 1]\). EVB accumulates both a positive foreground prototype \(\mu^+\) and a negative background prototype \(\mu^-\) using confidence weighting:

\[\mu^+ = \frac{\sum_{p,q} w(p,q) F(p,q)}{\sum_{p,q} w(p,q)}, \quad \mu^- = \frac{\sum_{p,q} (1 - w(p,q)) F(p,q)}{\sum_{p,q} (1 - w(p,q))}\]

where \(F(p, q)\) denotes the fused RGB feature, and \(\mu^+\) is \(\ell_2\)-normalized to eliminate scale bias from varying event counts. Before ingesting new frame statistics, existing votes in the bank undergo exponential temporal decay via \(\rho = 0.6\). Consequently, coherent target boundaries receiving recurring high-confidence votes are retained, while transient hot pixels and background clutter are rapidly forgotten. Finally, morphological OPEN and CLOSE operators eliminate isolated noise islands and bridge fragmented contours, supplying a pristine foreground prior mask for matching.

4. Purely Attenuative ReWeight Gating: Penalizing Unreliable Candidate Pairs

Rather than employing a learned scorer that attempts to predict absolute correspondence probabilities (which frequently destabilizes pre-calibrated similarity distributions), EVEE introduces ReWeight as a tiny, frozen 2-layer MLP gating function with parameters \(\phi\). For any candidate correspondence pair \((i, j)\) between keypoints \(x_i\) and \(x_j\), ReWeight takes the base appearance logit \(S_{ij}^{(0)}\), a soft Gaussian spatial displacement bias \(g_{ij}^{\text{geo}}\), and EVB foreground reliability priors \(b_i, b_j \in [0, 1]\) to generate an attenuation scalar \(w_{ij} \in [10^{-3}, 1]\). The final matching logit is formulated as:

\[S_{ij} = S_{ij}^{(0)} + \log(w_{ij} + \epsilon), \quad \epsilon = 10^{-6}\]

Because \(w_{ij} \le 1\), the additive adjustment \(\log(w_{ij} + \epsilon)\) is strictly non-positive (bounded above by \(\log(1 + 10^{-6}) \approx 0\)). Thus, ReWeight functions exclusively as an attenuation penalty, never artificially boosting low-confidence matches. Candidates residing in noisy background areas or violating geometric consistency are severely penalized, allowing subsequent dual-softmax normalization and dustbin competition to cleanly suppress false-positive correspondences.

Key Experimental Results

Main Results

Quantitative evaluations are conducted across the NAVI-MultiView benchmark (~25K image pairs under wide viewpoints using Prophesee-simulated events), TUD-L (~24K sequential frames exhibiting extreme illumination shifts), and the physical Event-Camera Dataset. Performance is measured via pose accuracy (ACC) and area under the pose error curve (AUC) at \(5^\circ, 10^\circ, \text{and } 20^\circ\).

Table 1: Quantitative comparison on NAVI-MultiView and TUD-L benchmarks (Source: Table 2 in original paper)

Method	NAVI ACC (5°/10°/20°)	NAVI AUC (5°/10°/20°)	TUD-L ACC (5°/10°/20°)	TUD-L AUC (5°/10°/20°)	Overall Average	Runtime (FPS)
SuperPoint + SuperGlue	54.3 / 79.3 / 87.2	20.9 / 45.9 / 65.0	72.7 / 75.4 / 78.8	69.9 / 72.0 / 74.6	66.3	42.5
SIFT + LightGlue	52.9 / 76.4 / 87.0	21.8 / 45.0 / 63.8	70.7 / 75.7 / 82.9	67.2 / 70.2 / 74.9	65.7	4.2
DISK + LightGlue	46.1 / 70.4 / 81.0	18.1 / 39.9 / 58.3	71.8 / 75.0 / 79.5	68.5 / 71.0 / 74.2	62.8	4.1
ALIKED + LightGlue	50.3 / 73.8 / 84.2	20.8 / 43.1 / 61.6	71.5 / 76.0 / 82.5	67.9 / 70.9 / 75.2	64.8	7.1
SuperPoint + LightGlue	55.1 / 78.4 / 88.3	22.5 / 46.4 / 65.3	72.0 / 76.1 / 81.7	68.7 / 71.4 / 75.2	66.8	6.9
OmniGlue (CVPR'24)	49.2 / 73.8 / 81.7	18.6 / 42.1 / 60.4	74.5 / 77.0 / 80.2	71.5 / 73.7 / 76.2	64.9	9.1
EVEE (Ours)	54.6 / 81.4 / 89.5	20.3 / 46.4 / 66.3	77.0 / 78.9 / 81.0	74.8 / 76.4 / 78.3	68.7	51.3

Table 2: Quantitative pose evaluation on physical sensor data from the Event-Camera Dataset (Source: Table 4 in original paper)

Method	shapes-6DoF (Seen) Mean	boxes-6DoF (Unseen) ACC@10° / AUC@10°	poster-6DoF (Unseen) ACC@10° / AUC@10°	Unseen Sequences Mean	All Sequences Overall Mean
SuperPoint + SuperGlue	53.0	75.2 / 59.4	58.6 / 46.2	59.8	57.6
DISK + LightGlue	63.9	78.2 / 63.2	60.4 / 47.7	62.4	62.1
SuperPoint + LightGlue	53.6	77.6 / 63.3	60.3 / 47.8	62.3	59.3
OmniGlue	54.6	72.7 / 57.9	60.0 / 47.6	59.5	57.6
EVEE (Ours)	64.0	80.6 / 63.3	75.6 / 56.2	67.2	66.5

Ablation Study

Table 3: Ablation of the four core system components on NAVI-MultiView (Source: Table 5 in original paper)

Variant	ACC@5°	ACC@10°	ACC@20°	AUC@5°	AUC@10°	AUC@20°	Matching Speed (FPS)	Pair Rate
Full Model (EVEE Full)	54.6	81.4	89.5	20.3	46.4	66.3	51.31	97.88%
w/o Event Voting Bank (non_EVB)	53.2	79.8	87.5	19.4	45.3	64.9	52.33	97.14%
w/o Online Adaptation (non_Learning)	53.9	80.8	89.0	20.0	46.0	65.9	51.86	97.68%
RGB-only (non_Event)	51.7	79.9	88.3	18.4	44.4	64.7	54.49	96.83%
w/o Proxy Event Supervision	53.1	80.1	87.6	19.5	45.4	65.0	51.39	96.70%

Key Findings

• Events as proxy supervision outperform naive input fusion: As shown in Table 3 of the original paper, naively injecting event-derived masks directly into SuperPoint + LightGlue causes NAVI ACC@10° to drop from 78.4% to 77.8% (AUC@10° falls from 46.4% to 45.6%). Similarly, SuperEvent + SuperGlue achieves only 71.4% ACC@10°, lagging behind EVEE by 10.0%. This confirms that raw event noise corrupts fixed feature representations, whereas utilizing events as adaptive supervision drives pure performance gains.
• TraW multi-threading and thresholding prevent drift: Fine-grained module ablations (Table 6 in the original paper) reveal that single-threaded adaptation without loss thresholding yields 80.4% ACC@10°. Introducing \(\tau=0.0001\) to discard converged frames improves accuracy to 80.7%, while scaling up to 23 threads with best-weight broadcasting boosts it to 81.4% across 250 test-time updates.
• Conv3D temporal fusion is crucial for stability: In the pre-trained feature extractor, removing Conv3D temporal fusion produces the most severe drop across all ablations, with NAVI ACC@10° plummeting from 81.4% to 74.9% (AUC@10° dropping from 46.4% to 42.2%), demonstrating that temporal aggregation is necessary to resolve high-speed sensor jitter.

Highlights & Insights

• The "Events-as-Supervision" Paradigm: EVEE rethinks event cameras not as an additional input sensor channel, but as an online physical oracle that provides motion-resilient proxy supervision for adapting an RGB detector at test time.
• Non-Positive ReWeight Logit Penalty: Constraining the ReWeight multiplier to \(w_{ij} \in [10^{-3}, 1]\) mathematically guarantees that the adjustment \(\log(w_{ij}+\epsilon)\) is non-positive, elegantly avoiding the common pitfall of false-positive boosting in learned matchers.
• Deployment-Grade Efficiency: Running at 14.52 FPS during online adaptation and 51.3 FPS during online matching, EVEE demonstrates that test-time adaptation can be achieved in real time on standard robotic computing hardware.

Limitations & Future Work

• Performance Gap Under Extreme Viewpoint Changes: On the tightest \(5^\circ\) error threshold on NAVI (ACC@5° of 54.6%), EVEE slightly trails heavy cross-attention matchers like LightGlue (55.1%), indicating that lightweight MLP gating cannot fully replace comprehensive self- and cross-attention when visual perspective changes drastically.
• Domain Gap in Event Synthesis: Experiments on NAVI and TUD-L rely on synthesized events generated via the Prophesee simulator, which may not capture subtle sensor phenomena like dark current and thermal hysteresis.
• Future Directions: Exploring rotation-invariant relative positional embeddings inside ReWeight, and extending EVB with instance-level clustering to handle complex multi-object scenarios.

Related Work & Insights

• vs SuperEvent (ICCV 2025): SuperEvent relies on consecutive-frame pseudo-labels during training, causing severe performance collapse when frame gaps widen (ACC@10° of 71.4% vs EVEE's 81.4%); EVEE maintains high accuracy over wide baselines through MagicPoint prior alignment and online proxy supervision.
• vs LightGlue (ICCV 2023): LightGlue utilizes frozen attention weights without target-specific adaptation; EVEE adapts keypoint representations dynamically via TraW, surpassing LightGlue by ~2.1% on NAVI ACC@10° while running over 7x faster (51.3 FPS vs 6.9 FPS).
• vs OmniGlue (CVPR 2024): OmniGlue incorporates foundation model guidance at a substantial computational cost (running at only 9.1 FPS); EVEE achieves superior pose accuracy on challenging lighting benchmarks (TUD-L ACC of 77.0% vs 74.5%) with a 5x speedup.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneers the use of event streams as test-time proxy supervision for online adaptation in feature matching]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive validation across simulated benchmarks, illumination stress tests, public real event datasets, and physical robotic rig deployments]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulations, coherent structural hierarchy, and rigorous ablations]
• Value: ⭐⭐⭐⭐⭐ [Offers a practical, low-latency paradigm for unknown-target tracking and geometric vision in robotics]

Related Papers

• [ECCV 2026] High-Throughput Event-Based Feature Detection and Tracking on an Embedded CPU
• [ECCV 2026] Match-Any-Events: Zero-Shot Motion-Robust Feature Matching Across Wide Baselines for Event Cameras
• [ICML 2026] TEMPORA: Characterising the Time-Contingent Utility of Online Test-Time Adaptation
• [CVPR 2025] Event Ellipsometer: Event-based Mueller-Matrix Video Imaging
• [ECCV 2026] Multi-Hypothesis Test-Time Adaptation to Mitigate Underspecification