Skip to content

Detect by Track: Making Detector-Free Matcher Trackable

Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: Detector-free matching, feature tracking, multi-view consistency, pose estimation, incremental SLAM

TL;DR

Addressing the core multi-view inconsistency and track fragmentation inherent to implicit keypoint selection in detector-free matchers, this paper proposes Detect-by-Tracking (DeT), an inference-time steering mechanism that enables off-the-shelf detector-free matchers to directly generate sub-pixel connected tracks without any retraining.

Background & Motivation

In Structure from Motion (SfM), Visual Odometry (VO), incremental SLAM, and visual object tracking, establishing long-term, multi-view consistent 2D feature trajectories is essential for accurate geometric estimation. Traditional detector-based pipelines follow the classic detect-describe-match paradigm (e.g., SuperPoint coupled with LightGlue). Although their explicit keypoint extraction naturally yields connected tracks across multiple viewpoints, the decoupled heuristic detection often struggles in texture-poor regions, repetitive structures, or under dramatic illumination changes. Furthermore, the quadratic computational complexity of attention-based matchers relative to the number of keypoints fundamentally limits their efficiency in semi-dense configurations.

Modern detector-free matchers (such as LoFTR, EDM, and JamMA) circumvent explicit detection bottlenecks through dense cross-image interactions in feature space alongside coarse-to-fine matching pipelines, delivering superior two-view correspondence accuracy and speed. However, their underlying mechanism—where keypoints emerge implicitly as a by-product of pairwise matching—leads to severe multi-view inconsistency. A point detected in an anchor image paired with one view drifts when paired with another view, resulting in fragmented tracks that cannot be directly integrated into standard online SLAM or tracking systems.

Existing remedies, such as Detector-Free SfM and Dense-SfM, mitigate track fragmentation by coarsely quantizing matches onto discrete grids and subsequently running heavy multi-view global optimizations (e.g., transformer-based pose/point refinement or Gaussian-splatting post-processing). These multi-stage batch pipelines are computationally intensive and tailored exclusively for offline SfM, rendering them unsuitable for real-time incremental SLAM or visual tracking where observations arrive sequentially. The critical insight here is that the coarse-to-fine local similarity matrix in detector-free matchers exhibits translational equivariance, and its row-wise softmax naturally defines a conditional correspondence probability distribution over target pixels. The core idea is to introduce a Detect-by-Tracking (DeT) mechanism that uses motion-consistent pseudo-tracks to steer target feature crops and leverages the translational equivariance of the similarity matrix to bilinearly resample fine features at sub-pixel queries, enabling off-the-shelf detector-free matchers to output connected sub-pixel tracks in a zero-shot, training-free manner.

Method

Overall Architecture

A standard coarse-to-fine detector-free matcher extracts coarse features \(C_A, C_B\) and fine features \(F_A, F_B\) via a joint backbone. Coarse matches define window centers \(\bar{p}_A, \bar{p}_B\), around which \(\kappa \times \kappa\) fine crops are compared using a mixing function \(\mathcal{M}_F\) and correlation to form a local similarity matrix. Peak selection via dual softmax determines the pairwise correspondence, allowing keypoint locations on the anchor image to drift freely across pairs.

DeT shifts this paradigm from pair-matching detection to detect-by-tracking. Given an established sub-pixel track query \(q_A\) from the previous frame and an incoming image pair \((I_A, I_B)\), DeT first computes a pseudo-track from the nearest pair-matching correspondence to steer the target crop center towards the true match location. It then bilinearly resamples the anchor fine features centered directly at the sub-pixel location \(q_A\). By extracting the row corresponding to zero offset from the resulting local similarity matrix, DeT obtains a connection probability map \(P_q\) over target pixels, locating the discrete maximum and refining it with the pre-existing regression head to output the continuous sub-pixel track \(q_B\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Image Pair (IA, IB) and Sub-Pixel Query qA"] --> B["Steer Target Crop<br/>Derive pseudo-track qB_ps via nearest pair match and crop FB"]
    B --> C["Steer Anchor Crop<br/>Bilinearly resample FA at sub-pixel location qA"]
    C --> D["Similarity Steering & Probability Mapping<br/>Extract row-wise softmax to construct connection map Pq"]
    D --> E["Sub-Pixel Track Update<br/>arg max grid peak + regression head refinement to yield qB"]
    E --> F["Output: Connected Sub-Pixel Track qB (Input to Next Frame)"]

Key Designs

1. Steer Target Crop: Guaranteeing Receptive Field Coverage via Pseudo-Tracks In standard matchers, local correlation is evaluated within a \(\kappa \times \kappa\) window centered at coarse match \(\bar{p}_B\). For an arbitrary historical query \(q_A\), the ground-truth correspondence \(q_B^{\text{GT}}\) may drift outside this window due to disparity and motion. To ensure that the ground truth remains within the fine receptive field, DeT constructs a pseudo-track based on local translational consistency. It finds the nearest detect-by-pair-matching correspondence \((p_A, p_B)\) to \(q_A\). Assuming local depth homogeneity and translational motion, the target crop center is steered to: $\(q_B^{\text{ps}} = \Delta_f \cdot \left\lfloor \frac{q_A + (p_B - p_A)}{\Delta_f} \right\rceil\)$ where \(\Delta_f\) is the fine feature stride (e.g., 2 px). By centering the target crop \(F_B^{q_B^{\text{ps}}}\) around this predicted location, the ground-truth match is encompassed within the window with high probability.

2. Steer Anchor Crop: Sub-Pixel Anchoring via Translational Equivariance Even with the target window properly centered, standard similarity matrices index anchor locations on a discrete grid of stride \(\Delta_f\), preventing direct querying of arbitrary floating-point coordinates \(q_A\). DeT leverages the translational equivariance of feature crops: shifting the anchor feature crop by an offset in image space shifts the resulting row-wise probability distribution equivariantly. Exploiting this property, DeT directly performs bilinear sampling on the continuous fine feature map \(F_A\) at \(q_A\) to extract \(F_A^{q_A}\): $\(P_q = \operatorname{softmax}\left( \operatorname{Corr}\left(\mathcal{M}_F(F_A^{q_A}), \mathcal{M}_F(F_B^{q_B^{\text{ps}}})\right) \big|_{[0,0]} \right)\)$ This formulation elegantly replaces discrete anchor coordinate lookups with continuous feature alignment, ensuring that the row corresponding to \([0,0]\) represents the exact matching probability distribution conditioned on the sub-pixel query \(q_A\).

3. Sub-Pixel Track Update and Newborn Track Seeding Once the connection probability map \(P_q\) is established, the fine-grid prediction is computed as \(\hat{q}_B = \arg\max P_q\). DeT then feeds the fine crop representations into the base matcher's pre-trained sub-pixel regression head to estimate the continuous residual offset \(\delta\), yielding the updated track coordinate \(q_B = \hat{q}_B + \delta\). To replenish tracks lost due to out-of-view motion, occlusions, or camera zoom, DeT optionally injects confident detect-by-pair-matching peaks as newborn tracks into the tracking pool at each frame, ensuring long-term tracking stability.

Loss & Training

DeT operates purely at inference time and requires zero training or fine-tuning. Both DeT-JamMA and DeT-EDM directly utilize off-the-shelf pre-trained weights from their official releases. The additional operations—bilinear feature interpolation, local feature transformation, and argmax extraction—incur minimal computation. Compared to the heavy backbone feature extraction and attention/SSM operations, DeT increases total FLOPs by only approximately 2%, maintaining real-time tracking performance.

Key Experimental Results

Main Results

The method was evaluated on 5-frame sequences from the MegaDepth and IMC benchmarks, assessing continuous tracking survival, multi-view camera pose estimation via essential matrix decomposition, and initial incremental SLAM reconstruction without bundle adjustment (reporting AUC@\(5^\circ\)).

Table 1: Multi-View Pose Estimation Accuracy (AUC@\(5^\circ\), Source Table 1)

Category Method IMC Reichstag IMC Sacre C. IMC St-Peters MegaDepth 0015 MegaDepth 0022
Detector-based SIFT + LightGlue 69.0 81.8 60.7 46.3 48.2
Detector-based SuperPoint + LightGlue 75.0 72.6 47.0 58.0 37.8
Detector-free + Naive Association NN-JamMA 35.0 45.0 11.0 24.4 18.3
Detector-free + Naive Association NN-EDM 19.0 36.0 5.0 12.6 15.9
Detect by Track (Ours) DeT-JamMA 84.0 77.0 61.0 72.4 46.3
Detect by Track (Ours) DeT-EDM 29.0 52.0 12.0 17.3 19.5

Table 2: Incremental SLAM Initial Pose Accuracy (AUC@\(5^\circ\), Source Table 2)

Category Method IMC Reichstag IMC Sacre C. IMC St-Peters MegaDepth 0015 MegaDepth 0022
Detector-based SIFT + LightGlue 64.9 82.3 56.3 61.8 34.7
Detector-based SuperPoint + LightGlue 46.0 70.9 41.1 43.4 24.6
Detector-free + Naive Association NN-JamMA 59.8 83.0 61.9 58.0 41.1
Detector-free + Naive Association NN-EDM 55.1 79.7 57.4 51.9 34.3
Detect by Track (Ours) DeT-JamMA 68.7 88.0 69.6 65.5 44.9
Detect by Track (Ours) DeT-EDM 55.4 79.5 59.4 55.2 35.7

Ablation Study

To quantify the trade-off of anchoring detector-free matching to historical tracks versus unconstrained pairwise matching, an ablation study on two-view relative pose estimation was conducted on MegaDepth using an initial homography-synthesized pair.

Table 3: Two-View Relative Pose and Trackability Ablation (Source Table 3)

Method Trackable Detected Pairs/Tracks Correct (%) Pose AUC@\(5^\circ\) Pose AUC@\(10^\circ\) Pose AUC@\(20^\circ\)
EDM Baseline No 2395 93.0 64.5 77.7 86.8
JamMA Baseline No 4021 88.9 64.1 77.4 86.5
DeT-EDM (Ours) Yes 2393 93.0 63.6 77.0 86.3
DeT-JamMA (Ours) Yes 2427 90.9 61.4 75.1 84.9

Key Findings

  • In multi-frame tracking benchmarks across 5 consecutive views, naive nearest-neighbor association (NN-JamMA/NN-EDM) suffered rapid track loss due to sub-pixel drift. In contrast, DeT retained more than five times as many correct tracks (epipolar error \(< 10^{-3}\)) compared to SuperPoint + LightGlue over challenging sequences.
  • In incremental SLAM benchmarks, DeT-JamMA attained the highest pose AUC across all tested scenes and registered over four times as many valid 3D points as sparse detector baselines, paving the way for direct semi-dense 3D mapping.
  • The two-view ablation confirms that converting a detector-free matcher into a trackable model incurs only a negligible drop in pose AUC@\(5^\circ\) (from 64.1 to 61.4 on JamMA), while fully eliminating multi-view track fragmentation.

Highlights & Insights

  • Repurposing Translational Equivariance: The method identifies and verifies the empirical translational equivariance of post-mixer correlation matrices, bridging the gap between discrete grid matching and continuous sub-pixel querying without changing network weights.
  • Plug-and-Play Zero-Shot Conversion: As a pure inference-time modification, DeT seamlessly integrates into distinct detector-free backbones (such as JamMA's Mamba architecture and EDM's correlation injection) with less than 2% computational overhead.
  • Unifying Matching and Tracking: It resolves the historic dichotomy between high-accuracy detector-free pairwise matching and multi-view geometric consistency, demonstrating that dense image-space interactions can natively support online incremental tracking.

Limitations & Future Work

  • Translational Motion & Depth Homogeneity Assumption: Pseudo-track prediction relies on the assumption that the nearest neighbor share similar depth and motion. Near depth discontinuities or under severe out-of-plane rotations, large motion violations can place the true target outside the fine window \(\kappa \times \kappa\).
  • Matching Conflict Under Zoom Scenarios: Detector-free matchers typically enforce one-to-one matching per coarse patch; during zoom-out motions, track counts drop significantly due to physical resolution bottlenecks.
  • Closed-Loop Consistency & End-to-End Tuning: The current formulation focuses on open-ended chain tracking; incorporating multi-frame temporal losses or loop closure constraints into a fine-tuned variant presents a promising future direction.
  • vs. Detector-Free SfM / Dense-SfM: Prior methods adapt detector-free matching to SfM via coarse grid discretization followed by heavy transformer optimization or Gaussian splatting post-processing, which are fundamentally offline batch algorithms. DeT directly yields connected sub-pixel tracks online, enabling real-time incremental VO and SLAM.
  • vs. CoTracker / TAPIR (Point Tracking): While point tracking models excel at smooth video streams with small baselines, they struggle with wide-baseline, large-disparity, unordered image collections. DeT inherits the large receptive fields and robust similarity maps of detector-free matchers, making it ideal for wide-baseline geometric reconstruction.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant exploitation of local similarity translational equivariance to solve multi-view fragmentation in detector-free matchers.
  • Experimental Thoroughness: ⭐⭐⭐⭐☆ Comprehensive evaluation spanning low-level track survival, multi-view pose estimation, incremental SLAM, and two-view ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulation, insightful failure mode analysis, and intuitive visual illustrations.
  • Value: ⭐⭐⭐⭐⭐ Provides an immediate, training-free mechanism to deploy state-of-the-art detector-free matchers in practical real-time robotic SLAM and tracking pipelines.