Skip to content

Ex-Sim(3)-Reg: 2D-3D Correspondence Pruning via Extended Sim(3) Registration

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/anpei96/ex-sim3-demo
Area: Model Compression
Keywords: Image-to-point-cloud registration, Sim(3) registration, monocular depth noise, 2D-3D correspondence pruning, compatibility graph

TL;DR

Addressing the vulnerability of depth-prior-based 2D-3D correspondence pruning to scale noise, this paper reformulates pruning as an extended Sim(3) registration problem with correspondence-specific scale perturbations, and introduces Ex-Sim(3)-Reg to convert it into a high-inlier SE(3) registration via scale-parameterized compatibility graphs, achieving up to 24.7% improvement in registration recall.

Background & Motivation

Image-to-point-cloud (I2P) registration is a foundational technology for 3D visual localization, large-scale mapping, and cross-sensor calibration. While recent learning-based I2P matching models demonstrate strong capabilities on clean benchmark sets, they suffer severe degradation in real-world scenarios characterized by unseen environments, feature corruptions, or extreme outlier ratios. Under such stress conditions, feature matchers output immense numbers of spurious 2D-3D correspondences. Classical RANSAC-P3P solvers fail catastrophically when the inlier ratio drops sharply, whereas branch-and-bound global optimizations remain prohibitively slow, and learning-based pruning networks require dense training supervision or heavy compute budgets.

To bypass these hurdles, an emerging and elegant paradigm lifts 2D-3D correspondences into 3D-3D point pairs using monocular depth priors from visual foundation models such as Depth Anything. With metric depth available, correspondence pruning can be formulated as an unconstrained similarity registration—specifically, a standard Sim(3) problem solvable via voting and compatibility graph optimization without neural networks. However, depth priors estimated from monocular images suffer inherently from scale ambiguity and spatially non-uniform errors. Standard Sim(3) algorithms enforce a single, rigid global scale factor across all correspondences; this brittle rigidity assumption causes true inliers perturbed by local depth noise to be wrongly discarded as outliers.

This paper tackles this bottleneck by directly confronting and formalizing the noise distribution of monocular depth priors. Core idea: model monocular depth noise as correspondence-specific scale perturbations, reformulate pruning as an extended Sim(3) registration problem, and solve it by searching for maximum cliques across a family of scale-parameterized compatibility graphs to reduce the underdetermined problem into robust SE(3) registration.

Method

Overall Architecture

The Ex-Sim(3)-Reg pipeline operates across four coordinated stages: lifting candidate 2D-3D correspondences into 3D-3D pairs via monocular depth estimation, sampling discrete scale hypotheses around an adaptive global voting anchor, constructing second-order extended compatibility graphs to detect scale-consistent maximum cliques, and transforming the scale-corrected correspondences into standard SE(3) space for initial pose estimation followed by reprojection-filtered RANSAC-P3P refinement.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: 2D-3D correspondences + monocular depth prior"] --> B["Scale Hypothesis Sampling<br/>adaptive voting determines localized scale search range"]
    B --> C["Compatibility Graph Maximum Clique Search<br/>build second-order graphs and extract maximum cliques"]
    C --> D["Scale-Correction Function<br/>scale 3D points and aggregate inlier candidate set"]
    D --> E["Pose Estimation & Reprojection Refinement<br/>SE(3) registration prior + RANSAC-P3P refinement"]
    E --> F["Output: optimized camera pose and inlier correspondences"]

Key Designs

1. Scale Hypothesis Sampling: Discretizing the Underdetermined Scale Solution Space

In extended Sim(3) registration, each inlier satisfies \(p_i = s_i R q_i + t\) with \(s_i = s \cdot \delta_i\) capturing both a global scale \(s\) and a local scale perturbation \(\delta_i\). This formulation introduces \(2N+6\) unknowns, rendering continuous global optimization underdetermined. To make the problem tractable, the algorithm first employs an adaptive voting scheme based on pairwise pairwise Euclidean distance ratios to determine a coarse global scale estimate \(\tilde{s}\). Rather than performing an intractable exhaustive search over \((0, +\infty)\), the search space is constrained to a localized interval \((\tilde{s} - \Delta s, \tilde{s} + \Delta s)\), from which \(K\) discrete scale hypotheses \(\{\zeta_k\}_{k=1}^K\) are uniformly sampled. This bounds the hypothesis search to the high-probability neighborhood of the true scene scale.

2. Compatibility Graph Maximum Clique Search: Topology-Based Outlier Rejection

For each sampled scale hypothesis \(\zeta_k\), a first-order extended compatibility graph \(\mathcal{G}_{\text{FOG}}(\zeta_k)\) is constructed with an adjacency matrix checking pairwise Euclidean distance consistency: $\(W_{\text{FOG}}(\zeta_k)_{ij} = \mathbb{I}\left( \big| \|p_i - p_j\|_2 - \|\zeta_k(q_i - q_j)\|_2 \big| \le d_{\text{thr}} \right)\)$ To suppress accidental first-order compatibility edges between outliers, a second-order extended compatibility graph is computed via Hadamard product and matrix multiplication: \(\mathbf{W}_{\text{SOG}}(\zeta_k) = \mathbf{W}_{\text{FOG}}(\zeta_k) \circ (\mathbf{W}_{\text{FOG}}(\zeta_k) \cdot \mathbf{W}_{\text{FOG}}(\zeta_k))\). The algorithm then searches for the maximum clique \(\mathrm{Mac}(\mathcal{G}_{\text{SOG}}(\zeta_k))\) using FastMAC. Theoretical analysis guarantees that the subset of true inliers consistent with scale \(\zeta_k\) is contained within the maximum clique of the practical graph, effectively isolating geometrically consistent clusters from dense outlier contamination.

3. Scale-Correction Function: Reducing Extended Sim(3) to Rigid SE(3)

For correspondences belonging to the maximum clique at scale \(\zeta_k\), denoted \(d_i = (q_i, p_i) \in \mathrm{Mac}(\mathcal{G}_{\text{SOG}}(\zeta_k))\), the 3D point \(q_i\) still possesses scale distortion. The scale-correction function \(\varphi(d_i, \zeta_k) = (\zeta_k \cdot q_i, p_i) = e_i\) rescales the back-projected 3D point using its matching hypothesis \(\zeta_k\). Taking the union across all \(K\) scale hypotheses produces the unified correspondence set: $\(\{e_j\}_{j=1}^M = \bigcup_{k=1}^K \varphi\big(\mathrm{Mac}(\mathcal{G}_{\text{SOG}}(\zeta_k)), \zeta_k\big)\)$ This transformation eliminates the correspondence-specific scale factors \(s_i\) and reduces the complex extended Sim(3) formulation into a standard rigid SE(3) transformation \(p_i = R e_i + t\). The resulting set \(\{e_j\}_{j=1}^M\) demonstrates a dramatically improved inlier ratio (boosted from ~20% to over 50%–70%) while enforcing Euclidean rigidity.

4. Pose Estimation & Reprojection Refinement: Coarse-to-Fine Geometric Optimization

On the corrected candidate set \(\{e_j\}_{j=1}^M\), an off-the-shelf SE(3) registration solver (such as SVD-based pose estimation) computes initial camera rotation \(\tilde{R}\) and translation \(\tilde{t}\). Because this estimate is derived from a high-inlier consensus set, it sits in close proximity to the ground truth. This pose is subsequently supplied as a prior to RANSAC-P3P: an initial pre-filtering stage removes candidate correspondences exceeding a reprojection error threshold \(\delta_{\text{thr}}\), followed by low-iteration P3P optimization over the surviving clean correspondences to yield the refined camera pose \((\tilde{R}_{\text{opt}}, \tilde{t}_{\text{opt}})\) and definitive inlier assignments \(\{w_i\}_{i=1}^N\).

Key Experimental Results

Main Results

The method was evaluated on 7-Scenes, TUM, ScanNet, and RGBD-v2 using correspondences generated by MATR under three difficult regimes: disturbed, unseen domain transfer, and low-inlier density. The table below presents inlier ratio (IR) and registration recall (RR, 5cm threshold) on 7-Scenes under adversarial feature distortion:

Method Formulation / Solver Inlier Ratio (IR) ↑ Registration Recall (RR) ↑ Gain over Baseline
Baseline RANSAC-P3P (no pruning) 0.266 0.292 -
Sim(3)-FastMAC Standard Sim(3) + FastMAC [38] 0.333 0.283 -0.9%
Sim(3)-SC2-PCR Standard Sim(3) + SC2-PCR [11] 0.451 0.275 -1.7%
Sim(3)-SC2-PCR++ Standard Sim(3) + SC2-PCR++ [10] 0.268 0.073 -21.9%
Sim(3)-MAC Standard Sim(3) + MAC [37] 0.333 0.275 -1.7%
Sim(3)-TurboReg Standard Sim(3) + TurboReg [29] 0.333 0.343 +5.1%
Ex-Sim(3)-Reg (Ours) Extended Sim(3) + Compatibility Graphs 0.548 0.472 +18.0%

Under low-inlier conditions on RGBD-v2 where standard geometric constraints collapse, Ex-Sim(3)-Reg achieves state-of-the-art performance with a 24.7% recall gain:

Dataset Setting Metric Baseline Second Best Baseline (Sim(3)-SC2-PCR) Ex-Sim(3)-Reg (Ours) Net Gain
7-Scenes (Disturbed) IR / RR 0.266 / 0.292 0.451 / 0.275 0.548 / 0.472 +28.2% / +18.0%
TUM (Disturbed) IR / RR 0.230 / 0.117 0.378 / 0.077 0.485 / 0.302 +25.5% / +18.5%
7-Scenes → ScanNet (Unseen) IR / RR 0.409 / 0.431 0.579 / 0.260 0.662 / 0.635 +25.3% / +20.4%
RGBD-v2 (Low-inlier) IR / RR 0.166 / 0.295 0.249 / 0.355 0.327 / 0.542 +16.1% / +24.7%

Ablation Study

Ablation experiments on 7-Scenes validate the impact of the scale interval count \(K\), pixel reprojection threshold \(\delta_{\text{thr}}\), and individual algorithmic components (from Tables 8, 9, and 10 of the original paper):

Configuration Setting / Component IR ↑ RR ↑ Runtime / RTE Note
Scale intervals \(K\) \(K=4\) 0.505 0.362 57.6 ms Undersampling leaves inliers uncovered
Scale intervals \(K\) \(K=6\) 0.542 0.394 65.4 ms Optimal balance of speed and coverage
Scale intervals \(K\) \(K=8\) 0.548 0.472 73.6 ms Recommended standard setting
Scale intervals \(K\) \(K=12\) 0.682 0.427 89.5 ms Excessive hypotheses introduce noise
Threshold \(\delta_{\text{thr}}\) \(\delta_{\text{thr}}=8\) px 0.513 0.380 0.072 m Overly strict; discards valid inliers
Threshold \(\delta_{\text{thr}}\) \(\delta_{\text{thr}}=16\) px 0.548 0.472 0.067 m Best trade-off across IR, RR, and RTE
Component Ablation FastMAC alone (w/o \(\varphi\)) - 0.311 72.4 ms Lack of scale correction degrades pose
Component Ablation MAC [37] + correction \(\varphi\) - 0.485 204.8 ms High recall but 2.78x slower
Component Ablation FastMAC [38] + correction \(\varphi\) - 0.472 73.6 ms Full proposed model, efficient and accurate

Key Findings

  • Rigid Sim(3) Vulnerability: Standard Sim(3) methods frequently underperform the baseline RANSAC-P3P under noisy depth priors (e.g., SC2-PCR RR drops from 0.292 to 0.275; SC2-PCR++ drops to 0.073) because rigid scale assumptions over-prune valid correspondences affected by monocular scale ambiguity.
  • Necessity of Scale Correction: Removing the scale correction function \(\varphi(\cdot)\) precipitates a 16.1% drop in registration recall (0.472 → 0.311), confirming that compensating for point-wise scale perturbations is the primary mechanism driving performance gains.
  • Plug-and-Play Generalizability: Integrating Ex-Sim(3)-Reg into state-of-the-art I2P matchers (FreeReg, Top-I2P, MinCD, Diff-I2P) boosts cross-domain registration recall on ScanNet to 93%–96%, offering consistent gains of 4%–15% over default matching pipelines.

Highlights & Insights

  • Correspondence-Specific Scale Perturbations: Rather than treating monocular depth predictions as noise-free or relying on a single scalar factor, modeling depth noise as \(s_i = s \cdot \delta_i\) grounds the geometric uncertainty of monocular depth models in a mathematically rigorous framework.
  • Tractable Clique Union for Underdetermined Problems: Confronted with an underdetermined optimization with \(2N+6\) variables, the authors bypass complex non-convex continuous solvers by constructing a family of scale-parameterized compatibility graphs, using the union of discrete maximum cliques to approximate the ideal solution.
  • Training-Free Lightweight Architecture: Operating completely without learned parameters, the algorithm executes in ~73 ms and acts as a generic geometric filter compatible with any upstream 2D-3D matching network.

Limitations & Future Work

  • Theoretical Inlier Incompleteness: As proved by the authors, the recovered inlier set is bounded by the union \(\bigcup_{k=1}^K F_k\). Even as \(K \to +\infty\), this remains a strict subset of the true inlier set, leaving out correspondences with extreme scale disparities.
  • Sensitivity to Coarse Voting Initialization: The search window depends on an initial scale estimate \(\tilde{s}\) from adaptive voting. If extreme outlier contamination invalidates this initial anchor, the sampled range may miss the true underlying scale.
  • Future Directions: Exploring non-uniform adaptive scale sampling or formulating continuous spectral relaxations for joint scale and pose recovery represents a promising avenue for further improvement.
  • vs Standard Depth-Prior Sim(3) (Yu et al. / Bie et al.): Prior works assume uniform metric depth scaling, which breaks down under realistic depth noise. Ex-Sim(3)-Reg relaxes this to per-correspondence perturbations, preserving dense inlier consensus.
  • vs Point Cloud Graph Solvers (MAC / SC2-PCR): Classical methods depend on strict Euclidean distance conservation. Ex-Sim(3)-Reg generalizes compatibility graphs to multi-scale spaces, establishing a valuable tool for cross-modal registration with scale ambiguities.

Rating

  • Novelty: ⭐⭐⭐⭐☆ [First to formulate depth-prior pruning as an extended Sim(3) problem with scale-parameterized compatibility graphs]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive validation across four benchmarks under feature corruption, domain shifts, and extreme low-inlier regimes]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical derivation, rigorous theoretical graph proofs, and cohesive narrative]
  • Value: ⭐⭐⭐⭐☆ [Training-free, highly efficient, and offers substantial plug-and-play improvements for visual localization and I2P registration]