OCA: ODE-Driven Cross-Attention for Image-to-Point-Cloud Registration

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/anpei96/oca-i2p-demo
Area: 3D Vision
Keywords: Image-to-point-cloud registration, cross-attention, ordinary differential equations, attention ambiguity, cross-modal feature interaction

TL;DR

Addressing the fundamental attention ambiguity caused by cross-modal discrepancy in image-to-point-cloud (I2P) registration, this paper formulates cross-attention as continuous assignment ordinary differential equations (ODEs) and introduces a non-parametric, plug-and-play module (OCA) that iteratively purifies 2D-3D correspondences via dynamical propagation with negligible runtime overhead (+6ms).

Background & Motivation

Image-to-point-cloud (I2P) registration is a core foundation for computer vision and robotic spatial perception, aiming to establish reliable pixel-to-point correspondences between an unaligned 2D RGB image and a 3D point cloud, thereby estimating the 6-DoF camera pose within the point cloud coordinate system. The task underpins key real-world systems ranging from visual camera relocalization and autonomous mobile robot state estimation to large-scale point cloud colorization and cross-modal SLAM. In recent years, deep learning pipelines equipped with Transformer-based feature interaction (notably 2D3D-MATR and successors) have become the predominant paradigm, leveraging cross-attention layers to bridge the semantic and geometric gap between 2D texture representations and 3D irregular coordinates.

Despite considerable empirical success, conventional cross-attention suffers from severe attention ambiguity when transferred from intra-modal matching to cross-modal scenarios. Because 2D images and 3D point clouds differ drastically in modality structure, density, and feature distribution, naive soft cross-attention matrices frequently assign deceptively high correlation scores to false correspondences and geometric outliers. While recent literature has introduced manifold alignment, feature uncertainty correction, or covariance constraints to alleviate noisy weights, these designs mostly rely on empirical discrete heuristics and lack a rigorous dynamic understanding of how ideal cross-modal feature interactions evolve over time.

From a mathematical perspective, standard Transformer cross-attention can be viewed as an explicit Euler discretization of an underlying continuous-time dynamical system. By defining the stationary state of ideal correspondence matching through continuous differential equations, one can analyze the exact stability conditions required to drive the assignment matrix toward the ground-truth permutation. Core idea: reformulate 2D-3D cross-attention as continuous-time assignment ODEs, derive sparsity and initialization conditions from dynamical convergence analysis, and construct a lightweight, non-parametric ODE-driven cross-attention (OCA) module that iteratively suppresses cross-modal attention ambiguity.

Method

Overall Architecture

The OCA framework comprises two cohesive components: the theoretical formulation of continuous assignment ODEs and their practical discrete algorithmic realization. Designed as a modular, plug-and-play unit, OCA is inserted directly after the feature interaction module of existing I2P registration architectures without requiring any structural alterations to the upstream feature extractors or downstream pose estimators. Given the patch-level or point-level feature tokens \(x[0] \in \mathbb{R}^{N \times c}\) from the 2D image and \(y[0] \in \mathbb{R}^{M \times c}\) from the 3D point cloud, OCA outputs purified, modality-invariant representations.

The operational flow consists of two consecutive stages: first, attention initialization constructs a metric-aligned, temperature-pruned correlation matrix \(\rho_{\text{sparse}}(\mathbf{A}[0])\) via unit \(L_2\) feature normalization; second, attention propagation iteratively advances the correlation matrix and the features across discrete ODE integration steps, utilizing intra-modal Gram covariances to prune ambiguous correspondences before performing weighted residual fusion.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input features: 2D image tokens x[0] and 3D point tokens y[0]"] --> B["Assignment ODE Formulation<br/>Model ideal feature interaction and derive Euler equivalence"]
    B --> C["Attention Initialization<br/>L2 normalization inner-product + temperature-based pruning"]
    C --> D["Attention Propagation Iteration<br/>Discrete ODE update guided by covariance operators"]
    D --> E["Dynamic Feature Purification<br/>Cross-step feature integration and residual weighting"]
    E --> F["Purified output features: Feed to downstream camera pose solver"]

Key Designs

1. Assignment ODE Formulation: Unifying Continuous Feature Dynamics with Cross-Attention

Existing cross-attention layers lack a principled theoretical limit describing ideal interaction. Assuming \(\mathcal{C}\) denotes the true set of 2D-3D correspondences and \(\mathbf{A}_{gt} \in \{0, 1\}^{N \times M}\) represents the ground-truth binary assignment matrix, an ideal interaction mandates identical feature representations \([\mathbf{x}']_i = [\mathbf{y}']_j\) for all \(\langle i, j \rangle \in \mathcal{C}\). To mimic this target in an unsupervised continuous formulation, the authors model features and assignment weights as continuous-time trajectories, defining the assignment ODEs:

\[ \frac{d\mathbf{x}(t)}{dt} = \rho(\mathbf{A}(t))\mathbf{y}(t), \quad \frac{d\mathbf{y}(t)}{dt} = (\rho(\mathbf{A}(t)))^T \mathbf{x}(t) \]

where \(\mathbf{A}(t) = \mathbf{x}(t)\mathbf{y}(t)^T\) and \(\rho(\cdot)\) denotes a row-normalization operator such as softmax. Applying explicit forward Euler discretization to this continuous system yields a discrete update step that is mathematically equivalent to the standard scaled dot-product cross-attention layer used in Transformers. This derivation establishes that cross-attention is an empirical single-step numerical solver of continuous assignment ODEs, opening the door to analyzing attention stability using dynamical systems theory.

2. Dynamical Convergence Analysis: Deriving Dual Conditions for Ambiguity Suppression

Eliminating attention ambiguity is mathematically equivalent to requiring the dynamical system to converge toward the ground truth as \(t \to +\infty\), meaning \(\rho(\mathbf{A}(+\infty)) \approx \mathbf{A}_{gt}\). Taking the time derivative yields \(\frac{d\rho(\mathbf{A}(t))}{dt} = \rho'(\mathbf{A}(t)) \left( \rho(\mathbf{A}(t))\mathbf{Y}(t) + \mathbf{X}(t)\rho(\mathbf{A}(t)) \right)\), where \(\mathbf{X}(t) = \mathbf{x}(t)\mathbf{x}(t)^T\) and \(\mathbf{Y}(t) = \mathbf{y}(t)\mathbf{y}(t)^T\) represent intra-modal covariance matrices. Analysis of the stationary point \(\frac{d\rho(\mathbf{A}(t))}{dt} = 0\) reveals two vital convergence criteria: condition (C1) where \(\rho'(\mathbf{A}(t)) = 0\), which holds when \(\mathbf{A}(t)\) approaches a sparse permutation matrix, and condition (C2) where the bracketed Sylvester equation vanishes (which does not yield practical benefits in registration).

These findings motivate two operational strategies: (S1) enforcing row-wise sparsity on \(\rho(\mathbf{A}(t))\) to push \(\rho'(\mathbf{A}(t))\) toward zero, and (S2) ensuring that the initial matrix \(\rho(\mathbf{A}(0))\) is as close as possible to the ground-truth assignment. Standard Transformers fail on both counts because dense softmax preserves noisy diffuse attention and unnormalized inner products violate \(L_2\) correspondence metrics.

3. Attention Initialization and Numerical Propagation: Efficient Parameter-Free Numerical Updates

To translate strategies (S1) and (S2) into a tractable module, OCA executes an initialization step followed by iterative propagation. During initialization, features are strictly unit-normalized to align with downstream \(L_2\) Euclidean matching, and a temperature scaling hyperparameter \(\gamma \ge 1\) sharpens row distributions to achieve differentiable soft pruning:

\[ \mathbf{A}[0] = \mathbf{x}_{\text{norm}}[0] \mathbf{y}_{\text{norm}}[0]^T, \quad \rho_{\text{sparse}}(\mathbf{A}[0]) = \text{softmax}(\gamma \mathbf{A}[0]) \]

Next, attention propagation computes discrete updates over \(T\) iterations using a step size \(\tau \in [0, 1]\). Corrected covariance matrices \(\mathbf{X}_{\text{norm}}[k] = \mathbf{x}_{\text{norm}}[k]\mathbf{x}_{\text{norm}}[k]^T\) and \(\mathbf{Y}_{\text{norm}}[k] = \mathbf{y}_{\text{norm}}[k]\mathbf{y}_{\text{norm}}[k]^T\) dynamically steer matrix trajectories:

\[ \mathbf{A}[k+1] = \mathbf{A}[k] + \tau \left( \rho_{\text{sparse}}(\mathbf{A}[k])\mathbf{Y}_{\text{norm}}[k] + \mathbf{X}_{\text{norm}}[k]\rho_{\text{sparse}}(\mathbf{A}[k]) \right) \]

\[ \mathbf{x}[k+1] = \mathbf{x}[k] + \tau \rho_{\text{sparse}}(\mathbf{A}[k+1])\mathbf{y}[k], \quad \mathbf{y}[k+1] = \mathbf{y}[k] + \tau \rho_{\text{sparse}}(\mathbf{A}[k+1]^T)\mathbf{x}[k] \]

Through this continuous integration process, spurious 2D-3D pairings lacking mutual geometric support are decayed, while true structural correspondences are strengthened. The final representation is produced via a residual weighted combination between the initial and propagated features with factor \(\omega\), without introducing a single learnable weight.

Loss & Training

Because OCA introduces zero learnable weights, it relies on a staged training protocol to ensure strategy (S2) holds (i.e., providing high-quality initialization for the dynamical trajectory): 1. Stage 1 (Warm-up Base Training): Setting time step \(\tau = 0\) deactivates the ODE propagation, training the host registration architecture from scratch for 25 epochs using its standard Circle Loss and overlap classification objectives. 2. Stage 2 (Dynamical Fine-tuning): Restoring the nominal step size (e.g., \(\tau = 0.10\)), the entire network is fine-tuned for 7 additional epochs (32 epochs in total). This allows the feature representation to adjust to the ODE propagation dynamics without destabilizing early representation learning.

Key Experimental Results

Main Results

Evaluations were conducted across four recognized indoor benchmarks: 7-Scenes (standard in-domain protocol), RGBD-v2, TUM, and ScanNet. Five representative state-of-the-art baselines were tested: Matr (ICCV'23), Flow-I2P (IJCV'25), Bridge (AAAI'25), CA-I2P (ICCV'25), and LDF-I2P (TIM'25). The primary evaluation metrics are Inlier Ratio (IR, threshold 5cm) and Registration Recall (RR, threshold 10cm). To ensure the performance boost is genuinely attributable to ODE dynamics rather than prolonged training schedules, an extended-training control group (X+Ref) was trained with the exact same epoch count (32 epochs) and optimizer configurations.

The table below summarizes standard evaluation on 7-Scenes and transfer fine-tuning on TUM:

Dataset / Scene	Method Variant	Inlier Ratio (IR)	Registration Recall (RR)	Relative Improvement
7-Scenes (Standard Split)	Matr [16]	0.453	0.472	-
7-Scenes	Matr + Ref	0.475	0.501	Baseline prolonged training
7-Scenes	Matr + OCA	0.501	0.552	RR +5.1% vs Ref
7-Scenes	Flow-I2P [2]	0.469	0.511	-
7-Scenes	Flow-I2P + Ref	0.459	0.562	-
7-Scenes	Flow-I2P + OCA	0.530	0.596	RR +3.4% vs Ref
7-Scenes	Bridge [6] + OCA	0.529	0.550	RR +3.0% vs Ref
7-Scenes	CA-I2P [7] + OCA	0.524	0.565	RR +3.1% vs Ref
7-Scenes	LDF-I2P [21] + OCA	0.525	0.591	RR +4.0% vs Ref
TUM (Transfer Fine-tuning)	Matr [16]	0.568	0.472	-
TUM	Matr + Ref	0.629	0.647	-
TUM	Matr + OCA	0.703	0.705	RR +5.8% (vs Baseline +23.3%)
TUM	LDF-I2P [21] + Ref	0.643	0.675	-
TUM	LDF-I2P + OCA	0.710	0.764	RR +8.9% vs Ref

In zero-shot cross-dataset evaluation (training solely on 7-Scenes and directly testing on ScanNet without fine-tuning), Flow-I2P + Zero-OCA improves Registration Recall from 26.3% to 41.4% (+15.1% absolute gain). Similarly, Bridge + OCA achieves a +11.8% recall boost, validating superior generalizability under cross-domain distribution shifts.

Compared against generative propagation baselines on TUM: - Matr + Simple DDPM: IR 0.465 / RR 0.460 - Matr + Simple FM: IR 0.448 / RR 0.459 - Diff-Reg (ECCV'24): IR 0.623 / RR 0.602 - Matr + OCA: IR 0.703 / RR 0.705 (markedly superior to both diffusion and flow-matching formulations).

Ablation Study

Ablation investigations conducted with Matr on 7-Scenes validate the dynamical hyperparameters:

Parameter Studied	Tested Configurations	Inlier Ratio (IR)	Registration Recall (RR)	Dynamical Mechanism Analysis
Iterations \(T\) (\(\tau=0.10\))	\(T = 1\)	0.571	0.489	Under-propagated; noisy cross-modal outliers remain
Iterations \(T\)	\(T = 2\)	0.554	0.510	Steady convergence trajectory
Iterations \(T\)	\(T = 3\) (Default)	0.501	0.552	Optimal trade-off; high-confidence inliers govern pose solve
Iterations \(T\)	\(T = 4\)	0.433	0.575	Modest recall bump at the expense of aggressive correspondence culling
Iterations \(T\)	\(T = 5\)	0.415	0.539	Overfitting to dominant pairs degrades overall accuracy
Time Step \(\tau\) (\(T=3\))	\(\tau = 0.05\)	0.562	0.495	Step size too conservative; integration incomplete
Time Step \(\tau\)	\(\tau = 0.10\) (Default)	0.501	0.552	Stable numerical integration regime
Time Step \(\tau\)	\(\tau = 0.20\)	0.400	0.497	Truncation error induces numerical oscillation and divergence
Temperature \(\gamma\)	\(\gamma = 1\)	0.531	0.527	Insufficient sparsity leads to non-vanishing derivative norm
Temperature \(\gamma\)	\(\gamma = 2\) (Default)	0.501	0.552	Optimal balance between selective pruning and smooth gradients
Temperature \(\gamma\)	\(\gamma = 8\)	0.435	0.532	Extreme sharpness traps integration in suboptimal local modes

Key Findings

• Stability Regime of ODE Dynamics: As propagation iterations \(T\) increase from 1 to 3, the Inlier Ratio decreases slightly as low-confidence matches are progressively culled, but Registration Recall surges due to heightened metric precision. Exceeding \(T = 4\) or \(\tau = 0.15\) triggers numerical instability and representation drift.
• Negligible Computational Overhead: On an NVIDIA RTX 3080 GPU, the pure feed-forward inference of Matr takes 0.132 seconds, while Matr+OCA requires 0.138 seconds. The entire 3-step ODE integration adds merely 6 milliseconds, proving exceptionally cost-effective.
• Mechanism of Zero-Shot Robustness: In zero-shot transfer, initial representations contain substantial noise. OCA leverages intra-modal Gram matrices \(\mathbf{X}\) and \(\mathbf{Y}\) as physical self-consistency constraints, systematically extinguishing isolated cross-modal noise and boosting zero-shot recall by up to 15%.

Highlights & Insights

• Continuous Dynamical Perspective on Cross-Attention: Moves beyond viewing cross-attention solely as discrete matrix multiplications, establishing that it is an explicit Euler discretization of assignment ODEs, thereby establishing a solid analytical bridge between neural attention and differential equations.
• Non-Parametric Geometry Refinement: Unlike methods that append massive pre-trained foundation models (e.g., SAM or DINO) or complex multi-layer decoders, OCA is entirely non-parametric and operates via pure matrix operations, seamlessly adapting to various upstream backbones.
• Intra-Modal Gram Guidance: Uses single-modality self-correlation Gram matrices \(\mathbf{x}\mathbf{x}^T\) and \(\mathbf{y}\mathbf{y}^T\) to regularize the derivative of the cross-attention matrix, enforcing spatial and structural coherence across modalities.

Limitations & Future Work

• Vulnerability to Severe Texture Deprivation: In completely textureless environments (e.g., bare featureless walls), initial feature correspondences degenerate into pure noise. Under such corrupt initial conditions, ODE propagation lacks valid attractors, causing inlier ratios to plummet (as low as 7.6%).
• First-Order Numerical Truncation: The current implementation utilizes standard first-order forward Euler discretization with fixed step size \(\tau\). Incorporating adaptive-step higher-order Runge-Kutta integrators could provide superior stability across uneven loss surfaces.
• Absence of Closed-Loop Feedback: ODE propagation currently operates as an open-loop roll-out. Introducing dynamic confidence estimation to modulate the integration horizon \(T\) dynamically could enhance robustness against local traps.

Related Work & Insights

• vs 2D3D-MATR [16]: MATR relies on static Transformer cross-attention, which remains vulnerable to ambiguous cross-modal similarities; OCA upgrades this interaction into continuous ODE propagation, lifting 7-Scenes recall from 47.2% to 55.2% with zero extra parameters.
• vs Flow-I2P [2]: Flow-I2P introduces Beltrami flow for feature smoothing in image coordinate space; OCA directly establishes ODE dynamics over the bipartite cross-modal assignment matrix, offering broader conceptual generality and providing an additive +3.4% recall boost when combined with Flow-I2P.
• vs Diff-Reg [30]: Diff-Reg uses a parametric score-based diffusion model on doubly stochastic matrices, incurring heavy training overhead and slow multi-step reverse sampling; OCA relies on a deterministic, closed-form 3-step ODE requiring only 6ms, consistently outperforming diffusion alternatives.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant mathematical unification of cross-attention with continuous assignment ODEs.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-dataset, zero-shot, ablation, and runtime benchmarks across five SOTA baselines.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous theoretical derivations, intuitive visual motivations, and objective discussions.
• Value: ⭐⭐⭐⭐⭐ Plug-and-play, non-parametric, and adding only 6ms of compute, offering immediate utility for real-time robotic localization and cross-modal registration.