RoMa v2: Harder Better Faster Denser Feature Matching¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/Parskatt/RoMaV2
Area: Dense Feature Matching / Multi-View Geometry
Keywords: frozen visual features, multi-view matching, subpixel refinement, predictive covariance, resolution robustness

TL;DR¶

RoMa v2 combines multi-view coarse matching over frozen DINOv3 features, separately trained efficient refiners, and pixel-wise uncertainty prediction to improve challenging-viewpoint matching and subpixel precision, reducing RoMa's AerialMegaDepth EPE from 25.05 to 4.12 pixels without claiming universal superiority in extreme cross-modal generalization or absolute speed.

Background & Motivation¶

Visual localization and 3D reconstruction require identifying pixels in two photographs that observe the same spatial point, rather than merely judging whether the photographs are semantically similar. Sparse methods first detect repeatable keypoints and then establish correspondences, which can leave texture-poor regions uncovered; dense matching instead predicts a target position and confidence for every pixel. RoMa improves robustness to appearance changes through frozen foundation-model features, but its coarse matching primarily relies on a Gaussian process and single-view decoding, limiting cross-image contextual interaction. Reliable coverage and local precision remain difficult under extreme viewpoints, aerial-to-ground changes, and motion of small objects.

UFM shows that training inspired by optical flow can substantially accelerate dense correspondence estimation and improve small-baseline cases. However, fine-tuning the pretrained encoder can weaken transfer under extreme appearance changes, and its subpixel precision trails RoMa. Geometric estimation can amplify this difference: correspondences that look plausible but have small positional errors may substantially reduce camera-pose accuracy at strict thresholds. The paper therefore addresses coarse matching robustness, refinement bias, and execution cost together, rather than pursuing only more matches or a faster network.

The authors retain a frozen encoder, add learnable multi-view interaction, and assign distinct training and computational responsibilities to the stages. They also expand the training distribution beyond RoMa's MegaDepth-only setting to a mixture of wide and small baselines, reducing specialization to a single type of geometric change. Core Idea: locate the correct region with more informative coarse matching supervision, then recover trustworthy subpixel correspondences through efficient, low-bias refinement with directional error estimates.

Method¶

Overall Architecture¶

The inputs are two images of the same scene, and the outputs comprise dense coordinate warps in both directions, covisibility confidence, and a pixel-wise precision matrix for two-dimensional errors. The warp specifies where a source pixel maps in the target image, confidence indicates whether the spatial point is visible in the other view, and the precision matrix describes positional uncertainty along different directions. Training distribution and resolution design govern training; at inference time, multi-view coarse matching produces a stride-4 coordinate field, followed by efficient refinement at strides 4, 2, and 1 with directional uncertainty prediction. Camera pose is not a direct network output: a downstream geometric estimator can recover it from the correspondences and uncertainty estimates.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Train["Training Distribution<br/>and Resolution"] -.->|Training sampling and scale conventions| Match["Multi-View Coarse Matching"]
    Images["Two input images"] --> Match
    Match -->|Stride-4 warps and confidence| Refine["Efficient Refinement"]
    Train -.->|Training sampling and displacement scales| Refine
    Refine --> Uncertainty["Directional Uncertainty"]
    Refine -->|Bidirectional warps and covisibility confidence| Output["Dense correspondences<br/>and geometric estimation"]
    Uncertainty -->|Pixel-wise precision matrices| Output

Key Designs¶

1. Training Distribution and Resolution: cover both coarse localization and fine motion

Wide-baseline examples require finding the same structure despite appearance and viewpoint changes, whereas small-baseline examples supervise fine motion, boundaries, and tiny displacements. The authors mix wide-baseline sources including MegaDepth, AerialMD, BlendedMVS, Hypersim, TartanAir v2, Map-Free, and ScanNet++ v2 with small-baseline sources UnrealStereo4k, Virtual KITTI 2, and FlyingThings3D. Table 3 reports 5069 scenes in total; the listed wide-baseline datasets each have sampling weight 1, while the final 3 datasets have weights 0.01, 0.01, and 0.5, respectively. These are relative sampling weights, not percentages, and should not be read directly as each dataset's percentage of training. Aerial data addresses rotations and aerial-to-ground changes, while small-baseline data supplies small moving objects and fine structure; diversity thus changes the available supervision rather than merely adding volume.

Transferring this capability across input sizes also requires position representations that are not tied to the training pixel grid. Coarse matching uses rotary position embeddings on normalized coordinates and applies them only to frame-wise attention, not global attention. Match embeddings use low-frequency absolute position encodings: RoMa's learnable scale initialized at 8 is replaced with a fixed scale of 1 to reduce interpolation difficulties. Refinement convolutions remain tied to the pixel grid, so input displacements are rescaled relative to a canonical resolution instead of mixing raw pixel displacements from arbitrary resolutions. The coarse matcher trains across resolutions and aspect ratios, while refinement training uses a fixed \(640\times640\) size; the stages do not use identical scale augmentation.

2. Multi-View Coarse Matching: learn correlation and continuous coordinates together

Each image independently passes through a frozen DINOv3 ViT-L, and the resulting features enter a ViT-B multi-view Transformer that alternates frame-wise and global attention. Each position can therefore use structure within its own image and candidate correspondences in the other image, while freezing preserves the pretrained backbone's transfer capabilities. Simply inserting a multi-view Transformer before RoMa's Gaussian process did not work: the authors observed insufficiently informative gradients through the Gaussian process and unstable training. They replace it with single-headed attention and construct a similarity matrix between all patches of the two images. Final coordinate errors alone may still provide inadequate supervision for this matrix, so the model receives an explicit negative log-likelihood target for the correct destination patch.

Let \(S\) denote the patch similarity matrix and \(n_m^*\) the target patch nearest the ground-truth warp of source patch \(m\); Equation (1) can be written as:

\[ \mathcal{L}_{\mathrm{NLL}}=-\sum_{m=1}^{M}\log\bigl(\operatorname{Softmax}(S_{m,:})_{n_m^*}\bigr). \]

The target index is determined by ground truth, not by treating the model's current highest-similarity prediction as its own label. Attention weights also aggregate destination position embeddings, passing information about where the match lies into a Dense Prediction Transformer (DPT) head. DPT jointly processes the final tokens, aggregated position embeddings, and DINOv3 features to output continuous coordinates and covisibility confidence rather than using discrete patch classification as the final answer. Correlation supervision learns an effective matching structure, while robust coordinate regression learns geometric positions; both constrain the coarse matcher. In Table 2's held-out Hypersim experiment, this matching architecture achieves PCK@1px of 30.5 versus 11.2 for the UFM architecture; this is an architecture comparison under the specified training setup, not an overall ranking of complete systems across datasets.

3. Efficient Refinement: reduce scale overhead and remove training-time subpixel bias

Coarse matching now reaches stride 4 directly, compared with RoMa's stride 14, so only three refiners at strides 4, 2, and 1 are required. Each is a UNet-like convolutional module conditioned on the previous warp and confidence, predicting displacement corrections and confidence increments. Local correlation searches nearby features around the current match, allowing correction of coarse localization without repeating a global candidate comparison. A custom CUDA kernel for local correlation primarily reduces intermediate memory consumption, and power-of-two channel dimensions further improve execution efficiency. The entire speed gain should not be attributed to this kernel: in Table 8 it raises throughput only from 30.3 to 30.9 pairs/s, while its clearer effect is reducing memory from 5.6 to 4.8 GB.

Refinement accuracy is also affected by a bias that is difficult to see in qualitative outputs: predictions drift by approximately \(\pm0.1\) pixels during training, observed at \(640\times640\) resolution. Because this bias fluctuates approximately irregularly over training, the authors smooth model weights with an exponential moving average (EMA) using decay 0.999. This averages parameter states across training, rather than mixing correspondence predictions from several images at inference time. Table 9 shows a larger benefit on MegaDepth-1500, which strongly depends on subpixel precision, and smaller gains on ScanNet-1500. Refiners train separately after the coarse matcher has completed training and been frozen, also avoiding a complete joint-training rerun for every refinement experiment.

4. Directional Uncertainty: distinguish visibility from localization precision

A covisible point can still be poorly localized: motion blur may make its location uncertain along one direction while preserving localization in the perpendicular direction. A single covisibility probability cannot express that difference, so refiners additionally predict a pixel-wise \(2\times2\) precision matrix \(P=\Sigma^{-1}\). The network outputs three values forming a lower-triangular Cholesky factor \(L\), applies Softplus plus \(10^{-6}\) to its diagonal entries, and sets \(P=LL^\top\) to guarantee positive definiteness. Precision increments are accumulated across refinement scales, using the additive-information property of the precision parameterization. Instead of assigning an equal-sized error circle to every match, this permits a direction-dependent uncertainty ellipse.

Let \(r\) denote the two-dimensional predicted-minus-ground-truth warp residual; the Gaussian negative log-likelihood in Equation (4) can equivalently be written as:

\[ \mathcal{L}_{\mathrm{prec}}=\frac{1}{2}r^\top P r-\frac{1}{2}\log\det P+\log(2\pi). \]

The first term aligns residuals with predicted precision, while the second prevents avoiding the error penalty by simply lowering all precision estimates. This term is supervised only at covisible locations with residuals below 8 pixels, and gradients through the residual are stopped so uncertainty learning does not alter the coordinate predictor's optimization objective. Its effective range therefore primarily concerns localization error near successful matches, not a probability model guaranteed to remain calibrated for arbitrary large mismatches. Downstream geometry can use the matrices to weight residuals, either in pose refinement after RANSAC or in model scoring inside RANSAC. The diagram separates uncertainty to distinguish its output semantics; implementation-wise it is produced by the refiners, not by an additional independent matching network.

A Worked Example¶

Consider a wide-baseline pair containing a road and buildings, using \(640\times640\) images to illustrate the internal data flow; this is a mechanism example, not an additional experiment. Frozen features and multi-view interaction first locate approximate building correspondences and produce a stride-4 warp field. Refiners then correct positions at strides 4, 2, and 1, precisely localizing building edges while updating covisibility confidence over the road. If the target image is blurred in one direction, predicted covariance should allow larger errors along that direction rather than merely marking the entire point invisible. For downstream pose estimation, geometric filtering must still reject mismatches; directional error weighting can then prevent the remaining blurred directions from becoming equally trusted geometric constraints.

Loss & Training¶

The coarse matcher trains for 300k steps with batch size 128, seeing approximately 38M image pairs; it is then frozen while the refiners train for 300k steps with batch size 64, seeing approximately 19M pairs. These totals count training exposures rather than mutually distinct image pairs; both stages use learning rate \(4\times10^{-4}\). The coarse objective combines the correlation NLL above, robust coordinate regression, and an overlap loss weighted by 0.1. The refinement objective sums robust coordinate regression, pixel-wise covisibility binary cross-entropy, and precision NLL across three scales, weighting the latter two by \(10^{-2}\) and \(10^{-3}\). Ground-truth covisibility comes from depth consistency in multi-view stereo data or warp cycle consistency in flow data. The paper uses generalized Charbonnier regression, but its expression is corrupted in the text extraction; this note does not reconstruct uncertain characters into a purported exact author equation.

Key Experimental Results¶

Main Results¶

The following values are selected from Tables 6 and 7, pages 12 and 13: EPE is mean Euclidean endpoint error in pixels, lower is better; PCK@1px is the percentage of matches below 1 pixel error, higher is better. Evaluation images are \(640\times640\); RoMa and RoMa v2 use this size directly, whereas UFM first resizes to its recommended \(560\times420\) and bilinearly upsamples predictions back to the evaluation size.

Dataset	Method	EPE (pixels)	PCK@1px (%)
MegaDepth	RoMa	2.34	74.8
MegaDepth	UFM	3.15	55.3
MegaDepth	RoMa v2	1.47	79.6
AerialMegaDepth	RoMa	25.05	39.0
AerialMegaDepth	UFM	17.44	29.3
AerialMegaDepth	RoMa v2	4.12	55.9
FlyingThings3D	RoMa	5.68	78.0
FlyingThings3D	UFM	1.33	83.4
FlyingThings3D	RoMa v2	0.93	89.4

Relative camera-pose evaluation uses a different setting: coarse resolution \(800\times800\), fine resolution \(1024\times1024\), and RoMa-style bidirectional sampling with density balancing. In Table 4, page 12, RoMa v2 obtains MegaDepth-1500 AUC@5°/10°/20° of 62.8/77.0/86.6, compared with RoMa's 62.6/76.7/86.3. These modest gains do not contradict the large EPE improvements on difficult data above: the tasks, distributions, and metrics differ.

Ablation Study¶

Table 8, page 13, measures throughput and memory on an H200 with batch size 8; RoMa v2 uses \(640\times640\) inputs, while RoMa and UFM use \(644\times644\) because their patch size is 14. K denotes the custom local-correlation CUDA kernel, and throughput is not single-pair end-to-end latency.

Method	Throughput (pairs/s)	Memory (GB)
UFM	43.0	16.2
RoMa	18.5	4.7
RoMa v2, without K	30.3	5.6
RoMa v2, with K	30.9	4.8

The EMA ablation in Table 9, page 14, uses the same pose metrics as Table 4; triplets represent AUC@5°/10°/20° in that order. This comparison changes refiner EMA, rather than replacing the training data or backbone.

Config	MegaDepth-1500	ScanNet-1500
Without EMA	61.4 / 75.8 / 85.7	33.6 / 56.1 / 73.5
EMA	62.8 / 77.0 / 86.6	33.6 / 56.2 / 73.8

Key Findings¶

Difficult geometric changes improve more substantially than the already mature standard pose benchmark: AerialMegaDepth EPE decreases from 25.05 to 4.12, described in the paper as an approximately 84% reduction.
Runtime gains have clear boundaries: 30.9 pairs/s exceeds RoMa's 18.5 but trails UFM's 43.0; memory of 4.8 GB also does not undercut RoMa's 4.7 GB.
Directional errors translate into geometric gains: Table 12, page 15, reports Hypersim AUC@1° of 54.9 without covariance, 75.8 when using it only in refinement, and 76.4 when using it in both RANSAC scoring and refinement.
Generalization does not improve everywhere: Table 11, page 14, reports WxBS mAA@10px of 60.8 for RoMa, 55.4 for RoMa v2, and 42.3 for UFM; the text identifies continued difficulty on the IR-to-RGB subset.

Highlights & Insights¶

The coarse matcher jointly learns where to attend and how to regress precise locations, but gives the former explicit correlation supervision rather than expecting useful intermediate matches to emerge solely from final errors.
EMA is more than a generic generalization trick here: it cancels fluctuating subpixel bias during training, emphasizing that tiny systematic errors can matter more than visual appearance in geometry tasks.
Covisibility probabilities and covariance serve different purposes. Separating visibility from geometric measurement precision lets downstream estimators use the network outputs more fully.
Training distribution, position encodings, and execution kernels jointly determine practical utility. The improvements cannot be reduced to replacing the backbone with DINOv3.

Limitations & Future Work¶

The authors explicitly acknowledge slightly worse robustness than RoMa under extreme modality changes; a stronger frozen backbone does not guarantee improvement on every out-of-distribution task.
Precision learning excludes residuals greater than or equal to 8 pixels, so uncertainty for severe mismatches still requires separate calibration and evaluation; this limitation follows from the training range.
Runtime measurements use an H200 and batch size 8, and the custom CUDA kernel depends on its execution environment; the same gains cannot be assumed on consumer hardware, batch size 1, or other backends.
The available cache contains only the main paper, so finer network specifications, data-construction details, and covariance experiment settings deferred to the appendix could not be verified; this note retains only information supported by the main text.
Further work could examine how cross-modal training distributions affect fine localization and evaluate covariance calibration across real blur and large mismatches, rather than comparing pose AUC alone.

RoMa / DKM: the coarse-to-fine dense matching structure remains, but multi-view interaction and explicit correlation supervision replace Gaussian-process-dominated coarse matching, and training is genuinely split into stages.
UFM: the work adopts decoupled matching-then-refinement training and a wide/small-baseline mixture, but retains a frozen encoder while adding a correlation auxiliary objective and finer geometric precision modeling.
LoFTR: both exploit cross-image context and matching supervision; this paper additionally retains continuous warp regression and full-resolution refinement rather than ending with semi-dense correspondences.
VGGT / MASt3R: multi-view structures provide architectural inspiration, but the paper directly optimizes two-dimensional dense correspondences without making feed-forward 3D reconstruction a necessary intermediate step.
Transferable insight: optical flow and geometric registration could separately audit visibility, local precision, and error directionality instead of asking one confidence score to represent every failure mode; this is a reader inference, not an additional experiment in the paper.

Rating¶

Novelty: 4/5. The contribution is primarily targeted system design, with explicit correlation supervision and directional precision outputs making it more than a backbone replacement.
Experimental Thoroughness: 4/5. Evaluation spans pose, localization, dense matching, and efficiency, while large-mismatch covariance calibration and deployment across hardware remain gaps.
Writing Quality: 4/5. Problems, mechanisms, and ablations are clearly connected, although some reproduction details depend on supplementary material absent from the cache.
Value: 5/5. The method directly benefits localization and reconstruction pipelines requiring precise correspondences while documenting clear performance boundaries.