Register Any Point: Scaling 3D Point Cloud Registration by Flow Matching¶
Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/PRBonn/RAP
Area: 3D Vision
Keywords: point cloud registration, flow matching, rigidity constraints, multi-view fusion, cross-domain generalization
TL;DR¶
RAP reformulates multi-view point cloud registration as point-coordinate generation conditioned on local geometry, stabilizes flow sampling with stepwise rigidity enforcement, and recovers per-view poses from generated points, achieving 85.9% edge success, 77.3% graph success, and an average runtime of 8.9 seconds on the authors' zero-shot cross-domain benchmark.
Background & Motivation¶
Multi-view registration takes several partial scans without initial poses and places them in a common coordinate system. Conventional systems first find local feature correspondences and estimate relative poses for scan pairs, then use pose graph optimization (PGO) to enforce global consistency. The problem is not merely that exhaustive pairwise processing grows quadratically with the number of views. With little overlap, local matching may be unable to resolve geometric ambiguity, leaving the backend with incorrect relative poses. Structures visible in other views could help disambiguate the alignment, but are not fully exploited during independent pairwise estimation.
Generative approaches offer another route: instead of first deciding which points correspond, jointly predict where all input points belong in a shared scene. RPF explored Euclidean point flows for object-level pose estimation and part assembly, but moving to roads, buildings, and large maps introduces much larger point counts and scale differences. A subtler difficulty is that independently moving points can stretch or bend the geometry within an individual scan. A final rigid fit may repair the output, but cannot undo the influence that earlier sampling drift has already had on subsequent velocity predictions.
RAP therefore combines local geometric representations, cross-domain training, and physical constraints during sampling. Local descriptors provide transferable shape cues, a shared scale and coordinate convention makes heterogeneous training targets compatible, and multi-view attention lets weakly overlapping scans draw on other observations. Core Idea: jointly infer where each view belongs in Euclidean point space, then project its predicted endpoint back onto that scan's rigid orbit at every sampling step, combining global generation with local geometric fidelity.
Method¶
Overall Architecture¶
The input consists of at least two unordered point clouds from the same scene, without initial poses, whose overlap relationships are assumed to form a connected graph. RAP builds a compact local geometry representation, generates conditional point flows in canonical coordinates, and enforces rigidity during sampling. Finally, it fits a pose between each view's input keypoints and their generated locations, then transforms the original dense point cloud.
Here, generation does not mean inventing an arbitrary new scene. Every generated keypoint retains its input identity and view membership, so input and output keypoints are already paired; SVD does not need another cross-scan correspondence search. Likewise, single-stage refers to eliminating the external pairwise-registration-plus-PGO pipeline, not to executing only one neural network forward evaluation by default.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
Input["Multi-view point clouds<br/>without initial poses"] --> Local["Compact Local Geometry"]
Local --> Flow["Canonical Conditional Point Flow"]
Noise["Gaussian noise and time"] --> Flow
Truth["Ground-truth registered keypoints<br/>training only"] -.->|Velocity supervision| Flow
Flow --> Rigid["Stepwise Rigidity Enforcement"]
Rigid -->|Update state and repeat sampling| Flow
Rigid -->|Sampling complete| Output["Recover poses with SVD<br/>transform dense point clouds"]
Key Designs¶
1. Compact Local Geometry: avoid feeding every dense scan point into the generator
The pipeline first applies voxel downsampling, then farthest point sampling to obtain keypoints covering the scene, allocating keypoints across views in proportion to their metric scales. For each keypoint, it queries a neighborhood in the downsampled cloud with a radius of 20 times the voxel size, normalizes the patch, and feeds it to pretrained MiniSpinNet for a 32-dimensional rotation-invariant descriptor. This backbone is pretrained on 3DMatch. The generator consequently receives local surface geometry rather than isolated coordinates stripped of context.
Keypoints represent global layout, descriptors represent local structure, and the original dense clouds are retained for transformation after pose recovery. This avoids repeatedly generating every original point without reverting to descriptor nearest-neighbor matching. Voxelization and local normalization help absorb sensor-dependent density differences before the model reasons about the shared scene behind multiple partial observations. The severe drop when descriptors are removed shows that scalability is not simply a consequence of using a larger Transformer.
2. Canonical Conditional Point Flow: jointly generate view locations under a shared coordinate convention
Each view's keypoints are centered at their own centroid. A shared scale factor is the longest bounding-box edge of the view with the most points; the clouds are scaled accordingly and randomly rotated in 3D. Training targets are formed by merging keypoints aligned with ground-truth poses, recentering them, and applying the reference view's rotation and the same scale factor. Ground-truth relative poses are not needed as input; they only construct supervision. The purpose is to place millimeter-scale objects and large maps in comparable numerical ranges while defining the target's overall orientation, so the model need not simultaneously solve relative registration and guess an arbitrary global coordinate convention.
Conditioning concatenates 32-dimensional local descriptors with 63-dimensional Fourier coordinate encodings and maps them linearly to 512 dimensions. The flow model has 10 alternating-attention blocks, 8 attention heads, and 73M parameters. Within-view self-attention consolidates individual scan structure, while global attention exchanges information across all views. View indices are not embedded in the condition, avoiding dependence on a fixed camera set or training view count. View grouping is nevertheless retained for within-view attention and subsequent rigid fitting.
During training, the model receives a linear interpolation between registered keypoints and standard Gaussian noise and learns the velocity associated with that state. At inference, it progressively transports noise back toward a registered point cloud. Local coordinates and descriptors remain fixed conditions; the generated coordinates change with time. Because all views update jointly through global attention, an ambiguous wall corner or road segment can use more complete structures in other scans, without first committing to an incorrect pairwise transformation.
3. Stepwise Rigidity Enforcement: constrain every predicted endpoint, not only the final output
Unconstrained Euclidean point flow can deform points belonging to a single wall. At every step, RAP extrapolates a clean endpoint from the current state and predicted velocity. It then applies Kabsch/SVD separately to each view to find the best rigid transformation from its input keypoints to that endpoint. Applying this transformation to the input keypoints produces an endpoint estimate that preserves the scan's internal geometry, which is then used to construct the next time state. This inference-time operation needs no test ground truth and performs no additional cross-view correspondence search.
An important distinction is that the predicted endpoint is rigidified; the method does not require every noisy intermediate state itself to be a rigid copy of the input. The corrected endpoint is fed back into the sampling trajectory. By default, 10 uniform Euler steps move from \(t=1\) to \(t=0\), with fresh prediction and projection at every step to limit accumulation of implausible deformation. The authors also describe rigidity error as a criterion for choosing among multiple generations, but the main text does not specify a complete candidate count and selection protocol, so this note does not treat it as a confirmed default configuration behind the main results.
Once sampling finishes, rigid poses are recovered between each view's canonical input keypoints and generated points, composed with input canonicalization, and applied to the original dense clouds. The global metric scale is then restored. This preserves the detailed original observations: the generator predicts alignment locations rather than replacing every measured sensor point.
A Worked Example¶
Consider three building scans: the first observes an entrance hall, the second sees both the hall and a corridor, and the third mainly observes the corridor. This is an explanatory example, not a separately reported experiment. The first and third scans may have no direct overlap, provided the second connects the overall overlap graph. Keypoints and local descriptors are extracted from each scan, and the network receives all three sets of conditions together rather than solving three scan pairs independently.
At an intermediate sampling step, the hall and corridor keypoints may have a plausible layout while the third scan's wall is slightly bent. Stepwise rigidity enforcement fits that prediction using the third scan's original wall geometry, corrects it to a rigid placement, and passes the corrected estimate into later sampling. After 10 steps, the recovered keypoint poses transform all dense points in their respective views. The example illustrates why global context complements within-view rigidity, and why graph connectivity does not require every scan pair to overlap.
Loss & Training¶
The objective is conditional flow matching: under sampled times and noise, predicted velocities should match the target velocities of the linear interpolation path. Supervision requires accurate scan poses to construct registered points in a common coordinate system, but no manually annotated keypoints or cross-scan correspondences. Equations (1) through (6) are damaged in the cached text extraction, so this note does not reconstruct their exact mathematical form and instead retains the interpolation, velocity supervision, and projection mechanisms established by the prose.
Training combines 17 datasets and over 100k registration instances: 12 outdoor LiDAR datasets, 4 indoor depth datasets, and ModelNet. Each instance contains 2 to 16 views. The training/validation split is approximately 9:1, and test sequences from common registration benchmarks are excluded from training. Samples include individual scans and accumulated submaps, deliberately retaining difficult low-overlap combinations that remain globally connected. No correspondence annotations does not mean fully unsupervised learning, because accurate poses are still required to construct the generative target.
Training uses 32 A100 GPUs with 80 GB memory each for approximately 120k iterations over 3 days. Muon uses initial learning rates of 0.002 for matrix-like parameters and 0.0002 for vector-like parameters. Dynamic batching accommodates variable view and token counts, with at most 110,000 tokens per GPU batch. Data-scaling experiments show that RAP remains weaker than baselines when trained on only 1 or 3 datasets. The full cross-domain mixture is therefore an important prerequisite for generalization under this formulation's weaker geometric inductive bias.
Key Experimental Results¶
Main Results¶
The cross-domain multi-view benchmark covers objects, indoor scans, outdoor scans, terrestrial laser scanning (TLS), and maps. The authors state that its datasets do not overlap with the training mixture. Valid edges connect scans with nonzero overlap. Edge success measures successful registration over these edges, while graph success requires every valid edge in an instance to succeed. Success requires rotation error below 15 degrees and translation error, normalized by the longest ground-truth bounding-box edge, below 2.5%. These are coarse-registration criteria, not direct evidence of precision-measurement accuracy.
Table 1 excerpts the original Table 2. BUFFER-X* is retrained on the same extended data as RAP and followed by PGO; RAP uses 10 sampling steps by default. Runtimes are instance averages. The paper evaluates runtime on one NVIDIA A5000 with 24 GB memory, not on the 32 A100 GPUs used for training.
| Method | Overall Edge Success (%) | Overall Graph Success (%) | Mean Runtime (seconds) |
|---|---|---|---|
| KISS-Matcher + PGO | 46.2 | 29.2 | 18.4 |
| BUFFER-X + PGO | 50.4 | 32.4 | 36.0 |
| BUFFER-X* + PGO | 57.7 | 36.5 | 36.0 |
| SGHR | 24.3 | 17.2 | 49.7 |
| RPF | 1.0 | 0.6 | 11.5 |
| RAP | 85.9 | 77.3 | 8.9 |
Against same-data-retrained BUFFER-X* + PGO, RAP improves edge success by 28.2 percentage points and graph success by 40.8 percentage points. Across the five categories, RAP's indoor graph success is 59.6%, compared with 91.6% for maps. Its aggregate advantage should not obscure the remaining difficulty of indoor scenes.
Pairwise results require separate interpretation. The original Table 1 reports mean success of 94.92% across ModelNet, 3DMatch, 3DLoMatch, NSS, ETH, and KITTI, compared with 92.46% for BUFFER-X. These benchmarks use their respective established RMSE or rotation/translation thresholds, and learning-based baselines do not all use the same training data. For example, RAP obtains 78.78% on 3DLoMatch, below PARENet's 80.50%. The supported conclusion is best average performance, not best performance on every benchmark.
Ablation Study¶
Table 2 excerpts the original Table 3. Pairwise columns retain their benchmark-specific protocols; multi-view columns use the cross-domain benchmark and the 2.5%/15-degree thresholds above. Full RAP has 10 Transformer blocks, 10 sampling steps, and stepwise rigidity enforcement. The original two-decimal precision is retained rather than mixed with the main table's one-decimal values.
| Config | 3DMatch (%) | 3DLoMatch (%) | Multi-View Edge Success (%) | Multi-View Graph Success (%) |
|---|---|---|---|---|
| Full RAP | 95.90 | 78.78 | 85.92 | 77.31 |
| Without stepwise rigidity enforcement | 94.65 | 74.63 | 83.57 | 72.88 |
| Only 1 generation step | 90.35 | 61.05 | 72.87 | 50.32 |
| Without local feature extraction | 83.28 | 50.05 | 43.07 | 26.85 |
| Reduced to 8 Transformer blocks | 93.24 | 71.63 | 81.08 | 72.19 |
| Reduced to 6 Transformer blocks | 90.90 | 57.59 | 65.64 | 54.35 |
Key Findings¶
- Local descriptors have the largest effect: removing them lowers multi-view graph success from 77.31% to 26.85%, a drop of 50.46 percentage points. Generative global inference still needs strong local geometric conditions.
- Rigidity enforcement raises multi-view graph success from 72.88% to 77.31%, a gain of 4.43 percentage points. The original Table 2 reports 8.9 seconds for both variants, so no additional runtime is visible at the reported precision.
- One-step generation averages 5.8 seconds but achieves only 50.32% graph success. In the original Figure 5, 10 and 50 steps achieve edge success of 85.9% and 86.8%, respectively, indicating diminishing returns from further sampling steps.
Highlights & Insights¶
- Registration can be obtained by predicting each input point's final position rather than directly predicting a transformation matrix. Retaining point identity connects generation to mature rigid-fitting algorithms without another correspondence search.
- Geometric constraints need not all be encoded in network architecture or training losses. Placing rigid projection inside the sampling loop constrains test trajectories while retaining Euclidean generative expressiveness, acting more directly than final-only projection.
- Same-data retraining provides an important control: BUFFER-X improves from 50.4% to 57.7% edge success, still below RAP's 85.9%. This supports a joint role for data, representation, and collective generation rather than attributing the entire gain to more data.
Limitations & Future Work¶
- The authors explicitly note that the model cannot determine whether scans come from the same environment and will attempt to register every input jointly. Practical systems need scene-consistency checks or rejection mechanisms; the title should not be read as unconditional support for unrelated scans.
- The authors identify non-rigid changes and longer streaming sequences as future directions. The current rigid sampling configuration is not evidence that dynamic objects, SLAM drift, or nonmetric reconstructions are already handled.
- This note's assessment: nearly linear measured scaling with view count does not establish a universal linear-complexity guarantee. Global attention remains quadratic in token sequence length, and flow inference becomes the bottleneck at high token counts. The paper also reports that RAP is slightly slower than KISS-Matcher and BUFFER-X* for two views.
- This note's assessment: the main 15-degree and 2.5% thresholds concern coarse registration, while training requires substantial compute and accurate poses. The main text points to stricter-threshold results in supplementary material, which is absent from the assigned cache; unseen values should not be supplied or used to claim fine-registration precision.
Related Work & Insights¶
- vs RPF: Both perform flow matching in Euclidean point space. RAP adds local feature conditioning for large scenes, cross-scale canonicalization, cross-domain training, and rigidity enforcement during sampling. RPF's weak performance on this cross-domain benchmark does not invalidate its original object-assembly setting.
- vs BUFFER-X + PGO: The latter relies on pairwise correspondences followed by pose graph integration, whereas RAP processes all views jointly at the point level. RAP predictions can still initialize downstream optimization incorporating IMU, GNSS, or other constraints; the method does not imply that every optimization backend should be removed.
- vs VGGT / FUSER: RAP borrows alternating within-view and global attention, but takes point clouds and local geometric descriptors as input. Relative to the main text's description of FUSER, its emphasis is Euclidean point flow and cross-domain coverage rather than transformation-group diffusion refinement following indoor multi-view prediction.
Rating¶
- Novelty: 4/5. Builds on RPF and alternating attention, with clear advances in stepwise rigid sampling and cross-domain registration.
- Experimental Thoroughness: 4/5. Includes a cross-domain benchmark, same-data retraining, and mechanism ablations, while some strict-threshold and reproduction details require supplementary material.
- Writing Quality: 4/5. The method and results are clearly organized, although single-forward-pass wording can be confused with the default multi-step sampler.
- Value: 5/5. Unifies object, scan, and map registration in one model and offers a strong direction for low-overlap multi-view geometry.