---

title: >- [Paper Note] PGCR: Pose–Geometry Coupled Reasoning for Image-to-Point Cloud Registration description: >- [ECCV 2026][3D Vision][Image-to-Point Cloud Registration] Reformulates 2D-3D registration into pose-geometry coupled reasoning with progressive filtering and dense geometric feedback, bypassing fragile explicit overlap prediction. tags: - ECCV 2026 - 3D Vision - Image-to-Point Cloud Registration - Geometric Reasoning - Cross-Modal Alignment date: 2026-09-19 content_hash: 24b89d4a6afd4c30

PGCR: Pose–Geometry Coupled Reasoning for Image-to-Point Cloud Registration

Conference: ECCV 2026
Paper: ECCV Official Link
Code: Pending release (no public repository link provided in paper)
Area: 3D Vision / Multimodal Cross-Modal Registration
Keywords: Image-to-point cloud registration, cross-modal geometry, pose-geometry coupling, progressive refinement, closed-loop feedback

TL;DR

Addressing the vulnerability of conventional overlap-then-registration pipelines where inaccurate overlap prediction irreversibly discards valid regions, PGCR re-conceptualizes 6-DoF image-to-point cloud registration as coupled pose-geometry reasoning, achieving 100% registration accuracy and 0.15 m / 0.44° precision on KITTI with a fast 0.17 s runtime.

Background & Motivation

Image-to-point cloud registration estimates the 6-degree-of-freedom (6-DoF) camera pose \([R \mid t] \in \mathrm{SE}(3)\) of a query 2D RGB image with respect to a pre-built 3D point cloud coordinate system. Serving as an indispensable capability for visual localization, autonomous vehicle mapping, and collaborative SLAM, this task is fundamentally challenged by the severe modality discrepancy between regular 2D pixel grids rich in photometric texture and unorganized 3D point clouds dominated by sparse spatial geometric topology.

Prevailing learning-based approaches (such as DeepI2P, CorrI2P, VP2P-Match, and CoFiI2P) adhere to an "overlap-then-registration" paradigm. These frameworks extract features independently and perform an explicit binary classification step to isolate mutually visible regions before conducting correspondence matching or PnP optimization within that restricted subset. While this sequential decomposition narrows the search space, it introduces a fragile bottleneck: in large-scale outdoor LiDAR environments characterized by sparse beams, dynamic occlusions, and repetitive structural patterns, early overlap estimation frequently errs. Once geometrically consistent structures are prematurely discarded, downstream pose solvers cannot recover from the truncated evidence, resulting in catastrophic failure.

Under physical observation principles, camera pose and scene geometry exhibit a fundamental bidirectional dependency: the 6-DoF pose defines which 3D surface elements project within the sensor's field of view, while observed surface geometries conversely bound and prune the feasible camera pose manifold. Decoupling them into a unidirectional pipeline contradicts this mutual physical conditioning. Core idea: discard explicit and brittle overlap pre-filtering and reformulate image-to-point cloud registration as a tightly coupled pose-geometry reasoning problem, where camera pose, dense depth, and world point maps are jointly inferred within a closed-loop Transformer that dynamically restricts cross-modal attention via intermediate pose hypotheses and refines pose via geometric feedback.

Method

Overall Architecture

PGCR operates as an end-to-end closed-loop reasoning framework. The input consists of a query RGB image \(I \in \mathbb{R}^{H \times W \times 3}\) and an unaligned 3D point cloud \(P = \{p_i \in \mathbb{R}^3\}_{i=1}^N\). The architecture employs a self-supervised DINO Vision Transformer to extract 2D patch tokens prepended with a learnable global camera token \(f_{\mathrm{cam}}^{(0)}\), alongside a KPConv backbone to extract geometry-aware 3D point features encoding local shape primitives.

The fused representations are processed across stacked Transformer layers alternating between intra-modal self-attention and cross-modal cross-attention. Throughout this iterative process, intermediate camera pose hypotheses are extracted periodically to dynamically prune non-overlapping and geometrically inconsistent 3D points via camera frustum and translation tolerance masks. At the network head, the architecture simultaneously predicts 6-DoF camera pose, dense metric depth, and dense world-frame 3D point maps. Finally, a Geometry-aware Pose Refinement module projects and fuses all dense geometric fields with the camera token to produce the refined, geometrically grounded 6-DoF pose.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: 2D Image I & 3D Point Cloud P"] --> B["Multimodal Feature Encoding<br/>DINO Image Patches + KPConv Point Features"]
    B --> C["Pose-guided Progressive Refinement (PPR)<br/>Intermediate pose frustum filtering to prune point sets"]
    C --> D["Multi-Task Dense Geometric Prediction<br/>Parallel regression of Pose Token, Depth Map, and Point Map"]
    D --> E["Geometry-aware Pose Refinement (GPR)<br/>Projection of dense fields and AvgPool closed-loop correction"]
    E --> F["Output: Refined 6-DoF Camera Pose [R | t]"]

Key Designs

1. Pose-guided Progressive Refinement (PPR): dynamically contracting the 2D-3D interaction space Performing unconstrained dense cross-attention across the full 3D point cloud and 2D pixel tokens incurs substantial computational overhead and introduces severe geometric noise from regions outside the field of view. PGCR introduces a progressive stage-wise refinement strategy across its \(L_p \times L_c = 8\) Transformer layers. Every \(L_p = 2\) layers, an intermediate MLP camera head predicts a coarse 6-DoF pose hypothesis \((R^{(k)}, t^{(k)})\) from the updated camera token for stage \(k \in \{1, \dots, L_c\}\).

Using this intermediate pose, 3D points are projected onto the image plane. A point \(p_i\) is retained in the active point set \(\mathcal{M}^{(k)}\) if its projected pixel coordinates satisfy relaxed image boundaries: $\(-\delta_{\mathrm{pix}}^{(k)} \le u_i^{(k)} \le W + \delta_{\mathrm{pix}}^{(k)}, \quad -\delta_{\mathrm{pix}}^{(k)} \le v_i^{(k)} \le H + \delta_{\mathrm{pix}}^{(k)}\)$ or if its transformed Euclidean distance to the optical center is bounded: \(\|R^{(k)} p_i + t^{(k)}\|_2 \le \delta_{\mathrm{trans}}^{(k)}\). Both tolerance thresholds decay exponentially across stages: \(\delta^{(k)} = \delta^{(1)} \cdot \eta^{k-1}\) with decay factor \(\eta = 0.5\). This design starts with broad global context and progressively focuses cross-modal attention onto true co-visible regions as pose confidence increases, stabilizing convergence and eliminating redundant computation.

2. Multi-Task Dense Geometric Prediction: establishing dual metric geometric anchors Relying solely on a global latent vector or sparse keypoints to regress a 6-DoF transformation often leads to optimization instability and sensitivity to initial perturbations. PGCR promotes dense scene geometry to an explicit first-class regularizer. From the deeply fused tokens, parallel specialized heads generate: - Camera Pose Head: An MLP regressing rotation (via continuous 6D representations on \(\mathrm{SO}(3)\)) and 3D translation; - Depth Head: A Dense Prediction Transformer (DPT) decoding a full-resolution metric depth map \(D \in \mathbb{R}^{H \times W}\); - Point Map Head: A parallel DPT predicting a per-pixel 3D world coordinate map \(M \in \mathbb{R}^{H \times W \times 3}\).

Supervised directly via LiDAR projections under ground-truth poses, the depth map anchors view-space radial geometry while the point map provides global coordinate constraints. Together, they create continuous, spatially distributed reprojection gradients across the entire image grid, substantially expanding the basin of attraction during optimization.

3. Geometry-aware Pose Refinement (GPR): closing the loop between geometry and pose In standard Transformer backbones, compressing spatially distributed geometric evidence into a single global camera token inevitably discards high-frequency spatial constraints. Furthermore, predicting pose and dense geometry via disjoint output heads lacks an explicit mechanism to enforce mutual geometric consistency. To eliminate this optimization gap, PGCR deploys a Geometry-aware Pose Refinement (GPR) module.

GPR projects the predicted depth map \(D\) and point map \(M\) through dedicated MLPs into shared feature spaces \(E_{\mathrm{depth}}\) and \(E_{\mathrm{M}}\). Concurrently, the final image tokens are upsampled to \(E_{\mathrm{img}}\), and the final camera token \(f_{\mathrm{cam}}\) is projected and spatially replicated across the grid to form \(E_{\mathrm{cam}}\). The four representations are concatenated channel-wise, fused through an MLP, and aggregated via global average pooling: $\(z_{\mathrm{fuse}} = \mathrm{AvgPool}\Big(\mathrm{MLP}_{\mathrm{fuse}}([E_{\mathrm{depth}}, E_{\mathrm{M}}, E_{\mathrm{img}}, E_{\mathrm{cam}}])\Big)\)$ The final refined camera pose is regressed from \(z_{\mathrm{fuse}}\). By grounding pose estimation directly in dense spatial geometry, any inconsistency between predicted scene surfaces and camera orientation generates immediate corrective signals, yielding superior rotational and translational precision.

Loss & Training

The entire pipeline is trained end-to-end under a multi-task objective: $\(\mathcal{L} = \lambda_p \mathcal{L}_{\mathrm{pose}} + \lambda_d \mathcal{L}_{\mathrm{depth}} + \lambda_m \mathcal{L}_{\mathrm{point\_map}}\)$ with weights \(\lambda_p = 5.0, \lambda_d = 1.0, \lambda_m = 1.0\). The pose loss supervises both intermediate stage hypotheses and final GPR outputs with smooth \(L_1\) and geodesic rotation losses. The depth and point map losses are computed exclusively on valid LiDAR projection pixels. Optimization is conducted using AdamW with learning rate \(5 \times 10^{-5}\) and batch size 8 on a single NVIDIA RTX 3090 GPU for 20 epochs.

Key Experimental Results

Main Results

PGCR was evaluated on two outdoor autonomous driving benchmarks (KITTI Odometry, nuScenes) and two indoor benchmarks (RGB-D Scenes V2, 7-Scenes). Metrics comprise Relative Translation Error (RTE, in meters), Relative Rotation Error (RRE, in degrees), and Registration Accuracy (Acc, percentage of samples meeting \(\mathrm{RTE} < 2\,\text{m}\) and \(\mathrm{RRE} < 5^\circ\) for outdoor evaluation).

Primary comparative results from Table 1 (outdoor) and Table 2 (indoor) of the paper are summarized below:

Dataset	Method	RTE (m) ↓	RRE (°) ↓	Acc (%) ↑
KITTI	DeepI2P (2D) [CVPR'21]	\(3.59 \pm 3.21\)	\(11.66 \pm 18.16\)	25.95
	CorrI2P [TCSVT'22]	\(3.78 \pm 65.16\)	\(5.89 \pm 20.34\)	72.42
	VP2P-Match [NeurIPS'23]	\(0.75 \pm 1.13\)	\(3.29 \pm 7.99\)	83.04
	CFI2P [IEEE Sensors'24]	\(0.63 \pm 1.19\)	\(1.72 \pm 2.58\)	97.15
	CoFiI2P [RA-L'24]	\(0.31 \pm 0.20\)	\(1.24 \pm 0.84\)	—
	ICL [CVPR'25]	\(0.20 \pm 0.21\)	\(1.24 \pm 2.34\)	97.49
	GraphI2P† [CVPR'25]	\(0.32 \pm 0.81\)	\(1.65 \pm 1.32\)	99.61
	PGCR (Ours)	\(0.15 \pm 0.15\)	\(0.44 \pm 0.34\)	100.00
nuScenes	VP2P-Match [NeurIPS'23]	\(0.89 \pm 1.44\)	\(2.15 \pm 7.03\)	88.33
	CFI2P [IEEE Sensors'24]	\(0.86 \pm 1.47\)	\(1.83 \pm 1.28\)	95.23
	ICL [CVPR'25]	\(0.63 \pm 0.44\)	\(2.13 \pm 3.75\)	90.94
	GraphI2P† [CVPR'25]	\(0.49 \pm 1.22\)	\(1.73 \pm 1.63\)	99.48
	PGCR (Ours)	\(0.43 \pm 0.42\)	\(0.88 \pm 1.16\)	99.51
RGB-D Scenes v2	2D3D-MATR [ICCV'23]	\(0.077 \pm 0.066\)	\(3.026 \pm 2.649\)	—
	CA-I2P [ICCV'25]	\(0.061 \pm 0.055\)	\(2.559 \pm 2.190\)	—
	PGCR (Ours)	\(0.035 \pm 0.023\)	\(2.215 \pm 1.207\)	—
7-Scenes	CA-I2P [ICCV'25]	\(0.076 \pm 0.069\)	\(3.200 \pm 2.825\)	—
	PGCR (Ours)	\(0.049 \pm 0.029\)	\(2.622 \pm 1.603\)	—

(Note: † indicates methods that rely on external monocular depth estimation models)

Ablation Study

Table 5 in the paper evaluates the Pose-guided Progressive Refinement (PPR) components across stages, while Table 4 isolates the impact of Geometry-aware Pose Refinement (GPR).

Ablation 1: Progressive Refinement and Filtering Dynamics (Paper Table 5)

Configuration & Stage	KITTI RTE (m) ↓	KITTI RRE (°) ↓	nuScenes RTE (m) ↓	nuScenes RRE (°) ↓	Note
w/o PPR (One-shot)	\(0.33 \pm 0.26\)	\(0.94 \pm 1.19\)	\(0.74 \pm 0.73\)	\(1.66 \pm 2.55\)	Substantial error without progressive steps
Intermediate sup. only Stage-1	\(0.35 \pm 0.31\)	\(0.96 \pm 0.79\)	\(0.77 \pm 0.76\)	\(1.34 \pm 1.68\)	Global attention, unpruned points
Intermediate sup. only Stage-4	\(0.25 \pm 0.16\)	\(0.82 \pm 0.64\)	\(0.67 \pm 0.73\)	\(1.24 \pm 1.55\)	Progressive supervision benefits convergence
Full PPR (Filter + sup.) Stage-1	\(0.35 \pm 0.47\)	\(0.98 \pm 0.80\)	\(0.81 \pm 0.86\)	\(1.70 \pm 1.99\)	Wide initial tolerance window
Full PPR (Filter + sup.) Stage-2	\(0.27 \pm 0.35\)	\(0.74 \pm 0.55\)	\(0.70 \pm 0.73\)	\(1.54 \pm 1.70\)	Coarse outlier filtering takes effect
Full PPR (Filter + sup.) Stage-3	\(0.20 \pm 0.21\)	\(0.50 \pm 0.40\)	\(0.58 \pm 0.70\)	\(1.20 \pm 1.47\)	Effective suppression of off-view noise
Full PPR (Filter + sup.) Stage-4	\(0.17 \pm 0.18\)	\(0.49 \pm 0.36\)	\(0.50 \pm 0.66\)	\(1.06 \pm 1.33\)	Significant error reduction across all metrics

Ablation 2: Contribution of Geometry-aware Pose Refinement (Paper Table 4)

Configuration	KITTI RTE (m) ↓	KITTI RRE (°) ↓	nuScenes RTE (m) ↓	nuScenes RRE (°) ↓
w/o GPR (Camera token only)	\(0.17 \pm 0.18\)	\(0.49 \pm 0.36\)	\(0.50 \pm 0.66\)	\(1.06 \pm 1.33\)
w/ GPR (Fused geometric fields)	\(0.15 \pm 0.15\)	\(0.44 \pm 0.34\)	\(0.43 \pm 0.42\)	\(0.88 \pm 1.16\)

Key Findings

• Explicit overlap prediction is unnecessary: By steering cross-attention through progressive spatial frustum decay, PGCR reaches 100.00% registration accuracy on KITTI, eliminating the tail-end registration catastrophic failures that plague overlap-classification pipelines.
• Complementary geometric supervision: Dense depth supervision enforces metric radial scale while 3D point maps establish absolute world coordinate constraints; joint supervision outperforms any individual branch (Table 6).
• High inference efficiency: By pruning irrelevant 3D points layer by layer, PGCR reduces cross-attention token counts, achieving a runtime of 0.17 s per pair on an RTX 3090 (Table 3), faster than VP2P-Match (0.19 s) and CoFiI2P (0.18 s), while outperforming unpruned PGCR (0.26 s).

Highlights & Insights

• Active geometric attention modulation: Rather than leaving geometric consistency to post-processing PnP / RANSAC solvers, PGCR incorporates current pose hypotheses directly into the backbone's attention sparsity schedule, establishing a positive feedback loop between geometric reasoning and pose estimation.
• Differentiable multi-modal field fusion: Instead of brittle differentiable PnP solvers that suffer from gradient instability in ill-conditioned scenes, GPR unifies dense spatial fields and global pose tokens via channel projection and spatial pooling, producing clean gradient propagation.
• Transferable conceptual pattern: This pose-geometry closed-loop paradigm readily extends to monocular 3D perception, multi-sensor extrinsic calibration, and embodied agent relocalization against large-scale architectural scans.

Limitations & Future Work

• Reliance on accurate camera intrinsics: The projection filtering equations in PPR assume an accurately known calibration matrix \(K\); severe radial distortion or camera zoom requires future integration of self-calibration mechanisms.
• Sensitivity to raw point cloud density: While tolerant to moderate sparsity, the KPConv backbone relies on predefined voxelization grids; performance on ultra-sparse (e.g., 16-beam LiDAR) point sets requires further exploration.
• Degenerate geometric corridors: In geometrically degenerate environments such as symmetric hallways or featureless highway expanses, geometric consistency alone cannot resolve axial ambiguities, necessitating semantic-level foundation model priors.

Related Work & Insights

• vs CorrI2P / CoFiI2P: Prior methods depend on an upfront binary classifier to predict visual overlap masks, suffering irreversibly when points are misclassified; PGCR removes explicit overlap classification, permitting soft, continuous geometric refinement.
• vs VP2P-Match: VP2P integrates voxel-to-pixel correspondence with a differentiable PnP solver; PGCR eliminates PnP solver instability by directly decoding dense depth and point maps and rectifying pose via GPR.
• vs GraphI2P: GraphI2P relies on a separate, heavy pre-trained monocular depth model to bridge the domain gap; PGCR operates as a fully self-contained, end-to-end framework, lowering latency to 0.17 s while achieving superior accuracy on KITTI and nuScenes.

Rating

• Novelty: ⭐⭐⭐⭐☆ (Redefines image-to-point cloud registration from decoupled overlap-filtering into an end-to-end pose-geometry coupled reasoning loop with GPR and PPR)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluated across KITTI, nuScenes, RGB-D Scenes V2, and 7-Scenes with fine-grained ablations on stages, tolerances, and multi-task heads)
• Writing Quality: ⭐⭐⭐⭐⭐ (Clear mathematical formulation, cohesive narrative flow, and well-structured empirical validation)
• Value: ⭐⭐⭐⭐☆ (High practical utility for visual-LiDAR localization, autonomous driving, and embodied multi-modal spatial perception)

Related Papers

• [ECCV 2026] Make Geometry Matter for Spatial Reasoning
• [ECCV 2026] Cast and Attached Shadow Detection via Iterative Light and Geometry Reasoning
• [ECCV 2026] Generative Lane Topology Reasoning via Autoregressive Model with Geometry Prior
• [CVPR 2026] PointThinker: Point-Incentivized Parallel Thinking for Multimodal Large Language Model
• [ICLR 2026] Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMs