Geometry-Preserving in 3D Gaussian Splatting for LiDAR-Camera Extrinsic Calibration

Conference: ECCV 2026
arXiv: 2606.20103
Code: None
Area: Autonomous Driving
Keywords: 3D Gaussian Splatting, LiDAR-Camera Extrinsic Calibration, Targetless Calibration, Geometry Preserving, Gradient Decoupling

TL;DR

This work proposes GeoP-Calib, which mitigates the "geometric decay" issue in 3DGS calibration through two designs: Dense Depth Anchoring (DDA, which aggregates multi-view LiDAR point clouds to establish a dense depth prior) and Gradient Decoupling (GD, which blocks the backpropagation of photometric loss to Gaussian spatial parameters). It achieves translation errors significantly superior to existing targetless calibration methods on KITTI-360 and KITTI.

Background & Motivation

LiDAR-camera extrinsic calibration is fundamental for multi-modal perception fusion—cameras provide dense semantics, while LiDAR provides precise 3D geometry; however, they must be accurately aligned spatially to be effective. Traditional target-based methods offer high accuracy but rely on manual setup, rendering them unable to perform automatic recalibration during system operation. Targetless methods utilize cross-modal geometric features such as edges and structural lines in the scene for automatic calibration, avoiding human intervention, but they are limited by the sparsity of discriminative features in natural scenes, showing unstable performance in texture-poor or structurally repetitive environments.

Recent trends favor reconstructing the entire scene into a differentiable model (such as 3DGS), replacing sparse feature matching with dense pixel-level supervision to optimize extrinsics. Each Gaussian primitive in 3DGS can be initialized from LiDAR points, inheriting their precise metric positions, and then projected onto the camera imaging plane via differentiable rasterization, naturally acting as a "geometric proxy" between the LiDAR coordinate system and the camera image domain. However, 3DGS was originally designed for view synthesis, so optimization naturally favors rendering quality—the optimizer freely adjusts the positions and shapes of Gaussians to minimize photometric residuals, even at the expense of distorting the underlying geometric structure. Although existing methods constrain the proxy geometry using single-frame sparse LiDAR depth supervision, a single-frame projection covers only a tiny fraction of the image pixels. Most regions remain unconstrained in the metric space, allowing Gaussians in these regions to drift while still producing visually plausible rendering. This paper names this phenomenon Geometric Decay: the metric structure of the Gaussian proxy gradually loses its fidelity to the true LiDAR structure, leading to degradation in extrinsic calibration that relies on metric accuracy.

Core Idea: To protect the proxy geometry from both "forward reinforcement" and "backward blocking" pathways—using multi-view LiDAR aggregation to construct dense depth anchors with wider coverage (forward), and blocking the gradient backpropagation of photometric loss to Gaussian spatial parameters (positions, covariances) (backward). This ensures that texture-driven rendering optimization cannot be achieved at the expense of geometric fidelity.

Method

Overall Architecture

The core idea of GeoP-Calib is to decouple "shape preservation" and "appearance rendering" of the 3DGS proxy geometry: the geometric shape is driven solely by LiDAR depth supervision, while the appearance color is driven solely by photometric loss, avoiding mutual interference. The input consists of multi-view camera images, LiDAR point cloud sequences, and the world poses of the LiDAR, and the output is the optimized LiDAR-camera extrinsic matrix \(\mathbf{T}_{cl} \in SE(3)\).

Specific pipeline: 3D Gaussians are first initialized with LiDAR point clouds (means \(\mu\) directly inherit the 3D coordinates of LiDAR points), followed by a two-stage optimization. The geometric warm-up stage only uses the sparse depth loss \(\mathcal{L}_{depth}^{sparse}\) to establish a coarse proxy geometry; the refinement stage activates the dense depth anchoring loss \(\mathcal{L}_{depth}^{dense}\) (DDA module). Meanwhile, gradient decoupling (GD module) is applied to the photometric loss \(\mathcal{L}_{pho}^{decoupled}\) throughout the process, blocking its gradients to \(\mu, \Sigma\) and only allowing updates to appearance parameters \(\alpha, c\) and extrinsics \(\mathbf{T}_{cl}\). The reprojection loss \(\mathcal{L}_{rep}\) is not restricted by GD, as its gradients are mediated by depth-induced geometric reprojection relationships, primarily promoting view-consistent geometric structures rather than per-view texture fitting.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view LiDAR Point Clouds"] --> B["DDA: Dense Depth Anchoring<br/>Global Point Cloud Accumulation + Occlusion Soft Mask"]
    A --> C["Gaussian Initialization<br/>μ ← LiDAR 3D Coordinates"]
    C --> D["3DGS Proxy Geometry<br/>Differentiable Rendered RGB + Depth Map"]
    B --> E["Dense Depth Loss<br/>Constrains Spatial Parameters μ, Σ"]
    D --> E
    D --> F["GD: Photometric Loss Decoupling<br/>sg[μ, Σ] Blocks Spatial Gradients"]
    D --> G["Reprojection Loss L_rep<br/>Geometric Consistency Supervision"]
    E --> H["Joint Optimization<br/>Extrinsics T_cl"]
    F --> H
    G --> H

Key Designs

1. Dense Depth Anchoring (DDA): Building a Rigid Geometric Foundation via Multi-view Accumulation

Single-frame LiDAR depth supervision (Eq. 9) only takes effect on pixels \(\Omega^t\) covered by LiDAR point projections. These pixels account for a tiny fraction of the image, leaving the proxy geometry in other regions entirely free from metric constraints. The core idea of DDA is to transform and accumulate all multi-frame LiDAR point clouds into the global coordinate system using known LiDAR world poses \(\mathbf{T}_{lw}^t\): \(\mathcal{P}_{global} = \bigcup_t (\mathbf{T}_{lw}^t)^{-1} \mathcal{P}^t\). Then, projecting \(\mathcal{P}_{global}\) onto the imaging plane from the current LiDAR view produces a dense depth prior \(D_l^{dense}\) that covers a significantly larger pixel area than a single frame, providing greatly enhanced metric constraints for the Gaussian spatial parameters \((\mu, \Sigma)\).

Multi-view accumulation inevitably introduces occlusion ambiguities: LiDAR points from distant views might be wrongly projected onto foreground objects. This work observes that 3DGS inherently possesses implicit visibility awareness—the rendered depth \(D_{rend}\) represents the expected ray termination position, which can serve as a continuous occlusion detection signal. Based on this, a Volume Soft Mask (VSM) is proposed to map the difference between the rendered depth and accumulated depth to soft weights using a sigmoid function:

\[W_{vis}(\mathbf{p}) = \sigma\left(\beta \cdot \big(D_{rend}(\mathbf{p})(1+\tau) - D_l^{dense}(\mathbf{p})\big)\right)\]

where \(\tau\) is the depth ratio tolerance, and \(\beta\) controls the transition sharpness. If the accumulated point is far behind the rendered surface (\(D_l^{dense} \gg D_{rend}\)), the weight approaches 0, smoothly suppressing occluded projections. Finally, the dense depth anchoring loss computes the L1 residual in the inverse depth domain, weighted by VSM, and excludes pixels without rendered depth using a validity mask \(M(\mathbf{p})\):

\[\mathcal{L}_{depth}^{dense} = \frac{\sum_{\mathbf{p}\in\Omega^{dense}} M(\mathbf{p}) W_{vis}(\mathbf{p}) \Delta d^{inv}(\mathbf{p})}{\sum_{\mathbf{p}\in\Omega^{dense}} M(\mathbf{p}) W_{vis}(\mathbf{p}) + \epsilon}, \quad \Delta d^{inv}(\mathbf{p}) = \left|\frac{1}{D_{rend}(\mathbf{p})+\epsilon} - \frac{1}{D_l^{dense}(\mathbf{p})+\epsilon}\right|\]

Designing the residual calculation in the inverse depth domain is a practical trick: it is more sensitive to close-range depth errors, and is more stable in parameterization when the depth range span is large.

2. Gradient Decoupling (GD): Blocking the Deforming Force of Photometric Textures on Proxy Geometry

In 3DGS calibration, the number of Gaussians is typically limited by the budget of LiDAR voxelization or point sampling to maintain metric correspondence with the LiDAR initialization. Under such restricted proxy resolution, detailed textures from photometric observations cannot be fully explained by appearance parameters (color \(c\), opacity \(\alpha\)). Consequently, the optimizer naturally modifies geometric parameters (mean \(\mu\), covariance \(\Sigma\)) to minimize photometric errors, sacrificing metric fidelity for visually plausible rendering—this is the driving mechanism of geometric decay.

The approach of GD is extremely simple: applying a stop-gradient operation to the spatial parameters along the backpropagation path of the photometric loss:

\[\mathcal{L}_{pho}^{decoupled} = \mathcal{D}_{photo}\big(I^t, \Phi_{color}(\text{sg}[\mu, \Sigma], \alpha, c, \mathbf{T}_{cl} \cdot \mathbf{T}_{lw}^t)\big)\]

In implementation, this is equivalent to treating \(\mu, \Sigma\) as constants during color rendering. The gradients of the photometric loss can only flow to \(\alpha, c\) (appearance) and \(\mathbf{T}_{cl}\) (extrinsics), preventing them from affecting the shape of the proxy geometry. To put it simply: if color matching is poor, the optimizer can only adjust the color or the extrinsics, rather than moving the Gaussian positions to "make things fit."

3. Selective Gradient Blocking Strategy

GD does not apply a one-size-fits-all block to gradients from all image-domain losses to spatial parameters. Although the reprojection loss \(\mathcal{L}_{rep}\) also uses a photometric distance metric, it compares differences in actual image intensities at depth-induced corresponding pixels, without involving the learned Gaussian color attributes \(c\). Its gradient indirectly affects \(\mu, \Sigma\) through the reprojected geometric relationship \(\mathbf{p}^{t} \rightarrow \tilde{\mathbf{p}}^{t+s}\) (Eq. 5), mainly promoting geometric consistency across views rather than per-view texture overfitting. Experimental evidence in Fig. 1(c) supports this: after the photometric loss is activated, the depth loss rises sharply, whereas subsequent activation of the reprojection loss does not cause similar degradation. Therefore, GD only blocks the gradient of \(\mathcal{L}_{pho}\) to \((\mu, \Sigma)\), preserving the full gradient path of \(\mathcal{L}_{rep}\) to utilize image-domain supervision while controlling geometric decay.

Loss & Training

The final optimization objective of GeoP-Calib is a weighted sum of four components:

\[\mathcal{L}_{total}^{GP} = \mathcal{L}_{pho}^{decoupled} + \lambda_{rep}\mathcal{L}_{rep} + \lambda_{depth}^{sparse}\mathcal{L}_{depth}^{sparse} + \lambda_{depth}^{dense}\mathcal{L}_{depth}^{dense}\]

where \(\mathcal{L}_{pho}^{decoupled}\) is the photometric alignment loss (pixel-level L1) processed by GD, \(\mathcal{L}_{rep}\) is the cross-view reprojection consistency loss, \(\mathcal{L}_{depth}^{sparse}\) is the single-frame sparse depth anchoring loss under rotation extrinsics (Eq. 9), and \(\mathcal{L}_{depth}^{dense}\) is the DDA dense depth anchoring loss (Eq. 13).

A two-stage optimization scheduling is adopted: (1) The geometric warm-up stage only uses \(\mathcal{L}_{depth}^{sparse}\) along with \(\mathcal{L}_{pho}^{decoupled}\) and \(\mathcal{L}_{rep}\) to establish a stable coarse geometry; (2) The refinement stage activates \(\mathcal{L}_{depth}^{dense}\), leveraging dense depth anchors to lock down Gaussian spatial parameters. GD is active throughout both stages. The implementation is based on the HiGS-Calib codebase, averaging about 15.4 minutes per sequence on an RTX 4070 Ti.

Key Experimental Results

Main Results

Evaluated on KITTI-360 (5 sequences, averaged over 2 forward-facing cameras) and KITTI odometry (5 sequences, cam2 only), comparing with GST (traditional optimization), CLAIM (foundation model method), RobustCalib, and HiGS-Calib (both also 3DGS-based methods). Metrics are rotation error \(E_r\) (degrees) and translation error \(E_t\) (meters), with standard deviations in parentheses.

Method	KITTI-360 Er(°)	KITTI-360 Et(m)	KITTI Er(°)	KITTI Et(m)
GST	1.547 (1.570)	0.315 (0.099)	1.671 (0.892)	0.278 (0.085)
CLAIM	0.520 (0.303)	0.129 (0.126)	0.352 (0.074)	0.082 (0.022)
RobustCalib	0.255 (0.055)	0.097 (0.028)	0.352 (0.071)	0.082 (0.022)
HiGS-Calib	0.140 (0.059)	0.103 (0.055)	0.206 (0.053)	0.055 (0.008)
GeoP-Calib	0.121 (0.041)	0.063 (0.021)	0.188 (0.051)	0.044 (0.013)

GeoP-Calib achieves the best performance in 14 out of 20 sequence-metric combinations. On KITTI-360, it improves rotation by 0.019° and translation by 0.034 m compared to the second-best results; on KITTI, it improves rotation by 0.018° and translation by 0.011 m. The translation improvement is particularly notable—GeoP-Calib achieves lower translation errors than HiGS-Calib across all 5 sequences of KITTI-360, indicating that geometric fidelity contributes significantly more to the translation component (which relies on the metric accuracy of the depth scale).

Ablation Study

Component ablation on KITTI-360 (table below, averaged across 5 aggregated sequences). VSM is a sub-component of DDA and does not exist independently.

Configuration	GD	DDA	VSM	Er(°)	Et(m)	Time (s)
Base				0.122 (0.031)	0.077 (0.022)	875 (16)
+DDA		✓		0.117 (0.044)	0.068 (0.021)	923 (16)
+GD	✓			0.127 (0.042)	0.067 (0.022)	866 (21)
+GD+DDA (w/o VSM)	✓	✓		0.124 (0.045)	0.066 (0.024)	—
+GD+DDA+VSM (Full)	✓	✓	✓	0.121 (0.041)	0.063 (0.021)	922 (21)

Base \(\rightarrow\) +DDA: Translation decreases from 0.077 to 0.068 (a drop of 0.009 m), and rotation slightly drops to 0.117 (a drop of 0.005°); Base \(\rightarrow\) +GD: Translation drops to 0.067 (a drop of 0.010 m), while rotation slightly increases to 0.127 (an increase of 0.005°, which is negligible within standard deviation); the full model achieves 0.063 translation (a 0.014 m decrease relative to Base) and 0.121 rotation (a 0.001° decrease relative to Base). DDA adds approximately 50 seconds of additional running time. VSM contributes an additional 0.003° rotation improvement and 0.003 m translation improvement on top of GD+DDA, confirming the consistent benefits of occlusion handling for dense depth supervision.

Key Findings

• Translation gain outweighs rotation: Ablations on both GD and DDA show that the improvement in translation far exceeds that of rotation. This suggests that geometric decay primarily impacts the translation component (which depends on depth consistency), whereas the rotation component benefits more from texture cues.
• Rendering quality is not equal to calibration quality: The rendering PSNR of GeoP-Calib is lower than that of the Base pipeline, but its calibration accuracy is higher. This directly proves that photometric fidelity is not a sufficient condition for calibration accuracy—good rendering does not equate to precise alignment.
• Quantitative validation of geometric accuracy: Upon convergence, the depth MAE (rendered depth vs. LiDAR depth) is 0.244 for GeoP-Calib and 0.294 for Base, an absolute reduction of 0.05 (approximately 17%). Furthermore, the error distribution is more uniform, avoiding the geometric decay phenomenon where local errors spike sharply as seen in the Base model.
• Noise robustness: Within the initial noise range of 1-9° rotation perturbations and 0.1-0.9 m translation perturbations, GeoP-Calib converges robustly without experiencing divergence or accuracy collapse.

Highlights & Insights

• Precise problem diagnosis: The introduction and analysis of the "Geometric Decay" concept (via the three-panel illustration and quantitative depth loss curves in Fig. 1) are straightforward and powerful, clearly demonstrating how photometric optimization degrades geometric fidelity, which provides a clean target for subsequent design.
• Extremely simple yet complete solution: DDA (forward reinforcement) + GD (backward blocking) form a complete protective loop. These two designs are orthogonal and complementary—DDA provides rigid metric constraints, while GD prevents these constraints from being broken, making both indispensable. The ablation study perfectly validates this complementarity (Base 0.077/0.122, +DDA 0.068/0.117, +GD 0.067/0.127, combining both 0.063/0.121).
• Stop-gradient is a highly reusable trick: For any task where "scene representation acts as an optimization proxy" (such as pose estimation, SLAM, NeRF/3DGS-based registration), whenever there is a conflict between geometric accuracy and appearance quality, the GD concept can be integrated at low cost—simply by adding a single line of detach() in the rendering path for the relevant parameters.
• The clever "use a shield to block one's own spear" design philosophy of VSM: Using 3DGS's own rendered depth as an occlusion detection signal to guide the filtering of accumulated LiDAR point clouds eliminates the need for an extra visibility inference module. This is a self-consistent design where "scene representation acts as both a proxy and a filter."

Limitations & Future Work

• Limited rotation gain: The authors acknowledge that while GD effectively blocks geometric decay, it might also limit the extent to which the optimization can exploit texture cues, resulting in a relatively modest improvement in the rotation dimension. This is an inherent trade-off in the design.
• Sensitivity of dense accumulation to pose accuracy: DDA relies on LiDAR world poses \(\mathbf{T}_{lw}^t\) to transform multi-frame point clouds into the global coordinate system. If the SLAM/odometry poses have errors, the accumulated point clouds will suffer from misalignment, consequently polluting the accuracy of depth anchors. The paper does not discuss the impact of pose noise on DDA.
• Single-sequence calibration assumption: GeoP-Calib jointly optimizes the scene and extrinsics within the current sequence and does not share this optimization across sequences. This means that a new scene requires running the optimization independently (approx. 15 minutes), making real-time, instantaneous inference like regression methods impossible.
• Potential future directions: (1) Introduce adaptive GD—moderately allowing some spatial gradients in texture-rich areas while strictly blocking them in texture-sparse regions, thereby balancing geometric fidelity and rotation accuracy; (2) Use robust pose graph optimization instead of direct accumulation to mitigate the impact of SLAM drift on DDA; (3) Explore reusing the trained proxy geometry for multi-sequence calibration to amortize the optimization overhead.

Related Work & Insights

• vs HiGS-Calib [zhang2025higs]: As another 3DGS calibration method, HiGS adopts a hierarchical coarse-to-fine architecture and decouples scene modeling from extrinsic refinement, but does not explicitly address the geometric decay issue. The DDA+GD of GeoP-Calib can be viewed as a "geometric fidelity patch" for HiGS, which is orthogonal and stackable with HiGS's decoupled architecture. In fact, GeoP-Calib is built on top of the HiGS codebase.
• vs RobustCalib [zhou2025robust]: RobustCalib uses 2D Gaussian Splatting and refines extrinsics through reprojection and triangulation losses. The findings in GeoP-Calib—that reprojection loss does not exacerbate geometric decay (Fig. 1c)—indirectly support RobustCalib's design choices.
• vs NeRF-based Calibration [zhou2023inf, yang2024unical]: NeRF similarly faces the photometric-geometric conflict, but the computational overhead of volume rendering limits large-scale experimental validation. GeoP-Calib quantifies this conflict at a lower cost within the 3DGS framework, which also offers insights for NeRF-based calibration.
• vs 3DGS-based SLAM: 3DGS-SLAM also needs to maintain geometric consistency. The concept of GD (blocking texture gradients from interfering with map geometry) can naturally migrate to online SLAM scenarios.

Rating

• Novelty: ⭐⭐⭐⭐ — The "Geometric Decay" concept is accurately diagnosed and supported by experiments. The GD+DDA solution is concise, offering high conceptual value over technical complexity.
• Experimental Thoroughness: ⭐⭐⭐⭐ — Dual datasets with 10 sequences + complete ablation + noise robustness + rendering quality vs. calibration quality + depth MAE, offering comprehensive coverage.
• Writing Quality: ⭐⭐⭐⭐ — The problem definition (Sec 3) progresses clearly; the demonstration of geometric decay in Fig. 1 is intuitive and effective; formulas and intuition support each other in the method description.
• Value: ⭐⭐⭐⭐ — It resolves real pain points in 3DGS calibration. Both GD and DDA can be independently transferred to other pose optimization tasks that use 3DGS/NeRF as a geometric proxy.

Related Papers

• [CVPR 2026] ParkGaussian: Surround-view 3D Gaussian Splatting for Autonomous Parking
• [ECCV 2026] 2K Retrofit: Entropy-Guided Efficient Sparse Refinement for High-Resolution 3D Geometry Prediction
• [ICCV 2025] Splat-LOAM: Gaussian Splatting LiDAR Odometry and Mapping
• [CVPR 2026] Generalizing Visual Geometry Priors to Sparse Gaussian Occupancy Prediction
• [CVPR 2025] RC-AutoCalib: An End-to-End Radar-Camera Automatic Calibration Network