Ray-Path-Aware Virtual Point Removal on 2D Layer-Wise Nearest Point Map¶

Conference: ECCV 2026
Paper: ECCV Official Page
Full-Text Cache: ../paper_cache/ECCV2026/eccv-3932.txt
Area: Autonomous Driving / 3D Vision
Keywords: Virtual Point Removal, 3D LiDAR Point Cloud, Equirectangular Projection, Ray-Path-Aware Attention, Multi-Return Representation

TL;DR¶

Addressing the erroneous virtual mirror points generated when LiDAR scans reflective materials (such as glass walls), this paper departs from explicit 3D surface modeling and heuristic symmetry assumptions by proposing an end-to-end removal framework in the 2D equirectangular projection domain using a Layer-Wise Nearest Point Map and Ray-Path-Aware Attention, improving the F1 score to 0.7895–0.8062 with over two orders of magnitude computational speedup.

Background & Motivation¶

Light Detection and Ranging (LiDAR) sensors have become indispensable in autonomous driving and 3D mapping due to their capability to acquire accurate, high-resolution metric geometry. In particular, multi-echo LiDAR sensors capture multiple returns per laser pulse, greatly improving spatial penetration in complex environments. However, when a laser pulse encounters specular or semi-transparent materials such as glass facades, polished marble, or metallic surfaces, a portion of the ray undergoes specular reflection and strikes objects located along an altered trajectory. Because the sensor assumes straight-line time-of-flight (ToF) propagation, it records erroneous 3D returns that mimic physical structures, commonly referred to as "virtual points". Because these virtual points preserve plausible geometric structure, shape, and density, conventional distance filtering or statistical clustering algorithms cannot separate them from real surfaces, severely corrupting downstream tasks such as obstacle detection, point cloud registration, and mapping.

Prior approaches predominantly rely on multi-stage geometric pipelines consisting of reflective region extraction, explicit 3D reflective plane fitting, and cross-plane symmetry verification. Such multi-stage pipelines suffer from fundamental structural vulnerabilities: in real-world environments, reflective surfaces are often sparse, noisy, or contaminated, causing slight normal estimation biases and surface roughness to propagate into substantial reflection-angle deviations. Consequently, upstream surface fitting errors cascade into the downstream symmetry comparison step, destabilizing detection across different scenes. Furthermore, multi-view consistency methods require precise inter-scan registration and temporal synchronization, which frequently break down in single-scan scenarios or dynamic urban environments.

To resolve this limitation, this work circumvents explicit 3D surface modeling and rethinks the problem from the perspective of LiDAR projection geometry. Core idea: reformulate virtual point removal as ray-aligned geometric reasoning in the 2D equirectangular projection (ERP) domain, introducing a Layer-Wise Nearest Point Map that preserves geometric continuity across multi-return layers and a Ray-Path-Aware Attention mechanism that constrains feature aggregation along physically valid optical reflection trajectories, enabling the network to learn reflection-induced symmetry end-to-end without explicit surface reconstruction.

Method¶

Overall Architecture¶

The proposed framework takes as input a raw 3D LiDAR point cloud containing both real surfaces and reflection-induced virtual points. It first projects the irregular 3D point cloud into the 2D equirectangular projection (ERP) domain to construct a continuous Layer-Wise Nearest Point Map (LNPM). Then, a U-Net style encoder-decoder equipped with Symmetry Attention Blocks extracts multi-scale reflection-aware representations. The network outputs a 2D pixel-level probability map indicating whether each multi-layer return corresponds to a virtual reflection. Finally, a deterministic 2D-to-3D unprojection process maps the predicted labels back onto the original 3D points, producing a clean point cloud with virtual artifacts removed.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input 3D Point Cloud<br/>Real points and virtual reflections"] --> LNPM["Layer-Wise Nearest Point Map<br/>Aligns multi-returns per ray while preserving geometric continuity"]
    LNPM --> Enc["U-Net Encoder<br/>Strided conv downsampling for multi-scale feature extraction"]
    Enc --> SAB["Symmetry Attention Block<br/>Ray-Path-Aware Attention + Normal Estimator + LeFF + GPSA"]
    SAB --> Dec["U-Net Decoder<br/>Transposed conv upsampling to reconstruct multi-layer features"]
    Dec --> Pred["2D Multi-Layer Label Prediction<br/>Binary classification probability map for virtual points"]
    Pred --> Unproj["2D to 3D Unprojection<br/>Deterministic nearest reference distance assignment"]
    Out["Cleaned 3D Point Cloud<br/>Virtual points removed while preserving real physical geometry"]
    Unproj --> Out

Key Designs¶

1. Layer-Wise Nearest Point Map: Preserving multi-return geometric continuity across projection layers
Conventional 2D range maps store only the nearest return per ray, discarding the critical multi-return cues required to detect virtual points. On the other hand, naive multi-layer range images discretize radial distance into uniform, fixed intervals and assign points to static depth bins. This rigid binning cuts continuous surfaces across bin boundaries, introducing severe artificial fragmentation and scattering symmetric real-virtual pairs into disconnected layers. To solve this, the Layer-Wise Nearest Point Map (LNPM) introduces a layer-centric mapping strategy. For each pixel $(u, v)$ in the ERP grid corresponding to ray $R(u, v)$, the multi-return set $S(u, v)$ is gathered. Across $L$ predefined reference distance layers $\{d_i\}_{i=1}^L$, the value at layer $M_i(u, v)$ is filled by the 3D point whose range is closest to reference depth $d_i$: $$M_i(u, v) = \|\hat{p}_j\|, \quad \text{where} \quad \hat{p}_j = \arg\min_{p_j \in S(u, v)} |\|p_j\| - d_i|$$ For non-reflective rays containing a single surface, every layer samples the same physical point, preserving dense surface continuity. For reflective rays, different layers sample different returns along the ray. Crucially, when reference distance $d_i$ lies between real and virtual ranges, both real and virtual points appear simultaneously in adjacent layers, providing structural multi-depth correspondence for the subsequent attention mechanism.

2. Ray-Path-Aware Attention: Constraining feature aggregation along optical reflection trajectories
Standard self-attention mechanisms compute unconstrained pairwise affinities across the entire feature map, which is computationally expensive and prone to spurious associations with naturally symmetric physical structures (e.g., repeating windows or columns). To instill physical reflection constraints, the proposed module leverages the first layer $M_1(u, v)$—representing the first physical surface hit by the ray—to compute surface normals $N(u, v)$. Given the incident ray direction $r = R(u, v)$ and normal $n = N(u, v)$, the reflected ray direction is analytically computed via the law of reflection: $$\bar{r} = r - 2(r \cdot n)n$$ Within the observed return interval $[t_{\min}, t_{\max}]$ derived from $S(u, v)$, the 3D reflected ray is uniformly sampled as $Q(u, v) = \{o + t\bar{r} \mid t \in [t_{\min}, t_{\max}]\}$. Projecting $Q(u, v)$ back onto the ERP domain yields a curved 2D trajectory linking the reflective surface to its potential virtual counterparts. Attention aggregation is then strictly restricted to the $K$ nearest neighbors along this projected trajectory, transforming unguided attention into physics-constrained correspondence reasoning.

3. Symmetry Attention Block: Fusing local geometry, cross-layer interaction, and physical reflection
To jointly reason about local surface context and long-range optical paths, the Symmetry Attention Block unifies three complementary components. First, a Local Enhancement Feed-Forward (LeFF) module employs depthwise separable convolutions to enhance fine-grained local geometric details within each individual layer. Second, Global Position Self-Attention (GPSA) coordinates feature interactions across different reference depth layers. Finally, the Ray-Path-Aware Attention module steers feature aggregation along the physical reflection trajectories. This hybrid design forms a compact feature space capable of identifying mirror-symmetric patterns without requiring explicit 3D surface reconstruction.

4. 2D-to-3D Unprojection: Deterministic and dual label mapping
The network predicts a multi-layer label tensor $\hat{Y} \in \mathbb{R}^{H \times W \times L}$. To assign labels back to the original 3D point cloud $P$ without ambiguity, each 3D point $p$ is mapped to its nearest reference layer index $\hat{i} = \arg\min_i |\|p\| - d_i|$ and projected pixel coordinate $(u, v) = \Pi_{\text{ERP}}(p)$. The point is classified as a virtual reflection if the predicted probability $\hat{Y}_{\hat{i}}(u, v)$ exceeds a threshold of 0.5. Because this unprojection is strictly dual to the forward LNPM formulation, it eliminates the need for heuristic clustering or spatial post-processing.

Loss & Training¶

The network is supervised in the 2D ERP domain via Binary Cross-Entropy (BCE) loss. A validity mask $V \in \{0, 1\}^{H \times W \times L}$ is applied to filter out empty pixels where no LiDAR return exists: $$\mathcal{L} = -\frac{1}{|V|} \sum_{(u, v, i) \in V} \left[ y_i(u, v) \log \hat{y}_i(u, v) + (1 - y_i(u, v)) \log(1 - \hat{y}_i(u, v)) \right]$$ To overcome the scarcity and annotation difficulty of real-world reflection data, the model is pre-trained for 100 epochs on a large-scale synthetic dataset (3,000 point clouds generated via ray tracing on Semantic3D scenes) using AdamW, followed by 20 epochs of fine-tuning on real-world LiDAR scans.

Key Experimental Results¶

Main Results¶

Quantitative evaluations were performed on two real-world terrestrial LiDAR datasets: the UNIST LS3DPC dataset (11 scenes captured with RIEGL VZ-400) and the 3DRN dataset (12 scenes captured with RIEGL VZ-200i), benchmarked against three representative multi-stage methods (Yun & Sim [25], Lee et al. [11], and Fang et al. [3]).

Method	Fine-Tuning Data (FT)	UNIST Prec.	UNIST Rec.	UNIST F1	UNIST Avg. Time	3DRN Prec.	3DRN Rec.	3DRN F1
Yun and Sim (PAMI 2019)	-	0.7206	0.7475	0.7151	2h 5m	0.5361	0.7192	0.6143
Lee et al. (RA-L 2023)	-	0.6089	0.7906	0.6848	2h 2m	-	-	-
Fang et al. (ISPRS 2025)	-	0.7627	0.7534	0.7580	37m 43s	0.7436	0.8291	0.7840
Ours (Synthetic Only)	-	0.7959	0.7238	0.7541	8.2s	0.8043	0.7711	0.7874
Ours (Cross Fine-Tuning)	3DRN / UNIST	0.8194	0.7618	0.7895	8.2s	0.8239	0.7894	0.8062

(Note: Data taken directly from Table 1 of the original paper. In the cross fine-tuning setting, models evaluated on UNIST were fine-tuned on 3DRN, and vice versa, highlighting strong cross-domain generalization.)

Ablation Study¶

Ablations on projection representations, attention mechanisms, and normal estimation robustness conducted on the UNIST dataset:

Configuration / Attention Variant	Prec.	Rec.	F1	Notes (from Table 2a in paper)
Naive projection (fixed intervals)	0.1715	0.2768	0.2043	Fixed depth bins fragment surface continuity; severe performance collapse
ViT (global self-attention)	0.6456	0.6613	0.6533	Lacks physical geometry prior; prone to spurious matches
UFormer (window self-attention)	0.5456	0.7427	0.6290	Local windows fail to capture long-range mirror symmetry
H-win (horizontal window attention)	0.5331	0.8654	0.6597	High recall but excessively prunes valid physical points (low precision)
H-win + V-win (cross-shaped window)	0.6360	0.7293	0.6794	Rigid orthogonal axes cannot track arbitrary reflection trajectories
Ours (LNPM + Ray-Path Attention)	0.8194	0.7618	0.7895	Physical ray-path guidance yields the best precision-recall trade-off

Robustness to surface normal estimation errors (Table 2b): Replacing 3D PCA normals with 2D depth-gradient normals retains an F1 of 0.7705. Introducing $[0, 9^\circ]$ uniform angular noise lowers F1 from 0.7895 to 0.7516, while injecting 20% random outlier errors ($[30^\circ, 50^\circ]$) still preserves an F1 of 0.7306, confirming that the framework is resilient to imperfect normal inputs.

Key Findings¶

Representation determines feasibility: Replacing LNPM with naive interval-based range mapping causes F1 to collapse from 0.7895 to 0.2043 (a 74% drop). Maintaining surface continuity across layers is essential for dense representation learning on multi-return LiDAR data.
Dramatically faster inference: Multi-stage 3D pipelines take between 37 minutes and over 2 hours per scan due to iterative 3D spatial clustering and plane fitting. By operating in the 2D ERP grid, the proposed method runs in just 8.2 seconds per scan, achieving a 270x–900x acceleration.
Superior precision minimizes false removals: In safety-critical perception, mistakenly removing real obstacles is far more hazardous than missing some noise. The proposed method achieves the highest precision (0.8194 on UNIST, 0.8239 on 3DRN), ensuring physical structures are preserved.

Highlights & Insights¶

From explicit fitting to implicit optical reasoning: Rather than attempting fragile RANSAC plane fitting on noisy 3D point clouds, the paper maps Snell's law into a 1D attention trajectory in ERP space, elegantly simplifying 3D reflection modeling into 2D guided attention.
Layer-Wise Nearest Point Map: The layer-centric formulation ensures that non-reflective rays remain dense and continuous, while reflective rays naturally partition multi-returns across depth layers, offering a generalizable representation for multi-echo LiDAR processing.
Transferable physics-guided attention: Restricting attention receptive fields to physically computed optical paths can be readily extended to other perception domains, such as automotive windshield reflection removal, puddle and water surface glint filtering, and indoor specular SLAM.

Limitations & Future Work¶

Planar single-bounce reflection assumption: The ray trajectory formulation currently assumes single specular reflections from planar surfaces. Curved reflectors (e.g., automotive side mirrors) or multi-bounce reflections (e.g., mutually reflecting glass corridors) violate this simple 1D trajectory model.
Dependence on initial surface returns: If the reflective surface is completely transparent and yields zero first-return points in $M_1$, normal estimation cannot initialize the reflection path, causing the attention guidance to fail.
Online automotive frame-rate gap: While 8.2 seconds represents a massive speedup over prior multi-hour methods, real-time autonomous driving requires 10Hz (100ms) execution, necessitating further model compression, resolution pruning, and TensorRT acceleration.

vs Yun & Sim [24, 25]: Relies on explicit RANSAC glass plane fitting and rigid Euclidean symmetry checks, which are easily confused by natural architectural symmetry (e.g., repetitive pillars) and take over 2 hours. This paper uses global scene context and directional ray attention to achieve superior accuracy in seconds.
vs Lee et al. [11]: Introduced learned reflection detection but still retained a coupled multi-stage pipeline where surface fitting errors compound downstream. This work is purely single-stage and end-to-end.
vs Fang et al. [3]: Employs coupled optical-radiometric modeling requiring intensity attributes. It degrades or cannot run on density-sampled datasets lacking intensity (such as 3DRN). The proposed method operates purely on geometric multi-return coordinates, offering greater versatility.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ (First end-to-end LiDAR virtual point removal framework in the ERP domain with physics-guided optical attention.)
Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Comprehensive benchmarking on UNIST and 3DRN datasets, extensive cross-dataset evaluations, and detailed ablations on representations, attention variants, and normal noise.)
Writing Quality: ⭐⭐⭐⭐⭐ (Clear mathematical formulation, cohesive narrative, and well-designed Mermaid architecture diagram.)
Value: ⭐⭐⭐⭐⭐ (Substantially advances both accuracy and practical feasibility by reducing processing time from hours to seconds.)