Reflection-Aware Reasoning for Non-Line-of-Sight Pedestrian Localization¶
Conference: ECCV 2026
Paper: ECCV Paper
Project: Project Page
Code: GitHub
Area: Autonomous Driving / Multimodal VLM
Keywords: NLOS perception, mmWave radar, multipath reflection reasoning, cross-modal fusion, collision avoidance
TL;DR¶
Addressing the severe multipath distortion and clutter caused by ego-vehicle dynamics, this paper proposes an end-to-end reflection-aware multimodal learning and physics-guided ray tracing framework that fuses camera and 2D mmWave radar representations in BEV space to disentangle higher-order reflections and reconstruct reflective surfaces, reducing NLOS pedestrian localization error to 1.23 m while providing a 0.77 s advance warning on a moving testbed vehicle.
Background & Motivation¶
In dense urban driving environments, structural occlusions from buildings, walls, and sound barriers frequently create severe non-line-of-sight (NLOS) blind spots. At narrow alleys and blind intersections, pedestrians stepping into the roadway are completely concealed until they emerge right in front of the vehicle—often far exceeding the safe emergency braking distance. Statistically, more than 80% of pedestrian fatalities occur in complex urban environments, exposing the fundamental physical boundary of direct line-of-sight (LOS) perception systems.
To perceive around blind corners, three main paradigms have been explored: vehicle-to-everything (V2X) cooperative communication, acoustic wave sensing, and mmWave radar multipath exploitation. V2X requires extensive and costly roadside infrastructure; acoustic sensing suffers from low signal-to-noise ratio and severe doppler noise in active outdoor traffic; mmWave radar operating at 77 GHz offers robust penetration and precise range measurements capable of capturing indirect echoes bouncing off planar walls. However, conventional radar NLOS localization methods overwhelmingly rely on pre-mapped architectural geometries, orthogonal corner assumptions, or heuristic threshold rules, and are almost exclusively evaluated in stationary setups or controlled indoor chambers.
When mounted on an ego-dynamic vehicle navigating outdoor streets, radar observations undergo severe multi-frame motion distortion, high false-alarm clutter, and Doppler ambiguities, causing direct returns, background multipath, and hidden target returns to interleave chaotically. Moreover, because radar signals bounce multiple times before returning, NLOS pedestrian returns manifest as geometrically distorted virtual reflections in bird's-eye-view (BEV) space rather than physical coordinates. Core idea: couple multimodal data-driven representation learning with classical geometric reflection optics by using visual structural cues to guide point-wise radar reflection-order classification and anisotropic reflective surface heatmap estimation, followed by deterministic physics-guided ray tracing and geometric mirroring to analytically recover the true positions of occluded pedestrians.
Method¶
Overall Architecture¶
The proposed framework comprises two interconnected stages: Reflection-Aware Representation Learning and Physics-Guided Ray Tracing. The system inputs are synchronized front-view RGB images \(I_t\) and motion-compensated 2D mmWave radar point cloud sequences \(\{R_{t-\tau}\}_{\tau=0}^{K-1}\) accumulated over \(K\) frames using wheel odometry. In the representation learning stage, image features extracted via a Lift-Splat-Shoot (LSS) encoder are projected onto the BEV plane as \(F_{\mathrm{cam}}\), while ego-compensated radar points are encoded into \(F_{\mathrm{rad}}\). A cross-attention module queries visual geometry using sparse radar points to produce a cross-modal fused feature representation \(F_{\mathrm{fused}}\). From \(F_{\mathrm{fused}}\), the network jointly predicts point-wise reflection-order classification logits and a BEV reflective surface probability map. In the ray tracing stage, deterministic geometric algorithms shoot rays from the radar origin toward clustered third-order ghost reflections, identify the optimal reflecting surface boundary, and analytically mirror the virtual points across the estimated plane to determine the true coordinates of NLOS pedestrians.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Input: Front-view Image & Motion-Compensated Radar Point Clouds"] --> F1["Spatially Aligned Multimodal BEV Feature Fusion<br/>LSS frustum projection & radar-camera cross-attention"]
F1 --> F2["Reflection-Order-Aware Radar Point Segmentation<br/>Disentangling 1st-order LOS and 3rd-order NLOS echoes"]
F1 --> F3["Anisotropic Gaussian Splatted Surface Estimation<br/>Multi-frame temporal aggregation & continuous surface heatmap"]
F2 --> F4["Geometric Mirroring-Driven Physics Ray Tracing<br/>Ray-guided reflector searching & closed-form reflection recovery"]
F3 --> F4
F4 --> Out["Output: Clustered & refined physical coordinates of NLOS pedestrians"]
Key Designs¶
1. Spatially Aligned Multimodal BEV Feature Fusion: Bridging sensor heterogeneity and ego-motion distortion
Single-frame radar point clouds are sparse and noisy with limited angular resolution, whereas camera frames offer dense visual semantics and boundary layouts but lack metric depth behind obstacles. To establish cross-modal correspondence under ego-motion, past radar sweeps are motion-compensated into current coordinates using wheel odometry. The visual backbone utilizes an LSS formulation to predict per-pixel categorical depth distributions and pool features into a camera BEV tensor \(F_{\mathrm{cam}} \in \mathbb{R}^{C \times H_{\mathrm{bev}} \times W_{\mathrm{bev}}}\), while accumulated radar points are mapped into \(F_{\mathrm{rad}} \in \mathbb{R}^{C \times H_{\mathrm{bev}} \times W_{\mathrm{bev}}}\). To allow sparse radar points to attend to dense environmental structures, cross-attention employs radar features as queries and camera features as keys and values: $\(Q = W_Q F_{\mathrm{rad}}, \quad K = W_K F_{\mathrm{cam}}, \quad V = W_V F_{\mathrm{cam}}\)$ $\(F_{\mathrm{fused}} = \mathrm{Softmax}\left(\frac{Q K^\top}{\sqrt{d}}\right) V\)$ This cross-attention mechanism contextualizes sparse radar returns with visual road and wall boundaries visible at the edge of the driver's perspective, supplying rich structural priors for decomposing subsequent multipath bounces.
2. Reflection-Order-Aware Radar Point Segmentation: Decomposing entangled multipaths into interpretable reflection trajectories
Depending on the propagation path from transmitter to receiver, echoes undergo single or multiple specular bounces. Signals reflecting from wall to pedestrian and returning via wall constitute third-order reflections, which carry NLOS spatial signatures. Instead of heuristic filtering, the system extracts the fused BEV feature at the exact planar location of each radar point \(r_{t,i}\) and feeds it into an MLP segmentation head \(g_\theta\): $\(\mathbf{p}_{t,i} = g_\theta(F_{\mathrm{fused}}(r_{t,i})), \quad \hat{y}_{t,i} = \arg\max_{c \in \mathcal{C}} \mathbf{p}_{t,i}^{(c)}\)$ The label set \(\mathcal{C}\) categorizes returns not merely by bounce order (first-, second-, and third-order) but also by physical origin (object-originated vs. surface-originated). Isolating third-order object-originated points \(\mathcal{P}_t^{(3)}\) from static wall returns and noise allows the pipeline to reliably extract informative NLOS pedestrian returns from raw radar clutter.
3. Anisotropic Gaussian Splatted Surface Estimation: Non-parametric reconstruction of reflective boundaries
Identifying third-order reflection points is insufficient without knowing the exact geometry of the reflector that bounced the electromagnetic wavefront. Rather than assuming static Manhattan layouts, the model directly regresses a continuous BEV surface existence map \(\hat{S}_t \in [0, 1]^{H_{\mathrm{bev}} \times W_{\mathrm{bev}}}\) from \(F_{\mathrm{fused}}\) via a convolutional head \(h_\phi\). During training, pseudo-ground-truth surface maps are generated by aggregating first-order surface points across a temporal window \([-K_h, K_f]\) and applying anisotropic Gaussian splatting: $\(S_t(u, v) = \max_{j} \exp\left(-\frac{1}{2}\left[\frac{d_{\parallel, j}^2}{\sigma_\parallel^2} + \frac{d_{\perp, j}^2}{\sigma_\perp^2}\right]\right)\)$ where \(d_{\parallel, j}\) and \(d_{\perp, j}\) denote tangential and normal distances from grid point \((u, v)\) to the candidate surface point \(r_j\), configured with \(\sigma_\parallel > \sigma_\perp\). This anisotropic formulation ensures smooth, continuous building wall boundaries along their physical length while preventing artificial perpendicular dispersion.
4. Geometric Mirroring-Driven Physics Ray Tracing: Closed-form localization grounded in reflection optics
Rather than relying on uninterpretable neural network coordinate regression, the system applies a deterministic geometric operator \(\mathcal{T}\) grounded in reflection physics. First, DBSCAN clusters the identified third-order points \(\mathcal{P}_t^{(3)}\) into virtual cluster centers \(\bar{r}_m^{(3)}\). Because incoming multipath rays originate from planar walls, the unit directional vector connecting the ego-radar origin \(o\) and the virtual cluster center is formulated as: $\(\mathbf{d}_m = \frac{\bar{r}_m^{(3)}}{\|\bar{r}_m^{(3)}\|_2}\)$ The algorithm casts a ray along \(\mathbf{d}_m\) across the surface heatmap \(\hat{S}_t\) to find the intersection point \(c_m\) and the local surface normal \(n_m\) that maximize reflection probability. Using the planar reflection law, the true NLOS pedestrian position \(\hat{\mathbf{x}}_m^{(3)}\) is analytically reconstructed via geometric mirroring: $\(\hat{\mathbf{x}}_m^{(3)} = \bar{r}_m^{(3)} - 2\left((\bar{r}_m^{(3)} - c_m) \cdot n_m\right) n_m\)$ Combining directly observed first-order clusters with mirrored third-order estimates yields the global pedestrian set \(\hat{X}_t\), effectively converting virtual radar ghosts into verifiable physical targets.
Loss & Training¶
The framework is optimized end-to-end with a composite multi-task objective: $\(\mathcal{L} = \mathcal{L}_{\mathrm{pt}} + \lambda_{\mathrm{surf}} \mathcal{L}_{\mathrm{surf}} + \lambda_{\mathrm{sem}} \mathcal{L}_{\mathrm{sem}}\)$ - Point Segmentation Loss \(\mathcal{L}_{\mathrm{pt}}\): standard point-wise multi-class cross-entropy loss over \(N_t\) points: $\(\mathcal{L}_{\mathrm{pt}} = \frac{1}{N_t}\sum_{i=1}^{N_t} \ell_{\mathrm{CE}}(\mathbf{p}_{t,i}, y_{t,i})\)$ - Surface Estimation Loss \(\mathcal{L}_{\mathrm{surf}}\): binary cross-entropy (BCE) loss evaluated over all BEV grid cells: $\(\mathcal{L}_{\mathrm{surf}} = \ell_{\mathrm{BCE}}(\hat{\mathbf{S}}_t, \mathbf{S}_t)\)$ - Auxiliary Semantic Loss \(\mathcal{L}_{\mathrm{sem}}\): cross-entropy loss on an auxiliary semantic segmentation head applied to the image branch during training. This forces the visual backbone to learn sharp road, wall, and curb geometries without requiring metric depth supervision; the head is pruned at inference time with zero latency overhead.
Key Experimental Results¶
Main Results¶
Experiments were conducted in a real-world outdoor T-junction testbed equipped with a 77 GHz mmWave radar (TI AWR2944EVM), front-view RGB camera, LiDAR, and wheel encoders, benchmarked against top-down panoramic reference cameras. The evaluation incorporates both a blind T-junction blocked by a frontal obstacle wall (B1) and an open intersection (B2) under stationary (ego-static) and moving vehicle conditions (ego-dynamic, driving speeds up to 22.00 km/h, NLOS detection range up to 17.92 m). The evaluation metric is Average Euclidean localization error (AE, in meters).
| Method | Sensor Modality | Ego Static: LOS (m) | Ego Static: NLOS (m) | Ego Dynamic: LOS (m) | Ego Dynamic: NLOS (m) |
|---|---|---|---|---|---|
| Scheiner et al. [27] (CVPR 2020) | Radar + LiDAR | 1.02 | 2.36 | 1.36 | 4.74 |
| Park et al. [24] (IROS 2025) | Radar + Camera | 0.89 | 1.69 | 1.41 | 3.23 |
| Proposed Method | Radar + Camera | 0.44 | 1.01 | 0.54 | 1.23 |
Under ego-static conditions, the proposed method reduces the NLOS localization error from 1.69 m to 1.01 m (a 40.2% improvement). Under challenging ego-dynamic driving conditions where motion clutter degrades prior methods to 3.23 m and 4.74 m, our approach achieves a 1.23 m NLOS error—a performance improvement exceeding 61.9%.
Ablation Study¶
1. Point-wise Reflection Path Classification Benchmark
Assessing the model's capability to isolate third-order NLOS pedestrian echoes from environmental clutter and first-order returns.
| Method | Driving Condition | Accuracy | Macro-F1 | 3rd-bounce Precision | 3rd-bounce Recall | 3rd-bounce F1-score |
|---|---|---|---|---|---|---|
| Kraus et al. [14] (ITSC 2020) | Ego-static | 0.686 | 0.661 | 0.757 | 0.639 | 0.693 |
| Kraus et al. [14] (ITSC 2020) | Ego-dynamic | 0.677 | 0.635 | 0.553 | 0.863 | 0.674 |
| Wu et al. [34] (PTv3, CVPR 2024) | Ego-static | 0.930 | 0.890 | 0.884 | 0.931 | 0.907 |
| Wu et al. [34] (PTv3, CVPR 2024) | Ego-dynamic | 0.610 | 0.481 | 0.250 | 0.501 | 0.334 |
| Proposed Method | Ego-static | 0.946 | 0.915 | 0.897 | 0.935 | 0.916 |
| Proposed Method | Ego-dynamic | 0.899 | 0.814 | 0.744 | 0.718 | 0.731 |
2. Reflective Surface Geometric Accuracy & Runtime Breakdown
Evaluated against RANSAC-fitted LiDAR ground-truth planes via Reflection Point Error (RPE, in meters) and Reflection Angle Error (RAE, in degrees).
| Ego Status | Scenario | RPE (m) | RAE (deg) | Submodule | Latency (ms) | Overall System Metric |
|---|---|---|---|---|---|---|
| Static | B1 (Wall) | 0.280 | 7.175° | Representation Learning | 55.22 ms | Total Latency: 72.02 ms |
| Static | B2 (Open) | 0.415 | 6.698° | Physics Ray Tracing | 16.80 ms | Throughput: 13.85 FPS |
| Dynamic | B1 (Wall) | 0.285 | 7.207° | Model Parameters | 1.17 M | Compute: 55.04 GMACs |
| Dynamic | B2 (Open) | 0.357 | 5.997° | Auxiliary Semantic Head | 0 ms (pruned) | Early Warning: +0.77 s |
Key Findings¶
- Cross-modal fusion is critical for preserving dynamic point classification: While pure 3D point transformer architectures (Point Transformer v3 [34]) perform competitively in static environments (Macro-F1 0.890), their performance collapses under vehicle motion (Macro-F1 plummets to 0.481 and precision to 0.250). In contrast, cross-modal attention anchored by visual spatial structures maintains an F1-score of 0.731 under vehicle motion.
- Surface normal precision governs NLOS localization lower bounds: The paper demonstrates that accurate point classification alone is insufficient; angular misalignments in the reflector plane compound quadratically across mirror projections. Maintaining RPE within 0.28–0.42 m and RAE below 7.3° provides the mathematical foundation for sub-1.5m NLOS tracking.
- Substantial reaction buffer for active vehicle safety: By leveraging third-order multipath reflection paths, the framework detects and localizes hidden pedestrians an average of 0.77 seconds earlier than waiting for direct first-order line-of-sight radar returns, creating a crucial margin for automated emergency braking (AEB).
Highlights & Insights¶
- Decoupled data-driven representation and physical optics: Instead of forcing a neural network to blindly regress target coordinates through occluding walls, the model delegates classification and surface mapping to deep networks while leaving spatial coordinate reconstruction to deterministic geometric optics, preventing black-box hallucinations.
- Transforming radar multipath clutter into an active sensing asset: Conventional radar algorithms filter multipath returns as unwanted ghost artifacts. This work actively mines third-bounce echoes to form spatial signatures of hidden pedestrians, achieving perception beyond physical visibility boundaries.
- Ultra-lightweight edge architecture: With only 1.17M parameters and 55.04 GMACs, the framework executes in 72 ms (13.85 FPS on an RTX 3090 GPU), demonstrating that sophisticated multipath ray tracing can run in real time on automotive-grade computing platforms.
Limitations & Future Work¶
- Planar reflector assumption: The formulation assumes locally planar, specular reflection surfaces such as concrete building facades or acoustic barriers. Non-planar obstacles, irregular vegetation, or rough masonry surfaces that produce diffuse wave scattering can degrade surface normal estimation.
- Extreme velocities and multi-target associations: The testbed validation operates at speeds up to 22 km/h. At higher urban speeds (40–60 km/h), Doppler velocity ambiguities and motion blur intensify; moreover, associating multiple third-order clusters with crossing pedestrians remains prone to identity switches.
- Future directions: Integrating bidirectional reflectance distribution function (BRDF) physics for diffuse scattering and introducing multi-hypothesis Bayesian tracking filters downstream of the ray tracing module.
Related Work & Insights¶
- vs Scheiner et al. [27] (CVPR 2020): Scheiner et al. pioneered around-the-corner Doppler radar tracking using geometric heuristics and LiDAR mapping, but suffered from massive error blowup under ego-motion (4.74 m error). Our framework leverages multimodal cross-attention and continuous surface regression to achieve 1.23 m dynamic accuracy.
- vs Park et al. [24] (IROS 2025): Prior work utilized camera-derived road layouts to constrain radar interpretation via handcrafted rules. This work replaces rigid rule-based parsing with learned reflection-order classification and anisotropic surface modeling, improving dynamic accuracy by over 60%.
- vs Point Transformer v3 [34]: PTv3 is an exceptional point cloud representation model, but radar multipath clutter violates standard continuous 3D geometric assumptions. Cross-modal visual context is essential to disambiguate motion-degraded radar points.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering integration of deep multimodal representation learning with deterministic ray-tracing physics for ego-dynamic NLOS localization]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Full-scale moving vehicle testbed in outdoor T-junctions with top-down panoramic ground-truth, covering extensive static and dynamic comparisons]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical problem formulation bridging electromagnetic wave propagation physics and neural network architecture]
- Value: ⭐⭐⭐⭐⭐ [Provides a practical, real-time, infrastructure-free solution to the critical urban driving safety challenge of occluded darting-out pedestrians]