title: >- [Paper Note] SP-TransientBench: A Real-Captured Single Photon Perception Benchmark description: >- [ECCV 2026][Autonomous Driving][Single-Photon LiDAR] A real-captured multi-task single-photon LiDAR benchmark with full 256x192x672 transient histograms, calibrated poses, and histogram-domain multi-return annotations for depth estimation, 3D reconstruction, and 3D semantic segmentation. tags: - ECCV 2026 - Autonomous Driving - Single-Photon LiDAR - Transient Histograms - 3D Perception date: 2026-09-19 content_hash: a51d4653b16cd91f
SP-TransientBench: A Real-Captured Single Photon Perception Benchmark¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://huggingface.co/datasets/shuinb/SP-TransientBench
Area: Autonomous Driving
Keywords: Single-photon LiDAR, transient histograms, 3D perception, multi-return decomposition, benchmark
TL;DR¶
The first real-captured multi-task single-photon LiDAR (SPL) 3D perception benchmark, providing full \(256 \times 192 \times 672\) transient time-of-flight histograms, calibrated poses, and histogram-domain multi-return semantic annotations across depth estimation, multi-view reconstruction, and 3D semantic segmentation.
Background & Motivation¶
Single-photon LiDAR (SPL) based on single-photon avalanche diode (SPAD) arrays provides extreme single-photon sensitivity and picosecond-scale temporal resolution, offering unique capabilities for low-power active 3D perception, kilometer-scale ranging, and non-line-of-sight imaging. Unlike conventional pulsed LiDAR that records only a few discrete echo returns per pixel, SPL detects and time-stamps individual photon arrivals to construct full time-of-flight transient histograms. This rich representation preserves fine-grained light transport, multi-surface reflection, and semi-transparent penetration cues. However, real-world single-photon measurements suffer from low signal-to-background ratios (SBR) due to ambient solar illumination, coupled with detector dead time, pulse broadening, and multi-path scattering that generate complex multi-peak waveforms. When these raw transients are collapsed into single-depth maps or conventional point clouds, critical temporal cues are permanently discarded.
Constructing a real-world SPL perception benchmark faces two primary bottlenecks. On the hardware front, solid-state SPAD sensors require specialized, custom synchronization and acquisition pipelines that hinder scalable collection and standardized sharing. On the annotation front, existing 3D labeling tools operate on single-surface depth maps or consolidated point clouds, completely failing to handle multi-return transient structures where multiple distinct physical surfaces (e.g., foliage canopy, road, and vehicle) overlap along a single ray. Consequently, existing SPAD datasets are predominantly confined to synthetic simulations with idealized Poisson noise models or small-scale single-view captures, leaving systematic real-world evaluation across geometry and semantics unaddressed.
To bridge this gap, this paper constructs a multi-sensor capture platform and introduces a histogram-domain annotation methodology based on sequential peak peeling, resulting in SP-TransientBench (STB). The core idea is to preserve raw 3D transient histograms with realistic multi-return and ambient noise profiles, apply sequential peak peeling directly in the histogram domain to label multiple disjoint surfaces along each ray, and establish a standardized multi-task SPL benchmark spanning depth estimation, multi-view neural reconstruction, and 3D semantic segmentation.
Method¶
Overall Architecture¶
The SP-TransientBench framework integrates hardware co-acquisition, spatiotemporal geometric calibration, histogram-domain multi-return annotation, and unified multi-task evaluation protocols. The capture rig combines a solid-state SPAD-LiDAR (Adaps ADS6311 Hawk) with an auxiliary conventional LiDAR (Livox Avia) and an onboard IMU. Repeated laser pulses timestamped via time-correlated single-photon counting (TCSPC) form a raw transient tensor of size \(256 \times 192 \times 672\) with a 750 ps bin resolution. Auxiliary point clouds are processed via LiDAR-inertial SLAM to recover drift-reduced trajectories, which are transferred into the SPL coordinate frame via calibrated rigid extrinsics to produce metric poses and depth references. For 3D semantic segmentation, an iterative sequential peak peeling procedure resolves multi-return histograms along each ray into discrete semantic entities.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-Sensor Rig Co-Capture<br/>SPAD-LiDAR + Auxiliary Livox LiDAR"] --> B["Spatiotemporal Calibration & Poses<br/>LiDAR-SLAM Trajectory + Extrinsics"]
B --> C["Histogram Sequential Peak Peeling<br/>Peak Search→FWHM Assignment→Peeling"]
C --> D["Multi-Task Benchmark Protocols<br/>Depth / Neural Reconstruction / 3D Semantics"]
Key Designs¶
1. Spatiotemporal Calibration & Depth Reference Generation: Overcoming Textureless SPAD Localization Limits Raw SPAD histogram frames have limited spatial resolution (\(256 \times 192\)) and severe photon shot noise under daylight, making direct feature tracking and pose estimation unstable. The platform rigidly couples the SPAD-LiDAR with a Livox Avia LiDAR and IMU. Using LiDAR-inertial SLAM, the platform computes smooth, drift-reduced trajectories, which are mapped to the SPL frame via checkerboard extrinsic calibration. The system's instrument response function (IRF) is calibrated using a 99% Enhanced Specular Reflector (ESR) target to isolate pulse broadening and timing jitter. For depth benchmarking, temporally accumulated Livox point clouds are projected onto the SPL camera viewpoint, yielding dense, metrically consistent ground-truth depth maps.
2. Histogram-Domain Sequential Peak Peeling Annotation: Resolving Multi-Return Ambiguity Along Rays In real scenes with semi-transparent objects or depth discontinuities, a single pixel line-of-sight intercepts multiple surfaces, producing multi-peak transients. Conventional single-depth annotations cannot capture these stacked structures. The authors define ground-truth semantic labels as a bin tensor \(S \in \{0, \dots, C\}^{N \times B}\), where \(N = H \times W\) denotes pixels and \(B = 672\) represents time bins (\(\Delta t = 750\text{ ps}\)). The annotation algorithm operates iteratively: (1) detect the dominant return peak at time \(t_1\); (2) assign semantic labels to the peak-support interval defined by the full width at half maximum (FWHM); (3) peel (suppress) the signal around the peak to reveal weaker subsequent returns (\(t_2, t_3\)). Each labeled temporal bin is converted into metric range and back-projected into 3D using calibrated intrinsics, yielding volumetric semantic supervision with multiple classes per ray.
3. Unified Multi-Task Benchmark Tracks: Standardizing Evaluation Across Noise Regimes STB standardizes 168.7 GB of real data (10 depth samples, 9 multi-view reconstruction scenes with 20–40 views each, and 27 semantic sequences containing 10,297 frames) across three evaluation tracks. The depth estimation track benchmarks accuracy using Chamfer Distance (CD) and Recall under strict temporal tolerances (1, 3, and 5 bins). The multi-view reconstruction track supports 3-, 5-, and 10-view training to evaluate novel-view synthesis across intensity (SSIM, LPIPS), depth (L1), and raw transient histograms (PSNR). The 3D semantic segmentation track establishes a decoupled evaluation between histogram denoising preprocessing and point-cloud backbones across 13 indoor and outdoor categories (8,297 train / 2,000 test split).
Key Experimental Results¶
Main Results¶
All benchmarks evaluate models using SPAD-only inputs without RGB guidance or multimodal fusion.
Table 1 reports Task 1 depth estimation performance in Chamfer Distance (CD, m) and Recall (%) across discrete bin tolerances (750 ps per bin), evaluating optimization models (Shin, Rapp), photon-efficient methods (Li), and learning-based SSPINET.
| Method | Type | CD ↓ (m) | Recall@1bin ↑ (%) | Recall@3bins ↑ (%) | Recall@5bins ↑ (%) |
|---|---|---|---|---|---|
| Shin [37] | Physical Optimization | 1.6908 | 25.97 | 74.83 | 89.14 |
| Rapp [30] | Statistical Unmixing | 1.7486 | 29.17 | 68.62 | 83.88 |
| Li [18] | Ultra-Low Flux Ranging | 1.5506 | 34.60 | 77.20 | 88.60 |
| SSPINET [44] | Sparse Learning Prior | 1.4978 | 44.27 | 83.72 | 93.68 |
Table 2 reports Task 2 multi-view 3D reconstruction results under the 10-view training configuration, evaluating novel-view synthesis across intensity (SSIM, LPIPS), depth (L1 error, m), and raw transient histograms (PSNR, dB).
| Method | Intensity SSIM ↑ | Intensity LPIPS ↓ | Depth L1 ↓ (m) | Histogram PSNR ↑ (dB) |
|---|---|---|---|---|
| TransientNeRF [22] | 0.7717 | 0.2315 | 0.7836 | 45.4251 |
| Transientangelo [20] | 0.7597 | 0.2501 | 0.8710 | 41.5735 |
Table 3 details Task 3 3D semantic segmentation on STB across combinations of histogram preprocessing techniques and 3D point-cloud backbones (mean values across 3 random seeds).
| Preprocessing | PointNet++ (OA / mIoU) | PointMLP (OA / mIoU) | Point Transformer (OA / mIoU) | PointNeXt (OA / mIoU) |
|---|---|---|---|---|
| w/o Preprocessing | 89.95 / 49.16 | 89.90 / 49.11 | 90.23 / 49.51 | 88.45 / 41.05 |
| Thresholding | 90.53 / 50.58 | 90.81 / 52.22 | 89.96 / 48.59 | 89.05 / 42.86 |
| Matched Filtering [42] | 90.26 / 49.71 | 89.93 / 50.02 | 90.43 / 50.37 | 87.07 / 40.04 |
| PPC [10] | 90.50 / 50.41 | 90.59 / 51.50 | 90.35 / 48.95 | 88.43 / 41.26 |
| SSPINET [44] | 90.73 / 51.31 | 90.65 / 51.78 | 90.51 / 50.03 | 88.46 / 41.54 |
Ablation Study¶
Table 4 analyzes the simulation-to-real gap using PointNet++: comparing training from scratch purely on real data versus pretraining on synthetic ScanNet SPAD point clouds and fine-tuning across different fractions of real STB data.
| Real Data Ratio | Scratch (OA) | Scratch (mIoU) | Finetune (OA) | Finetune (mIoU) |
|---|---|---|---|---|
| 10% Real Data | \(83.80 \pm 0.79\) | \(33.19 \pm 1.37\) | \(85.58 \pm 0.34\) | \(36.57 \pm 0.61\) |
| 20% Real Data | \(86.16 \pm 0.68\) | \(37.98 \pm 1.09\) | \(87.52 \pm 0.25\) | \(41.39 \pm 1.42\) |
| 50% Real Data | \(89.09 \pm 0.36\) | \(45.94 \pm 1.37\) | \(89.55 \pm 0.56\) | \(47.98 \pm 0.93\) |
| 100% Real Data | \(91.02 \pm 0.11\) | \(52.24 \pm 0.36\) | \(90.61 \pm 0.81\) | \(51.20 \pm 1.41\) |
Key Findings¶
- Stricter Temporal Tolerances Reveal Reconstruction Gaps: Under a relaxed tolerance of 5 bins (~3.75 ns), all depth estimation baselines achieve 83%–93% Recall. However, at a 1-bin tolerance (750 ps), physical and statistical optimizers degrade significantly (Shin achieves 25.97%, Rapp 29.17%), whereas SSPINET maintains 44.27%, demonstrating the importance of learned priors in mitigating real sensor timing jitter and asymmetric peak broadening.
- Preprocessing Outweighs Backbone Selection in Point Cloud Semantics: Direct projection of raw, noisy histograms degrades point-cloud geometry, dropping PointNeXt mIoU to 41.05%. Simple thresholding or matched filtering suppresses background clutter and elevates PointMLP mIoU to 52.22%, proving that histogram-domain denoising is critical before 3D spatial reasoning.
- Diminishing Returns of Synthetic Pretraining: Simulation pretraining provides a significant boost (+3.38% mIoU) in low-data regimes (10% real data). When 100% real data is available, training from scratch slightly outperforms fine-tuning (52.24% vs. 51.20%), confirming that synthetic Poisson noise models fail to fully capture the complex multi-path and detector artifacts present in real SPL systems.
Highlights & Insights¶
- Full Waveform Release: Unlike prior datasets that discard raw transients to publish single-return depth or point clouds, STB provides full \(256 \times 192 \times 672\) photon histograms along with physical lux measurements, providing an authentic benchmark for waveform-level multi-path modeling.
- Peak Peeling Annotation Protocol: The sequential peak peeling tool provides a systematic approach for labeling multi-surface returns along individual rays, establishing an extensible methodology for non-line-of-sight and penetration imaging annotation.
Limitations & Future Work¶
- Reliance on External LiDAR SLAM Reference: Poses and reference depth rely on Livox Avia point clouds and LiDAR-inertial odometry, which introduces subtle SLAM drift and cross-sensor registration tolerances rather than absolute optical ground truth.
- Single Sensor Architecture: All real sequences are gathered using an Adaps ADS6311 Hawk solid-state SPAD device. Evaluating cross-sensor generalization across different SPAD architectures, laser wavelengths, and adverse weather conditions (fog, rain, direct sunlight glare) remains an open challenge.
Related Work & Insights¶
- vs. TransientNeRF & SPADNet: TransientNeRF contains only 20 real captures and lacks semantic labels; SPADNet relies purely on synthetic data for single-view depth. STB is the first large-scale real-captured benchmark providing calibrated poses, full waveforms, and dense 3D semantic annotations.
- vs. Conventional LiDAR Benchmarks (e.g., nuScenes, Waymo): Traditional automotive LiDARs discard the temporal photon arrival curve and output only discrete \((x, y, z, I)\) returns. STB introduces the high-dimensional transient axis to study single-photon perception in occluded, multi-layered, and photon-starved scenarios.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ (First real-captured SPL benchmark with full transient waveforms and multi-return semantic bin annotations)
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Systematic evaluation across depth estimation, novel-view neural rendering, and 3D semantics, backed by sim-to-real analysis)
- Writing Quality: ⭐⭐⭐⭐⭐ (Clear formulation of transient physics, rigorous calibration protocols, and structured experimental tables)
- Value: ⭐⭐⭐⭐⭐ (Fills a major gap in real-world benchmarking for single-photon LiDAR and active transient imaging)