EventSpecPS: Photometric Stereo with Multispectral Reflectance Using an Event Camera¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/Wujingqian/EventSpecPS
Area: 3D Vision
Keywords: Photometric Stereo / Event Camera / Multispectral Reflectance Estimation / Surface Normal / Computational Imaging
TL;DR¶
EventSpecPS introduces the first single-stage event-based multispectral photometric stereo system, leveraging spectrally-angularly multiplexed illumination and a constant-light bias to decouple geometry from spectral reflectance and resolve scale ambiguity, achieving low-bandwidth, robust joint recovery of surface normals and absolute multispectral reflectance.
Background & Motivation¶
Recovering both 3D surface geometry and multispectral reflectance is a foundational capability for physical scene understanding, playing indispensable roles in industrial quality inspection, cultural heritage preservation, and remote sensing. Surface normals provide fine-grained descriptions of surface geometry, whereas multispectral reflectance characterizes intrinsic, wavelength-dependent material properties. Conventional approaches typically rely on multi-view hyperspectral imaging or multispectral photometric stereo (PS) under repeated, sequential illuminations to untangle geometry from spectral signatures. However, frame-based multispectral systems demand dense exposure sequences and complex narrowband filtering, resulting in slow capture cycles and massive data bandwidth. Furthermore, traditional PS formulations fundamentally depend on the Lambertian assumption, rendering them fragile to specular highlights and attached shadows, which frequently necessitates ad-hoc outlier rejection schemes or additional polarization optics.
Event cameras, which asynchronously record per-pixel logarithmic intensity variations with microsecond-level temporal resolution, high dynamic range, and minimal bandwidth consumption, offer a transformative sensing paradigm for high-speed computational photography. Although pioneering studies have explored event-based photometric stereo (e.g., EventPS) and event-based spectral imaging (e.g., EventSpec) individually, combining them into a naive two-stage pipeline demands disparate illumination protocols and calibration procedures across stages. Such multi-stage pipelines substantially elevate hardware and algorithmic complexity, while also leaving the recovered spectral reflectance subject to an unknown global scale factor because event sensors only capture relative log-intensity transitions. Crucially, surface normals and multispectral reflectance are multiplicatively intertwined in the event domain, rendering naive joint inversion fundamentally ill-posed without dedicated illumination constraints.
This paper addresses this challenge from physical co-design, establishing the theoretical conditions required to completely decouple geometry from multispectral reflectance. Core idea: by deploying a Spectrally-Angularly Multiplexed Encoding Device that cancels reflectance via multi-angle observations within identical spectral bands and resolves scale ambiguity via an additive constant-light offset, an Event-based Spectral-Angular Decoder achieves single-stage, low-bandwidth, shadow- and specularity-robust joint recovery of surface normals and absolute multispectral reflectance.
Method¶
Overall Architecture¶
The EventSpecPS framework unites physical multiplexed illumination encoding with an inverse neural decoding formulation. The hardware apparatus comprises a Prophesee EVK4 event sensor, a 24-LED multispectral light ring, a calibrated constant white-light source co-axially coupled into the optical path via a 50/50 beam splitter, and an Arduino-based microcontroller. The illumination schedule rapidly cycles through individual LEDs at microsecond intervals, triggering asynchronous positive and negative event streams on the event camera. The backend decoder receives this stream along with the known illumination schedule, parameterizes spectral reflectance using a compact low-rank basis representation, applies angular gating to reject specular outliers, and iteratively reconstructs dense surface normals and absolute per-band reflectance.
The end-to-end data processing workflow is depicted below:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multispectral-Angular Illumination + Constant Light<br/>24-LED ring cycle + 50/50 beam splitter bias"] --> B["Event Camera Asynchronous Sensing<br/>Log-intensity changes trigger event stream"]
B --> C["Spectral-Angular Multiplexed Geometric Decoupling<br/>Same-band multi-angle log-ratio cancels reflectance"]
C --> D["Scale Ambiguity Resolution via Constant-Light<br/>Additive offset analytically yields absolute reflectance"]
D --> E["Event-based Spectral-Angular Decoder Optimization<br/>Low-rank reflectance + specularity gating + spatial regularizer"]
E --> F["Output: Dense 3D Surface Normal Map<br/>& 8-Channel Absolute Multispectral Reflectance"]
Key Designs¶
1. Spectral-Angular Multiplexed Geometric Decoupling: Canceling Reflectance across Angular Repeats In the event sensing domain, the instantaneous radiance equation is formulated as \(I(\mathbf{x}, t) = \max(0, \mathbf{n}_\mathbf{x}^\top \mathbf{l}(t)) \sum_{i=1}^N D_i R_{\mathbf{x}, i} S_i(t)\), where normal \(\mathbf{n}_\mathbf{x}\) and spectral reflectance \(R_{\mathbf{x}, i}\) are multiplicatively coupled. If each spectral channel were activated under only a single illumination direction, infinite degenerate solutions would exist. To decouple them, the authors distribute 8 spectral bands (\(N=8\)) across a circular ring of 24 LEDs, repeating each channel across three angular positions separated by 120° azimuth. When observing the same spectral band \(\lambda_i\) under different lighting vectors \(\mathbf{l}_i^{(1)}\) and \(\mathbf{l}_i^{(2)}\), the sensor spectral sensitivity and per-pixel reflectance terms cancel out entirely in the accumulated event ratio: $\(\rho = \frac{\mathbf{n}_\mathbf{x}^\top \mathbf{l}_i^{(1)}}{\mathbf{n}_\mathbf{x}^\top \mathbf{l}_i^{(2)}}\)$ This ratio establishes a reflectance-invariant Null Space Vector (NSV). With two independent direction pairs per band, the intersection of the two constraint planes uniquely locks the normal vector up to a sign ambiguity. Furthermore, because three repeats are spaced 120° apart and attached shadows (\(\mathbf{n}^\top \mathbf{l} \le 0\)) span at most a hemisphere (\(<180^\circ\)), at least one observation per band is guaranteed to remain shadow-free even under worst-case geometries. The ring elevation angle is also bounded to ensure specular highlights form localized non-overlapping regions, physically ensuring uncorrupted spectral availability.
2. Scale Ambiguity Resolution via Constant-Light Augmentation: Analytical Absolute Reflectance Because event sensors measure only differential log-intensity variations, existing event-based photometric techniques recover reflectance only up to an arbitrary positive scalar. Rather than adding auxiliary intensity sensors, EventSpecPS incorporates a uniform constant white-light source of calibrated radiance \(C_w\) co-aligned with the camera optical axis. Under combined directional and constant illumination, the measured radiance becomes \(I(\mathbf{l}) = \alpha_\mathbf{x} s(\mathbf{l}) + C_w\), where \(s(\mathbf{l}) = \max(0, \mathbf{n}_\mathbf{x}^\top \mathbf{l})\) represents geometric shading and \(\alpha_\mathbf{x}\) denotes diffuse albedo. Switching between two known illumination directions \(\mathbf{l}_1\) and \(\mathbf{l}_2\) produces an event-derived ratio: $\(\rho = \frac{I(\mathbf{l}_1)}{I(\mathbf{l}_2)} = \frac{\alpha_\mathbf{x} s_1 + C_w}{\alpha_\mathbf{x} s_2 + C_w}\)$ The additive constant term \(C_w\) breaks the scale invariance inherent to pure ratio measurements, yielding an exact analytical closed-form solution for absolute albedo: $\(\alpha_\mathbf{x} = \frac{C_w (\rho - 1)}{s_1 - \rho s_2}\)$ This mathematical guarantee enables direct identification of absolute physical reflectance without requiring absolute photometric sensors or iterative scale calibration.
3. Event-based Spectral-Angular Decoder: Low-Rank Prior and Specularity Gating The inverse decoder represents per-pixel multispectral reflectance through a rank-\(K\) linear basis decomposition \(R_{\mathbf{x}, i} = \sum_{k=1}^K a_{\mathbf{x}, k} b_{k, i}\) with \(K=8\), enforcing spectral smoothness while reducing optimization dimensionality. To prevent specular events from distorting normal estimation, the decoder dynamically computes an angular-gating weight based on the intermediate normal estimate and active light vectors, selectively downweighting measurements located near mirror reflection configurations. The system optimizes the variables \(\mathbf{n}\), \(\{a_{\mathbf{x}, k}\}\), and \(\{b_{k, i}\}\) using AdamW over 1,600 iterations, enforcing physical unit-norm (\(\|\mathbf{n}\|_2=1\)) and forward-facing (\(n_z > 0\)) constraints on normals, non-negativity on reflectance, and a Gaussian-blur consistency regularization (kernel size 21, \(\sigma=5\)) to suppress high-frequency noise.
Key Experimental Results¶
Main Results¶
The method was extensively evaluated on synthetic benchmarks using the Blobby dataset and Sculptures dataset across three levels of spectral complexity (Texture Bases = 1, 3, 8). Baselines include the two-stage event pipeline (EventPS + EventSpec) and leading frame-based multispectral PS methods (Heng et al. and Fyffe et al.). Evaluation metrics comprise normal Mean Angular Error (MAE in degrees), reflectance Spectral Angle Mapper (SAM in degrees), Root Mean Square Error (RMSE), and average per-scene input data volume in MiB.
| Dataset | Texture Bases | Method | Normal MAE (°) ↓ | Reflectance SAM (°) ↓ | Reflectance RMSE ↓ | Data Volume (MiB) ↓ |
|---|---|---|---|---|---|---|
| Blobs | Bases=1 | Event-based (EventPS + EventSpec) | 9.61 | 9.10 | 0.0302 | 13.31 |
| Blobs | Bases=1 | Frame-based (Heng et al.) | 16.01 | 7.58 | 0.0402 | 16.00 |
| Blobs | Bases=1 | Frame-based (Fyffe et al.) | 20.62 | 13.49 | 0.0591 | 16.00 |
| Blobs | Bases=1 | EventSpecPS (Ours) | 5.89 | 2.65 | 0.0164 | 12.66 |
| Sculptures | Bases=1 | Event-based (EventPS + EventSpec) | 18.39 | 10.55 | 0.0383 | 7.53 |
| Sculptures | Bases=1 | Frame-based (Heng et al.) | 21.96 | 15.28 | 0.0755 | 16.00 |
| Sculptures | Bases=1 | Frame-based (Fyffe et al.) | 27.41 | 21.07 | 0.1056 | 16.00 |
| Sculptures | Bases=1 | EventSpecPS (Ours) | 12.22 | 3.11 | 0.0171 | 7.26 |
| Blobs | Bases=3 | Event-based (EventPS + EventSpec) | 9.62 | 10.52 | 0.0633 | 13.31 |
| Blobs | Bases=3 | Frame-based (Heng et al.) | 22.65 | 19.71 | 0.0886 | 16.00 |
| Blobs | Bases=3 | Frame-based (Fyffe et al.) | 27.58 | 25.21 | 0.1058 | 16.00 |
| Blobs | Bases=3 | EventSpecPS (Ours) | 8.16 | 5.78 | 0.0394 | 12.80 |
| Sculptures | Bases=3 | Event-based (EventPS + EventSpec) | 18.38 | 10.13 | 0.0576 | 7.05 |
| Sculptures | Bases=3 | Frame-based (Heng et al.) | 28.44 | 26.50 | 0.1225 | 16.00 |
| Sculptures | Bases=3 | Frame-based (Fyffe et al.) | 33.06 | 32.53 | 0.1428 | 16.00 |
| Sculptures | Bases=3 | EventSpecPS (Ours) | 13.50 | 7.42 | 0.0452 | 7.16 |
| Blobs | Bases=8 | Event-based (EventPS + EventSpec) | 9.61 | 10.36 | 0.0663 | 13.31 |
| Blobs | Bases=8 | Frame-based (Heng et al.) | 24.07 | 22.05 | 0.1017 | 16.00 |
| Blobs | Bases=8 | Frame-based (Fyffe et al.) | 28.73 | 27.28 | 0.1182 | 16.00 |
| Blobs | Bases=8 | EventSpecPS (Ours) | 9.38 | 7.41 | 0.0471 | 13.17 |
| Sculptures | Bases=8 | Event-based (EventPS + EventSpec) | 18.41 | 10.61 | 0.0633 | 6.93 |
| Sculptures | Bases=8 | Frame-based (Heng et al.) | 29.25 | 27.99 | 0.1289 | 16.00 |
| Sculptures | Bases=8 | Frame-based (Fyffe et al.) | 34.02 | 34.14 | 0.1527 | 16.00 |
| Sculptures | Bases=8 | EventSpecPS (Ours) | 14.60 | 9.09 | 0.0541 | 7.00 |
Real-World Evaluation¶
The prototype setup (Prophesee EVK4 sensor, 24-LED multispectral ring, 50/50 beam splitter with white screen illumination) was validated across diverse physical specimens: 1. Fabricated CAD Objects for Normal Accuracy: - Ball: surface normal MAE achieved 14.07°. - Lion sculpture: surface normal MAE achieved 17.18°. 2. X-Rite ColorChecker Reflectance Accuracy (calibrated against an Optosky ATP2000H spectrometer): - Representative Patch 1: RMSE = 0.0623, SAM = 10.1°. - Representative Patch 2: RMSE = 0.1086, SAM = 17.6°. - Representative Patch 3: RMSE = 0.1119, SAM = 18.2°. - Representative Patch 4: RMSE = 0.1273, SAM = 20.7°. - Representative Patch 5: RMSE = 0.1310, SAM = 21.3°. - Representative Patch 6: RMSE = 0.1603, SAM = 26.1°. 3. Colorful Handwritten Document: Reconstruction across all 8 individual spectral bands clearly captured wavelength-dependent ink absorption profiles; a 3-channel composite image (channels 0/5/3) aligned closely with standard smartphone photographs.
Key Findings¶
- Robustness Across Complex Spectral Dimensions: Frame-based solvers assume low-rank spectral models; when texture bases increase from 1 to 8 on Blobs, Heng et al.'s normal MAE degraded from 16.01° to 24.07° and reflectance RMSE spiked from 0.0402 to 0.1017. In contrast, EventSpecPS maintained stable normal accuracy (9.38°) and superior reflectance RMSE (0.0471), underscoring the efficacy of exact spectral-angular decoupling.
- Lower Bandwidth and Eliminated Error Accumulation: The two-stage event baseline accumulates sequential reconstruction errors between stages and consumes 13.31 MiB. EventSpecPS operates as an integrated single-stage system requiring only 12.66~13.17 MiB, while cutting reflectance SAM from 9.10° down to 2.65° and halving RMSE (0.0164 vs 0.0302 on Blobs, Bases=1).
Highlights & Insights¶
- Reflectance-Invariant Geometric Constraints: Arranging 8 spectral bands across 24 LEDs with 120° azimuthal offsets ensures that event log-ratios between matching bands completely eliminate material reflectance, converting an ill-posed bilinear problem into a well-posed null-space intersection.
- Closed-Form Absolute Reflectance via Constant Light: By augmenting the event camera's differential measurement with a calibrated co-axial constant light, the formulation breaks multiplicative scale invariance and analytically derives absolute physical albedo without requiring external photometers.
- Guaranteed Physical Feasibility Under Shadows and Specularity: Theoretical analysis proves that 3 azimuthal repeats combined with bounded elevation angles guarantee that localized specular highlights and attached shadows cannot simultaneously occlude all three directional observations, ensuring full spectral availability for any surface orientation.
Limitations & Future Work¶
- Near-Lambertian Assumption: Although adaptive angular gating rejects prominent specular highlights, complex non-Lambertian phenomena such as strong inter-reflections, subsurface scattering, and anisotropic specular lobes remain unmodeled.
- Static Scene Constraint: The cyclic LED firing schedule requires the target object to remain stationary throughout the illumination cycle; unmodeled rigid motion would generate confounding event streams that corrupt photometric variations.
- Future Directions: Integrating spiking neural networks or event-based implicit neural representations could enable joint compensation for object motion, expanding the system toward dynamic, in-line industrial inspection.
Related Work & Insights¶
- vs Frame-based Multispectral PS (Fyffe et al. / Heng et al.): Frame-based methods require heavy data acquisition (~16 MiB) and long exposure sequences, degrading severely under higher spectral dimensionality and shadow/specular artifacts. EventSpecPS achieves single-stage high-speed capture with low bandwidth (~7-13 MiB) and substantially superior normal/spectral precision.
- vs Two-stage Event PS + Spec (EventPS + EventSpec): Prior event workflows decouple geometry and spectroscopy into disjoint systems with mismatched lighting protocols, resulting in operational complexity, error propagation, and scale ambiguity. EventSpecPS delivers an end-to-end single-stage solution that analytically identifies absolute multispectral reflectance.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Introduces the first single-stage event-based multispectral PS system, theoretically proving geometric decoupling under spectral-angular multiplexing and deriving a closed-form scale resolution via constant light.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous synthetic evaluations across varying texture complexities, alongside comprehensive physical prototype validation on CAD benchmarks and spectrometer-ground-truthed color charts.
- Writing Quality: ⭐⭐⭐⭐⭐ Lucid mathematical formulations, intuitive physical motivation, and elegant illustrations.
- Value: ⭐⭐⭐⭐⭐ Establishes a practical, high-speed, and bandwidth-efficient paradigm for advanced computational photography, industrial inspection, and cultural heritage analysis.