Spatiotemporal Flux Probing for Single-Photon Videography¶

Conference: ECCV 2026
Paper: ECCV Original
Code: https://jerukan.github.io/spt-flux-probing
Area: Image Restoration
Keywords: single-photon imaging / high-speed videography / spatiotemporal flux probing / velocity-selective imaging / Fourier analysis

TL;DR¶

This paper introduces a spatiotemporal flux probing theory that discards the conventional reliance on local space-time windows or isolated pixels, estimating the continuous intensity spectrum directly from extremely sparse asynchronous photon streams via global 3D Fourier probing functions and constant false alarm rate (CFAR) detection, enabling joint recovery of ultrafast scene motion and high-frequency illumination flicker at 10× to 30× lower photon budgets along with velocity-selective video refocusing.

Background & Motivation¶

Single-photon avalanche diode (SPAD) arrays and detectors record individual photon arrivals with picosecond timestamp precision or ultra-high-rate binary detection frames, opening new frontiers for capturing dynamic scenes beyond the exposure and motion blur limits of standard sensors. However, in high-speed videography (such as 100 kfps) and ultra-low-light environments, exposure intervals shrink drastically, rendering photon detections exceedingly sparse in both space and time—often averaging only 0.001 to 0.059 photons per pixel. Under such extreme physical sparsity, recovering a continuously and rapidly varying incident light intensity (the spatiotemporal flux $\phi(x, y, t)$) from asynchronous, discrete, and stochastic photon arrivals remains an open challenge in computational photography.

Existing methods for single-photon video reconstruction fundamentally depend on local spatiotemporal grouping. Spatiotemporal pooling schemes such as Quanta Burst Photography (QBP) and deep denoising frameworks such as bit2bit aggregate photons within local spatial neighborhoods or along estimated motion trajectories. When photon counts are drastically depleted and the scene undergoes rapid illumination modulation (such as 120 Hz AC lighting or tens of kHz modulated pulsed LEDs), local spatiotemporal windows fail to capture enough detections to build reliable statistical estimates, causing denoising priors to smooth temporal modulations into unnatural spatial blur. Conversely, per-pixel temporal ultra-wideband (UWB) probing can recover megahertz-range temporal modulation by estimating Fourier coefficients from per-pixel timestamps, but it ignores spatial correlations entirely; as a moving object crosses a pixel, its temporal flux resembles an ultra-wideband rect pulse that pixel-wise models fail to resolve under severe photon starvation. Both paradigms fail when fast motion and rapid illumination modulation coincide.

The crucial insight of this work is that even under extreme photon sparsity, the physical information governing both macro scene motion and temporal illumination dynamics is fully retained in correlations across the global spatiotemporal pattern of photon arrivals. Core idea: extend flux probing to the full 3D spatiotemporal volume, formulate discrete photon phasor projection onto orthogonal 3D Fourier bases under an inhomogeneous Poisson martingale noise model, isolate statistically significant spectral components via constant false alarm rate (CFAR) hypothesis testing, and exploit planar spectral energy concentration to achieve unsupervised velocity-selective video refocusing.

Method¶

Overall Architecture¶

The arrival of photons on the sensor plane is modeled as a 3D inhomogeneous Poisson point process governed by a rate function representing the spatiotemporal flux $\phi(x, y, t)$ over the observation volume $\Omega = [0, w] \times [0, h] \times [0, t_{\text{exp}}]$. The sensor records a stream of discrete detection events $\mathcal{P} = \{(x_i, y_i, t_i)\}_{i=1}^N$. The framework introduces deterministic continuous spatiotemporal probing functions $p(x, y, t)$ and unbiasedly estimates the inner product $\langle p, \phi \rangle$ directly from discrete photon events via phasor summation, driving subsequent spectrum thresholding, flux inversion, and motion-selective refocusing.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Photon Event Stream<br/>SPAD / Spike / Event Camera"] --> B["3D Fourier Basis Probing<br/>Discrete photon phasor projection"]
    B --> C["CFAR Frequency Detection<br/>Chi-square test filtering Poisson shot noise"]
    C --> D["Inverse Fourier Reconstruction<br/>Continuous spatiotemporal flux synthesis"]
    C --> E["Velocity Plane Detection & Focusing<br/>Spectral plane scanning & shear mapping"]
    E --> F["Output Velocity-Selective Video<br/>Target motion focused / other velocities blurred"]
    D --> G["Output Multimodal Reconstructions<br/>100 kfps video & high-frequency flicker"]

Key Designs¶

1. 3D Fourier Basis Probing and Martingale Noise Modeling: Unbiased Mapping from Sparse Photons to Continuous Spectra

Instead of counting photons in localized boxes, the approach defines probing measurements of discrete events $\mathcal{P}$ with respect to a continuous probing function $p(\mathbf{x})$ as $E = \sum_{\mathbf{x}_i \in \mathcal{P}} p(\mathbf{x}_i)$. From point process theory, this summation decomposes exactly into a deterministic flux probing integral and a zero-mean continuous stochastic martingale noise process $M_p(t_{\text{exp}})$ induced by photon shot noise: $$E = \int_{0}^{t_{\text{exp}}} \int_0^h \int_0^w p(x, y, t) \phi(x, y, t) \, dx \, dy \, dt + M_p(t_{\text{exp}})$$ Adopting normalized 3D complex exponential Fourier basis functions $p_{\mathbf{f}}(\mathbf{x}) = \frac{1}{\sqrt{v}} e^{-j 2\pi \mathbf{f}^\top \mathbf{x}}$ over the full volume $\Omega$ (with volume $v = wh t_{\text{exp}}$), the probing measurement $E_{\mathbf{f}} = \frac{1}{\sqrt{v}} \sum_{\mathbf{x}_i \in \mathcal{P}} e^{-j 2\pi \mathbf{f}^\top \mathbf{x}_i}$ acts as a global coherent phasor summation over all detected photons. When frequency $\mathbf{f}$ is physically present in $\phi$, phasors add coherently and yield a large magnitude $|E_{\mathbf{f}}|$; when absent, uncorrelated photon arrivals destructively interfere toward zero. Every captured photon across the sensor plane contributes globally to each spectral coefficient, maintaining asymptotic complex normality even under severe low-photon regimes down to tens of photons.

2. Constant False Alarm Rate (CFAR) Frequency Detection: Statistically Principled Noise Thresholding

Because photon arrivals are governed by Poisson statistics, $|E_{\mathbf{f}}|$ exhibits stochastic fluctuations even when no underlying signal exists at frequency $\mathbf{f}$. To distinguish true scene modulations from random shot noise without heuristic tuning, the paper characterizes the null hypothesis distribution ($\mu_{\mathbf{f}} = 0$). Under the null hypothesis, the real and imaginary components of $E_{\mathbf{f}}$ are independent zero-mean Gaussians, which makes the normalized probing energy follow a central chi-square distribution with two degrees of freedom $\chi_2^2$. This yields an analytical constant false alarm rate (CFAR) detector with threshold: $$c_\alpha = \text{CDF}_{\chi_2^2}^{-1}(1 - \alpha) \cdot \frac{|\mathcal{P}|}{2 v}$$ where $\alpha$ is the prescribed false alarm probability and $|\mathcal{P}|$ is the total photon count. A frequency $\mathbf{f}$ is declared detected and added to $\mathcal{F}_{\text{det}}$ only if $|E_{\mathbf{f}}|^2 \ge c_\alpha$. Inverse discrete Fourier series evaluation $\hat{\phi}(\mathbf{x}) = \frac{1}{\sqrt{v}} \sum_{\mathbf{f} \in \mathcal{F}_{\text{det}}} E_{\mathbf{f}} e^{j 2\pi \mathbf{f}^\top \mathbf{x}}$ over detected frequencies yields a continuous reconstruction of the flux while suppressing background noise.

3. Spectral Plane Consistency and Velocity-Selective Videography: Unsupervised Motion Decoupling and Refocusing

For an object translating linearly with constant 2D velocity $\mathbf{v} = [v_x, v_y]^\top$ such that $\phi(x, y, t) = I(x - v_x t, y - v_y t)$, Fourier optics dictates that its spatiotemporal spectral energy is confined strictly to a 2D plane passing through the origin in 3D frequency space: $$v_x f_x + v_y f_y + f_t = 0$$ The normal vector of this plane directly encodes the physical velocity. The paper formulates motion extraction as an unsupervised plane-consistency search across discrete velocity hypotheses $\{\mathbf{v}_i\}_{i=1}^M$. The cumulative energy within an $\epsilon$-tolerance slab of each candidate plane is computed as $E_i = \sum_{|\mathbf{v}_i'^\top \mathbf{f}| \le \epsilon} |E_{\mathbf{f}}|^2$. Because adjacent velocity planes overlap in frequency space, producing correlated scores, a nonparametric rank-CFAR test evaluates the relative ranking of $E_i$ within its local hypothesis neighborhood $\mathcal{N}_i$, robustly detecting true object velocities $\mathbf{v}_{\text{det}}$ at significance level $\alpha_{\text{vel}}$. Once detected, applying a frequency coordinate shear $f_t \leftarrow f_t + \mathbf{v}_{\text{det}}^\top \mathbf{f}_{x,y}$ rotates the corresponding plane onto the temporal DC axis. Performing inverse Fourier synthesis in a sliding temporal window reconstructs a video where objects at $\mathbf{v}_{\text{det}}$ appear static and sharp while background and other moving objects are blurred out.

4. Sensor-Agnostic Extension to Asynchronous Modalities: Generalizing to Spike and Event Cameras

The probing theory naturally accommodates other asynchronous sensors by accounting for their forward models. For spike cameras, which fire a spike when integrated flux crosses a preset threshold, the instantaneous spike rate is proportional to $\phi(x, y, t)$, so spike phasor summation directly provides scaled Fourier coefficients of the flux (paired with low-pass filtering to suppress periodic train harmonics). For event cameras, which trigger events with polarity $\sigma_i = \pm 1$ when temporal logarithmic intensity changes exceed a threshold $C$, probing the event stream computes the Fourier coefficients of $\partial_t \log \phi(x, y, t)$: $$E_{\mathbf{f}}^{\text{event}} = \sum_{(\mathbf{x}_i, t_i) \in \mathcal{P}} \sigma_i p_{\mathbf{f}}(\mathbf{x}_i, t_i)$$ Dividing by $j 2\pi f_t$ performs exact frequency-domain integration, which is fused with low-frequency intensity frames using a Kalman filter to eliminate log-intensity drift and recover absolute radiance.

A Worked Example¶

Consider capturing high-speed projectiles rupturing a balloon using a SPAD512 camera under ambient 120 Hz ceiling light and a 31 kHz modulated LED: 1. Phasor Accumulation: Binary frames arrive at 100 kHz with an extreme sparsity of 0.059 photons per pixel. Discrete events are projected onto candidate 3D Fourier frequencies across the observation volume, computing complex sums $E_{\mathbf{f}}$. 2. CFAR Thresholding: At temporal slices corresponding to 120 Hz and 31 kHz, $|E_{\mathbf{f}}|^2$ sharply exceeds $c_\alpha$, confirming the dual illumination flicker. Simultaneously, frequencies along the slanted velocity plane of the speeding bullet pass the CFAR test, while millions of empty noise bins are set to zero. 3. Velocity Detection & Refocusing: The rank-CFAR test identifies the dominant linear velocity of the bullet. The 3D spectrum is sheared along that velocity vector, mapping bullet dynamics to temporal DC. Sliding-window inverse Fourier transformation produces a 100 kfps video where the projectile trajectory remains crisp and motion-focused while shattered balloon fragments are smoothly motion-blurred.

Key Experimental Results¶

Main Results¶

Experiments were conducted on real physical data captured across four sensor hardware setups: a 100 kHz synchronous SPAD512 array, a single-pixel SPAD with 68 ps time-tagging resolution, a DAVIS240C event camera, and a high-speed spike camera. Comparisons include Quanta Burst Photography (QBP), self-supervised denoising (bit2bit), 1D temporal probing (UWB), and neuromorphic learning methods (EvINR, STIR).

Dataset / Sensor	Conditions / Light Level	Baseline Methods	Ours	Baseline Performance	Key Comparative Conclusions
Foam bullet & balloon (SPAD512)	100 kfps, 105~524 photons/frame (0.001~0.059 ppp)	QBP / bit2bit	Simultaneously recovers 100 kfps motion and 120 Hz / 31 kHz flicker	bit2bit oversmooths flicker; QBP shows heavy noise	Only the proposed method cleanly isolates fast nonperiodic events alongside periodic lighting
Rotating fan blades (SPAD512)	15 Hz rotation, 0.009 ppp extreme low light	UWB (Wei et al.)	Sharp blade edges and specular highlights recovered	UWB suffers catastrophic breakdown, severe aliasing	Exploiting spatial correlations enables reliable imaging with 10× fewer photons than UWB
Guitar string vibration (SPAD512)	0.011 ppp extreme sparsity	QBP (Ma et al.)	Preserves thin guitar strings and subtle motion trajectories	QBP exhibits severe motion blur and blocky noise	Local align-and-merge fails when window photon counts are insufficient for alignment
Rotating fan & laser pulse (Single-pixel SPAD)	6 GHz ultra-wideband / 100 kHz modulation	UWB (Wei et al.)	Resolves both 6 GHz laser wavefront and spinning fan blades	UWB blurs fan structure at low photon flux	Global spatiotemporal correlations match reconstruction quality at 30× lower photon counts
DAVIS240C event stream	1 kfps dynamic scenes	EvINR (Wang et al. ECCV 2024)	High spatial sharpness, temporally consistent, no drift	EvINR requires neural field training and tends to smooth	Competitive with state-of-the-art learning methods without requiring neural network training
High-speed spike camera	10 kfps fast car / balloon pop	STIR (Fan et al. NeurIPS 2024)	Crisp motion boundaries, periodic spike train artifacts suppressed	STIR achieves high quality via complex spatiotemporal learning	Direct orthogonal Fourier probing matches deep learning reconstruction fidelity

Ablation Study¶

The contribution of core framework components was analyzed across real captures and synthetic benchmarks under varying illumination and motion conditions:

Config	Target Phenomenon / Setting	Observed Performance & Metrics	Mechanism & Ablation Insight
Full model (3D Fourier Probing + CFAR)	Extremely low light (0.003 ppp), multi-frequency flicker	Accurately isolates 120 Hz and 31 kHz peaks; clean video	Global spatiotemporal coherent integration yields maximal effective SNR
w/o spatial probing (1D temporal only, UWB)	Rotating fan and translating objects	Motion structure lost; severe aliasing and temporal blur	Ignoring spatial correlations causes severe undersampling as motion broadens pixel bandwidth
w/o global support (local pooling, QBP-style)	Ultra-sparse flux (0.001~0.011 ppp)	Heavy blotchy artifacts; variance explodes in empty patches	Local windows lack sufficient photon events to satisfy statistical convergence
w/o CFAR detection (uniform inverse FFT)	Background noise regions	False alarm rate exceeds 40%; reconstructions corrupted by noise	Poisson shot noise injects widespread spurious high-frequency energy without chi-square gating
w/o Rank-CFAR (direct energy peak picking)	Multiple objects with crossing velocities	False velocity detections caused by overlapping candidate planes	Overlapping spectral planes require nonparametric rank testing to prevent mutual interference
w/o Kalman Filter (pure event integration)	Extended event camera video sequences	Severe brightness drift and contrast decay over time	Event temporal derivatives discard DC and low-frequency components, requiring intensity frame fusion

Key Findings¶

Global spatiotemporal coherence overcomes photon scarcity: Jointly projecting photon detections onto 3D spatiotemporal Fourier bases allows all detected photons across the entire sensor array to contribute constructively. This structural pooling provides an effective SNR boost that reduces the photon budget required by 10× for 2D SPAD arrays and by 30× for ultra-wideband single-pixel sensors compared to per-pixel UWB probing.
Harmonious separation of flicker and motion: In local or learning-based methods, periodic illumination flicker violates optical flow brightness constancy and is suppressed as noise. Under the 3D Fourier perspective, periodic illumination forms clean 1D frequency combs along the temporal axis, orthogonal to the 2D slanted planes formed by linear motion, enabling both phenomena to be recovered simultaneously without interference.
Unsupervised velocity selectivity: Leveraging the planar geometry of linear motion in the spatiotemporal frequency domain, the rank-CFAR detector accurately isolates velocities without requiring flow estimation or learning priors. Spectral shearing refocuses videos onto specific moving targets while naturally blurring background distractors.

Highlights & Insights¶

Treating photon streams as point-process samples on a continuous manifold: The framework bypasses discrete pixel-binning and local smoothness heuristics by formulating reconstruction as an unbiased continuous inner-product estimation problem under martingale noise, backed by rigorous point-process statistics.
Velocity refocusing via spectral coordinate shearing: Transforming the frequency grid via $f_t \leftarrow f_t + \mathbf{v}^\top \mathbf{f}_{x,y}$ maps slanted motion planes directly onto the temporal DC axis, rendering the target object virtually stationary in time while naturally diffusing non-matching velocities.
Unified mathematical foundation across asynchronous sensors: Demonstrating that SPAD timestamp streams, spike camera train densities, and event camera log-intensity derivatives all reduce to linear projections of continuous flux properties provides a common analytical ground for evaluating asynchronous imaging modalities.

Limitations & Future Work¶

Gibbs ringing artifacts from Fourier truncation: Because global 3D Fourier basis functions have infinite spatiotemporal support, sharp occlusion boundaries and abrupt temporal step changes (e.g., the instant of balloon rupture) spread energy across high frequencies, inducing ringing artifacts upon truncation. Exploring compact localized bases, such as spatiotemporal wavelets, could mitigate this effect.
Restriction to constant linear motion: The velocity-selective focusing model assumes approximately constant translation within the temporal reconstruction window. Highly non-linear, accelerating, or turbulent dynamics (such as fluid splashing or tumbling fragments) bend and disperse the spectral planes, necessitating higher-order kinematic probing functions such as polynomial-phase or chirplet bases.

vs QBP (Quanta Burst Photography, Ma et al. ACM TOG 2020): QBP relies on local motion estimation and align-and-merge pipelines. At extreme low light (<0.01 ppp) and under high-frequency illumination flicker, local motion estimation fails completely; the proposed method requires no local alignment and handles complex illumination dynamics robustly.
vs bit2bit (Liu et al. NeurIPS 2024): bit2bit applies self-supervised deep neural networks with spatial smoothness inductive biases, which tend to suppress rapid illumination flicker as noise; the proposed method preserves true high-frequency modulations up to tens of kilohertz via statistically grounded CFAR detection.
vs UWB (Passive Ultra-Wideband Imaging, Wei et al. ICCV 2023): UWB performs flux probing strictly per pixel along the 1D time axis, failing when scene motion traverses pixels; the proposed framework lifts probing to 3D space-time, achieving comparable GHz-scale bandwidth with 30× fewer photons.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ (Formulates global 3D spatiotemporal flux probing from first principles and introduces unsupervised velocity-selective videography)
Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluated across SPAD512 arrays, single-pixel SPADs, event cameras, and spike cameras from 100 kfps to 6 GHz)
Writing Quality: ⭐⭐⭐⭐⭐ (Rigorous stochastic derivations, clear physical motivations, and well-structured experimental analyses)
Value: ⭐⭐⭐⭐⭐ (Pioneers a foundational theoretical framework for low-light high-speed imaging, active lidar sensing, and neuromorphic video processing)