Learn to See the Unseen in Low-light Spike Streams¶

Conference: ECCV 2026
Paper: ECCV Official
Area: Image Restoration
Keywords: Spike camera, Diffusion model, Low-light vision, Neuromorphic sensor, Image reconstruction

TL;DR¶

To overcome the severe ill-posedness of reconstructing images from sparse and noisy spike streams under extreme low-light and high-speed conditions, this paper proposes Diff-SPK, a diffusion-based framework that couples generative priors with spike-derived structural constraints using an Enhanced Texture from Inter-spike Interval (ETFI) representation, tailored condition encoding, and full-trajectory feature fusion.

Background & Motivation¶

Spike cameras are bio-inspired neuromorphic sensors capable of microsecond-level temporal sampling. Unlike conventional frame-based cameras that suffer from motion blur or event cameras that only capture differential intensity changes, spike cameras continuously accumulate incoming photons at each pixel and fire asynchronous binary spikes whenever the integrated charge reaches a predefined threshold. This mechanism makes them uniquely suited for high-speed motion perception. However, when operating under challenging low-light conditions, photon starvation causes inter-spike intervals to expand dramatically while sensor read noise and hot pixel artifacts become prominent. Consequently, the captured spike stream becomes spatio-temporally sparse, noisy, and stripped of discernible fine textures.

Existing spike reconstruction techniques primarily rely on either empirical statistical formulas (such as TFI and TFSTP) or deep regression architectures (such as CNN-, RNN-, or Transformer-based networks like S2I, WGSE, and BSF). These deterministic pixel-regression models fundamentally depend on sufficient photon integration. When incoming signals fall below critical signal-to-noise ratios, the inverse problem turns severely ill-posed: regression networks tend to produce over-smoothed images that lose high-frequency geometric boundaries, or collapse into chaotic noise artifacts. While recent efforts have explored diffusion models (such as Liu et al.), they rely on average spike counts over broad temporal accumulation windows, introducing fatal motion blur in dynamic high-speed environments.

When fine-grained appearance cannot be uniquely determined by sparse observations alone, the paradigm must shift from deterministic pixel matching to generative perceptual inference anchored by physical structures. Core idea: build a two-stage diffusion framework, Diff-SPK, that first extracts structurally consistent, unclipped brightness cues (ETFI) from inter-spike intervals, and then steers the reverse denoising trajectory of a latent diffusion model using a dedicated residual encoder and cross-attention feature fusion, achieving visually plausible and physically grounded reconstruction under extreme low-light high-speed conditions.

Method¶

Overall Architecture¶

Diff-SPK adopts the Latent Diffusion Model with ControlNet (LDMC) as its generative foundation, divided into three key stages: spike-interval texture enhancement (ETFI), structural condition encoding, and all-timestep feature fusion. Given an input spike sequence $\mathbf{S}_{k,\delta t}$ within temporal window $2\delta t$, the pipeline first computes inter-spike intervals (ISI) and applies maximum-interval normalization to rescue underexposed regions, followed by a self-supervised temporal convolution block that aggregates the sequence into a single-channel structural cue $\tilde{EI}_k$. To avoid the severe high-frequency geometric distortion observed when passing non-natural spike representations through standard VAE encoders, a lightweight residual encoder projects $\tilde{EI}_k$ into multi-scale conditioning features. Finally, throughout the reverse diffusion trajectory, an ETFI-guided feature fusion module injects condition information directly into the noisy latents $z_t$ across timesteps to enforce rigid structural guidance.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Low-light Spike Stream<br/>S_{k,delta t}"] --> B["Enhanced Texture from ISI<br/>Global ISI normalization & ConvB temporal aggregation"]
    B --> C["Aggregated Structural Map<br/>tilde{EI}_k"]
    C --> D["Structure-Preserving Condition Encoding<br/>Lightweight residual blocks"]
    D --> E["ControlNet Backbone<br/>Multi-scale conditioning branch"]
    C --> F["All-Timestep Feature Fusion<br/>ETFI-guided cross-attention & Transformer"]
    F --> G["Latent Reverse Denoising<br/>Injected into latent variable z_t"]
    E --> G
    G --> H["VAE Decoder<br/>Perceptually restored clear image"]

Key Designs¶

1. Enhanced Texture from Inter-spike Interval (ETFI): Preventing signal cutoff and temporal aggregation In extreme low-light conditions, directly using the output of existing deep regression networks as generative conditions fails because regression outputs degrade unpredictably across illumination levels, misguiding diffusion priors. The authors propose ETFI to extract a representation with predictable degradation and structural consistency. ETFI first computes light intensity via global inter-spike intervals (ISI). To ensure that prolonged intervals in pitch-black areas remain perceptible within 8-bit dynamic range, it normalizes intensities using the maximum interval: $$ \hat{EI}k = \frac{\max(ISI_k)}{ISI_k} $$ This operation guarantees that minimal photon evidence is preserved rather than clipped to black. Subsequently, temporal stacks within window $T = 2\delta t + 1$, denoted as $\hat{\mathbf{EI}}_{k,\delta t} \in \mathbb{R}^{T \times H \times W}$, are compressed by a convolution block into a single-channel representation $\tilde{EI}_k \in \mathbb{R}^{1 \times H \times W}$ trained via a self-supervised temporal broadcasting loss: $$ \mathcal{L}}} = \mathbb{E{\hat{\mathbf{EI}}_k)|_2^2 \right] $$ where }} \left[ |\hat{\mathbf{EI}}_{k,\delta t} - \mathcal{B}(\tilde{EI$\mathcal{B}(\cdot)$ broadcasts the channel along the temporal dimension, smoothing quantization noise while preserving dynamic boundaries.

2. Structure-Preserving Condition Encoding: Overcoming VAE latent distortion Conventional diffusion-based restorers typically map input conditions into the latent space using a pre-trained VAE encoder. However, empirical inspection reveals that ETFI features exhibit distinct discrete quantization distributions and non-natural image statistics; directly running them through a standard VAE encoder leads to catastrophic spatial degradation (local SSIM drops to ~0.32). To retain sharp structural boundaries and edge topologies, Diff-SPK replaces the VAE path with a lightweight residual convolutional encoder: $$ \mathcal{F}{enc} = \text{Conv}}(\hat{\mathcal{F}{enc}) + \tilde{EI}_k, \quad \text{where } \hat{\mathcal{F}}_k)) $$ Here, } = \text{Res}(\text{Conv}(\tilde{EI$\text{Conv}_{init}$ is initialized with zero weights to ensure smooth training initialization, and $\hat{\mathcal{F}}_{enc}$ provides intermediate features for the fusion module. This design preserves critical contours while projecting ETFI into the multi-scale embedding space expected by ControlNet.

3. All-Timestep Feature Fusion Module: Strengthening continuous denoising guidance Standard ControlNet injects condition features only into the decoder stages of the denoising U-Net via skip residual additions. Under severe photon deprivation where input features are extremely sparse, this late-stage injection provides insufficient guidance during early heavy-noise stages, causing the reverse trajectory to drift toward unconstrained generative hallucinations. The authors introduce a dedicated fusion module that injects ETFI features directly into the latent state $z_t$ at every timestep $t$: $$ \mathcal{F}{ECA} = \text{Linear}(\text{ECA}(t}, \hat{\mathcal{F}{enc}, z_t)) + z_t $$ where ECA is an ETFI-guided cross-attention mechanism taking queries from timestep embedding $t_{emb}$ and latent $z_t$, and keys/values from conditional feature $\hat{\mathcal{F}}_{enc}$. Transformer blocks then refine the spatio-temporal interactions before a zero-convolution layer produces the final fused latent: $$ \mathcal{F}} = \text{Conv{init}(\text{Trans}(\mathcal{F})) + z_t $$ The fused latent $\mathcal{F}_{fuse}$ substitutes $z_t$ as input to the denoising U-Net, anchoring the full generative trajectory to real physical spike evidence.

Loss & Training¶

The framework is optimized in two consecutive stages. First, the ETFI temporal convolution module is pre-trained independently for 6,000 iterations. Second, the ETFI module is frozen while ControlNet, the condition encoder, and the feature fusion module are fine-tuned end-to-end for 30,000 iterations using synthetic low-light spike streams. The model initializes from Stable Diffusion 1.5, with ControlNet initialized from control_v11f1e_sd15_tile.ckpt. The noise prediction loss is formulated as: $$ \mathcal{L} = \mathbb{E}{z_t, t, \mathcal{F}}, c_{text}, \epsilon} \left[ |\epsilon_\theta(\mathcal{F{fuse}, t, \mathcal{F}) - \epsilon|_2^2 \right] $$ Training is conducted with AdamW at a learning rate of }, c_{text$10^{-5}$ and batch size of 96 across 4 NVIDIA A800-80GB GPUs. During inference, DDPM scheduling with 50 steps is applied under a classifier-free guidance (CFG) scale of 2, striking an optimal trade-off between natural perceptual quality and physical fidelity.

Key Experimental Results¶

Main Results¶

To establish a rigorous benchmark for low-light high-speed spike stream reconstruction, the authors introduce the synthetic SATSPK dataset alongside real benchmarks that are ~30x larger than existing real datasets (including IndoorSPK with controlled lux levels and OutdoorSPK captured from a driving vehicle at dusk). Diff-SPK is compared against traditional algorithms (TFI, TFP, TFSTP) and state-of-the-art deep regression and diffusion baselines (SSML, S2I, WGSE, BSF, Hu et al., Liu et al.). Full-reference metrics (PSNR, SSIM, LPIPS) and no-reference perceptual metrics (MUSIQA, MANIQA, CLIP-IQA+) are evaluated.

Dataset	Metric	TFSTP (Trad.)	Hu et al. (Reg. SOTA)	Liu et al. (Diff. Base)	Ours (Diff-SPK)	Gain / Improvement
SATSPK (Extreme dim synthetic)	PSNR (↑) SSIM (↑) LPIPS (↓) MUSIQA (↑)	15.315 0.342 0.688 36.175	16.876 0.542 0.439 36.551	20.346 0.460 0.527 28.394	23.499 0.683 0.223 56.565	+6.623 dB vs Hu +0.141 vs Hu -0.216 vs Hu +20.014 vs Hu
IndoorSPK (Extreme dim real indoor)	MUSIQA (↑) MANIQA (↑) CLIP-IQA+ (↑)	39.310 0.297 0.484	46.559 0.328 0.395	42.737 0.306 0.345	48.046 0.436 0.499	+1.487 vs Hu +0.108 vs Hu +0.104 vs Hu
OutdoorSPK (Real dusk driving)	MUSIQA (↑) MANIQA (↑) CLIP-IQA+ (↑)	48.690 0.374 0.469	47.777 0.342 0.422	34.146 0.305 0.295	49.494 0.408 0.480	+1.717 vs Hu +0.066 vs Hu +0.058 vs Hu
Dong et al. (Cross-camera real)	MUSIQA (↑) MANIQA (↑) CLIP-IQA+ (↑)	36.286 0.256 0.368	46.456 0.342 0.317	33.917 0.206 0.210	55.268 0.474 0.450	+8.812 vs Hu +0.132 vs Hu +0.133 vs Hu

Ablation Study¶

The ablation experiments in Table 4 isolate the contribution of ETFI conditioning, the lightweight encoder, and the all-timestep fusion module, while Table 3 analyzes behavior across different illumination levels.

Configuration	IndoorSPK MUS. (↑)	IndoorSPK MAN. (↑)	SATSPK PSNR (↑)	SATSPK SSIM (↑)	SATSPK LPIPS (↓)	Observation & Impact
Full Model (Ours)	48.046	0.436	23.499	0.683	0.223	Balanced perceptual quality and structural fidelity
Baseline 1 (w/o ETFI, conditioning on Hu et al.)	46.268	0.375	15.605	0.429	0.474	PSNR plunges by 7.894 dB; irregular degradation misleads diffusion
Baseline 2 (w/o custom encoder, using fine-tuned VAE)	39.917	0.372	21.891	0.583	0.353	VAE latent distortion destroys fine structures and edges
Baseline 3 (w/o feature fusion module)	47.867	0.408	23.365	0.679	0.233	Late U-Net injection allows trajectory drift, introducing artifacts

Under controlled indoor lighting tests (Table 3 across 40, 60, and 80 Lux), Diff-SPK ranks first across all metrics and illumination levels (e.g., reaching 48.223 MUSIQA and 0.445 MANIQA at 40 Lux), proving stable resilience against diverse photon deprivation regimes.

Key Findings¶

Predictable degradation trumps pseudo-clean regression: Baseline 1 proves that generative models do not demand superficially clean inputs; rather, they require structural inputs with predictable, stationary degradation patterns. Deep regression outputs in low light contain erratic artifacts that fatally misdirect diffusion priors.
Modality mismatch in VAE latents: Forcing neuromorphic spike-interval distributions through natural-image VAE encoders causes irreversible geometric smoothing and structural distortion.
Critical role of early trajectory intervention: Fusing condition features directly into noisy latents $z_t$ during early reverse steps prevents generative priors from hallucinating incorrect textures (e.g., resolving previously illegible keyboard letters clearly).

Highlights & Insights¶

Paradigm shift from regression to constrained generative inference: When photon flux drops below the theoretical reconstruction threshold, deterministic recovery is ill-posed. Diff-SPK elegantly leverages pre-trained generative priors to infer plausible visual appearances while keeping them strictly bounded by spike-derived structural invariants.
Adaptive interval normalization: Normalizing local intervals by the maximum interval within the frame prevents underexposed regions from falling below quantization floors, preserving full-scene dynamic range without complex exposure mapping networks.
Cross-modality adaptation blueprint: The combination of lightweight residual encoding and continuous cross-attention latent fusion provides a clean, generalizable paradigm for feeding unconventional neuromorphic inputs (spike streams, event volumes) into pre-trained visual foundation models.

Limitations & Future Work¶

High inference latency: Relying on a 50-step iterative DDPM sampling process requires several seconds per frame, conflicting with the sub-millisecond real-time demands of neuromorphic robotics. Distillation into few-step models (e.g., Consistency Models or SD-Turbo) is an essential next step.
Temporal coherence across video sequences: Processing independent temporal windows without inter-frame temporal attention layers may induce high-frequency temporal flickering across consecutive video frames.
Hallucination risks in extreme noise: Under zero-photon conditions, generative priors carry inherent risks of hallucinating plausible but non-existent details; integrating calibrated uncertainty estimation is critical for safety-critical autonomous driving.

vs Hu et al. (ECCV 2024): Hu et al. employ an RNN regression scheme that achieves high fidelity under moderate lighting, but breaks down in severe darkness (SATSPK PSNR of only 16.876 dB). Diff-SPK unlocks generative priors to surpass Hu et al. by over 6.6 dB PSNR and 20.0 points in MUSIQA under extreme low-light.
vs Liu et al. (ICCV 2025): While Liu et al. also explore diffusion models for spike streams, they feed temporal average spike counts across broad windows, suffering catastrophic motion blur in dynamic high-speed scenes. Diff-SPK preserves high-speed motion by grounding generation on fine-grained inter-spike intervals.
vs DiffBIR / SUPIR: Leading image restoration diffusion models assume RGB image inputs and rely on VAE encoders. This work reveals the fundamental incompatibility between neuromorphic spike representations and natural VAE latent spaces, delivering a tailored alternative.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Pioneers diffusion-based generative reconstruction for low-light high-speed spike streams with tailored structural representations and fusion modules]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Establishes a comprehensive benchmark 30x larger than prior datasets, spanning controlled indoor lux sweeps and real-world high-speed outdoor driving]
Writing Quality: ⭐⭐⭐⭐⭐ [Clearly articulates the failure modes of regression and VAE encoders, with sound theoretical motivation and crisp empirical proofs]
Value: ⭐⭐⭐⭐⭐ [Breaks photon flux barriers for neuromorphic vision, offering a vital framework for extreme night-vision robotics and high-speed perception]