---

title: >- [Paper Note] EvDiff: High Quality Video with an Event Camera description: >- [ECCV 2026][Image Generation][Event Camera] EvDiff proposes a one-step diffusion framework with Stable Diffusion 3 and a surrogate training pipeline, decoupling event video reconstruction with large-scale image data to reconstruct high-quality chromatic video from monochromatic events, reducing MSE and LPIPS by 32.2% and 8.5% on BS-ERGB and DSEC. tags: - ECCV 2026 - Image Generation - Event Camera - Diffusion Model - One-Step Diffusion - Video Reconstruction - Surrogate Training date: 2026-09-19 content_hash: 9af1835485a261fd

EvDiff: High Quality Video with an Event Camera

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/weilunli/EvDiff
Area: Image Generation
Keywords: event camera, one-step diffusion model, video reconstruction, surrogate training, efficient temporal fusion

TL;DR

EvDiff addresses the computational overhead and paired-data scarcity in event-based video reconstruction by combining Stable Diffusion 3 with a one-step diffusion backbone, a recurrent EvEncoder with Efficient Temporal Fusion, and a 3-stage surrogate training pipeline that distills natural image priors into the event domain, reconstructing high-fidelity, high-dynamic-range chromatic video from monochromatic event streams alone.

Background & Motivation

Event cameras, such as Dynamic Vision Sensors (DVS), mimic the biological neuromorphic sensing mechanism by asynchronously recording per-pixel brightness changes. They provide unique physical advantages over conventional frame-based cameras, including microsecond-level temporal resolution, ultra-high dynamic range (up to 140 dB), and low power consumption. However, event cameras only emit sparse, asynchronous event streams devoid of absolute intensity references, which cannot be directly ingested by standard frame-based computer vision architectures. Reconstructing continuous intensity frames from events is therefore essential, not only for visualizing high-speed scenes but also for bridging neuromorphic sensing with mainstream downstream computer vision algorithms.

Since the seminal work E2VID, dominant event-to-video reconstruction approaches have followed an end-to-end regression paradigm, employing U-Net style convolutional networks to learn a deterministic mapping from accumulated event voxel grids to intensity frames. Nevertheless, scaling up model capacity and training data within this paradigm has hit fundamental bottlenecks. On one hand, because event cameras natively operate at extremely high temporal rates, directly deploying standard multi-step diffusion models or spatio-temporal self-attention introduces quadratic computational complexity, which becomes prohibitive for rendering high-frame-rate video. On the other hand, real-world paired event-image datasets are orders of magnitude smaller than natural image collections, and synthetic simulators are bottlenecked by the availability of high-frame-rate video sources, leaving regression models prone to low-quality grayscale outputs marred by severe blur, ghosting, and block-like artifacts.

A deeper analysis of these regression artifacts reveals that models like E2VID exhibit highly structured, consistent degradation patterns—specifically edge disruption, localized blurring, and blotchy distortions caused by missing boundary conditions and limited spatial bandwidth. The authors observed that if these artifact patterns could be synthesized onto massive natural image datasets, the task could be effectively decoupled into degraded surrogate synthesis and general-purpose generative restoration, allowing powerful diffusion visual priors to act as high-capacity restorers. Core idea: decompose event video reconstruction into a one-step diffusion restoration and surrogate distillation pipeline, using an E2VID-style degradation model to train a DiT on large-scale natural images and distilling it into a temporal EvEncoder for efficient, single-step end-to-end chromatic video generation.

Method

Overall Architecture

During inference, EvDiff operates in a single-stage, end-to-end fashion. The overall pipeline consists of two principal components: an event encoder (EvEncoder) equipped with temporal consistency modeling, and a one-step diffusion backbone (OSDiff) based on Stable Diffusion 3 (SD3) accompanied by a pre-trained VAE decoder. Given an asynchronous event stream \(\{e_{\Delta t}\}\) within a time window \(\Delta t\), the events are first discretized into a synchronous voxel grid \(V \in \mathbb{R}^{H \times W \times T}\). The EvEncoder maps these event voxels along with recurrent hidden states into latent representations \(\{z_i\}_{i=1}^N\). The one-step diffusion model takes each \(z_i\) as the starting point and executes a single forward denoising pass to predict clean latents, which the VAE decoder subsequently decodes into high-quality chromatic video frames.

To eliminate the dependency on scarce event-image paired data, EvDiff adopts a three-stage Surrogate Training Pipeline: 1. Stage 1 (DiT Training): On a large-scale natural image dataset (Places365) without any real events, degraded low-quality counterparts are synthesized using an E2VID-style degradation model to train the DiT refiner and a trainable Surrogate VAE encoder; 2. Stage 2 (Surrogate Distillation): An off-the-shelf E2VID model serves as a surrogate bridge to generate coarse intensity images from events, which are converted to target latents via the Surrogate VAE encoder to supervise the lightweight EvEncoder via knowledge distillation; 3. Stage 3 (Joint Fine-Tuning): The entire model is jointly fine-tuned on small-scale real and synthetic event sequences, adapting to real sensor noise while enforcing temporal coherence via optical flow constraints.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    E["Input Event Stream {e_Δt}"] --> V["Voxel Grid V"]
    subgraph S1["Surrogate Training Pipeline and Alignment"]
        direction TB
        DEG["Degradation Modeling and Large-Scale DiT Pretraining<br/>Synthesize E2VID degradation on Places365"]
        ETF_NODE["Efficient Temporal Fusion EvEncoder<br/>Lightweight gated temporal recursion"]
        DIST["Surrogate Distillation and Joint Fine-Tuning<br/>E2VID latent distillation and flow consistency"]
        DEG --> ETF_NODE
        ETF_NODE --> DIST
    end
    V --> ETF_NODE
    DIST --> OSD["One-Step Diffusion Model OSDiff<br/>Single forward denoising with SD3 priors"]
    OSD --> DEC["VAE Decoder"]
    DEC --> OUT["High-Quality Chromatic Video Stream"]

Key Designs

1. Degradation Modeling and Large-Scale DiT Pretraining: Bridging the Paired Data Gap with Synthetic Artifacts

To circumvent the severe scarcity of paired event-video datasets and unleash modern generative capacity, the authors systematically broke down the characteristic failure modes of regression-based EVRMs into three distinct physical factors: Event Generation Mechanism (EGM), Sensor-Inherent Imperfections (SII), and Algorithmic Degradation (AD). EGM degradation accounts for the absence of absolute intensity in smooth regions by detecting low-variance areas and superimposing stochastic multi-scale polygons and irregular blotches. SII degradation captures pixel temporal bandwidth limits and circuit noise through localized edge blurring, stochastic boundary jitter, and Gaussian noise. AD degradation simulates motion blur from prolonged voxel accumulation alongside spatial resolution filtering via down-and-up-sampling. By applying this compound degradation to 1.8 million high-resolution images in Places365, the DiT backbone and Surrogate VAE encoder are trained to invert E2VID-like corruptions in one step under a joint latent and perceptual loss:

\[\mathcal{L}_{\text{DiT}} = \lambda \|\boldsymbol{z}_{\text{pred}} - \boldsymbol{z}_{\text{gt}}\|_2^2 + \text{LPIPS}(\hat{\mathbf{I}}, \mathbf{I}_{\text{gt}})\]

This design effectively transfers billions of natural image generative priors into the reconstruction task without requiring paired physical event ground truths.

2. Efficient Temporal Fusion EvEncoder: Lightweight Gated Recursion Balancing Temporal Flow and Computation

Standard video diffusion architectures that employ full spatio-temporal self-attention (e.g., SVD or ControlNet) incur a prohibitive computational complexity of \(O((THW)^2 \cdot C)\), conflicting with the high-frame-rate requirements of event vision. EvEncoder addresses this by introducing the Efficient Temporal Fusion (ETF) module within a recurrent framework, reducing the complexity to \(O((HW)^2 \cdot T \cdot C)\). Each ETF module utilizes three convolutional blocks to extract features from both the current event feature map \(x_i\) and the previous hidden state \(h_{i-1}\), computing an adaptive fusion gate \(g_i\) to blend temporal information:

\[\mathbf{y}_i = \mathbf{g}_i \odot \mathbf{x}_i + (1 - \mathbf{g}_i) \odot \mathbf{h}_{i-1}\]

This dynamic gating mechanism enables the network to preserve long-term background context while adapting rapidly to transient, high-speed motions, outperforming static fusion without introducing recurrent memory bloat.

3. Surrogate Distillation and Joint Fine-Tuning: Latent Space Alignment and Temporal Coherence

While the Stage 1 DiT operates effectively on latent representations derived from degraded RGB images, it cannot directly process raw event voxel grids. In Stage 2, an off-the-shelf E2VID model is used as a surrogate bridge: event streams are converted to coarse frames and mapped to teacher latents \(z_{\text{vae}}\) via the Surrogate VAE encoder, guiding the training of the student EvEncoder by minimizing latent divergence:

\[\mathcal{L}_{\text{distill}} = \text{KL}(q_\phi(\boldsymbol{z}|E)) + \|\boldsymbol{z} - \boldsymbol{z}_{\text{vae}}\|_2^2\]

In Stage 3, the full EvDiff model is end-to-end fine-tuned on paired sequences (REDS dataset augmented via the v2e simulator). To bridge the sim-to-real gap, online corruptions (event dropping, merging, polarity removal) are applied to simulated events, while an optical flow warping loss \(\mathcal{L}_{\text{flow}}\) enforces frame-to-frame temporal stability, ensuring clean convergence during single-step inference.

Loss & Training

The training proceeds through three distinct phases on a single NVIDIA H200 GPU, using the AdamW optimizer (\(\beta_1=0.9, \beta_2=0.999\), initial learning rate \(5 \times 10^{-6}\)) and LoRA (rank 64) on the diffusion backbone: - Stage 1 (DiT Training): Trained on Places365 cropped to \(512 \times 512\) for 180k iterations with batch size 10 using \(\mathcal{L}_{\text{DiT}}\). - Stage 2 (Surrogate Distillation): Trained on event sequences of length 40 for 12k iterations (batch size 1) optimizing \(\mathcal{L}_{\text{distill}}\). - Stage 3 (Joint Fine-Tuning): Trained on sequences of length 30 for 12k iterations (batch size 1) with the composite objective \(\mathcal{L}_{\text{joint}} = \lambda \mathcal{L}_{\text{DiT}} + \mathcal{L}_{\text{flow}} + \|\boldsymbol{z}_{\text{pred}} - \boldsymbol{z}_{\text{gt}}\|_2^2\).

Key Experimental Results

Main Results

Evaluations were conducted on two real-world benchmarks: BS-ERGB (high-speed dynamic real scenes) and DSEC (autonomous driving recorded with stereo event cameras). For fair comparison with grayscale-only baselines, EvDiff outputs were converted to grayscale when calculating MSE, SSIM, and LPIPS.

Dataset	Method	MSE ↓	SSIM ↑	LPIPS ↓	FID ↓	FVD ↓
BS-ERGB	E2VID (CVPR 2019)	0.1175	0.3313	0.5518	276	1688
	FireNet (WACV 2020)	0.0857	0.3302	0.5328	271	1909
	E2VID+ (ECCV 2020)	0.0717	0.3710	0.4397	238	1652
	ET-Net (ICCV 2021)	0.0683	0.3557	0.4566	270	1569
	HyperE2VID (TIP 2024)	0.0756	0.3489	0.4598	275	1673
	EvDiff (Ours)	0.0463	0.3394	0.4023	148	984
DSEC	E2VID (CVPR 2019)	0.1042	0.3393	0.5427	239	1541
	FireNet (WACV 2020)	0.0927	0.3478	0.5670	262	1492
	SPADE-E2VID (TIP 2021)	0.0601	0.4578	0.4911	234	1277
	ET-Net (ICCV 2021)	0.0737	0.2953	0.5232	259	1496
	HyperE2VID (TIP 2024)	0.0691	0.2977	0.5297	272	1445
	EvDiff (Ours)	0.0476	0.3677	0.4226	129	1491

In terms of model efficiency (original Paper Table 2), EvDiff has 2.19G parameters and requires only 2.18 TFLOPs at \(512 \times 512\) resolution. In contrast, the multi-step SD3 ControlNet counterpart has 8.23G parameters and requires 26.62 TFLOPs (10 steps), 49.93 TFLOPs (20 steps), and 96.56 TFLOPs (40 steps).

Ablation Study

Ablation results on BS-ERGB and DSEC validating the contributions of each pipeline stage and module (original Paper Table 3):

Configuration	BS-ERGB: MSE ↓	BS-ERGB: LPIPS ↓	BS-ERGB: FID ↓	DSEC: MSE ↓	DSEC: LPIPS ↓	DSEC: FID ↓	Notes & Analysis
Final Model	0.0463	0.4023	148	0.0476	0.4226	129	Complete 3-stage EvDiff framework
w./o. Stage 1 training	0.0726	0.4906	374	0.0630	0.4817	256	Lacks large-scale prior, performance drops 56.8%
w./o. E2VID-Style Degradation Model	0.0850	0.5267	396	0.0918	0.5286	450	Severe domain gap, FID deteriorates by 83.6%
w./o. EGM Degradation	0.0738	0.5000	276	0.0702	0.4937	260	Omitting smooth blotches harms intensity estimation
w./o. SII Degradation	0.0667	0.4743	277	0.0734	0.4897	238	Omitting sensor noise and edge dispersion degrades fidelity
w./o. AD Degradation	0.0747	0.5059	333	0.0861	0.5150	333	Omitting motion blur and downsampling filters
w./o. Surrogate Distillation	0.0660	0.4373	162	0.0778	0.4750	164	Without teacher latent alignment, performance drops 29.8%
w./o. Stage 3 finetune	0.0763	0.5740	224	0.0968	0.6546	189	Lacks adaptation to real sensor distribution; LPIPS spikes
w./o. ETF	0.0586	0.4234	159	0.0605	0.4482	152	Removing adaptive temporal gating increases MSE by 20.9%
w. ConvGRU	0.0500	0.4147	165	0.0561	0.4337	127	Comparable size but lower stability than ETF
w. ConvLSTM	0.0496	0.4156	160	0.0620	0.4403	127	Increased redundancy and slower convergence
w. E2VID + VAE-Encoder	0.0729	0.4827	231	0.0580	0.4691	184	Pipeline latency rises from 63ms to 84ms with worse quality

Key Findings

1. Surrogate degradation is the linchpin for domain transfer: Omitting the degradation model entirely caused the BS-ERGB FID to deteriorate from 148 to 396 and FVD from 984 to 3683. Among individual factors, algorithmic degradation (AD) and event generation mechanism (EGM) exerted the strongest impact on perceptual fidelity.
2. Substantial gains in perceptual quality and realism: On BS-ERGB, EvDiff improved MSE, LPIPS, FID, and FVD by 32.2%, 8.5%, 37.8%, and 37.3% respectively over the second-best method. On DSEC, FID dropped by 44.9% (from 234 down to 129).
3. Pioneering chromatic video synthesis with HDR preservation: EvDiff is the first method to generate realistic, full-color video from monochromatic events. In scenes with extreme dynamic range (such as sky and cloud formations), EvDiff successfully preserves cloud structures that are completely washed out in the overexposed Ground Truth frame.

Highlights & Insights

• Decoupling data scarcity via degradation surrogates: Rather than struggling with the near-impossibility of collecting massive event-RGB paired datasets, the authors reverse-engineered the failure modes of regression networks into physical degradation operators, transforming 1.8M natural images into a rich pretraining domain.
• One-step diffusion unlocked for real-time neuromorphic streaming: By reformulating one-step diffusion for high-frame-rate event restoration, the model slashes inference FLOPs to 2.18 TFLOPs—over an order of magnitude lower than multi-step ControlNet counterparts—bringing large generative models within reach of real-time event processing.
• Independence from scene-specific text prompts: Unlike ControlNet baselines that depend heavily on VLM-generated prompts from ground truth images, EvDiff operates entirely prompt-free during blind reconstruction, demonstrating genuine robustness for practical sensor applications.

Limitations & Future Work

• Residual temporal flicker: Because diffusion is applied frame-wise and temporal coherence is enforced via recurrent latent features and flow loss, minor illumination or hue variations can still appear across consecutive frames in dynamic scenes.
• Backbone-induced grid artifacts: As a consequence of the patch-based transformer architecture in the underlying SD3 backbone, faint periodic grid-like artifacts may occasionally surface in smooth background areas.
• Ill-posedness of chromatic colorization: Since monochromatic event streams completely lack spectral wavelength information, color reconstruction relies on statistical priors learned from natural scenes; for ambiguous objects or artificial textures, reconstructed colors may diverge from ground-truth reflectance.

Related Work & Insights

• vs E2VID / E2VID+ / HyperE2VID: These regression models map event voxels directly to intensity frames via U-Net architectures; constrained by limited training data and compact capacities, their outputs remain grayscale with pronounced blur and artifacts. EvDiff incorporates billion-parameter diffusion priors to achieve unprecedented perceptual quality and chromatic outputs.
• vs E2VIDiff: E2VIDiff conditioned multi-step Stable Diffusion on event features, but suffered from excessive sampling latency and weak physical fidelity, frequently hallucinating content. EvDiff employs one-step diffusion and surrogate distillation, achieving superior fidelity, lower latency, and zero text-prompt dependency.
• vs ControlNet (SD3): Adapting ControlNet to event reconstruction incurs prohibitive multi-step FLOPs (\(>26\) TFLOPs) and fails to reconstruct faithful details without explicit scene prompts; EvDiff is over \(12\times\) more compute-efficient and provides significantly higher temporal consistency.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant degradation modeling that decouples event video reconstruction from paired data limitations.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across BS-ERGB and DSEC with 8 competitive baselines and exhaustive multi-component ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear problem formulation, intuitive diagrams, and rigorous analysis of neuromorphic degradation factors.
• Value: ⭐⭐⭐⭐⭐ Establishes a foundational framework for porting large generative models to neuromorphic vision and low-level inverse problems.

Related Papers

• [CVPR 2026] AE2VID: Event-based Video Reconstruction via Aperture Modulation
• [CVPR 2026] Spk2VidNet: A Hierarchical Recurrent Architecture for High-Fidelity Video Reconstruction from Long Spike-Camera Streams
• [CVPR 2026] EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing
• [CVPR 2026] Preserving Source Video Realism: High-Fidelity Face Swapping for Cinematic Quality
• [CVPR 2026] STCDiT: Spatio-Temporally Consistent Diffusion Transformer for High-Quality Video Super-Resolution