title: >- [Paper Note] From Noise to Events: Conditional Diffusion for Event Data Augmentation description: >- [ECCV 2026][video_understanding][Event Data Augmentation] To address the corruption of spatiotemporal continuity and ambiguous label assignment in event camera augmentation, N2E introduces a conditional DDIM framework with dimension-decoupled representation, significantly boosting event classification and action recognition. tags: - ECCV 2026 - video_understanding - Event Camera - Data Augmentation - Diffusion Models date: 2026-09-19 content_hash: a796d8eaa0384564

From Noise to Events: Conditional Diffusion for Event Data Augmentation¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/rfww/n2e
Area: Video Understanding
Keywords: event data augmentation, conditional diffusion models, spatiotemporal continuity, noise optimization, dimension-decoupled representation

TL;DR¶

Addressing the issues of broken temporal continuity and ambiguous labels in conventional event cropping and mixing augmentations, N2E introduces a dimension-decoupled tensor representation and a conditional DDIM diffusion framework that generates physically faithful and diverse augmented event streams by steering denoising steps, stochastic latent noise, and condition embeddings.

Background & Motivation¶

Event cameras are bio-inspired neuromorphic visual sensors that asynchronously detect per-pixel logarithmic illumination changes with microsecond temporal resolution, high dynamic range, low latency, and minimal power consumption. These sensors demonstrate compelling advantages in challenging environments such as high-speed motion capture and extreme lighting variations. However, high hardware costs and tedious annotation make high-quality event datasets scarce. Consequently, standard event-based deep neural networks frequently suffer from severe overfitting when deployed to complex downstream perception tasks, underscoring the urgent necessity of robust data augmentation strategies.

Existing event data augmentation methods predominantly borrow heuristic geometric transformations or patch-mixing techniques from conventional frame-based computer vision, which introduces critical structural drawbacks. An event stream is fundamentally composed of asynchronous quadruplets $(x, y, t, p)$ describing spatial coordinates, timestamps, and binary polarities, where all dimensions are intrinsically coupled. Masking-based approaches like EventDrop stochastically discard events across spatial or temporal bins, inevitably severing temporal continuity and degrading downstream convergence. Conversely, mixing methods like EventMix utilize 3D spatial-temporal masks to blend different event streams; this causes unrealistic overlapping artifacts and assigns ambiguous, unbalanced labels based on event counts, severely impairing slow-motion samples. Simulated injection methods like ShapeAug also fail to capture realistic event scene statistics.

The essential insight of this work is to break away from rigid coordinate transforms and ambiguous interpolation by directly modeling the joint probabilistic distribution of event streams. Because sparse and asynchronous event streams naturally conform to probabilistic diffusion processes, data augmentation can be reframed as a controllable generative task of mapping noise to events. Core Idea: decouple the quadruplet event stream into a spatiotemporally lossless 4-channel tensor, leverage a conditional DDIM to invert original event instances into sample-specific latent Gaussian noise, and synthesize faithful, in-distribution augmented event streams by perturbing denoising steps, latent noise, and semantic conditioning vectors.

Method¶

Overall Architecture¶

The N2E pipeline comprises two core phases: conditional diffusion training and controllable event data augmentation. In the training phase, the asynchronous quadruplet event stream is partitioned into slices and reshaped into a dimension-decoupled 4-channel tensor. A condition extractor derives global semantic context vectors from this tensor, while an event decoder (based on conditional DDIM) learns to predict Gaussian noise across forward diffusion steps to reconstruct the event representation. In the inference/augmentation phase, the model runs a deterministic DDIM backward process on the input sample to infer its specific latent noise. Users then steer three independent perturbation axes (denoising steps, latent stochastic noise, and condition embeddings) to synthesize diverse event streams containing realistic motion blur and background noise.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Quadruplet Event Stream<br/>(x, y, t, p)"] --> B["Dimension-Decoupled Representation<br/>Slicing and reshaping to 4-channel 3D tensor"]
    B --> C["Condition Extractor & DDIM Inversion<br/>Extract context vector c & infer latent noise"]
    C --> D["Three-Axis Controllable Perturbation<br/>Denoising steps / latent noise / condition"]
    D --> E["Augmented Event Streams<br/>Fed into downstream event vision backbones"]

Key Designs¶

1. Dimension-Decoupled Representation: Preserving Fine-Grained Spatiotemporal Continuity Conventional dense event representations (such as Event Frames, Time Surfaces, and Event Spike Tensors) accumulate or project asynchronous events onto 2D grids, inevitably merging or dropping multiple events that share identical pixel coordinates within a temporal window. This lossy compression degrades temporal precision and widens the domain gap when applied to generative modeling. To overcome this, N2E designs a dimension-decoupled transformation: it partitions the raw event stream into consecutive sub-event slices of fixed length $\Delta n$ (set to 4096), yielding $n = \lfloor N / \Delta n \rfloor$ slices. Within each slice, the spatial coordinates $(x, y)$, normalized timestamps $t$, and polarity $p$ are decoupled and reshaped into compact 2D grids of size $h \times w$. Concatenating these yields a regular tensor $\mathcal{E} \in \mathbb{R}^{n \times 4 \times h \times w} = \mathcal{R}(\mathbf{E}, \Delta n)$. Involving only slicing, reshaping, and normalization, this conversion operates with minimal computational overhead while fully preserving temporal dynamics and sparsity without event loss.

2. Conditional DDIM Event Decoding & Latent Inversion: Preventing Hallucinations and Semantic Drift Because raw events exhibit significant asynchronous sparsity and micro-stochasticity, sampling directly from standard unconditioned Gaussian noise makes it difficult for a diffusion model to generate coherent object boundaries. N2E introduces a deep condition extractor $f_c(\cdot)$ built from residual convolution blocks and attention mechanisms to extract a semantic context vector $c = f_c(\mathcal{E})$. A UNet-based event decoder is trained to predict the noise $\epsilon_\theta(\mathcal{E}_\tau, c, \tau)$ at timestep $\tau$. To ensure that augmented samples strictly follow the semantic distribution of the source data, N2E inverts the input event sample into a sample-specific latent noise $\mathcal{E}_T$ via the deterministic DDIM backward trajectory: $$\mathcal{E}_{\tau+1} = \sqrt{\alpha_{\tau+1}} f_\theta(\mathcal{E}_\tau, c, \tau) + \sqrt{1 - \alpha_{\tau+1}} \epsilon_\theta(\mathcal{E}_\tau, c, \tau)$$ where $f_\theta(\mathcal{E}_\tau, c, \tau) = \frac{1}{\sqrt{\alpha_\tau}}(\mathcal{E}_\tau - \sqrt{1 - \alpha_\tau}\epsilon_\theta(\mathcal{E}_\tau, c, \tau))$. By using this sample-specific latent code as the generative starting point, N2E guarantees that subsequent synthesis remains tightly anchored to the original sample's manifold, completely eliminating class ambiguity.

3. Three-Axis Controllable Augmentation Strategy: Regulating Blur and Background Noise To introduce rich, physically realistic data diversity, N2E establishes three decoupled operations during reverse diffusion synthesis: $$\hat{\mathcal{E}} = \begin{cases} \text{Generate}(\mathcal{E}_T + \epsilon_s, c, T \to 0) & \text{# stochastic noise perturbation} \\ \text{Generate}(\mathcal{E}_{\hat{T}}, c, \hat{T} \to 0) & \text{# diffusion step adjustment} \\ \text{Generate}(\mathcal{E}_T, c + \epsilon_c, T \to 0) & \text{# condition noise perturbation} \end{cases}$$ The underlying mechanics operate across three complementary levels: - Diffusion Step Adjustment ($\text{N2E}_{\text{step}}$): Restricting the reverse denoising steps to $\hat{T} \in [50, 200]$ controls the trade-off between background noise and structural clarity. Fewer steps preserve realistic environmental stochastic activations, whereas larger steps generate sharp contours. - Latent Noise Perturbation ($\text{N2E}_{\text{s.noise}}$): Adding Gaussian noise $\epsilon_s$ (scaled by 0.1) to the inverted stochastic noise $\mathcal{E}_T$ smoothly perturbs high-frequency event edges, simulating realistic motion blur. - Condition Perturbation ($\text{N2E}_{\text{c.noise}}$): Injecting perturbation $\epsilon_c$ into the context embedding $c$ alters fine-grained event textures without disturbing the overarching object geometry.

Loss & Training¶

The event decoder and condition extractor are jointly trained with a dual $\ell_1$-$\ell_2$ denoising loss. At each diffusion timestep $\tau \in [1, T]$, a noisy state is formed via $\mathcal{E}_\tau = \sqrt{\bar{\alpha}_\tau} \mathcal{E}_0 + \sqrt{1 - \bar{\alpha}_\tau} \epsilon_\tau$, with $\epsilon_\tau \sim \mathcal{N}(0, \mathbf{I})$. The objective function is defined as: $$\mathcal{L} = \sum_{\tau=1}^T \mathbb{E}_{\mathcal{E}_0, \epsilon_\tau} \left[ \lambda_1 \|\epsilon_\theta(\mathcal{E}_\tau, c, \tau) - \epsilon_\tau\|_2 + \lambda_2 \|\epsilon_\theta(\mathcal{E}_\tau, c, \tau) - \epsilon_\tau\|_1 \right]$$ The diffusion steps are set to $T = 1000$, and the model is optimized for 1000 epochs using the Adam optimizer with a learning rate of $1\times 10^{-4}$ and batch size 64. For downstream augmentation, 30% of the training dataset is randomly sampled and augmented via N2E. Downstream classifiers are trained for 30 epochs with batch size 16 on a single NVIDIA A100 GPU.

Key Experimental Results¶

Main Results¶

The performance of N2E is evaluated across five deep network architectures (ResNet18, ResNet50, VGG16, ViT, Swin) and four event representations (Event Frame, Time Surface, Voxel Grid, Event Spike Tensor) on the N-Caltech101 classification benchmark (101 classes, 8,246 samples) and the DVS-Gesture action recognition benchmark (11 classes, 1,342 samples).

Representative Top-1 accuracy comparisons (in %) are summarized below:

Table 1: Classification accuracy comparison on N-Caltech101 (Excerpts from Table 1 of the paper)

Method	Voxel Grid + RN50	Voxel Grid + Swin	Time Surface + RN18	Time Surface + Swin	Event Frame + RN50
Original	85.8	91.3	75.4	88.7	79.1
DataAug [9]	88.4	91.4	84.3	89.5	83.6
EventDrop [12]	87.2	91.9	82.6	89.7	81.6
EventMix [38]	85.5	92.0	80.2	91.3	76.7
ShapeAug [4]	87.0	88.7	80.5	85.3	80.6
N2E (Ours)	90.5	94.9	88.0	94.3	90.6

Table 2: Action recognition accuracy comparison on DVS-Gesture (Excerpts from Table 2 of the paper)

Method	Voxel Grid + RN18	Voxel Grid + Swin	Voxel Grid + ViT	EST + RN50	Time Surface + RN18
Original	87.5	74.3	63.9	88.5	74.3
DataAug [9]	93.8	87.5	82.3	90.3	80.6
EventDrop [12]	92.0	16.7	80.2	69.1	NaN (collapsed)
EventMix [38]	87.9	79.2	81.6	87.5	71.2
ShapeAug [4]	89.9	71.9	59.4	88.9	72.2
N2E (Ours)	97.6	93.4	83.7	91.0	82.6

Ablation Study & Physical Fidelity Analysis¶

On the MVSEC driving dataset, the paper quantitatively evaluates the physical fidelity of augmented events against ground truth by computing event-level PSNR (ePSNR), root mean square error (RMSE), and frame-level perceptual similarity (LPIPS).

Table 3: Quantitative fidelity evaluation on MVSEC dataset (Table 3 of the paper)

Method	ePSNR (dB) $\uparrow$	RMSE $\downarrow$	LPIPS $\downarrow$	Key Characteristics
DataAug [9]	2.1627	0.3694	0.6342	Fixed spatial shifts; temporal/polarity mismatches
EventDrop [12]	2.1587	0.3700	0.6156	Dropping events corrupts temporal motion flow
EventMix [38]	1.2190	0.5704	0.6839	Spatial blending creates non-physical artifacts
ShapeAug [4]	2.1356	0.3740	0.6355	Synthetic shapes inject artificial optical flows
N2E (T=200, Ours)	11.9747	0.0040	0.1910	Accurate in-distribution generation with consistent optical flow

Key Findings¶

Broad Model Adaptability: N2E consistently outperforms competing augmentation techniques across all combinations of backbones and representations. On N-Caltech101, Swin + Voxel Grid achieves 94.9% accuracy. On DVS-Gesture, ResNet18 + Voxel Grid attains 97.6%, an improvement of +10.1% over the unaugmented baseline.
Robustness Against Negative Transfer: While EventDrop triggers severe training divergence on Time Surfaces (NaN loss) and drops Swin performance to 16.7%, and EventMix degrades ResNet18 performance on Event Frames from 73.7% to 53.0%, N2E delivers consistently positive gains across all settings without destabilizing training.
Superior Optical Flow and Motion Plausibility: On MVSEC, N2E achieves an ePSNR of 11.9747 dB (versus ~2.16 dB for baselines) and slashes RMSE to 0.0040 (nearly two orders of magnitude lower than competitors), confirming that the reverse diffusion process preserves underlying physical motion gradients.

Highlights & Insights¶

Decoupled Quadruplet Tensorization: Resolves the severe event loss and temporal flattening inherent to traditional 2D accumulation grids, providing a generic representation suitable for probabilistic generative modeling.
DDIM Inversion for Distribution Anchoring: By inverting source events into their specific latent codes rather than sampling unconditionally from random Gaussian white noise, N2E confines generation to the local data manifold and prevents class hallucination.
Multi-Factor Controllable Augmentation: Decoupling diffusion steps, latent noise, and condition embeddings enables fine-grained control over global background stochasticity, motion blur, and structural textures.

Limitations & Future Work¶

Computational Overhead: Because N2E relies on iterative multi-step diffusion sampling alongside backward DDIM inversion, the augmentation pipeline is slower than simple random cropping or affine matrix operations.
Hyperparameter Tuning: Determining the optimal diffusion step interval ($\hat{T} \in [50, 200]$) and noise scaling factors requires manual calibration depending on sensor event frequencies.
Future Directions: Developing distilled single-step diffusion variants (e.g., consistency models or flow matching) and simplified latent encoders to enable on-the-fly, real-time event stream augmentation.

vs EventDrop / NDA: Traditional event removal methods cut out event subsets abruptly, severing temporal trajectories. N2E uses diffusion denoising to synthesize smooth, continuous background variations while maintaining physical temporal order.
vs EventMix: EventMix overlays heterogeneous event streams via 3D masks and assigns linearly interpolated labels, often resulting in conflicting semantics. N2E applies self-conditioned perturbations that retain exact ground-truth class labels.
vs ShapeAug: ShapeAug introduces synthetic polygon occlusions that contradict natural optical flow. In contrast, N2E learns the authentic spatiotemporal dynamics directly from real sensor distributions.

Rating¶

Novelty: ⭐⭐⭐⭐☆ [First application of conditional diffusion models to lossless, continuous event stream data augmentation]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive validation across 5 architectures, 4 event representations, 2 recognition benchmarks, and physical motion fidelity metrics]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural organization, rigorous formulation, and well-designed qualitative visualizations]
Value: ⭐⭐⭐⭐☆ [Provides an effective generative solution to event camera overfitting, with open-source code for reproducibility]