EDM:Event-guided Diffusion Model for Video Shadow Detection in Complex Dynamic Scenes

Conference: ECCV 2026
Paper: ECCV Official
Code: None
Area: Video Understanding
Keywords: Video Shadow Detection, Event Camera, Diffusion Model, Motion Trajectory Attention, Dual-Modal Guidance

TL;DR

The first work introducing event cameras into video shadow detection, proposing an Event-guided Diffusion Model (EDM) with motion trajectory attention and bidirectional mask generation, alongside a new real-world benchmark EVSD to conquer severe motion blur and extreme illumination changes.

Background & Motivation

Video shadow detection (VSD) serves as a critical foundational task in computer vision and dynamic scene understanding. While shadows inherently contain geometric cues such as scene layout, depth contours, and light source directions, they frequently cause severe perceptual ambiguities in downstream applications. In autonomous driving and mobile robot perception, vehicle shadows cast across road surfaces are frequently misclassified by semantic segmentation or obstacle detection systems as physical entities, triggering false braking or erratic path planning. In visual object tracking, shadow adhesion to moving targets leads to severe localization drift. Accurately segmenting dynamic shadows under unconstrained environments is therefore vital for robust visual perception.

However, existing video shadow detection methodologies perform reliably only in illumination-stable, low-dynamic scenarios, suffering precipitous performance degradation under high-speed motion or drastic lighting transitions. Conventional frame-based RGB cameras operate under fixed exposure times and limited dynamic range. Consequently, rapid motion induces severe motion blur that obliterates penumbra and umbra boundaries, while strong backlighting, severe underexposure, or extreme overexposure wipes out intensity contrasts, causing dark object surfaces to be readily confused with true shadows. Although event cameras offer microsecond-level temporal resolution, high dynamic range (>120 dB), and absolute resilience against motion blur—making them an ideal complementary modality for tracking rapid shadow deformation—event streams consist strictly of asynchronous binary spikes lacking photometric color and static texture. Simply concatenating raw event streams with RGB feature maps yields erratic gains and can destabilize visual representation learning.

The core tension lies in effectively extracting coherent spatiotemporal motion priors from asynchronous event spikes and establishing fine-grained, cross-modal continuous constraints within a generative denoising paradigm. Core idea: exploit the microsecond temporal resolution of event streams to extract explicit pixel-wise motion displacement fields for cross-frame trajectory attention, synthesize dense intermediate mask priors via bidirectional temporal interpolation, and inject dual-modal guidance into a continuous diffusion denoising process for robust video shadow detection in complex dynamic scenes.

Method

Overall Architecture

The overall pipeline of EDM is organized into two primary stages: coarse preliminary mask prediction (Stage 1) and fine-grained diffusion-based mask prediction (Stage 2). In Stage 1, the network takes \(2i+1\) consecutive RGB frames alongside \(2i\) inter-frame event voxel segments. A lightweight motion encoder extracts 2D displacement fields directly from event slices, after which the Motion Trajectory Attention (MTA) module aggregates multi-frame RGB features along explicit patch trajectories to generate coarse RGB preliminary shadow masks. Next, the Bidirectional Mask Generation (BMG) module estimates intermediate state masks between adjacent frames driven by inter-frame event streams, doubling the temporal guidance resolution. In Stage 2, the raw event streams are reconstructed into intensity frames via Event-based Single Integration (ESI) to predict an event-based preliminary mask sequence. The Dual-Modal Temporal Guidance (DMTG) module then fuses RGB frames, dense interpolated RGB masks, and event masks into unified conditional embeddings. Finally, an Analog Bits diffusion formulation maps discrete segmentation labels into continuous representations, enabling a lightweight DDIM reverse process to denoise and decode the refined shadow mask in just 15 sampling steps.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-frame RGB + Inter-frame Event Streams"] --> B["Motion Trajectory Attention (MTA)<br/>Predict displacement fields and aggregate features along trajectories"]
    B --> C["Preliminary RGB Shadow Mask Prediction"]
    C --> D["Bidirectional Mask Generation (BMG)<br/>Bidirectional recurrent interpolation for dense mask sequences"]
    D --> E["Dual-Modal Temporal Guidance (DMTG)<br/>Integrate ESI-reconstructed event masks with dense RGB masks"]
    E --> F["Diffusion-based Mask Prediction<br/>Analog Bits continuous embedding + DDIM reverse denoising"]
    F --> G["Final Binary Video Shadow Mask"]

Key Designs

1. Motion Trajectory Attention (MTA): explicit cross-frame trajectory aggregation via event displacement fields

Conventional optical flow methods rely on brightness constancy assumptions that fail catastrophically across moving shadow boundaries, severe motion blur, and abrupt illumination shifts. MTA circumvents heavy optical flow computation by directly decoding instantaneous pixel-level motion from asynchronous event streams. A lightweight MotionEncoder comprising 5 stacked \(3\times3\) convolutional blocks with stride 2 extracts event-driven spatial displacement features, which are mapped through a multi-layer perceptron (MLP) to output 2D displacement fields:

\[(\Delta x, \Delta y) = \text{MLP}\Big(\text{MotionEncoder}(E_{t \to t+1})\Big)\]

Using these estimated fields, patch-level trajectories are iteratively traced across all \(N\) frames starting from coordinate \((x_0, y_0)\) in the initial frame, updating coordinates via \((x_{t+1}, y_{t+1}) = (x_t + \Delta x_t(x_t, y_t),\, y_t + \Delta y_t(x_t, y_t))\). For all spatial patch locations, the complete trajectory set \(\tau_{x_0, y_0}\) is constructed. Multi-head self-attention is then executed among patch embeddings lying on the same trajectory. The query vector \(z(x_k, y_k)\) attends strictly to other patch embeddings along its trajectory without additional positional embeddings, forcing the network to maintain invariant identity tracking across non-rigid shadow deformations.

2. Bidirectional Mask Generation (BMG): event-driven sub-frame mask interpolation and error mitigation

During diffusion conditioning, preliminary mask sequences from adjacent frames supply critical temporal boundary constraints. Under high-speed motion, however, large inter-frame spatial displacements degrade temporal continuity and introduce ghosting artifacts into the reverse diffusion trajectory. BMG leverages the sub-millisecond triggering capability of event cameras by utilizing inter-frame event streams \(E_{T+k+1/2}\) to synthesize intermediate preliminary masks between adjacent RGB frames.

To avoid introducing conflicting auxiliary loss gradients into the primary segmentation backbone, BMG adopts a recurrent ECOSNet architecture that is pre-trained independently on the target training split and frozen during main pipeline optimization. Because standard unidirectional LSTM rollouts accumulate drift over extended intervals, BMG implements a bidirectional prediction and fusion scheme: forward sequences are unrolled from the initial frame and backward sequences from the final frame, unified via time-dependent linear blending:

\[M_t^{\text{fusion}} = W_t \otimes M_t^{\text{forward}} + (1 - W_t) \otimes M_t^{\text{backward}},\quad W_t = 1 - \frac{t}{T}\]

Concatenating interpolated masks with preliminary frame predictions yields a temporal guidance sequence \(\{M_{T+k}\}_{k=-i}^i \cup \{M_{T+k+1/2}\}_{k=-i}^{i-1}\) with doubled temporal sampling rate, supplying dense and continuous trajectory priors to guide diffusion denoising.

3. Dual-Modal Temporal Guidance (DMTG): event intensity reconstruction and joint prior conditioning

Directly converting discrete event spikes into voxel grids or time surfaces introduces high-frequency noise and structural artifacts that destabilize diffusion conditioning. DMTG adopts Event-based Single Integration (ESI) to reconstruct local event streams into high-quality absolute intensity images \(E'_{T+k}\), preserving the ultra-wide dynamic range and sharp edge profiles captured by event sensors.

A lightweight ResNet backbone processes these reconstructed intensity images under binary cross-entropy supervision to produce an event-modality preliminary mask sequence \(\{M'_{T+k}\}_{k=-i}^i\). DMTG then encodes the dense RGB mask sequence, the event mask sequence, and original RGB image frames into a unified multi-modal conditioning embedding \(g\), simultaneously supplying semantic appearance, crisp edge geometry, and instantaneous motion cues to govern reverse diffusion.

4. Diffusion-based Mask Prediction: analog bits continuous embedding and fast sampling

Standard diffusion architectures rely on massive U-Net backbones to predict continuous Gaussian noise, incurring prohibitive computational latency on high-resolution dense video segmentation. EDM adopts a direct mask prediction diffusion formulation, leveraging Analog Bits to encode discrete binary segmentation targets \(y \in \{0, 1\}\) into continuous representations. The forward diffusion process systematically injects Gaussian perturbation governed by a predefined cosine schedule:

\[q(z_t \mid z_0) = \mathcal{N}\left(z_t;\, \sqrt{\bar{\alpha}_t}z_0, (1 - \bar{\alpha}_t)\mathbf{I}\right)\]

In the reverse Markov chain, network \(p_\theta(z_{t-1} \mid z_t, x, g)\) conditioned on visual frame \(x\) and DMTG guidance \(g\) iteratively recovers the clean continuous target \(z_0\). Inference utilizes a DDIM sampling strategy that achieves accurate shadow boundary reconstruction in only 15 denoising steps.

Loss & Training

The overall architecture is trained under an end-to-end multi-task objective. The final diffusion shadow mask \(\hat{y}\) is supervised with a combination of Binary Cross-Entropy (BCE) loss and Lovász-Hinge loss to handle foreground-background imbalance and maximize IoU alignment. Concurrently, auxiliary BCE losses supervise the initial RGB mask prediction \(\hat{r}\) and the ESI event mask prediction \(\hat{e}\):

\[\mathcal{L} = \mathcal{L}_{\text{BCE}}(y, \hat{y}) + \mathcal{L}_{\text{hinge}}(y, \hat{y}) + \lambda_1 \mathcal{L}_{\text{BCE}}(r, \hat{r}) + \lambda_2 \mathcal{L}_{\text{BCE}}(e, \hat{e})\]

where balancing weights are fixed to \(\lambda_1 = \lambda_2 = 0.5\). The RGB visual backbone uses MixTransformer-B3, optimized via AdamW with an initial learning rate of \(1 \times 10^{-5}\) and batch size of 2. Input clips contain 3 consecutive RGB frames and synchronized event streams resized to \(512 \times 512\). All models are trained across 8 NVIDIA A6000 GPUs for 18 epochs.

Key Experimental Results

Main Results

EDM was evaluated against 16 state-of-the-art methods spanning Image Shadow Detection (ISD), Video Object Segmentation (VOS), Event-RGB Semantic Segmentation (ERSS), and Video Shadow Detection (VSD) on both the real-world EVSD benchmark and the synthetic Event-ViSha dataset.

Dataset	Task Category	Representative Model	MAE ↓	\(F_\beta\) ↑	IoU ↑	BER ↓
Event-ViSha	ISD	SDDNet (ACM MM'23)	0.040	0.754	0.548	14.05
Event-ViSha	VOS	SAM 3 (2025)	0.031	0.799	0.632	13.47
Event-ViSha	ERSS	CMNeXt (CVPR'23)	0.043	0.773	0.623	9.87
Event-ViSha	VSD	TVSD (CVPR'21)	0.033	0.757	0.567	17.70
Event-ViSha	VSD	Scotch&SODA (2023)	0.029	0.793	0.640	9.07
Event-ViSha	VSD	TBGDiff (ACM MM'24)	0.023	0.797	0.667	8.58
Event-ViSha	VSD	DTTNet (2025)	0.016	0.849	0.718	6.45
Event-ViSha	Ours	EDM (Ours)	0.015	0.861	0.739	5.76
EVSD (Real-World)	ISD	SDDNet (ACM MM'23)	0.076	0.638	0.522	17.78
EVSD (Real-World)	VOS	SAM 3 (2025)	0.046	0.661	0.594	14.97
EVSD (Real-World)	ERSS	CMNeXt (CVPR'23)	0.043	0.673	0.625	13.78
EVSD (Real-World)	VSD	TVSD (CVPR'21)	0.076	0.658	0.584	18.97
EVSD (Real-World)	VSD	Scotch&SODA (2023)	0.073	0.625	0.586	18.05
EVSD (Real-World)	VSD	TBGDiff (ACM MM'24)	0.054	0.644	0.612	16.13
EVSD (Real-World)	VSD	DAS (TCSVT'24)	0.066	0.673	0.601	15.99
EVSD (Real-World)	Ours	EDM (Ours)	0.035	0.724	0.664	11.05

On the newly introduced real-world EVSD benchmark, EDM significantly outperforms the previous state-of-the-art video shadow detection method DAS, achieving a +6.3% absolute IoU improvement (0.664 vs 0.601) and lowering the Balanced Error Rate (BER) by 4.94 points (11.05 vs 15.99).

Ablation Study

Systematic ablations were performed to validate the contribution of core components, motion field estimation mechanisms, patch window sizes, and event representations.

1. Component Contribution Analysis (Table 3):

Config	MTA	BMG	DMTG	Event-ViSha IoU ↑	Event-ViSha BER ↓	EVSD IoU ↑	EVSD BER ↓	Note
Baseline	×	×	×	0.605	11.74	0.563	19.12	Pure static diffusion baseline
EDM w/o MTA	×	✓	✓	0.692	7.35	0.633	13.21	Omitting trajectory attention degrades BER by 2.16
EDM w/o BMG	✓	×	✓	0.706	6.64	0.645	11.98	Omitting intermediate dense mask sequences
EDM w/o DMTG	✓	✓	×	0.663	8.72	0.602	15.64	Omitting dual-modal temporal conditioning
EDM w/o event	-	-	-	0.621	9.88	0.585	17.32	Complete removal of event modality
EDM (Full Model)	✓	✓	✓	0.739	5.76	0.664	11.05	Optimal across all metrics

2. Motion Extraction, Directionality, and Event Representations (Tables 4, 6 & 7):

Experiment Group	Strategy Option	EVSD MAE ↓	EVSD \(F_\beta\) ↑	EVSD IoU ↑	EVSD BER ↓	Key Insight
Motion Field Extraction	RGB-based RAFT	0.048	0.689	0.615	13.97	Conventional flow fails under motion blur
Motion Field Extraction	Event-based E-RAFT	0.047	0.712	0.632	12.86	Flow network is heavy with suboptimal alignment
Motion Field Extraction	Event-based Motion (Ours)	0.035	0.724	0.664	11.05	Lightweight convolution delivers cleaner displacement
BMG Directionality	Unidirectional	0.046	0.707	0.643	11.47	Single-direction rollout accumulates drift
BMG Directionality	Bidirectional (Ours)	0.035	0.724	0.664	11.05	Forward-backward fusion cancels accumulation errors
Event Representation	w/o Event Mask	0.064	0.596	0.618	14.69	Lack of event guidance degrades dark-region boundaries
Event Representation	Voxel-based Mask	0.057	0.632	0.582	17.94	Voxel grid introduces discrete noise artifacts
Event Representation	TimeSurface-based Mask	0.035	0.693	0.652	13.24	Time surface discards absolute intensity dynamic range
Event Representation	ESI-based Mask (Ours)	0.035	0.724	0.664	11.05	Direct intensity integration retains sharp boundaries

Key Findings

• Superiority in Extreme Dynamic Conditions: Scenario-stratified analysis shows that in high-speed motion subsets, EDM achieves an IoU of 0.647 (an 11.5% lead over DAS's 0.532), and 0.682 in low-light subsets (a 10.9% lead over DAS's 0.573), highlighting the power of event streams against motion blur and lighting extremes.
• Critical Role of Dual-Modal Guidance: DMTG provides the single largest performance contribution on EVSD (removing it drops IoU from 0.664 to 0.602, while BER deteriorates by 4.59), confirming that continuous multi-modal constraints are vital for steering diffusion denoising.
• Patch Window Sensitivity: Trajectory tracking with a \(32\times32\) patch window achieves the optimal trade-off between spatial quantization precision and trajectory temporal stability (BER 5.76 vs 6.89 for \(16\times16\) and 6.23 for \(64\times64\)).

Highlights & Insights

• Targeted Multi-Modal Synergy: Converts asynchronous microsecond event spikes into continuous spatial displacement fields (MTA) and sub-frame intermediate guidance (BMG), using the high temporal resolution of event data to counteract RGB motion blur while leveraging RGB semantics to overcome event texture sparsity.
• Flow-Free Motion Trajectory Modeling: Avoids brittle brightness-constancy assumptions by utilizing a 5-layer convolutional encoder to regress 2D motion displacements directly from event slices, improving both inference efficiency and non-rigid deformation tracking.
• Analog Bits Formulation: Employs continuous label embedding to reformulate discrete semantic shadow segmentation as a lightweight diffusion denoising process, generating crisp, highly accurate shadow contours in only 15 DDIM steps.

Limitations & Future Work

• Hardware Co-Registration Dependency: The approach strictly requires precisely synchronized and spatially calibrated RGB-Event sensor setups, limiting immediate plug-and-play deployment on legacy pure-RGB surveillance and dashboard video pipelines.
• Event Sensor Quiescence in Static Scenes: Because event cameras trigger solely on logarithmic temporal brightness changes, in static camera setups with non-moving objects, event streams remain completely silent, reducing the multi-modal advantage.
• Future Direction: Investigating cross-modal semi-supervised distillation to transfer high-temporal event-guided motion priors into single-camera RGB video backbones during training.

Related Work & Insights

• vs TBGDiff (ACM MM'24): While TBGDiff first applied diffusion models to video shadow detection, it operates solely on RGB frames and spatial edge maps, frequently failing under extreme motion blur or low-light overexposure. EDM incorporates physical event streams to eliminate degradation at the sensing source.
• vs DAS (TCSVT'24) & SAM 3: Foundation segmentation models excel at static object delineation but lack physical spatiotemporal continuity constraints across subtle penumbra regions. EDM shows that specialized lightweight event-guided temporal tracking retains distinct advantages in efficiency and fine boundary precision.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneers the introduction of event sensors to video shadow detection with tailored trajectory attention and diffusion conditioning]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Introduces real-world EVSD benchmark alongside Event-ViSha, with comparisons across 16 competitive baselines and deep ablations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural flow, rigorous mathematical explanations, and well-designed experimental figures]
• Value: ⭐⭐⭐⭐⭐ [Provides a crucial breakthrough for autonomous driving and robotic visual perception under extreme lighting and high dynamic motion]