Event-driven Motion Deblurring via Trajectory-based Kernel Reconstruction

Conference: ECCV 2026
Paper: ECCV Official
Area: Image Restoration
Keywords: Image Deblurring, Event Camera, Trajectory Kernel Reconstruction, Deep Unfolding Network, Physics-based Vision

TL;DR

To overcome the severe ill-posedness of non-uniform motion deblurring, this paper estimates dense pixel-wise motion trajectories directly from event streams via a differentiable alignment objective, constructs physically grounded spatially varying point spread function (PSF) operators, and solves the joint deblurring problem using an ADMM-inspired deep unfolding network that achieves state-of-the-art restoration quality with low latency.

Background & Motivation

Single-image motion blur in dynamic scenarios is a ubiquitous degradation caused by relative motion between the camera and scene objects during the exposure window. In classical image deblurring, traditional optimization formulations predominantly rely on the assumption of spatially invariant blur, where a single blur kernel is shared across the entire image. However, in real-world scenes, 6-DoF camera shakes, non-rigid object movement, and abrupt depth discontinuities render the blur kernel spatially varying across individual pixels. This spatial variability significantly amplifies the ill-posed nature of blind deconvolution. While recent deep learning frameworks based on CNNs, Transformers, and diffusion models have made remarkable progress, most rely solely on blurry RGB frames. Because an identical blurry observation can originate from countless combinations of latent sharp images and spatially varying kernels, purely data-driven models without reliable exposure-time motion cues struggle with severe ambiguity, frequently generating over-smoothed textures, ringing artifacts, and physical inconsistencies.

Event cameras (or Dynamic Vision Sensors) present an appealing neuromorphic paradigm to resolve this fundamental bottleneck. Unlike standard frame-based sensors that aggregate light over a fixed integration period, event cameras record log-intensity brightness changes asynchronously at microsecond-level temporal resolution with high dynamic range and virtually zero motion blur. Because the continuous event stream captures high-frequency texture transitions along the true motion trajectory without temporal blur, it inherently preserves the physical motion field within the exposure interval.

Despite this potential, existing event-guided deblurring approaches exhibit two major shortcomings. One branch treats events purely as generic supplementary feature modalities and fuses them with frame features in an end-to-end black-box network, neglecting the underlying physical imaging relationship between event generation and blur formation. The second branch attempts to incorporate physical constraints but relies on oversimplified assumptions, such as uniform blur or linear constant-velocity trajectories, which collapse under complex dynamic scenes with depth variations and non-rigid rotations. Core idea: estimate dense pixel-wise motion trajectories directly from the event stream via a differentiable alignment objective, explicitly construct physically grounded spatially varying point spread function (PSF) operators, and solve the joint inverse problem within an ADMM-inspired deep unfolding network alternating between data-consistency updates and learned image priors.

Method

Overall Architecture

The proposed non-uniform motion deblurring framework comprises two synergistic systems: a physically grounded event-driven blur kernel estimation module and an ADMM-inspired unrolled optimization network. The complete restoration pipeline processes the observed blurry image \(B\) and the corresponding event stream \(E\) triggered during exposure \(T\). First, the system estimates a dense continuous motion field across the image plane by optimizing a differentiable event-alignment objective that compensates event coordinates to maximize edge sharpness. Next, continuous temporal trajectories are discretized and accumulated onto local kernel grids via bilinear splatting, yielding explicit per-pixel point spread functions (PSFs) that define the spatially varying degradation operator \(\mathcal{K}(K)\). Finally, deblurring is formulated as an energy minimization problem enforcing physical blur fidelity, event-guided anisotropic structural consistency, and deep image regularization. An ADMM unrolled network translates this joint optimization into multi-stage iterative updates, alternating between data consistency and learned image priors to reconstruct the latent sharp image \(S\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Blurry image B & Event stream E"] --> B["Differentiable event-alignment motion estimation<br/>Motion compensation & sharpness maximization"]
    B --> C["Spatially varying PSF kernel reconstruction<br/>Bilinear splatting of discrete trajectory points"]
    C --> D["Anisotropic structural consistency modeling<br/>Local neighborhood difference edge aggregation"]
    D --> E["ADMM deep unfolding iterative deblurring<br/>USM data fidelity + UZM learned prior + UUM dual update"]
    E --> F["Output: Multi-stage learnable weighted sharp image"]

Key Designs

1. Differentiable event-alignment motion estimation: inferring dense physical motion fields from events

Conventional optical flow methods fail catastrophically under severe blur, whereas asynchronous event streams capture sharp high-frequency motion cues. An event \(e_k = (\mathbf{x}_k, t_k, p_k)\) with polarity \(p_k \in \{-1, +1\}\) is triggered when the log-intensity variation reaches a contrast threshold \(C\). Assuming a continuous motion field \(v(\mathbf{x}, t)\) during exposure, any event can be motion-compensated to a reference time \(t_0\): $$ \tilde{\mathbf{x}}k = \mathbf{x}_k + \int}^{t_0} v(\mathbf{xk, \tau) \, d\tau $$ When the estimated velocity field accurately matches the true scene dynamics, compensated events originating from the same physical boundary align in spatial coordinates, maximizing the contrast and spatial gradient of the motion-compensated event image. The method formalizes this intuition by maximizing the sum of squared image gradients under an \(L_1\) motion smoothness regularizer \(\mathcal{R}_v(v) = \|\nabla v\|_1\) to suppress noise in textureless regions: $$ \hat{v} = \arg\min_v \left( -\sum_k) \right|^2 + \lambda_v |\nabla v|_1 \right) $$ Solving this differentiable objective yields pixel-wise continuous velocity vectors without requiring restrictive constant-velocity or planar motion assumptions.}} \left| \nabla \sum_k p_k \delta(\mathbf{x} - \tilde{\mathbf{x}

2. Spatially varying PSF kernel reconstruction: mapping discrete trajectories to physical degradation operators

Optical blur results from the temporal integration of the latent sharp scene \(S\) along the spatial motion trajectory \(\Phi_t(\mathbf{x})\) over exposure interval \(T\). To derive a tractable discrete operator, the exposure time is uniformly partitioned into \(M\) temporal samples \(\{t_m\}_{m=1}^M\). Driven by the estimated motion field, the coordinate displacement of each pixel is tracked, forming a discrete spatial trajectory. The local point spread function \(k_{\mathbf{x}}(\mathbf{d})\) for pixel \(\mathbf{x}\) at offset \(\mathbf{d}\) is defined by: $$ k_{\mathbf{x}}(\mathbf{d}) = \frac{1}{M} \sum_{m=1}^M \delta\left(\mathbf{d} - (\Phi_{t_m}(\mathbf{x}) - \mathbf{x})\right) $$ During implementation, sub-pixel displacements are splatted onto a discrete \(5 \times 5\) kernel grid via bilinear interpolation and normalized to enforce energy conservation \(\sum_{\mathbf{d}} k_{\mathbf{x}}(\mathbf{d}) = 1\). The resulting kernel field \(K = \{k_{\mathbf{x}}\}\) establishes the forward degradation operator \(\mathcal{K}(K)S\), faithfully reproducing the spatially varying blurring process \(B(\mathbf{x}) = \sum_{\mathbf{d}} k_{\mathbf{x}}(\mathbf{d}) S(\mathbf{x} + \mathbf{d}) + n(\mathbf{x})\).

3. Anisotropic structural consistency modeling: cross-modal high-frequency edge alignment

Relying solely on the blur fidelity term \(\|B - \mathcal{K}(K)S\|_F^2\) leaves room for high-frequency artifacts due to inversion ill-posedness and minor kernel estimation noise. Fortunately, the event stream also contains unblurred structural geometry. To inject this structural prior without inheriting sensor noise, the framework introduces an anisotropic structural response operator \(\mathcal{A}(\cdot)\). For any 2D input \(X\), its response at pixel \(i\) aggregates local differences over a neighborhood \(\mathcal{N}(i)\): $$ \mathcal{A}(X)i = \frac{1}{|\mathcal{N}(i)|} \sum (X_j - X_i) $$ This operator selectively emphasizes edge-aligned structural variations while canceling uncorrelated shot noise. Minimizing the structural discrepancy }(i)\(\|\mathcal{A}(S) - \mathcal{A}(E)\|_F^2\) forces the restored image to align its structural edges with the event stream, providing high-frequency spatial anchors.

4. ADMM deep unfolding iterative deblurring: interleaving physical forward models with learned priors

To overcome the expressiveness limitations of handcrafted regularizers (such as Total Variation) and the physical opacity of pure CNNs, the optimization formulation is split using an auxiliary variable \(Z\) and a scaled dual variable \(U\). The augmented Lagrangian is defined as: $$ \mathcal{L}_\rho(S, Z, U) = |B - \mathcal{K}(K)S|_F^2 + \mu |\mathcal{A}(S) - \mathcal{A}(E)|_F^2 + \lambda \mathcal{R}_S(Z) + \frac{\rho}{2} |S - Z + U|_F^2 $$ The problem is unrolled into \(N=6\) iterative stages, each executing three sub-modules: - USM (Update S Module): Performs a gradient descent step on \(S\) governed by the forward and adjoint physical kernel operators \(\mathcal{K}\) and \(\mathcal{K}^\top\), event consistency, and the quadratic penalty: \(S^{k+1} = S^k - \eta^k \nabla_S \mathcal{L}_\rho\), where \(\eta^k\) is a learnable step size; - UZM (Update Z Module): Implements the proximal mapping \(Z^{k+1} = \mathrm{prox}_{\frac{\lambda}{\rho} \mathcal{R}_S}(S^{k+1} + U^k)\) parameterized by a lightweight convolutional encoder-decoder network, replacing manual priors with deep image statistics; - UUM (Update U Module): Updates the dual variable algebraically via \(U^{k+1} = U^k + S^{k+1} - Z^{k+1}\).

The final reconstruction aggregates all intermediate stage predictions using learnable weights: \(\hat{S} = \sum_{i=0}^N w_i S_i\), realizing progressive, interpretable artifact removal.

Loss & Training

The complete unrolled network is trained end-to-end using the Charbonnier loss between the aggregated prediction \(\hat{S}\) and the ground-truth sharp image \(S^*\): $$ \mathcal{L}_{rec} = \sqrt{|\hat{S} - S^*|^2 + \epsilon^2} $$ with \(\epsilon = 10^{-3}\). Training is executed on four NVIDIA RTX 5090 GPUs using AdamW (\(\beta_1 = 0.9, \beta_2 = 0.99\)) with an initial learning rate of \(2 \times 10^{-4}\) decayed via cosine annealing. Inputs are randomly cropped into \(256 \times 256\) patches with a batch size of 8. Default hyper-parameters are fixed to \(M=8\) temporal bins, kernel size \(K=5 \times 5\), and \(N=6\) unrolled stages.

Key Experimental Results

Main Results

The framework is rigorously benchmarked across synthetic (GoPro), semi-synthetic (HS-ERGB), and real-world (REBlur) datasets. All competing algorithms are retrained from scratch on HS-ERGB and REBlur to eliminate pretraining bias.

Dataset	Metric	Ours	Prev. SOTA	Gain
GoPro [30]	PSNR (dB) SSIM	37.15 0.978	36.78 (CMTA-7) / 36.70 (Zhu et al.) 0.978 (MAT) / 0.977 (CMTA-7)	+0.37 dB Matched/Best
HS-ERGB [44]	PSNR (dB) SSIM	29.43 0.818	28.97 (MAT) / 28.11 (FFTFormer) 0.816 (MAT) / 0.813 (FFTFormer)	+0.46 dB +0.002
REBlur [40]	PSNR (dB) SSIM	37.24 0.971	36.97 (MAT) / 36.78 (MAENet) 0.969 (MAT) / 0.967 (MAENet)	+0.27 dB +0.002

Note: Against purely frame-based models, Restormer obtains 32.92 dB and Concertormer achieves 34.42 dB on GoPro; incorporating event-derived physical modeling grants our method a performance margin exceeding 2.7 dB.

Ablation Study

Component ablations conducted on the GoPro dataset validate the necessity of the physics-based blur operator and event structural guidance:

Config	Event-derived Blur Operator	Event Consistency	PSNR (dB)	SSIM	Note
Model 1 (Standard Conv)	\(\times\)	\(\checkmark\)	36.25	0.976	Replacing \(\mathcal{K}(K)\) with learnable standard convolution drops PSNR by 0.90 dB
Model 2 (w/o Event Consistency)	\(\checkmark\)	\(\times\)	36.60	0.977	Setting \(\mu=0\) removes structural supervision, reducing PSNR by 0.55 dB
Full Model (Ours)	\(\checkmark\)	\(\checkmark\)	37.15	0.978	Joint physical PSF operator and anisotropic event structural prior

Key hyper-parameter observations: - Temporal Bins \(M\): Increasing \(M\) from 2 to 8 steadily raises PSNR from ~34.5 dB to 37.15 dB, proving that dense temporal trajectory sampling is vital for accurate blur kernel modeling; performance saturates beyond \(M=8\). - Kernel Support \(K \times K\): Kernels of size \(1 \times 1\) and \(3 \times 3\) fail to capture large displacements; \(5 \times 5\) provides the optimal trade-off, with marginal returns observed for larger kernel footprints. - Unrolled Stages \(N\): Reconstruction fidelity improves progressively from stage 2 to 6; beyond \(N=6\), detail restoration converges while network parameters scale linearly.

Key Findings

• Explicit physics outperforms learned convolutions: The sharp drop of 0.90 dB in Model 1 confirms that generic convolution cannot approximate spatially varying blur kernels; explicit trajectory-driven PSFs are the primary source of competitive advantage.
• Favorable computational efficiency: Benchmarked on an NVIDIA RTX 5090 with \(480 \times 480\) resolution, the proposed network executes in ~50 ms with under 10M parameters, running substantially faster than heavy vision Transformer baselines such as MAT and Concertormer.
• Robustness in dynamic real-world environments: On real event captures in REBlur and HS-ERGB, reconstructed PSFs exhibit diverse spatial orientations that adapt to complex non-rigid motion, effectively eliminating ghosting artifacts.

Highlights & Insights

• Repurposing event data as a physical PSF generator: Rather than treating event streams as passive feature channels, the method leverages motion compensation sharpness principles to reconstruct the exact optical blur integration path.
• Lightweight proximal prior network: By offloading physical degradation modeling to explicit data-consistency operators, the proximal network (UZM) requires only a compact encoder-decoder, avoiding parameter bloat.
• Noise-tolerant edge transfer: The anisotropic difference operator extracts clean geometric edge responses from noisy asynchronous event streams without transferring high-frequency sensor noise into the restored image.

Limitations & Future Work

• Performance in textureless low-light scenes: When scene illumination is low or surfaces lack texture, event firing becomes extremely sparse. In such regimes, the event-alignment sharpness loss lacks sufficient gradient signal, potentially biasing estimated kernels toward isotropic spatial smoothness.
• Rigid spatio-temporal calibration reliance: The physical trajectory integration formulation assumes precise synchronization and spatial co-registration between the event sensor and frame-based camera.
• Future directions: Extending the framework to handle severe dynamic occlusions, non-coaxial hardware setups, and ultra-lightweight real-time edge deployment.

Related Work & Insights

• vs End-to-end event fusion networks (e.g., EFNet, MAENet, MAT): Prior learning methods use generic multi-scale feature concatenation or attention-based cross-modal fusion without explicit blur modeling. In contrast, this work establishes a rigorous mathematical formulation of the degradation process and embeds it into deep unfolding, achieving higher accuracy with fewer parameters.
• vs Classical physical event-based deblurring (e.g., Nakabayashi et al.): Existing physics-guided methods assume spatially invariant blur or linear constant motion. This work models spatially varying blur with non-rigid, pixel-wise trajectories and arbitrary PSF orientations, unlocking real-world applicability.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant physical formulation linking microsecond event trajectories to spatially varying PSFs within an ADMM unrolled network.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across three synthetic and real datasets with exhaustive hyper-parameter ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical derivation, clear architectural flow, and thorough qualitative analysis.
• Value: ⭐⭐⭐⭐⭐ Provides an exemplary blueprint for integrating physical degradation models with deep learning in neuromorphic computational photography.