Skip to content

High-speed Imaging through Turbulence with Event-based Light Fields

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Project: Project Page
Area: Computational Photography / Physics-based Vision (physics)
Keywords: Event Camera, Atmospheric Turbulence, Light Field, High-Speed Video Reconstruction, Kaleidoscope Optics

TL;DR

Addressing the severe motion blur and geometric wobble when imaging dynamic scenes through atmospheric turbulence, this work presents the first event-only light field imaging system. By exploiting kaleidoscope optics to capture 9 sub-aperture views, it physically decouples cross-view correlated scene motion from uncorrelated turbulence perturbations, enabling crystal-clear video reconstruction at up to 12,000 fps under strong turbulence.

Background & Motivation

Long-range terrestrial imaging—crucial for astronomy, remote sensing, and long-distance surveillance—is fundamentally impaired by atmospheric turbulence. Spatial and temporal fluctuations in air temperature and index of refraction introduce severe optical aberrations, resulting in geometric distortion, random image wander, and anisoplanatic blur. Conventional turbulence mitigation frameworks, ranging from hardware adaptive optics (AO) to software-based lucky imaging and deep multi-frame restoration networks, inherently depend on conventional frame-based sensors. These sensors integrate incident photons over a fixed exposure window, which irrecoverably blends microsecond-scale atmospheric fluctuations and smears fast-moving targets into intractable motion blur.

Event cameras offer an attractive alternative by recording logarithmic brightness changes asynchronously at microsecond temporal resolution, theoretically eliminating exposure-induced motion blur. However, applying a single event camera to dynamic scenes observed through turbulence encounters an intractable physical ambiguity. In an event sensor, both high-speed object motion and turbulent refractive index fluctuations trigger brightness variations, generating an indistinguishable mixture of event streams. In mathematical terms, scene radiance and turbulent point spread functions (PSFs) are multiplicatively coupled in a bilinear forward model. A single viewpoint cannot disambiguate whether an event was induced by true object dynamics or spurious atmospheric perturbation without relying on restrictive rigid-body assumptions. Prior hybrid efforts combining events with conventional intensity frames remain tethered to the low frame rate of standard sensors.

This paper tackles this fundamental bottleneck by introducing computational light field optics. When observing a scene at long distances, sub-aperture parallax is negligible, ensuring that target radiance remains practically identical across all viewpoints. In stark contrast, light rays entering different sub-apertures traverse spatially distinct atmospheric columns, each subject to an independent turbulence realization and phase aberration. Core idea: capture simultaneous sub-aperture views via a single-sensor kaleidoscope event light field camera, disentangling cross-view correlated scene dynamics from weakly correlated turbulence perturbations through channel-stacked recurrent neural video reconstruction to achieve tens-of-thousands fps clean video.

Method

Overall Architecture

The complete end-to-end event light field turbulence mitigation system integrates a custom optical pickup stage with a neural temporal reconstruction pipeline. On the optical side, light from a distant dynamic target propagates through anisoplanatic atmospheric turbulence and is gathered by an 800 mm telephoto objective lens. A pupil-matching relay lens couples the exit pupil of the main lens into the entrance pupil of a 4-mirror kaleidoscope. The kaleidoscope folds the optical paths through internal planar reflections, forming a non-overlapping \(3 \times 3\) grid of 9 distinct sub-aperture virtual views onto a single Prophesee EVK4 event sensor. On the algorithmic side, the asynchronous event stream from each sub-aperture is discretized into a spatio-temporal voxel grid. The 9 voxel grids are concatenated along the channel dimension and fed into a recurrent convolutional encoder-decoder (an extended multi-view E2VID), which recursively aggregates multi-view temporal consensus and produces clean, distortion-free intensity frames.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Distant Dynamic Scene S(u, t)"] --> B["Anisoplanatic Turbulence Medium<br/>Spatially Independent Phase Error h^(i)"]
    B --> C["Telephoto Lens + Pupil-Matching Relay"]
    C --> D["Kaleidoscope Optical Splitting<br/>Path Folding into 3x3 Sub-apertures"]
    D --> E["Single Event Sensor Capturing 9 Streams E^(i)"]
    E --> F["Spatio-Temporal Voxelization & Channel Stacking<br/>5 Bins per View into 45-Channel Tensor"]
    F --> G["Multi-View Recurrent Conv Encoder-Decoder<br/>ConvLSTM Accumulating Spatial Consensus"]
    G --> H["High-Speed Restored Video (Up to 12,000 fps)"]

Key Designs

1. Turbulent Light Field Bilinear Decoupling Model: Exploiting Cross-View Correlation to Disentangle Motion from Wobble

Under anisoplanatic turbulence, the instantaneous incoherent intensity recorded at the \(i\)-th sub-aperture follows: $\(I^{(i)}(\mathbf{x}, t) = \int_{\Omega} |h^{(i)}_\mathbf{u}(\mathbf{x}, t)|^2 S(\mathbf{u}, t) \, d\mathbf{u}, \quad i = 1, \dots, N\)$ where \(S(\mathbf{u}, t)\) is the scene radiance and \(h^{(i)}_\mathbf{u}(\mathbf{x}, t)\) is the complex coherent point spread function (PSF) along the optical path. In a monocular observation, \(S\) and \(|h|^2\) are bilinearly intertwined, making inversion ill-posed. Expanding to \(N\) simultaneous sub-apertures breaks this impasse: because parallax is negligible at long range, every view observes the exact same scene \(S(\mathbf{u}, t)\). However, because rays heading into separate sub-apertures traverse distinct atmospheric columns, each view experiences an uncorrelated instantaneous PSF \(h^{(i)}\). Consequently, event spikes triggered by object motion show robust cross-view spatio-temporal correlation, whereas event spikes caused by local atmospheric turbulence are spatially and temporally uncorrelated across views. This multi-view redundancy provides the fundamental physical constraint required to isolate genuine scene motion.

2. Single-Sensor Kaleidoscope Light Field Architecture: Eliminating Multi-Camera Synchronization Latency

Building a light field camera array using separate event sensors introduces prohibitive physical bulk, high cost, and severe vulnerability to microsecond-level clock drift and geometric calibration errors. The authors devise a compact, single-sensor kaleidoscope optical system. Light captured by a Canon RF 800 mm f/11 objective lens passes through a 100 mm pupil-matching relay lens into a 305 mm-long rectangular hollow kaleidoscope with a 12.6 mm entrance aperture. Internal specular reflections segment the main lens pupil, projecting a \(3 \times 3\) array of sub-aperture views side-by-side onto the active area of a single Prophesee EVK4 event sensor. A beam splitter positioned behind the kaleidoscope simultaneously routes identical rays to a Blackfly S CMOS camera for reference capture. Imaging all 9 views onto a single silicon die completely eliminates inter-sensor hardware synchronization drift and temporal jitter.

3. Multi-View Channel-Stacked Recurrent Reconstruction Network: Implicitly Learning Turbulence Statistics

To convert the light field optical advantage into high-fidelity video, the authors adapt the recurrent E2VID architecture with minimal overhead. For each sub-aperture stream \(\mathcal{E}^{(i)}\), events within a temporal window \(\Delta T_k\) are discretized into a voxel grid \(E_k^{(i)} \in \mathbb{R}^{B \times H \times W}\) across \(B=5\) temporal bins. The \(N=9\) voxel grids are stacked along the channel dimension into a unified representation \(\tilde{E}_k \in \mathbb{R}^{(N \cdot B) \times H \times W}\) (a 45-channel tensor). The encoder uses early convolutional layers to cross-compare the 9 views, identifying spatial consensus to filter out uncorrelated turbulence events while amplifying coherent object motion. ConvLSTM modules in the bottleneck iteratively update recurrent hidden states to enforce temporal consistency and suppress frame-to-frame wander. Spectral normalization is applied to all ConvLSTM layers, preventing numerical instability and high-frequency artifacts from accumulating over extended rollouts.

Loss & Training

The network is trained end-to-end on synthetic turbulence data minimizing the Learned Perceptual Image Patch Similarity (LPIPS) loss: $\(\mathcal{L} = \frac{1}{K} \sum_{k=1}^K \operatorname{LPIPS}(\hat{V}_k, V_k)\)$ LPIPS is chosen over standard mean squared error (MSE) because MSE penalizes minor spatial shifts excessively, producing blurry reconstructions. Key training configurations include: - Optimization: Trained from scratch for 200 epochs with batch size 8 using the Adam optimizer with a fixed learning rate of \(10^{-4}\) on a single NVIDIA RTX A6000 GPU. - Data Augmentation: Employs random \(112 \times 112\) spatial crops and horizontal flips (\(p=0.5\)); injects Gaussian noise \(\mathcal{N}(0, 0.4)\) into voxel grids and simulates hot-pixel noise \(\mathcal{N}(0, 0.1)\) on up to 0.01% of pixels; applies pause augmentation (\(p=0.05\) to zero input events, holding state with \(p=0.9\)), training the ConvLSTM cells to sustain clear frame outputs during static intervals without event triggers.

Key Experimental Results

Main Results

Quantitative evaluations were performed on synthetic sequences generated from the REDS dataset using the physics-based P2S turbulence simulator (Table 1), as well as real-world tabletop experiments observing an LCD monitor through heat-induced turbulence (Table 2). Baselines include single-view E2VID trained on the central sub-aperture (\(256 \times 256\)) and an equal-pixel single-view baseline (\(768 \times 768\)). Evaluation metrics comprise PSNR, SSIM, LPIPS, Inter-frame Transformation Fidelity (ITF, assessing temporal stability), and warping error (\(E_{\text{warp}}\)).

Table 1: Quantitative results on simulated dataset (REDS + P2S turbulence simulator) (Source: Table 1 in paper)

| Method | Resolution / Configuration | PSNR (dB) ↑ | SSIM ↑ | LPIPS ↓ | ITF (dB) ↑ | \(E_{\text{warp}}\) ↓ | Note | |---|---|---|---|---|---|---| | Single-view | \(768 \times 768\) (matched total pixels) | 14.84 | 0.501 | 0.5185 | 22.71 | 0.0050 | High res, but plagued by turbulence distortion | | Single-view | \(256 \times 256\) (center view) | 14.84 | 0.461 | 0.4607 | 22.72 | 0.0047 | Baseline single viewpoint | | Light field (Ours) | \(3 \times 3 \times 256 \times 256\) | 15.40 | 0.505 | 0.4332 | 25.16 | 0.0022 | Substantial stability gain: ITF +2.45 dB, \(E_{\text{warp}}\) cut by >50% |

Table 2: Quantitative results on tabletop experiment (REDS video through heat turbulence) (Source: Table 2 in paper)

| Method | PSNR (dB) ↑ | SSIM ↑ | LPIPS ↓ | ITF (dB) ↑ | \(E_{\text{warp}}\) ↓ | Note | |---|---|---|---|---|---| | Single-view | 12.37 | 0.351 | 0.543 | 20.87 | 0.0085 | Edges warped, spatial dimming artifacts | | Light field (Ours) | 12.99 | 0.367 | 0.531 | 23.30 | 0.0051 | Maintains straight trajectories; \(E_{\text{warp}}\) reduced by 40% |

Ablation Study & High-Speed Physical Verification

To verify that the performance gains stem from multi-view angular diversity rather than simple pixel count, and to evaluate high-speed limits, the authors conducted specific ablation comparisons:

Dimension / Test Case Configuration Comparison Key Metrics & Observed Phenomena Insight & Mechanism
Light Field vs Equal-Pixel Single View \(3 \times 3 \times 256^2\) vs \(1 \times 768^2\) LPIPS: 0.4332 vs 0.5185; \(E_{\text{warp}}\): 0.0022 vs 0.0050 Even when single-view has 9× more spatial pixels, its temporal stability is far inferior. Confirms that performance gains stem from angular diversity sampling uncorrelated phase errors, not spatial oversampling.
Motion Blur Mitigation vs SOTA Ours vs MambaTM (ECCV 2024 SOTA) Fast motion synthesis (4-frame accumulation) MambaTM suffers severe motion blur and catastrophic structural distortion under frame accumulation; our event light field ingests unintegrated microsecond spikes, restoring crisp edges.
Rotating Reflective Stripe (600 fps) Single-view vs Light field \(\theta-t\) spatio-temporal slice linearity Single-view produces an oscillatory trajectory due to unmitigated turbulence; light field yields a straight, linear trajectory reflecting constant angular velocity.
Hypersonic Nerf Dart (12,000 fps) Ours vs $100k High-Speed Camera (Photron UX100) Dart speed: 16,000 px/s; Reconstruction: 12,000 fps Photron UX100 (4,000 fps) requires 2–45× intense illumination and still exhibits motion blur; our system reconstructs a clean, straight trajectory at 12,000 fps under standard lighting.

Key Findings

  • Angular diversity decisively outperforms spatial pixel count: Even when matched to the same aggregate pixel count (\(768 \times 768\)), single-view models exhibit high warping error (\(E_{\text{warp}} = 0.0050\)) and poor perceptual scores (LPIPS = 0.5185). The \(3 \times 3\) light field drives \(E_{\text{warp}}\) down to 0.0022 and boosts ITF by +2.45 dB, confirming that uncorrelated wavefront sampling across multiple sub-apertures is the true driver of turbulence disambiguation.
  • Event streams unlock high-speed imaging without excessive light: In the 16,000 pixel/second Nerf dart test, conventional high-speed cameras (\(100k, 1.5 kg, 2-second buffer) suffered from illumination starvation and motion smearing. The event light field camera (\)5k, 40 g sensor) operated continuously without exposure limits, reconstructing sharp 12,000 fps frames under modest laboratory lighting.

Highlights & Insights

  • Elegant physical synergy between event sensing and light fields: Exploiting the physical reality that long-range targets exhibit negligible parallax while atmospheric phase errors decorrelate rapidly across sub-apertures, the authors formulate a clean, blind inverse solution without requiring complicated wavefront reconstructions.
  • Single-sensor kaleidoscope engineering simplicity: Sidestepping the micro-lens manufacturing tolerances and inter-camera microsecond clock synchronization challenges, the 4-mirror kaleidoscope delivers 9 perfectly synchronized sub-aperture views onto one event chip.
  • New paradigm for through-medium high-speed sensing: This methodology demonstrates that asynchronous neuromorphic sensing coupled with angular light field diversity can overcome severe optical scattering and turbulence, offering immense promise for airborne defense, autonomous driving in fog/smoke, and deep-tissue intravital microscopy.

Limitations & Future Work

  • Author-admitted limitations: The system is constrained by the native resolution of event sensors (~1 MP), which is divided across the \(3 \times 3\) sub-apertures, leaving each view at modest spatial resolution (\(256 \times 256\)). The optical infinity assumption breaks down for near-range biomedical imaging with significant parallax. In addition, the current model has an inference latency of 3.4 ms at \(176 \times 176\) resolution, necessitating model compression for ultra-high-resolution real-time deployment.
  • Potential blind spots: The model is trained exclusively on synthetic data (REDS + P2S); extreme real-world lighting conditions or unmodeled event sensor non-idealities (e.g., refractory period variations, acoustic vibrations altering kaleidoscope geometry) may degrade performance. Furthermore, sub-aperture registration relies on static pre-calibration with a homography, which could drift under thermal expansion or structural flex in field environments.
  • vs EvTurb (CVPR 2024) / EGTM (CVPR 2024): EvTurb and EGTM rely on hybrid inputs (frames + events) and are bounded by the low frame rate and exposure window of conventional CMOS sensors. In contrast, this paper is entirely event-only, reaching up to 12,000 fps.
  • vs MambaTM (ECCV 2024): MambaTM processes conventional frame sequences with state space models; when targets move rapidly, exposure motion blur causes severe reconstruction failure. The event light field operates directly on asynchronous spikes, entirely avoiding integration blur.
  • vs Hardware Adaptive Optics (AO): Standard AO requires expensive wavefront sensors, deformable mirrors, and bright guidestars (natural or laser). This system operates completely guidestar-free and passively at kilohertz-to-ten-kilohertz frame rates with an order-of-magnitude lower cost.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ (Pioneering fusion of event cameras and light field optics for through-turbulence dynamic video reconstruction)
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Exemplary experimental validation: synthetic turbulence, indoor flame/heater testbed, 120 fps to 12,000 fps video, and head-to-head comparison with a $100k high-speed camera)
  • Writing Quality: ⭐⭐⭐⭐⭐ (Crisp physical formulation, elegant motivation, restrained mathematical notation, and compelling spatio-temporal visualizations)
  • Value: ⭐⭐⭐⭐⭐ (Substantial contribution to computational imaging, neuromorphic vision, and remote sensing in scattering media)