Event-based Sparse-view Background-Oriented Schlieren Tomography¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Physics
Keywords: Background-Oriented Schlieren, Event Camera, Neural Implicit Field, Physics-informed Regularization, Sparse-view Tomography
TL;DR¶
Addressing the high data bandwidth and intense illumination demands of frame-based background-oriented schlieren (BOS) tomography, this paper proposes the first event-based sparse-view BOS tomography framework (EventBOS), coupling a continuous 4D neural implicit fluid field with differentiable refractive ray tracing and thermal-convection Navier-Stokes regularization to reconstruct high-speed airflow dynamics that outperform 1000 fps frame-based baselines under single- and orthogonal dual-view setups.
Background & Motivation¶
In scientific and engineering applications such as aerodynamic testing, HVAC ventilation diagnostics, and transient combustion analysis, non-invasive 3D quantitative measurement of high-speed unsteady density fields in transparent media is essential. Conventional optical schlieren techniques image refractive-index variations caused by density fluctuations. Among them, background-oriented schlieren (BOS) has gained widespread adoption due to its simple and low-cost setup, requiring only imaging a textured background through the working fluid and inferring deflection from optical displacement. However, capturing rapid flow dynamics with frame-based BOS (FrameBOS) tomography requires high-speed camera arrays. This introduces severe challenges in microsecond multi-camera synchronization and places enormous burdens on transmission and storage bandwidth. Furthermore, maintaining a large depth of field requires small lens apertures, which demands high-intensity external lighting during short exposure times and restricts practical deployment.
Neuromorphic event cameras provide asynchronous per-pixel brightness change measurements with microsecond temporal resolution, high dynamic range, and low bandwidth, presenting an unprecedented opportunity for continuous schlieren flow sensing under ambient illumination. Nevertheless, porting event cameras to 3D BOS tomography faces two fundamental hurdles. First, event sensors encode temporal logarithmic radiance variations rather than absolute intensity frames. Conventional FrameBOS relies on computing 2D displacement fields between reference and distorted image pairs; under constant illumination, event cameras register only inter-timestamp relative changes, and BOS event streams are sparse and noisy, making direct 2D displacement estimation using standard optical flow unreliable. Second, deploying dense event camera arrays is cost-prohibitive and calibration-intensive, demanding tomographic reconstruction under extreme sparse-view configurations (such as a single view or orthogonal dual views) where line-of-sight refractive projections exhibit severe spatial depth ambiguities.
To circumvent the pitfalls of two-stage pipelines that explicitly estimate 2D displacement fields, this paper develops an end-to-end analysis-by-synthesis inverse-rendering framework. Core idea: represent the spatiotemporal 4D density, velocity, and pressure fields as a continuous neural implicit coordinate network, formulate a differentiable forward model linking non-linear refractive ray tracing with the event generation process, optimize the network directly against measured event accumulation, and incorporate Navier-Stokes thermal-convection physical priors and ambient boundary conditions to resolve ill-posed depth ambiguities under sparse views.
Method¶
Overall Architecture¶
The proposed EventBOS tomography method operates within an analysis-by-synthesis neural inverse rendering paradigm. The framework comprises four tightly coupled components: neural fluid field representation, differentiable refractive ray tracing, event log-brightness increment rendering, and physics-informed regularization. The inputs consist of asynchronous event streams \(\mathcal{E}\) captured by one or two event cameras against a known undistorted background reference image \(I_{\text{ref}}\); the outputs are the continuous 4D density field \(\rho(\mathbf{x}, t)\) along with implicitly inferred velocity \(\mathbf{u}(\mathbf{x}, t)\) and pressure \(p(\mathbf{x}, t)\).
Given a spatiotemporal coordinate \((\mathbf{x}, t)\), the coordinate-based MLP predicts density, velocity, and pressure. Density is linearly transformed into refractive index \(n(\mathbf{x}, t)\) and its gradient \(\nabla n\) via the Gladstone–Dale relation. For any discrete time interval \((t_1, t_2]\), a quasi-linear ray-tracing approximation queries refractive properties along optical paths, rendering the distorted background images \(\hat{I}_{\text{def}}(t_1)\) and \(\hat{I}_{\text{def}}(t_2)\) and their logarithmic intensities. The difference between rendered log-brightnesses is matched directly to the accumulated event image \(E(t_1, t_2)\) via a mean-squared error rendering loss. In parallel, numerical differentiation computes spatial and temporal derivatives to penalize dimensionless residuals of the Navier-Stokes mass, momentum, and heat-convection conservation laws, which together with ambient boundary constraints and spatial total variation (TV) regularize the network end to end.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: 4D coordinates (x, t)<br/>and background reference I_ref"] --> B["Neural Fluid Implicit Field<br/>predict density/velocity/pressure"]
B --> C["Differentiable Refractive Ray Tracing<br/>quasi-linear Hamilton adjoint tracing"]
C --> D["Event Log-Brightness Rendering<br/>match accumulated events with log-diff"]
D --> E["Thermal Convection Physics & Boundary Constraints<br/>Navier-Stokes residuals + ambient calibration"]
E --> F["Output: High-Temporal-Resolution 4D Density<br/>and accurate reprojected optical flow"]
Key Designs¶
1. Neural Fluid Implicit Field & Differentiable Refractive Ray Tracing: Bounded Memory Non-linear Light Deflection
In conventional BOS tomography, spatial voxelization and frame-by-frame discretization cause memory footprint explosions and temporal incoherence. The method models the flow field as a continuous spatiotemporal function \(\mathcal{F}_\theta: (\mathbf{x}, t) \mapsto (\rho, \mathbf{u}, p)\) using a coordinate MLP with sinusoidal positional encodings. Light propagation through inhomogeneous refractive media follows Hamilton's ray equations: $$ \frac{\mathrm{d}\mathbf{x}}{\mathrm{d}s} = \frac{\mathbf{v}(s)}{n(\mathbf{x}(s))}, \quad \frac{\mathrm{d}\mathbf{v}}{\mathrm{d}s} = \nabla n(\mathbf{x}(s)) $$ where \(s\) is the arc-length parameter and \(\mathbf{v}(s) \triangleq n(\mathbf{x})\frac{\mathrm{d}\mathbf{x}}{\mathrm{d}s}\) denotes optical momentum. Numerically integrating these curved trajectories with dense MLP queries would incur unacceptable computational and memory costs. The method adopts a quasi-linear ray-tracing approximation: evaluating refractive properties along a straight-line proxy \(\bar{\mathbf{x}}(\tau) = \mathbf{x}_0 + (\tau - t_n)\mathbf{v}_0\) within the phase object, and computing the integrated deflection angle \(\boldsymbol{\varepsilon} = \frac{1}{n_0}\int \nabla_{\perp} n(\bar{\mathbf{x}}(s))\mathrm{d}s\). Combined with adjoint non-linear ray tracing, the gradient of the rendering loss with respect to the sampled refractive field is computed backwards without storing full ray trajectory histories, maintaining training within a 12 GB GPU memory envelope.
2. Event Log-Brightness Increment Rendering: Bypassing Explicit 2D Optical Displacement
Event sensors asynchronously fire when temporal logarithmic brightness changes \(\Delta L(\boldsymbol{\xi}; t_k, t_{k-1}) = \log I(\boldsymbol{\xi}, t_k) - \log I(\boldsymbol{\xi}, t_{k-1})\) exceed contrast thresholds \(C_{\pm}\). Because schlieren-induced intensity modulations are subtle and corrupted by sensor noise, estimating 2D displacement vectors directly from event streams via traditional optical-flow algorithms yields severe inaccuracies and noise artifacts. This method establishes an end-to-end differentiable mapping from 3D density to 2D accumulated event images. Using BOS camera geometry, image-plane displacement \(\boldsymbol{\delta}(\boldsymbol{\xi}, t) = \frac{M z_{\text{D}}}{\psi_{\text{pix}}}\boldsymbol{\varepsilon}\) is computed to warp the undistorted reference background into a synthesized distorted frame \(\hat{I}_{\text{def}}(\boldsymbol{\xi}, t) = I_{\text{ref}}(\boldsymbol{\xi} + \boldsymbol{\delta}(\boldsymbol{\xi}, t))\), with logarithmic brightness \(\hat{L}(\boldsymbol{\xi}, t) = \log \hat{I}_{\text{def}}(\boldsymbol{\xi}, t)\). Over any time window \((t_1, t_2]\), the rendered log-difference is supervised against the accumulated event image \(E(\boldsymbol{\xi}; t_1, t_2) = \sum_{e_k} p_k C_{p_k} \delta_{\text{D}}(\boldsymbol{\xi} - \boldsymbol{\xi}_k)\): $$ \mathcal{L}{\text{render}} = \mathbb{E}; t_1, t_2) \right)^2 \right] $$ This design completely eliminates intermediate flow estimation errors, allowing the neural implicit field to directly absorb microsecond-level continuous flow dynamics.}, t_1, t_2)} \left[ \left( \big(\hat{L}(\boldsymbol{\xi}, t_2) - \hat{L}(\boldsymbol{\xi}, t_1)\big) - E(\boldsymbol{\xi
3. Physics-informed Navier-Stokes Constraints & Boundary Grounding: Resolving Sparse-View Ambiguity
Under single-view or orthogonal dual-view configurations, recovering 3D fields from line-of-sight integrated gradient measurements is severely ill-posed and susceptible to scale and depth ambiguities. To constrain the solution space, the authors incorporate two physical priors: first, governing equations for low-Mach, nearly isobaric thermal convection in air, comprising mass conservation, momentum Navier-Stokes equations, and heat-transfer conservation: $$ r_{\text{mass}} \triangleq \partial_t \rho + \nabla \cdot (\rho \mathbf{u}), \quad r_{\text{mom}} \triangleq \rho (\partial_t \mathbf{u} + (\mathbf{u}\cdot\nabla)\mathbf{u}) + \nabla p - \mu \nabla^2 \mathbf{u} - (\rho - \rho_0)\mathbf{g} $$ coupled with the ideal gas law \(T(\rho) = \frac{P_0}{R\rho}\) for heat transfer residual \(r_{\text{heat}}\), all non-dimensionalized to balance gradient scales; second, an ambient boundary condition that anchors undisturbed boundary regions \(\partial\Omega\) to ambient air density \(\rho_0\) via \(\mathcal{L}_{\text{bnd}}^{\text{amb}} = \mathbb{E}_{(\mathbf{x}, t)} [(\rho(\mathbf{x}, t) - \rho_0)^2]\). This anchor fixes the DC density offset, providing physical rigidity to reconstruct accurate 3D geometry from minimal viewpoints.
Loss & Training¶
The overall optimization objective minimizes a weighted sum: $$ \mathcal{L} = \lambda_{\text{render}}\mathcal{L}{\text{render}} + \lambda}}\mathcal{L{\text{bnd}} + \lambda}}\mathcal{L{\text{tv}} + \lambda $$ where }}\mathcal{L}_{\text{phy}\(\mathcal{L}_{\text{bnd}} = \mathcal{L}_{\text{bnd}}^{\text{amb}} + w_{\text{in}}\mathcal{L}_{\text{bnd}}^{\text{in}}\) optionally includes inlet velocity/density priors (\(w_{\text{in}} \in \{0, 1\}\)), and \(\mathcal{L}_{\text{tv}} = \mathbb{E}[\|\nabla_\mathbf{x}\rho\|_1]\) promotes spatial smoothness. Loss weights are set to \(\lambda_{\text{render}} = 500, \lambda_{\text{phy}} = 100, \lambda_{\text{bnd}} = 5, \lambda_{\text{tv}} = 1\).
The MLP architecture contains 8 layers with 256 hidden units and SiLU activations for stable higher-order derivative calculations. Positional encoding utilizes 7 spatial and 9 temporal frequency bands with a coarse-to-fine annealing schedule that gradually enables higher frequencies. Optimization is performed using Adam with a learning rate of \(5 \times 10^{-4}\) for 80,000 iterations per scene, sampling 16,384 pixels and 160 ray-tracing steps per ray per iteration. Training requires approximately 12 GB GPU memory and takes about 5 hours on an NVIDIA RTX 3090 GPU.
Key Experimental Results¶
Main Results¶
Quantitative evaluations are conducted on synthetic thermal flow datasets generated using OpenFOAM (buoyantPimpleFoam solver with LES turbulence modeling, recorded at \(5 \times 10^{-4}\text{ s}\) intervals). Four distinct flow dynamics are evaluated: Heated air jet, Hot airflow, Heated air puff, and Cold airflow. Reconstructions are assessed under 1-view and orthogonal 2-view setups using Root Mean Squared Error (RMSE \(\downarrow\)) and Pearson Correlation Coefficient (PCC \(\uparrow\)) against FrameBOS baselines (at 25 fps and 1000 fps) and NeRIF* (a 4D spatiotemporal extension of NeRIF).
| View Setup | Method | Heated air jet (RMSE↓ / PCC↑) | Hot airflow (RMSE↓ / PCC↑) | Heated air puff (RMSE↓ / PCC↑) | Cold airflow (RMSE↓ / PCC↑) |
|---|---|---|---|---|---|
| 1 View | Frame (25 fps) | 0.0243 / 0.912 | 0.0247 / 0.913 | 0.0361 / 0.716 | 0.0267 / 0.898 |
| 1 View | Frame (1000 fps) | 0.0184 / 0.946 | 0.0234 / 0.921 | 0.0154 / 0.825 | 0.0251 / 0.913 |
| 1 View | Ours (Event) | 0.0183 / 0.947 | 0.0229 / 0.928 | 0.0148 / 0.852 | 0.0236 / 0.924 |
| 2 Views | NeRIF* (1000 fps) | 0.0209 / 0.937 | 0.0221 / 0.929 | 0.0133 / 0.884 | 0.0341 / 0.826 |
| 2 Views | Frame (25 fps) | 0.0216 / 0.929 | 0.0199 / 0.940 | 0.0317 / 0.769 | 0.0194 / 0.944 |
| 2 Views | Frame (1000 fps) | 0.0172 / 0.955 | 0.0152 / 0.967 | 0.0129 / 0.899 | 0.0143 / 0.971 |
| 2 Views | Ours (Event) | 0.0149 / 0.967 | 0.0144 / 0.971 | 0.0126 / 0.917 | 0.0141 / 0.972 |
Ablation Study¶
Ablation experiments conducted on the dual-view Hot airflow sequence evaluate the necessity of each constituent term:
| Config | RMSE ↓ | PCC ↑ | Note |
|---|---|---|---|
| Full model (Ours) | 0.0144 | 0.971 | full model with all losses active |
| W/o TV | 0.0152 | 0.967 | drops spatial total-variation regularization |
| W/o Phy | 0.0171 | 0.958 | drops Navier-Stokes physical constraints |
| W/o In | 0.0209 | 0.949 | drops inflow boundary supervision |
| W/o | 0.0242 | 0.915 | drops both inflow and fluid physics priors |
| W/o Rend | 0.0539 | 0.547 | drops event inverse-rendering supervision |
| W/o Bnd | 0.5260 | 0.672 | drops ambient boundary density grounding |
Key Findings¶
- Superiority of Event Streams over High-Speed Frames: Leveraging microsecond temporal resolution, the proposed event-based pipeline consistently surpasses not only 25 fps frame baselines but also 1000 fps high-speed camera inputs. While 25 fps frames suffer from inter-frame dead times that force non-physical interpolation, event cameras densely sample active refraction boundaries, delivering continuous and noise-suppressed density evolutions.
- Critical Importance of Boundary Anchoring and Physics Regularization: Removing the ambient boundary loss (
W/o Bnd) causes the RMSE to jump by an order of magnitude to 0.5260, confirming that line-of-sight gradient integration cannot determine absolute density offsets without a boundary anchor. Similarly, physics-informed regularization (W/o Phy) resolves directional ambiguities along sparse sightlines. - Accurate Reprojected Optical Flow Without Outliers: Reprojecting the 3D density reconstruction to camera image-plane optical flow yields an Average Endpoint Error (AEE) of 0.075 on the Hot airflow sequence, with a zero percentage of outlier pixels (%Out > 1 px = 0.00%). This significantly outperforms the dedicated event-schlieren method EBOS (AEE = 0.660, %Out = 20.35%) and closely matches the classical Farnebäck baseline computed on ideal image pairs.
Highlights & Insights¶
- Unified Differentiable Event-Schlieren Formulation: Seamlessly connects non-linear light refraction and event sensor threshold-trigger mechanisms, directly training a neural field from raw asynchronous polarity streams and avoiding error-prone intermediate 2D displacement estimation.
- Practical Quasi-Linear Adjoint Ray Tracing: By sampling refractive gradients along straight-line proxies and deriving adjoint analytical gradients, the algorithm captures refractive bending while fitting complex 4D neural inverse optimization into a single 12 GB consumer GPU.
- Sparse-View Tomography via Physics Prioirs: Demonstrates that pairing fluid dynamic conservation laws (Navier-Stokes and heat transfer) with neural representations enables high-fidelity 3D reconstruction from only 1 or 2 viewpoints, drastically reducing hardware overhead compared to traditional 6-30 camera rings.
Limitations & Future Work¶
- Reduced Sensitivity to Static or Slow Flows: Because event sensors fire only in response to temporal radiance variations, steady-state or extremely slow fluid dynamics generate very few events, leading to data starvation in the rendering loss.
- Domain-Specific Physical Modeling: The current physics-informed regularizer assumes low-Mach, isobaric thermal convection in air. Adapting to supersonic compressible flows with shock waves, chemical combustion, or multi-species gas mixtures requires formulating alternative conservation models.
- Dependence on Ambient Boundary Conditions: Absolute density recovery relies on an undisturbed ambient boundary with known density \(\rho_0\). In fully enclosed or saturated hot volumes where boundaries are perturbed, absolute value estimations may experience global drift.
Related Work & Insights¶
- vs EBOS (Shiba et al., CVPR 2024): EBOS is restricted to 2D event-based schlieren optical flow estimation and relies on co-located synchronized intensity frames; this work accomplishes full 4D spatiotemporal volumetric tomography without requiring auxiliary intensity frames during dynamic capture.
- vs NeRIF (Hu et al., ICCV 2023): NeRIF performs neural BOS tomography at static time instants using conventional frames; this work extends tomographic reconstruction to time-continuous 4D space-time, replacing high-speed frame cameras with low-bandwidth event sensors and incorporating Navier-Stokes physical regularization.
- vs PINN-BOS (Cai et al., JFM 2021 / Teh et al., 2024): Existing physics-informed BOS approaches rely on multi-view frame cameras or are restricted to steady-state flows; this work unites PINN regularizers with event sensing to tackle unsteady high-speed flows under extreme sparse views.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ The first work to achieve event-based 4D BOS tomography, introducing a novel paradigm for neuromorphic sensor-driven volumetric fluid measurement.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation spanning LES OpenFOAM simulations across four dynamic flows, rigorous loss ablations, optical flow benchmark comparisons, and real-world single- and dual-view experiments.
- Writing Quality: ⭐⭐⭐⭐⭐ Well-structured, mathematically rigorous, and clearly conveying both physical principles and neural rendering nuances.
- Value: ⭐⭐⭐⭐⭐ Provides a practical, cost-effective, and low-bandwidth solution for 3D measurement in aerodynamics, thermal testing, and high-speed flow diagnostics.