Hybrid Event-Frame Sensors: Unified Noise Modeling, Calibration, and Simulation

Conference: ECCV 2026
arXiv: 2511.18037
Code: https://yunfanlu.github.io/HESIM
Area: Autonomous Driving
Keywords: Hybrid Sensors, Event Cameras, Noise Modeling, Sensor Calibration, H-ESIM

TL;DR

Proposes the first unified statistical noise model and calibration pipeline for hybrid event-frame sensors, and constructs H-ESIM, a simulator capable of generating RAW frames and event streams with noise distributions matching real sensors, validating its sim-to-real transfer capabilities on video frame interpolation and deblurring tasks.

Background & Motivation

Due to their asynchronous triggering mechanism, event cameras naturally possess high dynamic range (HDR) and microsecond-level low-latency characteristics, making them more robust than conventional frame cameras in extreme scenarios like high-speed motion and low light. However, pure event cameras can only output brightness change signals, failing to provide absolute spatial intensity information. Frame cameras are the exact opposite—they capture dense spatial intensity information but suffer from motion blur during fast movements. Combining their complementary advantages has long been pursued in computer vision. Traditional approaches use a beam splitter to divide the same light path onto an event camera and a frame camera, but this requires precise optical alignment, resulting in bulky systems unsuitable for mobile platforms. Recent semiconductor manufacturing advances have enabled hybrid sensors (hybrid event-frame sensors): integrating event pixels (EVS) and active pixels (APS) onto a single chip, achieving compact, naturally spatially-aligned dual-modality output. However, the shared circuitry between APS and EVS introduces complex noise patterns—fixed row noise, dark current drift, photon shot noise, and quantization noise are mutually coupled. Existing camera noise models and event simulators (such as ESIM, v2e) are only designed for pure frame or pure event cameras, failing to describe the coupled noise in hybrid architectures.

The core challenge is that the noise of hybrid sensors is not a simple superposition of APS and EVS noise. The shared photodiode and readout circuits couple the noise from both modalities across multiple dimensions (e.g., illumination, exposure time, temperature). For instance, the fixed pattern noise of APS affects the triggering probability of EVS through the shared bias circuitry, while the dark current noise of EVS manifests as additional dark non-uniformity in APS RAW frames. To accurately simulate this coupling, a unified statistical model is needed to describe the noise behavior of both APS and EVS, along with a calibration method to estimate model parameters from real sensor data. The key insight of this paper is: since APS and EVS share the same optical signal (scene irradiance), one can start from the physical source, using the shared signal as a link to build individual noise models for both sensors, and then isolate the parameters of each noise component through controlled variable calibration experiments. Core Idea: Propose the first unified statistical imaging noise model for hybrid event-frame sensors, covering photon shot noise, dark current noise, fixed pattern noise, and quantization noise. Based on this, build a calibration pipeline and the H-ESIM simulator, making the noise statistical characteristics of the simulated data highly consistent with real hybrid sensor outputs.

Method

Overall Architecture

The overall architecture of H-ESIM is an end-to-end simulation pipeline starting from high-frame-rate video input and synchronously outputting APS RAW frames and EVS event streams. It relies centrally on two preceding modules: the unified noise model and the calibration pipeline. The noise model defines which noise components affect APS and EVS, and how these components are represented as parameterizable statistical distributions. The calibration pipeline estimates these model parameters from real sensors via controlled-variable data collection (dark field + multi-exposure + multi-illumination). With these foundations, H-ESIM's simulation runs in two parallel paths. The APS path takes the input high-frame-rate sRGB video, applies inverse Gamma, inverse color matrix, inverse white balance, and Bayer rearrangement, injects calibrated fixed pattern noise and light/exposure-dependent random noise pixel-by-pixel, and outputs synthetic RAW frames. The EVS path maps the same pixel intensities to the voltage domain, calculates log-intensity differences, injects noise-parameterized random samples, and ultimately outputs ON/OFF events through threshold comparison. Using noise parameters derived from a single calibration pipeline for both paths ensures cross-modal noise consistency.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: 3200fps Video (sRGB)"] --> B["Inverse Color Mapping<br/>Gamma → Color Matrix → Bayer"]
    B --> C["APS Simulator"]
    B --> D["EVS Simulator"]
    C --> E["Inject Fixed Noise<br/>N_FP(Row Bias+BLC)<br/>+ Δt·N_DP"]
    E --> F["Inject Random Noise<br/>Light/Exposure-dep Variance<br/>Sample N(0,σ²)"]
    F --> G["Synthetic RAW Frame"]
    D --> H["Intensity→Voltage Mapping<br/>V_hat + V_PD ≈ β₁Ic+β₂"]
    H --> I["Log Difference ΔV<br/>Signal S + Noise Ne"]
    I --> J["Threshold Comparison<br/>ΔV > θ → ON<br/>ΔV < -θ → OFF"]
    J --> K["Synthetic Event Stream"]
    G --> L["Downstream Apps<br/>VFI / Deblurring"]
    K --> L

Key Designs

1. Unified Statistical Noise Model: Harmonizing APS and EVS Noise under a Shared Signal Framework

Hybrid sensor's APS and EVS share the same optical system (lens, aperture); hence, the ideal electrical signal \(I_c(t;\Delta t)\) is the shared input to both streams. APS integrates and samples this signal to output RAW frames, while EVS performs log-domain difference-threshold comparison to output events. Their fundamental difference lies in the sampling mechanism rather than the signal source. Based on this, the APS output is modeled as the ideal signal superimposed with three noise terms: illumination-dependent photon shot noise \(N_{\text{shot}}\) (Poisson-to-Gaussian approximation), exposure-time-dependent dark current noise \(N_{\text{time}}\) (containing linear drift and shot noise), and fixed pattern noise \(N_{\text{fixed}}\) (including row bias, black level correction, read noise, and quantization noise), yielding \(I_o = I_c + N_{\text{shot}}(I_c) + N_{\text{time}}(\Delta t) + N_{\text{fixed}}\). For EVS, the threshold condition \(\Delta V_{t_1} > \theta\) is decomposed into a signal term \(S\) (determined by ideal voltage) and a noise term \(N_e\) (composed of shot, dark current, and fixed offset noise). A QQ-function \(Q(x)=\frac{1}{\sqrt{2\pi}}\int_x^\infty e^{-t^2/2}dt\) is introduced to explicitly connect the event trigger probability \(P_+ = P(\Delta V > \theta) = Q((\theta-(S+\mu_n))/\sigma_n)\) with illumination and dark current parameters. The key contribution of this model is that it provides, for the first time, an analytical relationship between EVS noise, scene brightness, and dark current, rather than relying on empirical manual parameters.

2. Controlled-Variable Calibration Pipeline: Layer-by-Layer Separation of Noise Parameters from Real Sensor Data

Calibration is based on two types of data: dark field data (fully covered lens) and illuminated data (static multi-brightness patterns like checkerboards, color charts, and resolution charts). The pipeline is divided into two steps: calibrating the APS noise parameters first, followed by calibrating the EVS noise parameters on this basis. During APS calibration, linear regression is performed on exposure times utilizing the mean of multi-exposure dark field frames to isolate the dark current drift \(N_{\text{DP}}\) and fixed pattern noise \(N_{\text{FP}}\) (further decomposed into row bias \(N_{\text{row}}\) and pixel-wise black level \(N_{\text{BLC}}\)). For illuminated data, bias correction is first applied using calibrated dark components. Then, a second-order polynomial \(\text{Var}(N_a) = \beta_0 + \beta_1 I_c + \beta_2 \Delta t + \beta_3 I_c^2 + \beta_4 I_c \Delta t + \beta_5 \Delta t^2\) is fitted using least squares on the pixel-wise variance of corrected frames, yielding 16 groups of position-dependent coefficients within the Quad-Bayer grid. The core challenge of EVS calibration is cross-domain mapping: the APS intensity domain \(I_c\) must be mapped to the EVS voltage domain \(\hat{V}_t\). An affine mapping \(\hat{V}_t + V_{\text{PD}} \approx \beta_1 I_c + \beta_2\) is employed, while modeling the EVS noise parameters as \(\sigma_{\text{shot}} = \beta_3 I_c\), \(\sigma_{\text{DCSN}} = \beta_4\), and \(\rho = -\beta_5\). These are substituted into the inverse QQ-function form \(Q^{-1}(P)\) for gradient descent optimization, estimating the parameter set \(\boldsymbol{\beta}_e\) in a single run. This calibration pipeline yields not only noise parameters but also a bad pixel mask (handling defective pixels where \(\mu_n \neq 0\)), forming the foundation of subsequent simulation accuracy.

3. H-ESIM Simulator: Calibration Parameter-Driven Collaborative Noise Injection for APS and EVS

Once calibration parameters are obtained, the simulation flow of H-ESIM follows a "shared-first, branched-later" structure. In the shared phase, input sRGB frames are mapped to the linear intensity domain via inverse Gamma, inverse color matrix, and inverse white balance, and then rearranged into single-channel RAW formats following the Quad-Bayer pattern. In the APS branch, fixed noise \(N_{\text{FP}} = N_{\text{row}} + N_{\text{BLC}}\) and dark current drift \(\Delta t \cdot N_{\text{DP}}\) are injected first, followed by random noise sampled from light/exposure-dependent Gaussian distributions according to Eqs. (9)(10), before quantizing to bit depth to output RAW frames. In the EVS branch, the per-pixel intensity \(I_c\) from the shared phase is reused, yielding voltage \(\hat{V}_t + V_{\text{PD}}\) via affine mapping, calculating the log difference \(\Delta V_{t_1}\) of adjacent time steps, superimposing noise \(N_e \sim \mathcal{N}(\mu_n, \sigma_n^2)\) onto the signal term \(S\), and finally comparing against the threshold \(\theta\) to trigger events (\(+\!1\) or \(-\!1\)). Utilizing the same \(I_c\) input for both APS and EVS ensures that the noise of both outputs remains consistent in their illumination dependence—an essential difference from methods that simulate separately and then fuse. Furthermore, H-ESIM utilizes high-frame-rate 3200 fps videos as input, avoiding temporal artifacts caused by frame interpolation seen in ESIM/v2e.

Loss & Training

As a simulator, H-ESIM does not involve training loss functions. Downstream networks (such as TimeLens-XL, HR-INR, eSL, EFNet) use their respective original loss functions (e.g., L1 + perceptual loss for VFI, adversarial loss for deblurring) when fine-tuned on the simulated data generated by H-ESIM. Parameter estimation during calibration utilizes least-squares regression (for APS) and gradient descent optimization (for EVS), with target functions aiming to minimize the error between model-predicted noise variance/event trigger probability and measured values.

Key Experimental Results

Main Results

The authors evaluated H-ESIM on two hybrid sensors (AlpsenTek GEN2 and Eiger), measuring simulated data fidelity from both the temporal (video frame interpolation, VFI) and spatial (video deblurring) dimensions. The VFI task adopts a frame-skipping setup (removing intermediate frames and predicting with adjacent frames + events). Due to the lack of ground-truth sharp frames, the deblurring task uses no-reference image quality metrics (CLIP-IQA, MUSIQ, NRQM).

Table 1: Video Frame Interpolation Results (Eiger Sensor)

Metric	TimeLens (ESIM)	TL-XL w/o	TL-XL w (H-ESIM)	HR-INR w/o	HR-INR w (H-ESIM)
PSNR↑	29.99	31.93	33.87	32.41	35.24
SSIM↑	0.579	0.841	0.863	0.782	0.886
LPIPS↓	0.404	0.302	0.168	0.198	0.079

On Eiger, HR-INR fine-tuned with H-ESIM achieves a 2.83 dB PSNR gain over the non-fine-tuned counterpart, and LPIPS drops from 0.198 to 0.079, demonstrating that the simulated data effectively bridges the domain gap. Results on GEN2 follow a consistent trend, with HR-INR PSNR rising from 34.26 to 35.52. TimeLens-XL's PSNR also improves from 32.30 to 33.87.

Table 2: Deblurring Results (Eiger Sensor, No-Reference Metrics)

Method	Simulator	CLIP-IQA↑	MUSIQ↑	NRQM↑
eSL w/o	[wu2025mipi]	0.307	18.74	4.091
eSL w	+H-ESIM	0.435	22.97	5.899
EFNet w/o	[wu2025mipi]	0.221	16.83	3.797
EFNet w	+H-ESIM	0.425	21.52	5.108

After fine-tuning with H-ESIM, eSL's CLIP-IQA jumps from 0.307 to 0.435 (+42%), and EFNet increases from 0.221 to 0.425 (+92%), showing that spatial improvements brought by the simulated data are equally significant.

Ablation Study

Noise Component Ablation (Qualitative Analysis)

Configuration	Key Findings
Full H-ESIM model	Highest VFI PSNR and fewest visual artifacts on both sensors
Only APS noise (no EVS noise modeling)	Isolated noise spots appear around reconstructed edges in VFI, resembling non-fine-tuned TL-XL
Only EVS noise (no APS fixed noise)	Overall brightness shift in reconstructed images (fixed pattern noise is uncompensated)
Using ESIM/v2e (ignoring hybrid sensor noise)	Lowest deblurring CLIP-IQA (~0.221~0.330), largest domain gap

Key Findings

• EVS-APS stacking significantly amplifies fixed pattern noise: Calibration results indicate that row noise on Eiger is much stronger than on GEN2. This is because Eiger embeds 4 event pixels with color filters within each Quad-Bayer grid, causing larger interference in the readout path.
• The impact of illumination on EVS noise can be analytically characterized using the QQ-function: Experiments display that event triggering probability monotonically increases with brightness (consistent with model predictions), and the ON/OFF event ratio slightly leans towards ON under low light while varying with brightness under high light—reflecting non-linear threshold-brightness coupling, which the model successfully reproduces.
• Physics-based simulation is more effective than pure data-driven simulation: Even when only used for fine-tuning, the domain adaptation gains of H-ESIM outperform direct training with larger quantities of real data, suggesting the model captures the core noise mechanism of hybrid sensors.

Highlights & Insights

• Using a shared signal as a link is a highly elegant design choice: Instead of modeling APS and EVS noise separately and merging them, the authors start from the source (the shared optical signal \(I_c\)) for parallel modeling. This naturally guarantees intrinsic consistency in noise parameters. This "one root, two branches" modeling philosophy is a valuable reference for addressing any multi-modal sensor fusion problem.
• Introducing the QQ-function into EVS noise analysis: Event triggering is fundamentally a probabilistic process (voltage noise is always present). Expressly correlating threshold, noise variance, and event probability via the QQ-function is physically and statistically more rigorous than the manual threshold tuning in ESIM/v2e. This tool can be extended to model any neuromorphic sensor relying on threshold comparisons.
• 3200 fps high-frame-rate video input eliminates interpolation artifacts: The low-frame-rate inputs of ESIM/v2e require temporal interpolation, and artifacts from interpolation are amplified through the event-triggering mechanism. H-ESIM avoids this issue by directly employing high-speed video data, which represents a simple yet practical engineering contribution.
• Open-source ISP library: H-ESIM's accompanying NumPy ISP implementation (including black level correction, demosaicing, color correction, and Gamma/Tone Mapping) makes the RAW \(\leftrightarrow\) sRGB conversion completely transparent and reproducible, aiding the community in conducting deeper sensor modeling work.

Limitations & Future Work

• Extreme conditions uncovered: The accuracy of the model under extreme boundaries of high dynamic ranges (e.g., extremely low light, very high temperatures, bandwidth limitations) remains unverified. Under these conditions, the Gaussian approximation may fail, prompting future work to jointly model Poisson distributions and multiplicative noise.
• Calibration pipeline requires dedicated datasets: Each sensor requires collecting a dedicated dataset of dark fields, multi-brightness, and multi-exposure, which makes the pipeline complex. Designing a "zero-shot" calibration scheme (e.g., relying solely on factory parameters or a few casual captures) remains an important direction for practicality.
• PRNU not modeled separately: Photo Response Non-Uniformity (pixel-to-pixel sensitivity variation) is absorbed into the position-dependent coefficients \(\boldsymbol{\beta}_a\) rather than being explicitly isolated. For applications requiring precise spatial consistency (e.g., scientific imaging, high-precision 3D), individual modeling of PRNU may be necessary.
• Downstream validation focused on VFI and deblurring: Sim-to-real transfer has yet to be verified in other hybrid sensor tasks (e.g., HDR reconstruction, optical flow estimation, SLAM). H-ESIM's generalizability warrants further testing.

Related Work & Insights

• vs ESIM / v2e: ESIM and v2e are pure event camera simulators that generate events using manual thresholds and frame interpolation. They do not model APS frame noise or support noise parameter consistency. H-ESIM is the first simulation framework to unify APS and EVS under a statistical model.
• vs DVS-Voltmeter / ADV2E / PECS: These physical simulators focus on different physical details of event-triggering mechanisms (e.g., Brownian motion, second-order circuit models) but omit hybrid sensor architectures and APS-EVS noise coupling. H-ESIM extends the fidelity of physical modeling to multi-modal scenarios.
• vs [wu2025mipi]: The collection efforts of the MIPI hybrid sensor dataset provided some test data for this work. This paper gains noise modeling and simulation capabilities on top of it, transitioning from "data-providing" to "model-driven".
• vs Traditional Camera Noise Models (such as [wei2020physics]): Standard camera noise modeling typically targets RAW frames only. This work extends it to hybrid sensor scenarios, introducing analytical modeling for EVS noise and QQ-function-related calibration.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [First to propose a unified statistical noise model, complete calibration pipeline, and end-to-end simulator for hybrid event-frame sensors, filling a gap in the field]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations of VFI and deblurring performed on two sensors, including quantitative comparisons, ablation analyses, and parameter stability analysis, validating the approach thoroughly]
• Writing Quality: ⭐⭐⭐⭐ [Clear framework and comprehensive equation derivations, though annotations for some symbols could be more intuitive]
• Value: ⭐⭐⭐⭐⭐ [Provides theoretical models and reproducible toolchains for hybrid sensor research, directly advancing fields heavily using event cameras, such as autonomous driving, robotics, and mobile vision]

Related Papers

• [ECCV 2026] HilDA: Hierarchical Distillation with Diffusion for Advancing Self-Supervised LiDAR Pre-training
• [ECCV 2026] FLM-Occ: Feed-forward Likelihood Maximization for Efficient Indoor Occupancy Prediction
• [ECCV 2026] LaGen: Towards Autoregressive LiDAR Scene Generation
• [ECCV 2026] DriveVA: Video Action Models are Zero-Shot Drivers
• [ECCV 2026] ASTAD: 非对称风格迁移实现自动驾驶合成到真实域的适应