Hybrid Event-Frame Sensors: Unified Noise Modeling, Calibration, and Simulation

Conference: ECCV 2026
arXiv: 2511.18037
Code: https://yunfanlu.github.io/HESIM
Area: Autonomous Driving
Keywords: Hybrid Sensors, Event Cameras, Noise Modeling, Sensor Calibration, H-ESIM

TL;DR

Proposes the first unified statistical noise model and calibration pipeline for hybrid event-frame sensors, and constructs H-ESIM, a simulator capable of generating RAW frames and event streams with noise distributions consistent with real sensors. Its sim-to-real transferability is validated on video frame interpolation and deblurring tasks.

Background & Motivation

Event cameras, due to their asynchronous triggering mechanism, inherently possess high dynamic range (HDR) and microsecond-level low latency, making them more robust in extreme scenarios such as high-speed motion and low light compared to traditional frame-based cameras. However, pure event cameras only output luminance change signals and cannot provide the absolute intensity information of the scene. In contrast, frame-based cameras capture dense spatial intensity information but suffer from motion blur during rapid movement. Harnessing their complementary advantages has long been pursued in computer vision. Traditional approaches use a beam splitter to split the same light beam to both the event camera and the frame camera, but this method requires precise optical alignment, resulting in bulky systems unsuitable for mobile platforms. Recent advances in semiconductor technology have spawned hybrid sensors (hybrid event–frame sensors) that integrate event pixels (EVS) and active pixels (APS) onto a single chip, achieving compact, inherently spatiotemporally aligned dual-modal outputs. However, the shared circuitry layer between APS and EVS introduces complex noise patterns—such as fixed-row noise, dark current drift, photon shot noise, and quantization noise—which are mutually coupled. Existing camera noise models and event simulators (e.g., ESIM, v2e) are designed only for pure frame or pure event cameras and fail to describe the coupled noise in hybrid architectures.

The Key Challenge is that the noise of a hybrid sensor is not a simple superposition of APS and EVS noise. The shared photodiode and readout circuits couple the noise of both modalities across multiple dimensions, including illumination, exposure time, and temperature. Specifically, the fixed pattern noise of APS affects the triggering probability of EVS through the shared bias circuit, while the dark current noise of EVS manifests as additional dark-region non-uniformity in the RAW frames of APS. To accurately simulate this coupling, a unified statistical model that describes both APS and EVS noise behavior is required, along with a calibration method to estimate model parameters from real sensor data. The Key Insight of this work is: since APS and EVS share the same optical signal (scene irradiance), one can start from the physical source and use the shared signal as a bridge to establish separate noise models for both sensors, then estimate each noise component's parameters individually through controlled variable calibration experiments. Core Idea: Propose the first unified statistical imaging noise model for hybrid event-frame sensors, covering photon shot noise, dark current noise, fixed pattern noise, and quantization noise. Based on this, construct a calibration pipeline and the simulator H-ESIM to match the noise statistical characteristics of simulated data highly with those of real hybrid sensors.

Method

Overall Architecture

The overall architecture of H-ESIM is an end-to-end simulation pipeline starting from high-frame-rate video input to the synchronized output of APS RAW frames and EVS event streams. It relies on two pre-requisite modules: a unified noise model and a calibration pipeline. The noise model defines which noise components affect APS and EVS and how these components are represented as parameterizable statistical distributions. The calibration pipeline estimates these model parameters from real sensors via controlled-variable data acquisition (dark field + multi-exposure + multi-illumination). With these prerequisites, H-ESIM simulates along two parallel paths. The APS path passes the input high-frame-rate sRGB video through inverse Gamma correction, inverse color matrix, inverse white balance transform, and Bayer rearrangement, then injects the calibrated fixed noise and illumination/exposure-dependent random noise pixel-by-pixel to output synthetic RAW frames. The EVS path maps the same pixel intensity to the voltage domain, calculates the log-intensity difference, injects noisy random samples, and finally outputs ON/OFF events through threshold comparison. The noise parameters for both paths originate from the same calibration process, ensuring cross-modal noise consistency.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: 3200fps Video (sRGB)"] --> B["Inverse Color Mapping<br/>Gamma → Color Matrix → Bayer"]
    B --> C["APS Simulator"]
    B --> D["EVS Simulator"]
    C --> E["Inject Fixed Noise<br/>N_FP (Row Bias + Black Level)<br/>+ Δt·N_DP"]
    E --> F["Inject Random Noise<br/>Illum/Exp-Dependent Variance<br/>Sample N(0, σ²)"]
    F --> G["Synthetic RAW Frames"]
    D --> H["Intensity-to-Voltage Map<br/>V_hat + V_PD ≈ β₁Ic+β₂"]
    H --> I["Log Difference ΔV<br/>Signal S + Noise Ne"]
    I --> J["Threshold Compare<br/>ΔV > θ → ON<br/>ΔV < -θ → OFF"]
    J --> K["Synthetic Event Stream"]
    G --> L["Downstream Applications<br/>VFI / Deblurring"]
    K --> L

Key Designs

1. Unified Statistical Noise Model: Incorporating APS and EVS Noise into a Unified Framework via the Shared Signal

The APS and EVS of hybrid sensors share the same optical system (lens, aperture), so the ideal electrical signal \(I_c(t;\Delta t)\) is the shared input of both sensors. APS integrates and samples this signal to output RAW frames, while EVS performs log-domain difference-threshold comparisons to output events. The fundamental difference lies in their sampling mechanisms, not the source signal. Consequently, this paper models the APS output as the ideal signal overlaid with three noise terms: illumination-dependent photon shot noise \(N_{\text{shot}}\) (Poisson-to-Gaussian approximation), exposure-time-dependent dark current noise \(N_{\text{time}}\) (comprising linear drift and shot noise), and fixed pattern noise \(N_{\text{fixed}}\) (including row bias, black-level correction, read noise, and quantization noise), yielding \(I_o = I_c + N_{\text{shot}}(I_c) + N_{\text{time}}(\Delta t) + N_{\text{fixed}}\). For EVS, the electrical voltage difference driving the trigger condition \(\Delta V_{t_1} > \theta\) is decomposed into a signal term \(S\) (determined by the ideal voltage) and a noise term \(N_e\) (composed of shot noise, dark current, and fixed offset). The QQ function \(Q(x)=\frac{1}{\sqrt{2\pi}}\int_x^\infty e^{-t^2/2}dt\) is introduced to explicitly relate the event triggering probability \(P_+ = P(\Delta V > \theta) = Q((\theta-(S+\mu_n))/\sigma_n)\) to the illumination and dark current parameters. The key contribution of this model is that it provides the first analytical relationship between EVS noise and scene illumination/dark current, rather than relying on empirically hand-tuned parameters.

2. Controlled-Variable Calibration Pipeline: Decoupling Noise Parameters Layer-by-Layer from Real Sensor Data

Calibration is based on two types of data: dark-field data (fully covered lens) and illumination data (static multi-luminance patterns, such as checkerboards, color charts, and resolution charts). The pipeline operates in two steps: first calibrating the APS noise parameters, and then using them as a foundation to calibrate the EVS noise parameters. During APS calibration, a linear regression on exposure time is performed using the mean of multi-exposure dark-field frames to isolate dark current drift \(N_{\text{DP}}\) and fixed pattern noise \(N_{\text{FP}}\) (further decomposed into row bias \(N_{\text{row}}\) and pixel-wise black level \(N_{\text{BLC}}\)). For illumination data, the calibrated dark components are first used for bias correction. The pixel-wise variance of the corrected frames is then fitted with a second-order polynomial \(\text{Var}(N_a) = \beta_0 + \beta_1 I_c + \beta_2 \Delta t + \beta_3 I_c^2 + \beta_4 I_c \Delta t + \beta_5 \Delta t^2\) using least squares, yielding 16 sets of position-dependent coefficients within the Quad-Bayer array. The core challenge of EVS calibration is cross-domain mapping: mapping the APS intensity domain \(I_c\) to the EVS voltage domain \(\hat{V}_t\). This work employs an affine mapping \(\hat{V}_t + V_{\text{PD}} \approx \beta_1 I_c + \beta_2\) while modeling the EVS noise parameters as \(\sigma_{\text{shot}} = \beta_3 I_c\), \(\sigma_{\text{DCSN}} = \beta_4\), and \(\rho = -\beta_5\). These are substituted into the inverse QQ function \(Q^{-1}(P)\) for gradient descent optimization to estimate the parameter set \(\boldsymbol{\beta}_e\) in a single pass. This calibration pipeline outputs not only noise parameters but also a bad-pixel mask (to handle defective pixels where \(\mu_n \neq 0\)), forming the foundation of subsequent simulation accuracy.

3. H-ESIM Simulator: Calibrated-Parameter-Driven Coordinated Noise Injection for APS and EVS

Once the calibration parameters are obtained, the simulation flow of H-ESIM adopts a "shared-first, branched-later" structure. Shared Stage: Input sRGB frames are mapped to the linear intensity domain via inverse Gamma, inverse color matrix, and inverse white balance, and then rearranged into a single-channel RAW format following the Quad-Bayer pattern. APS Branch: Fixed noise \(N_{\text{FP}} = N_{\text{row}} + N_{\text{BLC}}\) and dark current drift \(\Delta t \cdot N_{\text{DP}}\) are injected first, followed by random noise sampling from the illumination-/exposure-dependent Gaussian distribution according to Eqs. (9) and (10), and finally quantized to the target bit depth to output RAW frames. EVS Branch: The per-pixel intensity \(I_c\) from the shared stage is reused, mapped to voltage \(\hat{V}_t + V_{\text{PD}}\) via affine mapping, and used to calculate the log-difference of successive moments \(\Delta V_{t_1}\). Noise \(N_e \sim \mathcal{N}(\mu_n, \sigma_n^2)\) is superimposed onto the signal term \(S\), which is then compared with the threshold \(\theta\) to decide whether to trigger an event (\(+\!1\) or \(-\!1\)). The use of the same \(I_c\) input for both APS and EVS means that the noise of both outputs remains consistent in terms of illumination dependence—a fundamental difference from methods that simulate separately and then fuse. Moreover, H-ESIM uses ultra-high-frame-rate videos at 3200 fps as input, avoiding the temporal artifacts caused by frame interpolation in ESIM/v2e.

Loss & Training

H-ESIM itself is a simulator and does not involve training loss functions. Downstream networks (such as TimeLens-XL, HR-INR, eSL, EFNet) use their original respective loss functions (e.g., L1 + perceptual loss for VFI, adversarial loss for deblurring) when fine-tuned on simulation data generated by H-ESIM. Parameter estimation in the calibration phase uses least-squares regression (for APS) and gradient descent optimization (for EVS), with the objective function minimizing the discrepancy between the model's predicted noise variance or event trigger probability and the actual measurements.

Key Experimental Results

Main Results

The authors evaluated H-ESIM on two hybrid sensors (AlpsenTek GEN2 and Eiger), measuring the fidelity of Simulated data in two dimensions: temporal accuracy (video frame interpolation VFI) and spatial fidelity (video deblurring). The VFI task adopts a frame-skipping setup (removing intermediate frames and predicting using two adjacent frames + events), while the deblurring task, lacking ground-truth sharp frames, uses no-reference image quality metrics (CLIP-IQA, MUSIQ, NRQM).

Table 1: Video Frame Interpolation Results (Eiger Sensor)

Metric	TimeLens (ESIM)	TL-XL w/o	TL-XL w (H-ESIM)	HR-INR w/o	HR-INR w (H-ESIM)
PSNR↑	29.99	31.93	33.87	32.41	35.24
SSIM↑	0.579	0.841	0.863	0.782	0.886
LPIPS↓	0.404	0.302	0.168	0.198	0.079

On Eiger, HR-INR fine-tuned with H-ESIM shows a PSNR improvement of 2.83 dB compared to the non-fine-tuned version, and LPIPS drops from 0.198 to 0.079, demonstrating that the simulated data effectively bridges the domain gap. Similar trends are observed on GEN2, where HR-INR's PSNR improves from 34.26 to 35.52. The PSNR of TimeLens-XL also increases from 32.30 to 33.87.

Table 2: Deblurring Results (Eiger Sensor, No-Reference Metrics)

Method	Simulator	CLIP-IQA↑	MUSIQ↑	NRQM↑
eSL w/o	[wu2025mipi]	0.307	18.74	4.091
eSL w	+H-ESIM	0.435	22.97	5.899
EFNet w/o	[wu2025mipi]	0.221	16.83	3.797
EFNet w	+H-ESIM	0.425	21.52	5.108

After fine-tuning with H-ESIM, eSL's CLIP-IQA jumps from 0.307 to 0.435 (+42%), and EFNet's improves from 0.221 to 0.425 (+92%), showing that the improvement in the spatial domain from simulated data is equally significant.

Ablation Study

Noise Component Ablation (Qualitative Analysis)

Configuration	Key Findings
Full H-ESIM Model	Highest VFI PSNR and fewest visual artifacts on both sensors
APS Noise Only (No EVS Noise Modeling)	Isolated noise points appear on VFI reconstruction edges, similar to non-fine-tuned TL-XL
EVS Noise Only (No APS Fixed Noise)	Overall brightness shift in the reconstructed image (fixed pattern noise is not compensated)
Using ESIM/v2e (Ignoring Hybrid Sensor Noise)	Lowest deblurring CLIP-IQA (~0.221–0.330) and the largest domain gap

Key Findings

• EVS-APS stacking significantly amplifies fixed pattern noise: Calibration results indicate that row noise in Eiger is much stronger than in GEN2, as Eiger embeds 4 event pixels with color filters within each Quad-Bayer, which causes greater interference to the readout path.
• The effect of illumination on EVS noise can be analytically described using the QQ function: Experiments show that the event triggering probability increases monotonically with brightness (consistent with model predictions), and the ratio of ON/OFF events is slightly biased toward ON at low illumination, shifting with brightness changes at higher illumination—reflecting a non-linear coupling of threshold and brightness, which the model successfully reproduces.
• Physics-based simulation is more effective than pure data-driven simulation: Even when only used for fine-tuning, the improvement in domain transfer brought by H-ESIM is far greater than direct training with more real data, indicating that the model captures the core noise mechanism of hybrid sensors.

Highlights & Insights

• The design choice of using the shared signal as a bridge is highly elegant: Instead of modeling APS and EVS noise separately and performing a hard integration, starting from the source (shared optical signal \(I_c\)) and modeling along separate branches inherently guarantees the internal consistency of the noise parameters. This "one root, two branches" modeling philosophy is a valuable reference for handling any multi-modal sensor fusion problem.
• Introduction of the QQ function for EVS noise analysis: Event triggering is essentially a probabilistic process (voltage noise is always present). Explicitly linking the threshold, noise variance, and event probability using the QQ function is physically and statistically more rigorous than the hand-tuned threshold schemes in ESIM/v2e. This tool can be extended to model any neuromorphic sensor that relies on threshold comparisons.
• 3200 fps high-frame-rate video as input eliminates interpolation artifacts: The low-frame-rate inputs of ESIM/v2e require frame-to-frame interpolation, and the artifacts introduced by interpolation are amplified by the event triggering mechanism. H-ESIM avoids this issue entirely by utilizing high-speed video data, which is a simple yet practical engineering contribution.
• Open-source ISP library: The NumPy-based ISP implementation accompanying H-ESIM (comprising black-level correction, demosaicing, color correction, and Gamma/Tone Mapping) makes the RAW \(\leftrightarrow\) sRGB conversion completely transparent and reproducible, aiding the community in conducting deeper sensor modeling work.

Limitations & Future Work

• Extreme conditions are not covered: The accuracy of the model under extreme high dynamic range edge cases (such as extremely low light, extremely high temperatures, or bandwidth limits) has not been verified. Under these conditions, the Gaussian approximation may fail, and the joint modeling of Poisson distribution and multiplicative noise remains to be addressed in future work.
• Calibration pipeline requires dedicated datasets: Each sensor requires the collection of dedicated datasets representing dark field + multi-luminance + multi-exposure conditions, making the pipeline relatively complex. Designing "one-shot" or "zero-shot" calibration schemes (e.g., relying solely on manufacturer datasheets or a few casual captures) is an important direction for practicality.
• PRNU is not modeled separately: Photo Response Non-Uniformity (pixel-to-pixel sensitivity variation) is absorbed into the position-dependent coefficients \(\boldsymbol{\beta}_a\) rather than being explicitly isolated. For applications requiring precise spatial consistency (e.g., scientific imaging, high-precision 3D), separate modeling of PRNU may be necessary.
• Downstream task validation is concentrated on VFI and deblurring: Sim-to-real transfer on more hybrid sensor tasks (such as HDR reconstruction, optical flow estimation, and SLAM) has not yet been verified. The generalizability of H-ESIM awaits further testing.

Related Work & Insights

• vs ESIM / v2e: ESIM and v2e are pure event camera simulators that generate events using hand-tuned thresholds and frame interpolation. They do not model APS frame noise, nor do they support cross-parameter consistency. H-ESIM is the first simulation framework that unifies APS and EVS within a statistical model.
• vs DVS-Voltmeter / ADV2E / PECS: These physical simulators focus on different physical details of the event-triggering mechanism (e.g., Brownian motion, second-order circuit models) but do not address hybrid sensor architectures and APS-EVS noise coupling. H-ESIM extends the accuracy of physical modeling to multi-modal scenarios.
• vs [wu2025mipi]: The data collection work for the MIPI hybrid sensor dataset provided partial test data for this paper. This work adds noise modeling and simulation capacities on top of it, moving from "data-driven" to "model-driven".
• vs Traditional camera noise models (e.g., [wei2020physics]): Standard camera noise models are typically tailored for RAW frames only. This work extends them to hybrid sensor scenarios, introducing analytical modeling of EVS noise and QQ-function-related calibration methods.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [First to propose a unified statistical noise model, complete calibration pipeline, and end-to-end simulator for hybrid event-frame sensors, filling a gap in the field]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [VFI and deblurring evaluated on two sensors, including quantitative comparisons, ablation studies, and parameter stability analysis; highly thorough validation]
• Writing Quality: ⭐⭐⭐⭐ [Clear framework and complete mathematical derivations, though some notation explanations could be more intuitive]
• Value: ⭐⭐⭐⭐⭐ [Provides theoretical models and reproducible toolchains for hybrid sensor research, directly driving fields that heavily utilize event cameras such as autonomous driving, robotics, and mobile vision]

Related Papers

• [ECCV 2026] HilDA: Hierarchical Distillation with Diffusion for Advancing Self-Supervised LiDAR Pre-training
• [ECCV 2026] FLM-Occ: Feed-forward Likelihood Maximization for Efficient Indoor Occupancy Prediction
• [ECCV 2026] LaGen: Towards Autoregressive LiDAR Scene Generation
• [ECCV 2026] DriveVA: Video Action Models are Zero-Shot Drivers
• [ECCV 2026] ASTAD: 非对称风格迁移实现自动驾驶合成到真实域的适应