Depth-guided Multi-view Exposure Bracketing for HDR Robot Vision

Conference: ECCV2026
Paper: Official page · PDF
Project: https://divisonofficer.github.io/dmeb
Area: Robot Vision / HDR Imaging
Keywords: Multi-view exposure bracketing, metric depth, confidence-aware fusion, tone mapping, robotic perception

TL;DR

DMEB distributes different exposures across synchronized cameras and uses external depth to align and fuse complementary observations for single-shot HDR reconstruction, achieving 39.40 dB PSNR-µ with three cameras on real modest-DR data while introducing evaluation data covering 121 real scenes and 20 synthetic videos.

Background & Motivation

When robots encounter headlights, shadows, and bright skies, ordinary cameras often retain information in only part of the brightness range. Traditional exposure bracketing captures short, medium, and long exposures sequentially with one camera before merging them into HDR. Once objects or the camera move, however, the same image location can represent different content across frames. Optical flow, attention, and alignment networks can reduce this mismatch, but they require matchable texture across exposures: widening the exposure gap preserves more highlights and shadows while making saturated and low-signal regions harder to match.

Synchronized multi-camera capture removes the inter-frame motion component but still introduces parallax. Existing multi-camera HDR methods that find correspondences through appearance must likewise restrict exposure differences. This paper exploits the depth sensors already present on robots and some phones, assigning correspondence estimation to geometry so that camera exposures can cover a wider dynamic range. The authors also identify a missing benchmark configuration: synchronized, strongly differently exposed, calibrated multi-view imagery paired with depth. The dataset and the reference reconstruction system are therefore equally important contributions.

Core idea: use external metric depth to remove the dependence of cross-exposure matching on visual similarity, let synchronized cameras cover different brightness ranges, and choose reliable observations using exposure confidence and geometric visibility.

Method

Overall Architecture

The inputs are synchronized multi-view low-bit-depth images, camera exposure parameters, calibration, and aligned depth measurements; the output is an HDR image in a selected reference-camera view. The system first updates current exposures from the previous frame, then performs depth-confidence fusion, and finally applies multi-tone-map refinement to residual errors. Here, single-shot means that the images used for the current HDR estimate come from simultaneous multi-camera observations. It does not mean that exposure control ignores previous frames or that the system uses only one camera.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    previous["Previous images and exposures"] --> exposure["Multi-view exposure control"]
    exposure --> capture["Synchronized multi-exposure images"]
    capture --> fusion["Depth-confidence fusion"]
    depth["Depth measurements and calibration"] --> fusion
    fusion --> refinement["Multi-tone-map refinement"]
    refinement --> output["Reference-view HDR"]

Key Designs

1. Multi-view exposure control: assign cameras to different brightness ranges

Rather than automatically exposing every camera to the same mean brightness, the system assigns geometrically spaced target means between normalized intensities of 0.05 and 0.8. Each camera compares its previous-frame mean with its target, updates exposure proportionally, and clips the result to hardware limits. Cameras with lower brightness targets preserve intense light sources, while cameras with higher targets capture shadows. Feedback also lets the exposures adapt as illumination changes.

For camera \(i\), the paper's exposure update can be written as:

\[ \xi_i\leftarrow\operatorname{clip}\!\left(\xi_i\frac{m_i}{m_i^*},\xi_{\min},\xi_{\max}\right). \]

Here, \(m_i\) is the target mean, \(m_i^*\) is the measured previous-frame mean, and \(\xi_i\) is exposure. Geometrically spaced target means do not imply that actual shutter times always follow a fixed ratio, since view contents and measured intensities can differ. Exposure is then decomposed into time and gain: exposure time is limited by \(\tau_{\max}\), with gain increased when needed and capped by \(g_{\max}\). The main paper gives approximately 50 ms as the maximum stable exposure time for an 8 FPS acquisition configuration. This acquisition rate should not be confused with the network inference rates reported later.

2. Depth-confidence fusion: geometry aligns observations and reliability selects them

External depth measurements do not usually cover every RGB pixel. DMEB first projects raw measurements into each camera view, then uses a pretrained Depth Anything V2 monocular prior to obtain dense depth. Sparse measurements provide metric-scale constraints through scale estimation. Thus, the method does not directly treat relative monocular depth as physical distance: measurements anchor the scale, while the prior fills in spatial structure. Each view obtains a dense depth map to support geometric reprojection.

Correct alignment alone does not make a pixel useful: one observation may be saturated, while another may be dominated by underexposure noise. The authors multiply an explicit intensity weight by a learned weight:

\[ C_i=f_{\mathrm{trapezoid}}(I_i)\,f_{\mathrm{CNN}}(I_i,\widehat D_i). \]

The trapezoidal intensity function suppresses excessively dark or bright observations, while the CNN combines image and dense-depth information to estimate more complex unreliability. Reference-view dense depth and calibration then warp source images, depth maps, and confidences into the reference view. The system first takes a confidence-weighted average of the aligned depths to obtain fused depth, then checks whether each projected depth agrees with that fused geometry. Geometric alignment and pixel reliability have distinct roles; neither replaces the other.

Soft visibility follows the paper's depth-consistency formulation:

\[ V_{i\to t}=\sigma\!\left(\frac{(\dot D_t+\delta)-\widehat D_{i\to t}}{\tau_{\mathrm{rel}}(\dot D_t+\epsilon)}\right)+w_{\mathrm{floor}}. \]

Here, \(\dot D_t\) is fused depth, \(\widehat D_{i\to t}\) is reprojected source depth, and \(\sigma\) is the logistic function. The margin \(\delta=1\,\mathrm{m}\) absorbs cross-view misalignment, \(\tau_{\mathrm{rel}}\) controls relative-depth tolerance, \(\epsilon\) prevents division by zero, and \(w_{\mathrm{floor}}=0.05\) avoids completely hard-cutting contributions. A source surface substantially behind the fused surface receives less weight. This is a soft fusion weight, not a probability guaranteed to lie strictly between 0 and 1.

HDR fusion additionally normalizes source intensities by exposure time and gain to bring differently exposed observations onto a comparable radiometric scale, then performs a normalized weighted combination using confidence multiplied by soft visibility. Invalid projections and geometrically unreliable observations make only restricted contributions; the operation cannot create information for non-overlapping regions. Equation (5) is misaligned in the cached text extraction, so this note retains its operational meaning rather than assembling an unverified full equation from corrupted typography.

3. Multi-tone-map refinement: present an extreme brightness range at manageable scales

Initial fusion can retain depth-boundary misalignment and local artifacts. Moreover, the large range of linear HDR values makes different brightness regions difficult to process under a single compression scale. The authors apply three µ-law tone mappings to the same fused estimate, with \(\mu_1=10^3\), \(\mu_2=5\times10^4\), and \(\mu_3=10^6\), and use a Transformer to combine and refine them. These are different compressed representations of the same radiometric information, not three additional captures or another cross-view correspondence estimator.

The network result is returned to the linear intensity domain through inverse µ-law mapping to produce the final HDR image. The main paper delegates the confidence CNN architecture, refinement-network internals, and additional fusion details to supplementary material. The local cache contains only the main paper, so layer counts, channel widths, attention configurations, and inverse-mapping branch details are not supplied here.

A Worked Example

Consider three cameras observing headlights and a vehicle in shadow. The paper's target-mean rule gives targets of 0.05, 0.2, and 0.8; this is a worked derivation of the rule, not an additional experimental measurement. The low-target view preserves the headlights, the high-target view captures shadowed bodywork, and the middle view provides ordinary-brightness content.

After synchronized capture, measured depth and the monocular prior generate dense metric depth for each view. The system projects vehicle and background observations into the reference view, suppresses saturated headlight pixels and background observations occluded by the vehicle, normalizes the remaining complementary observations by exposure, fuses them, and refines the result through three tone mappings. If a surface is visible in only one camera and that observation is also saturated, this pipeline provides no guarantee of recovering its true details.

Loss & Training

The trainable components are the confidence network and HDR refinement network. The paper uses a 9:1 train/test split, first pretraining on 40k synthetic training scenes for 50 epochs. HDR supervision combines \(\ell_1\), SSIM, and gradient losses in the tone-mapped domain, while depth supervision uses \(\ell_1\) and SILog losses. Both networks are then fine-tuned on 10k real training scenes for 10 epochs using only HDR reconstruction losses. Optimization uses Adam with a learning rate of \(10^{-4}\).

The 40k/10k training counts are retained as stated, but the main paper does not explain how they are sampled from the collected scenes and video frames described below. They should not be added as independently captured scenes. Loss weights, training resolution, and split granularity are likewise not invented where the current cache does not specify them.

Key Experimental Results

Main Results

The robotic platform uses six 12-bit LDR cameras, two 24-bit HDR cameras supplying pseudo-ground-truth references, a RealSense D455 active-stereo sensor, and an Ouster OS1 LiDAR. The real modest-DR dataset contains 31 scenes and 15,000 frames captured with robot motion; the ultra-DR dataset contains 80 static scenes with HDR ground truth constructed through temporal exposure bracketing. The iPhone 13 Pro contributes another 10 scenes, giving 121 real scenes in total. CARLA data contains 20 videos across six environments, totaling 15,000 frames, with rendered ground truth exceeding 150 dB effective dynamic range. This is not a claim that reconstructed outputs achieve that range.

The following table selects the single-shot, three-camera configuration from the paper's Table 1. PSNR-µ and SSIM-µ are evaluated in the µ-law tone-mapped domain; higher values are better for these metrics and HDR-VDP. The authors state that baselines are trained from scratch with matched splits, resolution, exposure settings, valid masks, and metric domains. However, DMEB additionally uses depth, making this a system-level comparison rather than a strictly identical-sensor-input network comparison.

Method	FPS	Modest PSNR-µ	Modest SSIM-µ	Modest HDR-VDP	Ultra PSNR-µ	Ultra SSIM-µ	Ultra HDR-VDP
HDR Transformer	1.189	33.61	0.872	9.64	38.74	0.848	9.40
SAFNet	53.937	30.96	0.644	7.72	27.39	0.769	9.06
HDRFlow	77.519	27.09	0.541	6.29	27.44	0.617	8.95
AFUNet	0.795	32.25	0.819	9.18	34.69	0.877	9.39
DMEB	13.073	39.40	0.916	9.69	39.17	0.868	9.16

DMEB exceeds HDR Transformer by 5.79 dB PSNR-µ on modest-DR data, but only by 0.43 dB on ultra-DR data. Its ultra-DR SSIM-µ is below AFUNet's, and its HDR-VDP is below both HDR Transformer and AFUNet. It is neither uniformly best across all datasets and metrics nor the fastest method.

The extreme-illumination synthetic evaluation in the paper's Table 3 provides another perspective:

Method	FPS	PSNR-µ	SSIM-µ	HDR-VDP
HDR Transformer	1.19	32.65	0.830	8.20
SAFNet	53.94	28.21	0.687	6.25
HDRFlow	77.52	26.33	0.475	7.12
AFUNet	0.80	33.52	0.830	8.02
DMEB	9.76	34.72	0.885	8.29

DMEB leads all three quality metrics on synthetic data, but its 9.76 FPS does not support a blanket claim of exceeding 10 FPS in every configuration. The main paper does not provide full timing hardware and overhead breakdowns alongside this table.

Ablation Study

The paper's Table 2 compares geometry sources on real modest-DR data. The gap column is calculated from the original values relative to the full configuration, in dB.

Geometry configuration	PSNR-µ	Gap to full configuration	Meaning
Flow	35.84	3.56	Optical flow without depth input
Mono	37.09	2.31	Monocular depth
LiDAR-SV	37.90	1.50	Single-view LiDAR-anchored depth
LiDAR-MV	39.40	0.00	Full multi-view LiDAR-guided fusion

This ablation supports progressive gains from photometric correspondence to metric geometry and then multi-view fusion. It does not isolate the contributions of the confidence CNN, soft visibility, or Transformer, because the main paper does not report separate removal experiments for those components.

Key Findings

• The same model can benefit from synchronized multi-camera inputs: on modest-DR data, HDR Transformer improves from 31.26 dB with multi-shot single-camera inputs to 33.61 dB; AFUNet improves from 29.64 to 32.25 dB, and DMEB from 30.72 to 39.40 dB. The temporal baseline uses sequential frames synthesized at different exposures from HDR ground truth and should not be treated as an identical physical acquisition process.
• In the paper's Table 4, iPhone common-overlap PSNR-µ is 23.35 for HDR Transformer, 25.21 for AFUNet, and 26.12 for DMEB. On the external Choi two-view data, the corresponding values are 26.33, 25.39, and 33.00. The zero-shot, no-retraining designation belongs to DMEB's 33.00 dB result and should not automatically be assigned to all baselines.
• Increasing the camera count from 3 to 8 expands recoverable dynamic range while keeping SSIM relatively stable in Figure 6, eventually encountering shutter hardware limits. The eight inputs include six LDR images and two LDR-converted HDR-camera images. Individual plotted values are not reliably recoverable from the cached figure extraction, so no point-by-point table is invented.
• YOLOv8 detection on 90 manually annotated frames gives DMEB Recall/F1 of 0.83/0.73, versus 0.78/0.63 for HDRFlow and 0.81/0.67 for AFUNet. This suggests that reconstruction improvements help perception, but the evaluation is too small to establish full driving or robotic closed-loop benefits.

Highlights & Insights

• Depth sensors do more than supply distance: they allow cameras to adopt more aggressive exposure differences. Sensor fusion benefits acquisition and reconstruction, rather than merely adding features at the end.
• Explicit exposure confidence, learned confidence, and depth visibility have distinct responsibilities. An observation with normal brightness can still project onto the wrong surface, which explains why exposure weighting alone is insufficient.
• Robot, consumer-phone, and extreme-illumination synthetic data help distinguish sensor-configuration benefits from network-specific gains. Positioning DMEB as a reference system for the dataset better captures the contribution than emphasizing a single leaderboard number.

Limitations & Future Work

• The authors explicitly require shared fields of view across cameras and depth sensors; quantitative iPhone evaluation covers only common-overlap regions. They propose accumulating HDR information in a 3D scene representation to explore modeling beyond a single reference view.
• Mechanistically, calibration errors, synchronization errors, sparse-depth failures, and occlusion boundaries can still affect fusion. Monocular priors and soft weights mitigate these issues without guaranteeing recovery of all missing information. Depth-noise, calibration-perturbation, and near-range occlusion-boundary sensitivity would be useful further tests.
• The main paper lacks independent confidence, visibility, and refinement ablations and does not explain the relationship between 40k/10k training counts and collected data. Supplementary material is absent from the cache, so this does not establish that every such detail is missing from the complete paper.
• Ultra-DR perceptual metrics are not uniformly superior, the iPhone dataset has only 10 scenes, and downstream detection uses only 90 frames. Broader dynamic phone scenes, cross-device testing, and temporal-stability evaluation are needed for stronger deployment claims.

Related Work & Insights

• Versus HDR Transformer, SAFNet, HDRFlow, and AFUNet: these methods address alignment and fusion in different ways, whereas DMEB introduces external depth so that geometric correspondence need not depend on texture similarity across extreme exposures. The advantage comes with additional depth hardware and calibration requirements.
• Versus Choi et al., Dual Exposure Stereo for Extended Dynamic Range 3D Imaging: both exploit complementary multi-camera exposures. This paper builds data and a scalable multi-view HDR pipeline around external depth guidance and reports zero-shot transfer to that two-view dataset.
• Connection to Depth Anything V2 and sparse-depth scale adaptation: pretrained priors supply dense structure while sensors supply metric anchors. The transferable principle is to use learned priors for coverage and real measurements for constraints that determine geometric scale.
• Resources: the project address comes from the main paper's public-resource statement; its release status was not checked online for this note. No confirmed standalone code repository is provided in the cache, so no guessed repository address is added.

Rating

• Novelty: 4/5. Integrates synchronized complementary exposures, external depth, and a supporting benchmark into a complete system, with the main contribution in sensor configuration and geometric fusion.
• Experimental Thoroughness: 4/5. Covers real, synthetic, phone, external, and downstream evaluations, but component ablations and large-scale deployment evidence remain limited.
• Writing Quality: 3/5. The pipeline is clear, but training counts need explanation and some qualitative-comparison text points to methods inconsistent with figure captions, requiring cross-checks against tables.
• Value: 4/5. A useful reference for robots already equipped with multiple cameras and depth sensors, with benefits constrained by overlap, calibration, and capture conditions.