DiffHDR: Re-Exposing LDR Videos with Video Diffusion Models

Conference: ECCV2026
Paper: Official paper page · PDF
Area: Video Generation
Keywords: HDR reconstruction, video diffusion, Log-Gamma mapping, exposure masks, controllable generation

TL;DR

DiffHDR treats clipped highlights and noise-obscured shadows in a single 8-bit LDR video as radiance-inpainting regions, uses Log-Gamma mapping to generate HDR in the latent space of a frozen LDR video VAE, reaches 6.89 FOVVDP versus SingleHDR's 6.56 on Cinematic Video, and supports text- or reference-guided reconstruction.

Background & Motivation

Most consumer and generated videos are still stored as 8-bit LDR. Quantization and clipping to \([0,1]\) do more than compress the luminance range: they irreversibly erase structure in blown-out skies and lights or bury it beneath noise in dark shadows. Multi-exposure fusion can recover complementary observations, but requires a specialized capture process. Single-image HDR networks instead infer missing radiance from one LDR input, usually through deterministic pixel-to-pixel regression, and tend to produce flat highlights, artifacts, or temporal flicker under severe saturation.

The inverse problem is intrinsically one-to-many. An identical white patch in LDR may hide clouds, the sun, or entirely different bright texture, so the original signal does not determine a unique answer. Video diffusion models have spatiotemporal generative priors learned from vast LDR corpora and are well suited to synthesizing plausible, non-unique detail. Their VAE, however, has only seen the standard LDR distribution; directly encoding linear HDR values far above 1 causes severe reconstruction error. Meanwhile, paired high-quality LDR-HDR video is too scarce to train a large model from scratch.

DiffHDR therefore adapts the signal to the VAE rather than replacing the VAE, then performs parameter-efficient adaptation of the diffusion backbone. Core idea: align HDR radiance with a pretrained LDR latent space through Log-Gamma mapping, treat exposure-damaged regions as video radiance inpainting targets, and control temporally coherent synthesis with smoothed exposure masks plus region-specific text or image conditions.

Method

Overall Architecture

Training pairs are first built by rendering static 16K HDRIs into linear HDR videos with camera motion, then simulating exposure shifts, correlated sensor noise, quantization, and clipping to form LDR inputs. At inference, the LDR video is linearized, Log-Gamma mapped, and encoded by Wan-2.1-VAE into VACE's context branch. Starting from random noise, the main branch generates an HDR latent conditioned on the LDR latent, exposure masks, and optional text or reference images; VAE decoding and inverse mapping finally recover linear HDR.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input 8-bit LDR video"] --> B["Paired HDRI synthesis<br/>construct training supervision"]
    B --> C["Log-Gamma latent adaptation<br/>linearize→map→VAE encode"]
    C --> D["Exposure-mask-guided<br/>dual-branch diffusion inpainting"]
    D --> E["Context-focused control<br/>text or reference image"]
    E --> F["VAE decode + inverse map<br/>linear HDR video"]

Key Designs

1. Paired HDRI synthesis: turn static environment maps into temporally consistent HDR supervision

Because paired HDR video is scarce, DiffHDR starts from roughly 800 16K Polyhaven HDRIs, places the camera at the center of a Blender skybox, and renders three trajectories: zooms toward the brightest region, zooms toward the darkest region, and three \(120^\circ\) rotation segments covering a full panorama. Zooms sample the starting focal length from 18--30 mm and the ending length from 50--70 mm. The result is about 5,400 linear Rec.709 HDR clips of 81 frames each. Although a static HDRI cannot model true object motion, it supplies continuous radiance and controlled camera motion.

The paired LDR is not produced by one fixed tone curve. The pipeline first samples an exposure offset \(\Delta\in[-2,2]\) stops and scales the linear signal by \(2^\Delta\), then adds signal-dependent and stationary sensor noise. Instead of drawing noise independently per frame, an AR(1) field preserves temporal correlation:

\[ \epsilon_t=\rho\epsilon_{t-1}+\sqrt{1-\rho^2}u_t,\qquad \rho=0.5. \]

Conversion to sRGB, clipping to \([0,1]\), and 8-bit quantization complete the degradation. This chain teaches highlight recovery, shadow denoising, and temporal stability rather than inversion of a single tone mapper.

2. Log-Gamma latent adaptation: carry HDR radiance without fine-tuning the VAE

Linear HDR lies outside the domain of an LDR VAE. Retraining that VAE would shift its latent space and risk weakening the pretrained video prior. DiffHDR instead normalizes radiance against a maximum representable value \(M\) with logarithmic compression and then applies gamma compression so that the resulting statistics better resemble natural LDR:

\[ \mathcal{T}(x)=\left(\frac{\log(1+x)}{\log(1+M)}\right)^{1/\gamma}. \]

The inverse transform recovers linear radiance after decoding. The point is not merely to squeeze values into \([0,1]\), but to present the VAE with a familiar luminance distribution. In the ablation, removing gamma yields only 25.38 dB PSNR, whereas the full mapping reaches 32.86 dB, showing that gamma substantially reduces color and high-contrast edge reconstruction errors.

3. Exposure-mask-guided dual-branch diffusion inpainting: invoke the generative prior only where information is lost

DiffHDR places the encoded LDR in VACE's context branch and generates HDR from Gaussian noise in the main branch. It computes linear Rec.709 luminance, marks pixels above \(\tau_{high}=0.95\) as overexposed and below \(\tau_{low}=0.05\) as underexposed, then suppresses frame-to-frame threshold jitter with an exponential moving average:

\[ \widetilde M_t=\alpha M_t+(1-\alpha)\widetilde M_{t-1},\qquad \alpha=0.7. \]

The masks tell diffusion where new detail must be synthesized, while the LDR context constrains reliable structure and color elsewhere. The Wan-2.1-VACE-14B backbone remains frozen; rank-32 LoRA adapters are inserted only into attention and feed-forward layers of the DiT blocks. This retains the general spatiotemporal prior and reduces overfitting to about 5,400 synthetic training clips.

4. Context-focused control: route highlight and shadow semantics to their respective regions

A global prompt cannot specify which words describe the bright sky and which describe a dark tree. DiffHDR structures captions as [overexposed: ...]; [underexposed: ...], computes base, highlight, and shadow cross-attention responses separately, and injects each differential residual only inside its mask:

\[ r=r_{base}+\alpha_{over}M_{over}\odot(r_{over}-r_{base})+\alpha_{under}M_{under}\odot(r_{under}-r_{base}). \]

This context-focused cross-attention modifies each attention output only at inference and changes neither weights nor training objectives. The same model can therefore synthesize a sun, clouds, or shadow texture locally while preserving global structure. For finer appearance control, a reference image is encoded by the same VAE and concatenated to the condition sequence along the temporal dimension.

A Worked Example

Consider an 8-bit clip with a blown-out window and underexposed interior columns. Luminance detection assigns the window to \(M_{over}\) and the columns to \(M_{under}\); EMA keeps those boundaries stable as the camera moves. After the Log-Gamma representation enters the context branch, the main branch generates the HDR latent from noise. Given a prompt saying that the overexposed region contains a cloudy sky and the underexposed region contains wooden columns, each phrase enhances attention only within its mask. Decoding and inverse mapping then let an editor inspect the sky at EV-4 and the column texture at EV+2 while regions that were never clipped remain constrained by the source video.

Loss & Training

Training follows rectified flow matching. For an HDR latent \(x_1\), Gaussian noise \(x_0\sim\mathcal N(0,I)\), and random \(t\in[0,1]\), the method samples an intermediate point on their line and asks the conditional velocity field to predict the constant direction from noise to data:

\[ x_t=t x_1+(1-t)x_0, \]

\[ \mathcal L=\mathbb E_{x_0,x_1,t}\left[\left\|u_\Theta(x_t,t,c)-(x_1-x_0)\right\|_2^2\right]. \]

The condition \(c\) contains the LDR latent, exposure masks, and optional text or reference image. The model is trained for 10,000 steps at a spatiotemporal resolution of \(33\times720\times1280\) with AdamW, a constant \(10^{-4}\) learning rate, and 8 NVIDIA A100 GPUs. DiT fine-tuning uses BF16, whereas the VAE stays in FP32 because BF16 decoding can introduce banding into continuous HDR tones.

Key Experimental Results

Main Results

Dataset	Method	Reference metric	DOVER ↑	MUSIQ ↑	CLIPIQA ↑
Cinematic Video	SingleHDR	FOVVDP 6.56	0.77	51.79	0.33
Cinematic Video	LEDiff	FOVVDP 3.75	0.63	47.90	0.28
Cinematic Video	DiffHDR	FOVVDP 6.89	0.81	58.38	0.41
Polyhaven Synthetic	SingleHDR	FOVVDP 7.48	0.66	57.60	0.46
Polyhaven Synthetic	LEDiff	FOVVDP 4.68	0.59	58.92	0.46
Polyhaven Synthetic	DiffHDR	FOVVDP 7.65	0.68	60.02	0.50

Dataset	Method	HDR-VDP3 ↑	PU21-PIQE ↓	FID ↓
SI-HDR	HDRCNN	6.82	24.30	19.26
SI-HDR	SingleHDR	7.37	26.64	27.55
SI-HDR	LEDiff	6.56	22.71	25.98
SI-HDR	DiffHDR	6.98	19.37	18.68

DiffHDR does not lead HDR-VDP3 on SI-HDR. The authors attribute this to generative inpainting: plausible reconstructed details need not match the one ground-truth image pixel for pixel and are penalized by a full-reference metric. It nevertheless leads PU21-PIQE and FID, and it wins every reported metric on the video datasets. The no-reference results remain consistent: it obtains DOVER 0.74, MUSIQ 55.79, and CLIPIQA 0.48 on 50 in-the-wild videos, and 0.61, 46.06, and 0.34 on 10 Veo2 videos.

Ablation Study

Configuration	PSNR ↑	SSIM ↑	LPIPS ↓
Direct linear HDR encoding	22.16	0.74	0.28
Pure Log mapping	14.61	0.74	0.57
Log-Gamma without gamma	25.38	0.75	0.34
Full Log-Gamma	32.86	0.86	0.15

Configuration	FOVVDP ↑	DOVER ↑	MUSIQ ↑	CLIPIQA ↑
Without data augmentation	7.57	0.67	59.86	0.49
Without exposure masks	7.58	0.67	59.42	0.48
Full model	7.65	0.68	60.02	0.50

Key Findings

• Log-Gamma is the strongest quantitative ablation. Relative to removing gamma, the full map gains 7.48 dB PSNR and 0.11 SSIM while reducing LPIPS from 0.34 to 0.15.
• Removing augmentation mainly leaves residual shadow noise; removing mask guidance damages shadow texture more clearly and lowers MUSIQ from 60.02 to 59.42. Their aggregate gaps are modest, but the visual failures differ.
• Context-focused prompting is ablated only qualitatively. Compared with a global prompt, it reconstructs the solar shape and shadowed trees more accurately, but the paper reports no isolated numerical gain for this component.

Highlights & Insights

• The central engineering insight is to adapt the representation rather than the VAE. Log-Gamma preserves a large LDR-trained video VAE and avoids allowing a small HDR dataset to disrupt its latent space and generative prior.
• Exposure masks do more than identify inpainting areas; they also route semantics into highlights and shadows. Reusing one signal for spatial localization, temporal smoothing, and conditional control reduces mismatch between training and inference interfaces.
• Generalization from static HDRIs to real dynamic video suggests that the generic video backbone supplies most motion knowledge, while synthetic supervision only needs to teach how radiance is degraded and recovered. The strategy could transfer to high-bit-depth, RAW, or relighting tasks with scarce ground truth.

Limitations & Future Work

• The authors explicitly note that training and evaluation are limited to 720p. Direct 4K inference remains constrained by memory, compute, and the train-test resolution gap, so this is not yet a drop-in 4K post-production tool.
• HDR training clips come from static panoramas and virtual camera motion, omitting non-rigid motion, dynamic illumination, real camera response, and compression artifacts. Real-video tests show some generalization, but no paired real HDR ground truth measures radiometric fidelity.
• Generative completion aims for plausibility rather than recovery of a unique fact. Prompts and references control ambiguity, but hallucinated highlight structure may be unacceptable in documentary or scientific imaging; uncertainty maps, a conservative mode, and explicit provenance masks would make the output safer.
• Context-focused prompting has only a qualitative ablation. Control strength, random-seed stability, and conflicting conditions require systematic evaluation, including user studies and cross-prompt consistency metrics.

Related Work & Insights

• vs SingleHDR: SingleHDR reverses the camera pipeline for one image and produces deterministic HDR. It is simpler, but lacks video temporal priors and a strong generator; DiffHDR handles severe clipping and temporal stability better at the cost of a 14B backbone and generative ambiguity.
• vs LEDiff: LEDiff performs HDR exposure generation in latent space, but trails DiffHDR in reported video quality and dynamic range. DiffHDR adds explicit Log-Gamma alignment to the pretrained VAE, exposure masks, and region-specific conditional control.
• vs multi-exposure HDR video fusion: Multi-exposure methods exploit real complementary observations and have stronger physical grounding, but require alternating exposures and motion alignment. DiffHDR accepts ordinary legacy or generated LDR video, while necessarily guessing information that was completely lost.

Rating

• Novelty: ⭐⭐⭐⭐⭐ The first framework to combine pretrained video diffusion, HDR latent mapping without VAE fine-tuning, and region-specific controllable reconstruction for single-video LDR-to-HDR conversion.
• Experimental Thoroughness: ⭐⭐⭐⭐ It spans synthetic, cinematic, in-the-wild, and generated video with multiple metrics, but lacks paired real HDR video, quantitative control ablations, and high-resolution evaluation.
• Writing Quality: ⭐⭐⭐⭐ The pipeline and visual comparisons are clear, although some mapping hyperparameters and control details require the supplement for full reproduction.
• Value: ⭐⭐⭐⭐⭐ It connects large stores of legacy and generated LDR video to HDR display, re-exposure, and grading workflows with direct research and production relevance.