---

title: >- [Paper Note] Video Generation Models Are Inherent Lighting Estimators description: >- [ECCV 2026][video_generation][Lighting Estimation] V-LITE reformulates dynamic HDR lighting estimation from in-the-wild videos as guided video inpainting, unlocking video diffusion models' intrinsic lighting priors via an HDR-aware VAE and LoRA tuning. tags: - ECCV 2026 - video_generation - Lighting Estimation - Video Diffusion Models - High Dynamic Range date: 2026-09-19 content_hash: 27c0d93b49119aa7

Video Generation Models Are Inherent Lighting Estimators

Conference: ECCV 2026
Paper: ECCV Official
Project: OpenReview / ECVA
Area: Video Generation / 3D Vision
Keywords: Dynamic Lighting Estimation, Video Diffusion Models, HDR Environment Map, Video Inpainting, Light Probe

TL;DR

V-LITE draws inspiration from VFX light probe practices to recast dynamic HDR lighting estimation from single in-the-wild videos into guided video inpainting, unlocking modern video diffusion models' implicit physical lighting priors via a log-domain HDR-aware VAE and LoRA fine-tuning.

Background & Motivation

Recovering complete, dynamic high dynamic range (HDR) environment maps from single casually captured in-the-wild videos is the cornerstone of photorealistic augmented reality, seamless virtual object insertion, and dynamic scene relighting. However, real-world footage presents severe challenges: cameras move rapidly, foreground objects frequently occlude the surroundings, and non-Lambertian surfaces scatter light unpredictably. As a result, the observable field of view contains only sparse, partial illumination cues. Classical physics-based inverse rendering methods are forced to rely on restrictive assumptions—such as static lighting, pure Lambertian reflectance, or known 3D geometry—which break down immediately in generic unconstrained videos.

Deep learning lighting estimation techniques have made notable strides, yet existing paradigms remain fundamentally bottlenecked. Single-image approaches (such as DiffusionLight and StyleLight), when applied to video sequences frame by frame, inevitably suffer from severe high-frequency flickering, shadow instability, and temporal color shifts because they lack any cross-frame coupling. Conversely, multi-view setups or video conditional translation models that generate multiple low dynamic range (LDR) exposures for subsequent HDR fusion incur massive computational costs (often taking more than a day per video) while treating the generative backbone as a naive pixel translator, frequently introducing exposure-fusion artifacts and color bias.

Recent foundation video generation models like Wan 2.1 demonstrate that unconstrained pretraining endows diffusion Transformers with an astonishing implicit grasp of complex physical lighting, shadow formation, and temporal coherence. Instead of training an external translator from scratch, the natural angle of attack is to directly unlock the model's inherent generative priors. Core idea: reframe dynamic HDR lighting estimation as a guided video inpainting task that inserts a synthetic reflective chrome ball into the scene, leveraging an HDR-aware log-domain VAE and LoRA fine-tuning to recover temporally coherent dynamic HDR environment maps in just 80 seconds per video.

Method

Overall Architecture

Built upon the flow-matching Wan 2.1-1.3B video foundation backbone, V-LITE executes a seamless three-stage pipeline: in the input stage, a central circular mask is applied to the input in-the-wild LDR video to decouple scene context and the probe inpainting region; in the generative stage, an HDR-aware VAE projects features into a stabilized log domain where a LoRA-adapted diffusion Transformer synthesizes a physically plausible reflective chrome ball guided by text prompts and visual condition latents; in the final reconstruction stage, an HDR decoder maps the latents into full HDR video frames, which are unprojected via equirectangular transformation into a 256×512 dynamic HDR environment map sequence.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input in-the-wild LDR video<br/>Apply central circular mask M"] --> B["Condition decoupling & encoding<br/>Separate environment and scene context"]
    B --> C["HDR-aware VAE mapping<br/>Convert to log domain via tonemap adapter"]
    C --> D["Inherent lighting inpainting<br/>LoRA diffusion Transformer fills chrome ball"]
    D --> E["HDR inverse mapping & decoding<br/>Reconstruct full spatiotemporal HDR frames"]
    E --> F["Equirectangular unwrapping<br/>Output dynamic HDR environment map sequence"]

Key Designs

1. Light Probe Video Inpainting: Reformulating Global Inversion into Local Physical Reflection Synthesis

Directly predicting a \(360^\circ\) panoramic environment map requires the model to aggressively outpaint unseen regions from a narrow field-of-view (FoV), typically leading to distorted geometry and hallucinated light sources. V-LITE borrows a proven industry technique from visual effects (VFX)—placing a reflective mirrored chrome sphere on set to record global illumination. The global lighting regression problem is thus converted into a localized video inpainting task. Given an input LDR video \(x_{\text{LDR}}\) and a central mask \(M\), the input is formatted as \(x_{\text{input}} = x_{\text{LDR}} \odot (1-M) + C \odot M\) with gray constant \(C\). Steered by the prompt "A perfect mirrored reflective chrome sphere, near the camera, on top of everything", the diffusion Transformer is compelled to observe ambient specular highlights, dynamic shadow motions, and environmental context over time, faithfully rendering the physical reflections across the ball's surface. Once synthesized, standard equirectangular unprojection \(f_{\mathcal{R}}\) extracts the dynamic HDR environment map sequence.

2. Log-Domain HDR-Aware VAE: Bridging LDR Pretrained Foundations and Scene-Linear HDR Radiance

Pretrained video generative backbones operate strictly within standard [0, 1] LDR color spaces and cannot natively encode linear HDR radiance that spans extreme orders of magnitude. Rather than relying on error-prone multi-exposure bracket generation and post-fusion, V-LITE preserves the architecture of the pretrained LDR VAE encoder \(E\) and decoder while inserting lightweight 3D convolutional tonemap adapters \(\mathcal{F}\) and \(\mathcal{G}\) at both ends. Operating in the scene-linear Rec. 2020 wide-gamut space, input HDR signals are transformed into the log domain with learnable scale \(\mathbf{s}\), bias \(\mathbf{b}\), and 3D convolution \(\mathcal{F}\):

\[x_{\log} = \mathbf{s} \cdot \log(x_{\text{HDR}} + \epsilon) + \mathbf{b} + \mathcal{F}(x_{\text{HDR}})\]

After the compressed latent is processed through the generative backbone, the HDR decoder \(D_{\text{HDR}}\) and inverse adapter \(\mathcal{G}\) reconstruct the linear HDR output \(x'_{\text{HDR}}\). A two-stage training scheme is employed: first, freezing the VAE weights to train adapters \(\mathcal{F}\) and \(\mathcal{G}\) to stabilize the log-domain manifold; second, unfreezing the decoder for joint end-to-end optimization, ensuring high photometric fidelity across highlights and shadows.

3. Mixed-Data Training and Modulated Flow Matching: Anchoring Physical Luminance and Spatiotemporal Consistency

To adapt the diffusion Transformer to the HDR latent distribution without suffering catastrophic forgetting of its rich spatiotemporal common sense, rank-32 LoRA adapters are integrated exclusively into the attention layers. Because real dynamic HDR videos are exceedingly scarce in the wild, training solely on tonemapped pseudo-HDR videos fails to capture accurate physical sun-to-sky intensity ratios. The authors construct the V-LITESet dataset, mixing 8K dynamic in-the-wild videos with 800 static high-fidelity real-world HDR panoramas (replicated into static video sequences as physical photometric anchors) at a 10:1 ratio. In the flow-matching objective, a spatial modulation mask \(M' = 1 + (\alpha - 1)M\) with \(\alpha = 5\) is introduced:

\[\mathcal{L} = \mathbb{E}_{z, \epsilon, t} \left[ \left\| M' \odot (v_{\theta}(z_t, \mathcal{C}, t) - v) \right\|_2^2 \right]\]

Weighting the central probe region 5× higher forces the model to focus primarily on generating precise physical reflections while simultaneously lifting ambient background frames to the HDR domain.

Loss & Training

The framework is built on Wan 2.1-1.3B and trained on 8 NVIDIA H100 GPUs for 100K steps with a global batch size of 64 using the AdamW optimizer with a learning rate of \(1 \times 10^{-5}\). The VAE two-stage adaptation consumes 10K and 5K steps respectively. Input video frames are processed at 480×832, and the resulting unwrapped HDR environment maps are produced at 256×512 resolution. Inference computes trajectories from \(t=1\) to \(t=0\) using standard ODE solvers, requiring only 80 seconds per complete video sequence.

Key Experimental Results

Main Results

Quantitative evaluations on the Editable Indoor rendering benchmark and the EnvMapNet dominant lighting direction benchmark are detailed below (matching Table 1 of the paper, including 10-video subset evaluation and full test comparisons):

Method	Editable Indoor MSE ↓	SI-MSE ↓	AER (rad) ↓	LS ↓	EnvMapNet AED (°) ↓	AS ↓	Inference Time (s) ↓
DiffusionLight† [29]	0.10	0.05	4.58	0.03	40.07	16.76	145,800
Ours† (V-LITE)	0.09	0.03	4.68	0.03	42.53	15.96	80
DiffusionLightTurbo [11]	0.11	0.05	4.90	0.04	37.74	16.68	713
StyleLight [37]	0.13	0.07	6.01	0.05	44.22	17.83	2,002
Ours (V-LITE Full)	0.10	0.03	4.77	0.03	41.63	16.30	80

Note: † indicates evaluation on the shared 10-video benchmark subset; LS (Lighting Stability) is the temporal standard deviation of SI-MSE, and AS (Angle Stability) is the standard deviation of AED; DiffusionLight takes over 40 hours (145,800s) per video, whereas V-LITE achieves an 1800× speedup.

Ablation Study

The lower section of Table 1 evaluates the impact of isolating individual components under the same protocol:

Config	Editable Indoor MSE ↓	SI-MSE ↓	AER (rad) ↓	LS ↓	EnvMapNet AED (°) ↓	AS ↓	Note
Full Model (Ours)	0.10	0.03	4.77	0.03	41.63	16.30	Full V-LITE model
LDR Baseline	0.11	0.04	5.45	0.03	44.71	15.87	Inpainting without HDR VAE/tuning; lifted via GT inverse tonemapping
Frozen Backbone	3.00	0.07	12.95	0.04	51.71	19.45	Direct integration of HDR VAE without DiT LoRA fine-tuning
Video-only	0.10	0.03	4.82	0.02	41.96	15.81	Trained exclusively on dynamic videos without 800 static HDR anchors

Key Findings

• Latent HDR Alignment Is Critical: Merely appending the HDR VAE to an untuned diffusion backbone (Frozen Backbone) triggers catastrophic degradation (MSE jumps to 3.00 and AER explodes to 12.95 rad), proving that LDR-trained DiT models cannot parse log-domain HDR distributions without targeted LoRA adaptation.
• Static HDR Anchors Ground Physical Radiance: Training exclusively on dynamic videos (Video-only) yields acceptable temporal smoothness (LS 0.02) but noticeably inferior lighting angular accuracy (AED 41.96° vs 41.63°) and photometric error, establishing that static uncompressed HDR images are indispensable as physical intensity anchors.
• Superior Insertion Realism in User Study: In a blinded user study involving 10 participants across 20 diverse videos with virtual 3D objects inserted using Blender, 64% of responses rated V-LITE's insertions as "Perfect" and 28% as "Acceptable" for perceptual realism; 66.5% rated lighting consistency as "Perfect", with a total failure rate of under 2%.

Highlights & Insights

• Ingenious Problem Formulation: Reframing dynamic lighting estimation as a localized inpainting task of a physical VFX chrome ball bypasses ill-conditioned direct panorama outpainting, playing directly to the core strengths of video diffusion models.
• Minimalist Yet Effective HDR Bridge: By pairing 3D convolutional tonemap adapters with log-domain compression, V-LITE enables standard LDR-trained DiT architectures to process and decode scene-linear Rec. 2020 HDR signals without architectural surgery or multi-exposure fusion.
• Orders-of-Magnitude Acceleration: Slashing inference runtime from 145,800 seconds (over 40 hours) down to 80 seconds per video while enhancing temporal stability transforms dynamic video lighting estimation from an academic curiosity into a practical production-grade tool.

Limitations & Future Work

• Sensitivity to Out-of-Distribution Illumination: Being data-driven, V-LITE can occasionally fail when encountering extreme, out-of-distribution conditions (e.g., severe overexposure, heavy lens flares, or atypical theatrical lighting), producing misaligned dominant light angles.
• Sequence Length Constrained by Generative Backbone: The underlying Wan 2.1 video foundation is bounded by its temporal context window, making it challenging to maintain global color temperature consistency over very long video clips without specialized long-horizon drift control.
• Single-Probe Spatial Non-Locality: A single probe placed near the camera reflects scene lighting from that specific vantage point; in environments with heavy geometric occlusion or deep spatial transitions (e.g., passing beneath tree canopies or tunnels), spatially varying lighting fields cannot be fully captured by a single probe.

Related Work & Insights

• vs DiffusionLight [29] / DiffusionLightTurbo [11]: Prominent image-based diffusion lighting estimators that suffer from severe temporal flicker and color jitter when applied per-frame to video, with DiffusionLight requiring up to 40 hours per sequence; V-LITE natively maintains inter-frame coherence via flow-matching spatiotemporal attention and runs in 80 seconds.
• vs StyleLight [37]: StyleGAN-based inversion struggles with arbitrary open-world camera motions and complex scene geometry; V-LITE's diffusion Transformer architecture delivers vastly superior generalization across diverse in-the-wild captures.
• Insights: Video generative models function not only as content synthesizers, but as implicit neural simulators of physical optics, material reflectance, and illumination transport. Directing generative models with canonical virtual probes (e.g., chrome, matte, or normal spheres) presents an exciting general paradigm for intrinsic video decomposition, dynamic depth estimation, and 3D inverse rendering.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Brilliant and intuitive reformulation of lighting estimation into virtual light probe video inpainting]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive comparisons across rendering benchmarks, ablation variants, and Blender user studies]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, elegant architecture diagrams, and rigorous mathematical formulation]
• Value: ⭐⭐⭐⭐⭐ [Eliminates temporal flicker while delivering an 1800× speedup, unlocking real-world AR and VFX workflows]

Related Papers

• [ECCV 2026] Video Generation Models are General-Purpose Vision Learners
• [CVPR 2026] SVBench: Evaluation of Video Generation Models on Social Reasoning
• [ICLR 2026] Vid2World: Crafting Video Diffusion Models to Interactive World Models
• [CVPR 2026] Diff4Splat: Repurposing Video Diffusion Models for Dynamic Scene Generation
• [CVPR 2026] ExPose: Reinforcing Video Generation Models for Extreme Pose Estimation