Temporally Stable Generative Illumination with a One-Step Diffusion Model¶
Conference: ECCV 2026
Paper: ECCV 2026
Area: Image Generation
Keywords: Neural Illumination, Global Illumination, One-Step Diffusion Model, Temporal VAE, Screen-Space Rendering
TL;DR¶
Addressing slow multi-step sampling and temporal flicker in neural global illumination, this paper proposes a single-step latent diffusion framework conditioning on intrinsic signals and sparse lighting hints via dual adapters, combined with a plug-and-play Temporal VAE decoder to synthesize geometrically aligned, flicker-free indirect illumination at 0.29s per frame.
Background & Motivation¶
Physically based rendering simulates the rendering equation to produce photorealistic images by modeling complex light transport. While path tracing delivers high fidelity, its computational cost scales heavily with geometric and material complexity. Screen-space approximations decouple rendering cost from scene complexity by operating on G-buffer representations. Recently, generative models—particularly large-scale diffusion models—have shown immense potential for neural rendering by synthesizing convincing lighting effects through rich learned priors.
However, adopting generative diffusion models for forward illumination rendering presents two critical bottlenecks. First, iterative multi-step denoising is too slow for real-time applications, while adopting video diffusion models incurs prohibitive memory and computational overhead. Second, per-frame 2D latent diffusion suffers from severe high-frequency temporal flickering across sequences, and naively concatenating dense condition images onto the noisy latent perturbs the pre-trained diffusion backbone's feature distribution, causing geometric misalignment and hallucination artifacts.
The key insight is to decouple single-step generative lighting inference from cross-frame temporal stabilization. Core idea: condition a single-step latent diffusion model (SD 2.1-Turbo) on intrinsic signals and sparse lighting cues via albedo demodulation and dual T2I-Adapters, and replace the frozen VAE decoder at inference time with a motion-reprojected Temporal VAE decoder to achieve high-fidelity, temporally stable forward global illumination without video diffusion costs.
Method¶
Overall Architecture¶
The proposed method formulates screen-space global illumination (GI) approximation as a multi-condition image generation task. Input signals comprise intrinsic geometry/material attributes (normals, albedo, roughness, metallic) and lighting hints (direct lighting, 1-spp noisy GI, and radiance cues). During input processing, direct lighting is demodulated by albedo and compressed via \(\mu\)-Law tonemapping before entering the VAE encoder. The noisy direct lighting latent is denoised in a single step by the diffusion UNet, guided by multi-scale features from dual T2I-Adapters. The generated latent is then decoded by the Temporal VAE decoder using reprojected history features, followed by inverse tonemapping and albedo modulation to restore the final full-dynamic-range global illumination.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Stream<br/>Direct lighting / G-buffers / Sparse GI hints"] --> B["Albedo Demodulation & Tonemapping<br/>Strip texture bias & compress HDR range"]
B --> C["Dual Adapter Feature Injection<br/>Decouple geometry/materials and lighting cues"]
C --> D["One-Step Diffusion UNet<br/>SD-Turbo latent space predicts indirect lighting"]
D --> E["Temporal VAE Decoder<br/>Motion-reprojected history fusion for smooth decoding"]
E --> F["Inverse Tonemapping & Output Modulation<br/>Restore dynamic range & surface textures"]
Key Designs¶
1. Albedo Demodulation and Non-Linear Tonemapping: Decoupling Surface Textures from Pure Illumination Directly predicting indirect illumination over textured, colored surfaces leads diffusion models into hallucinating texture-dependent lighting artifacts. To prevent this, the direct lighting hint is demodulated by dividing by albedo, removing surface texture patterns so the generative model strictly focuses on radiative light transport. Furthermore, because physical radiance exceeds the standard VAE distribution, input values are compressed from high dynamic range into \([0.0, 1.0]\) using \(\mu\)-Law tonemapping (\(\mu = 5011\), mapping up to 1000 nits to 1.0): $\(y = \frac{\ln(1 + \mu x)}{\ln(1 + \mu)}\)$ After the latent is decoded, an inverse \(\mu\)-Law tonemap converts the result back to linear HDR space, followed by albedo modulation to restore fine albedo textures before compositing with direct lighting.
2. Dual Adapter Multi-Scale Injection: Preventing Condition-Concatenation Distribution Shift In single-step distilled diffusion, concatenating numerous high-dimensional condition images (normals, roughness, metallic, sparse GI) directly into the noisy latent disrupts the model's pre-trained latent distribution. The proposed framework employs two dedicated T2I-Adapters: one for material and geometry conditions (normals, roughness, metallic) and another for lighting cues (1-spp noisy GI, radiance cues). Features extracted across hierarchical adapter scales are equally injected into the downsampling and upsampling blocks of the diffusion UNet. Passing sparse noisy GI through an adapter rather than the VAE encoder dramatically improves glossy reflections and specular highlights.
3. Temporal VAE Decoder (TVAE): Plug-and-Play Lightweight Sequence Stabilization Conventional 2D VAE decoders deconvolve latent vectors frame by frame independently, where imperceptible latent fluctuations amplify into pronounced pixel flicker. Instead of retraining the entire UNet with heavy 3D spatio-temporal attention, the framework replaces the frozen SD-VAE decoder at inference time with a fine-tuned Temporal VAE decoder. The TVAE integrates a convolutional History Encoder that takes the previous output frame warped by motion vectors and extracts multi-scale temporal context, injecting it into the VAE decoder via zero convolutions. Where current latents match reprojected historical context, pixel consistency is strongly enforced, eliminating flicker across infinite video sequences.
Loss & Training¶
The framework is optimized via a two-stage decoupled training protocol:
- Joint Diffusion UNet and Adapter Fine-Tuning: Keeping the pre-trained SD-VAE frozen, the UNet is initialized from SD 2.1-Turbo and fine-tuned jointly with the adapters. Noise is injected at timestep \(t = 499\) following a single-step image-to-image pipeline with the DDIMScheduler. Supervision is applied directly in pixel space on the composited output against ground-truth path-traced references using \(L_1\) and perceptual LPIPS losses:
$\(\mathcal{L}_{\text{diff}} = \mathcal{L}_1(I_{\text{pred}}, I_{\text{ref}}) + \lambda_{\text{LPIPS}} \mathcal{L}_{\text{LPIPS}}(I_{\text{pred}}, I_{\text{ref}})\)$
Optimized with AdamW (\(\beta_1 = 0.9, \beta_2 = 0.999\), weight decay \(0.01\), learning rate \(5 \times 10^{-5}\)) on AMD Instinct™ MI210 GPUs for 50 epochs with batch size 32.
- Temporal VAE Decoder Fine-Tuning: Initialized from sd-vae-ft-mse-original, the TVAE decoder is trained on frame sequences while keeping the VAE encoder frozen. Gaussian noise is added to the current frame latent to encourage reliance on the reprojected previous frame via the history encoder, optimized with \(L_1\) loss for 10 epochs. In parallel, standard VAE reconstruction is supervised on DIV2K with zeroed history to prevent cumulative color drift.
Key Experimental Results¶
Main Results¶
Evaluated on synthetic indoor test scenes with dynamic camera trajectories, moving lights, and diverse material shaders at \(512 \times 512\) resolution against neural GI baseline BCNN, forward diffusion baseline RGB↔X, and video diffusion baseline DiffusionRenderer (White Room, Kitchen, and Living Room are entirely unseen scenes):
| Scene | Metric | BCNN | RGB↔X | DiffusionRenderer | Ours |
|---|---|---|---|---|---|
| White Room | PSNR↑ / SSIM↑ / LPIPS↓ / FovVDP↑ | 33.18 / 0.97 / 0.04 / 6.8811 | 10.21 / 0.65 / 0.26 / 0.6669 | 18.70 / 0.79 / 0.13 / 3.0763 | 33.80 / 0.98 / 0.04 / 7.0661 |
| Kitchen | PSNR↑ / SSIM↑ / LPIPS↓ / FovVDP↑ | 21.93 / 0.88 / 0.11 / 6.0229 | 16.68 / 0.53 / 0.36 / 4.5451 | 13.46 / 0.58 / 0.38 / 3.8247 | 23.96 / 0.91 / 0.10 / 6.4124 |
| Living Room | PSNR↑ / SSIM↑ / LPIPS↓ / FovVDP↑ | 21.41 / 0.83 / 0.15 / 5.1514 | 14.43 / 0.58 / 0.28 / 3.3022 | 16.33 / 0.57 / 0.27 / 2.9937 | 23.63 / 0.86 / 0.14 / 5.8753 |
| Staircase | PSNR↑ / SSIM↑ / LPIPS↓ / FovVDP↑ | 25.86 / 0.95 / 0.05 / 6.1955 | 18.98 / 0.60 / 0.22 / 4.1449 | 13.84 / 0.50 / 0.27 / 3.1187 | 30.23 / 0.97 / 0.04 / 6.4422 |
| Shadow Room | PSNR↑ / SSIM↑ / LPIPS↓ / FovVDP↑ | 19.39 / 0.90 / 0.10 / 4.8756 | 14.34 / 0.79 / 0.16 / 2.6496 | 12.11 / 0.74 / 0.23 / 2.2668 | 21.12 / 0.95 / 0.05 / 5.8838 |
Ablation Study¶
Ablation on video perceptual quality metric CGVQM (sensitive to spatio-temporal distortions) demonstrates the impact of the TVAE decoder and albedo demodulation:
| Test Scene | Full Model (Ours) | w/o TVAE | w/o Albedo Demod. | w/o Both |
|---|---|---|---|---|
| White Room | 80.36 | 80.60 | 75.18 | 75.99 |
| Kitchen | 75.97 | 74.74 | 73.86 | 72.86 |
| Living Room | 75.08 | 73.95 | 74.02 | 73.39 |
| Staircase | 69.80 | 70.18 | 60.94 | 60.27 |
| Shadow Room | 87.23 | 86.55 | 87.66 | 87.91 |
Runtime benchmarks on an NVIDIA RTX™ 3090 GPU demonstrate that the proposed model executes in 0.29s per frame (8.68 GB VRAM), and compact model Ours-C in 0.284s (7.97 GB VRAM). In comparison, RGB↔X requires 3.25s, while DiffusionRenderer takes 52.09s (21.81 GB VRAM) for 24-frame chunks. The proposed method is 11× to 179× faster than competing generative baselines.
Key Findings¶
- Albedo demodulation is indispensable for texture preservation: in heavily textured scenes like Staircase, omitting demodulation drops CGVQM from 69.80 to 60.94 due to severe texture hallucinations.
- Temporal stability gains are substantial: while per-frame RGB↔X exhibits severe flicker (FovVDP 0.6669 on White Room) and DiffusionRenderer experiences 24-frame chunk boundary seams (FovVDP 3.0763), the TVAE achieves a smooth 7.0661 without video diffusion overhead.
- Dual adapters dedicated separately to lighting cues versus geometry/materials prevent distribution contamination, significantly improving specular reflections on glossy tables and metallic items.
Highlights & Insights¶
- Decoupled Temporal Stabilization at the VAE Stage: Rather than imposing costly temporal attention mechanisms inside diffusion UNet, temporal coherence is maintained post-hoc via motion-warped latent decoding with zero convolutions, achieving flicker-free video at single-frame latency.
- Demodulation-Modulation Synergy with Generative Priors: Factoring out albedo before the diffusion process forces the generative backbone to act purely as a radiative transfer solver, restoring complex surface details losslessly at the output.
- Robust Real-World Generalization: The compact variant (Ours-C) trained only on synthetic data operates robustly on real-world imagery with inverse-rendered intrinsics and RAFT optical flow, exhibiting high tolerance to noisy estimates.
Limitations & Future Work¶
- Distant Specular Reflections and Deep Occlusion Shadows: Indirect shadows in directly shadowed recesses and color fidelity in distant object reflections can be under-predicted due to synthetic dataset distribution boundaries.
- Reliance on Explicit Motion Vectors: The TVAE assumes reliable motion vectors; while RAFT optical flow provides reasonable fallback, inaccurate flow boundaries can introduce localized blurring. Developing motion-vector-free temporal VAE decoders represents an impactful next step.
Related Work & Insights¶
- vs RGB↔X (ICLR 2024): RGB↔X uses multi-step diffusion with concatenated condition latents, leading to prominent hallucinations and temporal flicker; the proposed one-step adapter architecture runs over 11× faster with strict geometric alignment.
- vs DiffusionRenderer (CVPR 2024): DiffusionRenderer relies on video diffusion chunking with 52s latency per 24 frames and noticeable chunk boundary jumps; the proposed model operates at 0.29s per frame with continuous temporal coherence.
- vs Neural GI (BCNN / SSGI): Screen-space GI fails under occlusion, and BCNN struggles with complex glossy and multi-bounce light transport; the proposed model leverages pre-trained generative priors to synthesize plausible specular highlights and indirect bounces.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Decouples one-step generative lighting inference from video temporal stability via an external TVAE decoder.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across synthetic and real-world scenes with PSNR, FovVDP, CGVQM, and runtime profiling.
- Writing Quality: ⭐⭐⭐⭐⭐ Cohesive structure, clear design motivations, and thorough ablation evidence.
- Value: ⭐⭐⭐⭐⭐ Provides a highly practical, computationally efficient blueprint for real-time generative neural rendering.