VSDiffusion: Taming Ill-Posed Shadow Generation via Visibility-Constrained Diffusion

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/Jadelingli/VSDiffusion
Area: Image Generation
Keywords: Shadow Generation, Image Composition, Diffusion Model, Visibility Prior, Conditional Control

TL;DR

Addressing the ill-posed inverse problem of foreground shadow generation in image composition, VSDiffusion introduces a two-stage visibility-constrained diffusion framework that combines residual control prior injection, shadow-gated cross-attention, and spatially soft prior-weighted loss to achieve geometrically consistent, sharp shadows with high computational efficiency.

Background & Motivation

In photorealistic image composition, synthesizing cast shadows for inserted foreground objects that are geometrically consistent with background environments is essential for perceived realism. Early physics-driven methods rely heavily on explicit 3D scene reconstruction and illumination estimation, which struggle to generalize to in-the-wild uncalibrated imagery. Recent deep learning approaches replace explicit physical rendering with data-driven neural pipelines, yet they predominantly depend on weak supervision such as binary masks or coarse bounding conditions. Consequently, models frequently learn superficial texture correlations rather than genuine physical shadow formation principles, leading to lighting mismatches, shape distortions, and floating artifacts.

From a physical and mathematical standpoint, shadow generation from a single composite image lacks explicit 3D geometry and source illumination parameters. This makes the task an inherently ill-posed one-to-many inverse problem where countless visually plausible shadow configurations might map to the same composite input. Physically, a shadow originates from an occlusion-induced visibility transition (\(V: 1 \to 0\)) along light rays between the light source and receiver surface. Modeling the relative spatial interplay among the light source, the occluding caster, and the receiver geometry provides the exact physical constraints needed to shrink the unconstrained solution space toward the plausible manifold.

Standard conditional diffusion pipelines that apply dense conditioning across all U-Net layers tend to over-constrain the generation process, erasing delicate background textures, and fail catastrophically in reference-free scenes (BOS-free) where background shadow cues are entirely absent. Core idea: decompose shadow generation into coarse localization and visibility-constrained diffusion stages, injecting inverse-rendered lighting and monocular depth priors into sparse U-Net anchors via a residual control encoder and shadow-gated cross-attention, coupled with a soft prior map that dynamically reallocates optimization gradients to error-prone boundary regions.

Method

Overall Architecture

VSDiffusion executes a two-stage generative pipeline designed to progressively shrink the feasible solution space. In Stage I, given the composite image \(I_c\), foreground mask \(M_{fo}\), and background shadow mask \(M_{bs}\), a dual-branch encoder with cross-attention integration predicts a coarse foreground shadow mask \(M_{fs}^{(1)}\), which is merged with \(M_{fo}\) to establish a composite spatial mask \(M_{comb}\). In Stage II, conditional diffusion denoising operates alongside a Visibility Control Branch (VCB). The VCB extracts spherical harmonics illumination maps \(I_{light}\) and monocular depth maps \(I_{depth}\), encodes them through a Residual Control Encoder (RCE), and sparsely injects structural guidance via Shadow-Gated Cross-Attention (SGCA) across three strategic U-Net anchor scales. Concurrently, a High-Frequency Guided Enhancement (HFGE) module refines penumbra transitions using shallow encoder edge features, while a lightweight U-Net generates a soft prior weight map to guide spatially reweighted optimization during training.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Input Data<br/>Composite Ic + Object Mask Mfo + Background Shadow Mbs"] --> B["Stage 1: Coarse Mask Prediction<br/>Dual Encoders + Cross-Attention Integration"]
    B --> C["Combined Spatial Mask Mcomb<br/>Logical OR of Mfs(1) and Mfo"]

    subgraph VCB["Visibility Control Branch VCB"]
        direction TB
        D1["Visibility Prior Estimator<br/>Inverse-Rendered Ilight + Monocular Depth Idepth"] --> D2["Residual Control Encoder RCE<br/>7 Residual Blocks + Zero-Conv Interface"]
    end

    C --> E["Stage 2: Diffusion Denoiser U-Net<br/>Image-Space Conditional Denoising Iteration t=T...1"]
    D2 -->|Multi-scale sparse conditioning| F["Shadow-Gated Cross-Attention SGCA<br/>Gated modulation at early/mid/late anchors"]
    F --> E

    E --> G["High-Frequency Guided Enhancement HFGE<br/>Shallow gradient/Laplacian cues into late decoder"]
    G --> H["Output Shadowed Image Ig' & Refined Mask"]

Key Designs

1. Visibility Control Branch: Dual Prior Extraction via Inverse Rendering and Monocular Depth Pure data-driven diffusion networks cannot reliably deduce 3D occlusion interactions from 2D pixel grids alone, resulting in directionally drifting shadows. To solve this, the Visibility Control Branch introduces an analytical inverse-rendering formulation and monocular depth estimation without requiring auxiliary network prediction for illumination. For global lighting, a shared encoder with three lightweight decoders decomposes albedo \(\alpha\), shadow weight \(s\), and surface normal \(n\). Formulated under Lambertian reflectance: $\(i(x, y) = \alpha(x, y) \odot s(x, y) \, B\bigl(n(x, y)\bigr) \, I\)$ where \(B(n)\) represents the second-order Spherical Harmonics (SH) basis functions and \(I\) denotes unknown SH lighting coefficients. Solving this linear least-squares problem analytically yields the global lighting map \(I_{light}\), while a pre-trained MiDaS model extracts the metric depth representation \(I_{depth}\). To avoid training instability and initial zero-gradient signal blocks common in standard ControlNet architectures, a lightweight Residual Control Encoder (RCE) with 7 residual blocks stably encodes condition features into the diffusion stream.

2. Shadow-Gated Cross-Attention: Sparse Multi-Scale Anchoring and Adaptive Feature Filtering Dense feature concatenation across all diffusion layers inevitably over-conditions the model, causing background texture smoothing and excessive computational overhead. SGCA circumvents this by selecting three strategic anchor resolutions within the U-Net (early, middle, and late stages) to handle geometric alignment, global layout, and fine boundary tuning, respectively. Taking U-Net feature \(X^{(s)}\) as Query and RCE conditional feature \(C^{(s)}\) as Key and Value, SGCA applies a lightweight gate predictor \(\psi(\cdot)\) to generate adaptive spatial weights: $\(G^{(s)} = \sigma\left(\psi(C^{(s)})\right)\)$ $\(\hat{X}^{(s)} = X^{(s)} + G^{(s)} \odot \text{unflat}\left(\text{MHA}(Q, K, V)\right)\)$ This spatial gating amplifies prior conditioning inside the projected shadow zone while suppressing it in irrelevant background regions, preserving native generative texture diversity while strictly enforcing global geometric consistency.

3. High-Frequency Guided Enhancement: Shallow Structural Residuals for Penumbra Refinement Diffusion models naturally exhibit an inverted frequency progression: high-frequency components are destroyed earliest during forward diffusion and recovered latest during reverse sampling, frequently leaving shadow contours blurry or jagged. The HFGE module extracts robust edge features directly from the shallow encoder representations \(F_e\). After spatial Gaussian filtering suppresses high-frequency pseudo-textures, horizontal and vertical Sobel gradients yield gradient magnitude \(G = \sqrt{g_x^2 + g_y^2 + \epsilon}\), which is combined with Laplacian second-order response \(L\) and normalized to form high-frequency feature \(H(F_e) = \text{Norm}(G + \alpha L)\). Aligned via \(1 \times 1\) convolutions, bilinear resizing, and a CBAM attention mechanism, this guidance is injected residually into the high-resolution decoder: $\(F_d' = F_d + \gamma \cdot \text{CBAM}\left(\phi(H(F_e))\right)\)$ This design sharpens shadow penumbras and boundary contours without compromising background surface textures.

Loss & Training

The framework is optimized in three steps: (i) pretraining Stage I independently for coarse spatial localization; (ii) training Stage II with Stage I frozen; and (iii) end-to-end joint fine-tuning with a reduced learning rate.

Stage I is supervised by binary cross-entropy and Dice loss \(\mathcal{L}_{s1}\). In Stage II, standard global \(L_1/L_2\) losses dilute gradients over thin shadow contours and penumbras. To address this, the Sprior-Weighted Loss (SWL) uses a lightweight U-Net \(G_p\) to predict a soft prior map \(S_{prior} \in [0, 1]\) conditioned on \(I_{light}, I_{depth}, M_{fo}, M_{fs}^{(1)}\). Mean normalization prevents gradient collapse into trivial zero-weight solutions: $\(\hat{S}_{prior}(x, y) = \frac{S_{prior}(x, y)}{\frac{1}{N}\sum_{i,j} S_{prior}(i,j) + \epsilon}\)$ Multiplying this weight against pixel-level image and mask reconstruction error \(\ell_{base}(x, y)\) yields the reweighted loss: $\(\mathcal{L}_{swl} = \frac{1}{N} \sum_{x, y} \hat{S}_{prior}(x, y) \, \ell_{base}(x, y)\)$ The complete Stage II objective is \(\mathcal{L}_{s2} = \mathcal{L}_{base} + \mathcal{L}_{swl}\). Finally, the joint training loss balances both stages: $\(\mathcal{L}_{joint} = \mathcal{L}_{s2} + \gamma \mathcal{L}_{s1}\)$ with hyperparameters set to \(\lambda_1=0.2, \lambda_2=0.5, \gamma=0.1\).

Key Experimental Results

Main Results

Evaluated on the DESOBAv2 benchmark (500 BOS samples with background shadow references and 250 BOS-free samples without references), reporting global/local RMSE (GR, LR), global/local SSIM (GS, LS), and global/local Balanced Error Rate (GB, LB; lower is better):

Evaluation Setting	Method	Global RMSE (GR)↓	Local RMSE (LR)↓	Global SSIM (GS)↑	Local SSIM (LS)↑	Global BER (GB)↓	Local BER (LB)↓
BOS Setting (with reference)	ShadowGAN	7.511	67.464	0.961	0.197	0.446	0.890
	AR-SG	7.335	58.037	0.961	0.241	0.383	0.682
	SGRNet	7.184	68.255	0.964	0.206	0.301	0.596
	SGDiffusion	6.098	53.611	0.971	0.370	0.245	0.487
	GPSDiffusion	5.896	46.713	0.966	0.374	0.213	0.423
	VSDiffusion (Ours)	6.422	46.542	0.964	0.377	0.182	0.360
BOS-free Setting (no reference)	ShadowGAN	17.325	76.508	0.901	0.060	0.425	0.842
	AR-SG	16.067	63.713	0.908	0.104	0.349	0.682
	SGRNet	15.596	60.350	0.909	0.100	0.271	0.534
	SGDiffusion	15.110	55.874	0.913	0.117	0.233	0.452
	GPSDiffusion	13.809	55.616	0.917	0.166	0.197	0.384
	VSDiffusion (Ours)	14.629	55.377	0.912	0.184	0.194	0.373

In terms of computational efficiency, VSDiffusion has 547.2M total parameters (300.4M trainable) and requires 1.6986 seconds per image for inference. In contrast, GPSDiffusion requires 1.479B parameters and 6.9049 seconds, while SGDiffusion requires 1.458B parameters and 8.0791 seconds. VSDiffusion cuts parameters by over 62% while speeding up inference by more than \(4\times\).

Ablation Study

Ablation experiments on DESOBAv2 isolating VCB, HFGE, and SWL components:

Config	VCB	HFGE	SWL	GR↓	LR↓	GS↑	LS↑	GB↓	LB↓	Note
Baseline	-	-	-	6.655	50.598	0.962	0.364	0.221	0.438	Base two-stage diffusion pipeline
+ VCB	✓	-	-	6.592	48.265	0.963	0.366	0.203	0.403	Lighting and depth structural priors
+ HFGE	-	✓	-	6.621	49.214	0.963	0.371	0.211	0.407	Shallow high-frequency residual enhancement
+ SWL	-	-	✓	6.580	48.876	0.963	0.369	0.204	0.398	Spatially weighted boundary loss
VCB + HFGE	✓	✓	-	6.571	48.147	0.964	0.374	0.193	0.381	Structural conditioning + edge cues
Full Model	✓	✓	✓	6.422	46.542	0.964	0.377	0.182	0.360	Complete model (best performance)

Detailed ablations on the SGCA mechanism show that removing spatial gating (w/o gated attention) degrades LB from 0.360 to 0.378, while replacing sparse injection with all-layer dense conditioning (w/o sparse injection) degrades LB to 0.383, validating the benefit of sparse anchor conditioning.

Key Findings

• Substantial Leap in Shadow Boundary Fidelity: On the critical local balanced error rate (LB), VSDiffusion achieves 0.360 on BOS (a 14.9% relative error reduction over GPSDiffusion's 0.423) and 0.373 on BOS-free, demonstrating precise contour alignment.
• Robust Generalization in Reference-Free Scenarios: In the challenging BOS-free setting where background shadow cues are absent, prior methods struggle with orientation ambiguity. VSDiffusion derives global spherical harmonics directly from scene illumination, preventing inverted shadow artifacts.
• Effective Adaptive Gradient Allocation: The learned soft prior map focuses high-loss weights precisely on penumbra transitions and thin boundaries, directly reducing local RMSE (LR) by 1.605 when added to VCB+HFGE.

Highlights & Insights

• Pragmatic Synergy between Inverse Rendering and Generative Diffusion: Rather than attempting full differentiable rendering within diffusion loops, the framework extracts low-order spherical harmonics and monocular depth as compact geometric priors, injecting them via sparse gated cross-attention to maintain high visual fidelity and strict physical plausibility.
• Budget-Preserving Adaptive Reweighting: The self-supervised soft prior network \(G_p\) paired with mean normalization autonomously detects boundary failure zones without explicit supervision, providing an effective blueprint for resolving gradient dilution in localized image editing tasks.
• High Performance with Compact Efficiency: By avoiding heavy dense conditioning modules, the complete model runs on 547M parameters and 1.7-second inference, offering a practical, production-ready solution for automated image composition pipelines.

Limitations & Future Work

• Complex Multi-Light and Non-Lambertian Scenarios: The second-order spherical harmonics assumption relies on diffuse Lambertian reflectance and distant illumination; scenes with high-frequency point lights, colored spotlights, or specular reflections can introduce bias into lighting estimation.
• Complex 3D Non-Planar Receivers: On highly irregular or stepped receiver geometries, 2D monocular depth maps cannot fully substitute for true 3D spatial ray-tracing.
• Future Directions: The authors suggest extending the visibility constraint framework to universal subject-driven image editing and developing reference-free adaptive shadow intensity calibration.

Related Work & Insights

• vs GPSDiffusion: GPSDiffusion uses bounding box prediction and discrete shape prototype retrieval to constrain shadows, relying on synthetic templates without physical visibility modeling; VSDiffusion directly models occlusion physics and inverse lighting, cutting model parameters by >62% while eliminating discrete retrieval bottlenecks.
• vs SGDiffusion: SGDiffusion integrates ControlNet across all layers, incurring significant inference latency and smoothing fine textures; VSDiffusion uses three-anchor sparse injection with adaptive shadow gating, delivering sharper contours at four times the inference speed.

Rating

• Novelty: ⭐⭐⭐⭐☆ Elegantly reduces the ill-posed shadow generation space using visibility and analytical inverse rendering priors within a diffusion model.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across BOS and BOS-free benchmarks, rigorous modular ablations, gradient heatmap visualizations, and efficiency profiling.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulations, intuitive physical motivations, and coherent architectural design.
• Value: ⭐⭐⭐⭐⭐ Establishes a highly efficient, accurate benchmark for physically consistent shadow generation in realistic image composition.