Cast and Attached Shadow Detection via Iterative Light and Geometry Reasoning¶

Conference: ECCV 2026
Paper: Official page · PDF
Project: Project page provided in the paper
Area: Shadow Detection / Image Segmentation / Geometry and Lighting Reasoning
Keywords: Cast shadows, attached shadows, surface normals, light estimation, iterative refinement

TL;DR¶

The paper couples three-class shadow segmentation with three-dimensional light-direction estimation through surface-normal-guided feedback, reducing attached-shadow BER from 19.49 for the strongest fine-tuned baseline to 12.92 after three passes, at the cost of richer geometry and class supervision and increased inference time.

Background & Motivation¶

Dark regions in a photograph do not all arise from the same mechanism. A cast shadow falls on an external receiving surface when an object blocks illumination, whereas an attached shadow occurs on the object itself, typically where its surface faces away from the light. Cast shadows often have connected regions and conspicuous boundaries. Attached shadows follow curved surfaces, can be fragmented and low-contrast, and are easily confused with dark materials. Predicting a single mask from darkness alone can miss these regions and leaves downstream removal systems unaware of which shadows are tied to object shape.

Existing methods have not entirely ignored attached shadows. SILT and the CUHK-related data include them, but generally do not learn cast and attached shadows as separate classes. The paper first examines pretrained detectors, then fine-tunes several baselines with full-shadow supervision on its new data. Attached shadows remain substantially harder even after this adaptation. The issue is therefore not simply a shortage of training images: a unified label does not explicitly constrain the relationship between surface orientation, incoming light, and shadow type.

Under a single dominant directional light, normals and light direction indicate whether a surface faces away from the source. Conversely, observed shadows constrain where the light could come from. The proposed reasoning is implemented by two visual networks, not by a language model producing textual explanations. Core idea: estimate illumination from current shadow predictions, use illumination and normals to generate an incomplete attached-shadow prior, and feed that prior back into segmentation so that physical cues and image evidence can refine one another.

Method¶

Overall Architecture¶

The inputs are one RGB image and its normal map; the output assigns pixels to background, cast shadow, or attached shadow. The normals are not sensor ground truth: relative depth is first predicted with Depth Anything V2 and then converted into normals. The shadow detector follows the SILT backbone, while the lighting module uses ConvNeXt-S.

Each pass performs three-class shadow segmentation, estimates a scene-wide three-dimensional light direction using normals and predicted shadows, and generates a local back-facing prior for the next pass. The first pass receives an all-ones prior, and the final configuration uses three passes. This initialization enables detection before a lighting estimate is available; it does not assert that every pixel is actually shadowed.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["RGB + estimated normals<br/>Initial prior: all ones"] --> B["Three-class shadow segmentation"]
    B --> C["Shadow-conditioned light estimation"]
    A --> C
    C --> D["Local back-facing prior feedback"]
    A --> D
    D -->|Prior for the next pass| B
    B -->|Output after the third pass| E["Cast and attached shadow masks"]

Key Designs¶

1. Three-class shadow segmentation: recover shadow extent and distinguish formation mechanisms

The detector jointly receives the image, normals, and current prior, and predicts logits for background, cast shadow, and attached shadow instead of merely shadow versus non-shadow. RGB supplies texture, material, and boundary evidence; normals supply surface orientation; the feedback prior indicates which surfaces should be back-facing under the current lighting explanation. This combination addresses a specific ambiguity: a dark surface facing the light should not be labeled an attached shadow solely because of its intensity, while a weakly contrasted back-facing patch should not be discarded merely because an appearance model misses it.

Supervision operates at two levels. A full-shadow objective keeps the union of the two shadow classes spatially correct, while type-specific supervision requires the classes to remain distinct. Predicting all shadows as one subclass can therefore no longer satisfy the entire objective. Unlike splitting a binary prediction with an object mask after inference, this network predicts the types directly. BiRefNet object masks are used only for annotation and evaluation, not as required inputs to the proposed detector.

2. Shadow-conditioned light estimation: infer an interpretable ray direction from the current segmentation

The lighting module estimates a scene-level three-dimensional vector from normals and predicted shadows. Its direction convention is essential: the vector points from the light source toward the scene, not from a surface toward the light. In camera coordinates, positive x points right, positive y points down, and positive z points into the scene. Ignoring this convention would reverse the inequality used to identify back-facing surfaces.

Not every direction target comes from precise lighting measurements. WSRD uses calibrated fixed illumination, from which the authors manually compute directions. For SOBA and CUHK, they use the displacement from an object centroid to its cast-shadow centroid for the image-plane direction, infer the sign of the depth component from relative object and shadow depths, and combine and normalize the components. This is a heuristic target based on directional-light and geometric approximations, not a calibrated three-dimensional light measurement for every image. It helps prevent the lighting branch from producing arbitrary vectors that assist segmentation without a meaningful physical interpretation.

3. Local back-facing prior feedback: guide complete segmentation without replacing it with a geometric rule

Given the current light direction, the method evaluates its dot product with each surface normal. The prose and the clearly labeled condition in Figure 3 support the following expression:

\[ \mathcal{M}_{\mathrm{attached}}(p)=\mathbf{1}\!\left[\boldsymbol{n}(p)\cdot\boldsymbol{\ell}>0\right]. \]

The positive sign follows from the inward ray convention: an outward surface normal aligned with light propagation faces away from the source. The cached equation has extraction damage; this expression only organizes the condition independently confirmed by the prose and figure, without supplying undocumented implementation details. The map is a partial prior because the dot product tests local orientation, not light-ray visibility or geometric blocking between surface regions. Correct orientation alone is insufficient to determine the complete shadow mask.

The next detector pass receives this prior and uses RGB evidence and normals to cover cases that the prior cannot represent. Its improved shadow predictions then condition another lighting estimate. During training, the prior is generated from the current predicted illumination rather than an always-perfect target, so the detector encounters imperfect physical guidance. The purpose is progressive improvement, not a guarantee that every metric improves monotonically at every pass. Indeed, the five-pass experiment shows that additional refinement can worsen type-specific results.

A Worked Example¶

Consider a curved object resting on the ground. This is an illustration of the mechanism, not an additional experimental case. The object casts a shadow onto the ground and has an attached shadow on its own back-facing side. The first pass uses RGB, normals, and an all-ones prior to produce a coarse segmentation. The lighting module then conditions on those predictions to estimate the incoming ray direction, and the dot-product rule marks locally back-facing surfaces.

The second pass now receives a direction-dependent prior rather than an all-ones map, potentially recovering low-contrast dark patches on the object while retaining the ground shadow as cast. A third update produces the final masks. If the estimated normals on the curved surface are flipped, however, feedback may repeatedly reinforce the wrong locations. Geometry is thus both a source of capability and an important failure pathway.

Loss & Training¶

For full-shadow supervision, log-sum-exp aggregates the cast and attached logits and combines them with the background logit into a binary shadow decision, supervised with BCE and a Dice regularizer. Type supervision uses three-class cross-entropy plus a class-conditional margin: the cast logit should exceed the attached logit at cast pixels, with the reverse requirement at attached pixels. The target margin is 0.2. This penalty further separates confusing types; it does not introduce a fourth prediction class.

Lighting supervision contains three terms: weighted BCE aligns the geometry-derived prior with attached-shadow annotations; an L1 direction loss pulls the prediction toward its target; and a unit-norm constraint controls the direction vector's scale. Shadow and lighting losses are optimized together. The reported weights for Dice, the class margin, attached-mask alignment, direction, and unit norm are 0.1, 0.2, 0.4, 0.5, and 0.1, respectively.

Implementation uses PyTorch and Adam with learning rate \(5\times10^{-4}\), 20 epochs, and batch size 4. The main text does not clearly specify the differentiable treatment of the hard-threshold prior when used with BCE. The available cache also does not sufficiently document loss aggregation across passes or cross-pass gradient handling. It would therefore be inappropriate to invent a sigmoid temperature, straight-through estimator, or stop-gradient configuration. Several extracted loss equations have missing symbols; only components and weights supported by the prose are retained here.

The dataset contains 1458 images: 220 from WSRD, 710 from SOBA, and 528 from CUHK, split into 1166 training images and 292 test images. For WSRD, color-space subtraction of shadow and shadow-free image pairs first supplies full-shadow masks, followed by manual type annotation. Other sources use object masks for manual annotation and refinement. Fine-grained types are labeled for selected foreground objects. Remaining shadows that cannot be attributed to those objects are undefined: they participate in full-shadow supervision and evaluation but are excluded from cast/attached supervision and evaluation, rather than being treated as background.

Key Experimental Results¶

Main Results¶

The following selection comes from Table 2. Lower BER and higher F1 are better; values retain the paper's percentage scale. For clarity, the standard metric definitions are:

\[ \mathrm{BER}=100\times\frac{1}{2}\left(\frac{\mathrm{FN}}{\mathrm{TP}+\mathrm{FN}}+\frac{\mathrm{FP}}{\mathrm{TN}+\mathrm{FP}}\right),\qquad \mathrm{F1}=100\times\frac{2\mathrm{TP}}{2\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}. \]

TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives within the applicable binary evaluation region. An asterisk indicates fine-tuning on the proposed training set with full-shadow supervision; unmarked SILT uses SBU-pretrained weights. Single-mask baselines are split using object masks. Cast evaluation excludes undefined regions, and attached evaluation is restricted to object regions. This is a comparison under the paper's protocol, not a module replacement under identical supervision.

Method	Full BER	Full F1	Cast BER	Cast F1	Attached BER	Attached F1
SILT	20.20	71.51	4.64	80.39	32.70	60.23
BDRAR*	8.15	83.95	5.18	72.86	20.75	82.69
FSDNet*	10.17	85.10	5.82	81.94	19.49	80.57
FDRNet*	8.62	83.00	4.65	73.27	23.11	81.42
SILT*	6.51	82.98	4.52	68.31	26.57	80.72
Ours	6.50	86.61	5.04	85.46	12.92	86.49

Attached BER improves over the strongest fine-tuned baseline, FSDNet, by 6.57 percentage points, or approximately 33.71% relatively. This is not a win on every metric: cast BER is 5.04 versus 4.52 for SILT, and the full-shadow BER difference of 6.50 versus 6.51 is very small. The principal advantages concern attached shadows and F1.

Ablation Study¶

The next table is drawn from Table 3. The non-iterative model still jointly learns shadows and lighting with normal maps; it simply does not feed the partial attached prior back into detection. It should not be described as a model without lighting.

Configuration	Full BER	Full F1	Cast BER	Cast F1	Attached BER	Attached F1
No normals	7.60	85.46	7.82	79.33	20.87	77.07
Non-iterative	7.57	85.99	6.14	87.75	15.10	83.91
2 passes	6.63	87.77	5.41	86.75	13.22	85.63
3 passes, final model	6.50	86.61	5.04	85.46	12.92	86.49
5 passes	6.11	87.14	5.83	85.57	13.51	85.74
No margin loss	7.19	86.25	6.16	85.69	13.45	85.38
No attached alignment loss	6.89	87.03	5.81	85.26	14.61	84.24
No direction loss	7.50	86.42	4.97	85.02	15.26	83.59

Key Findings¶

Normals are the most influential tested component: removing them worsens attached BER from 12.92 to 20.87, a 7.95-point increase. Geometry does more than sharpen boundaries; it helps identify the physical mechanism behind a dark region.
Feedback provides an additional gain: attached BER falls from 15.10 without iteration to 12.92 after three passes. It rises to 13.51 after five passes despite lower full-shadow BER. The evidence supports neither unlimited refinement nor simultaneous improvement of all class-specific metrics.
Among the listed individual loss ablations, removing direction supervision produces the largest attached-BER degradation, to 15.26. This supports directional supervision but does not establish that heuristic targets equal true illumination.
Per-image runtime rises from 0.18 seconds without iteration to 0.55 seconds for three passes, approximately 3 times slower. The cached main text does not adequately specify hardware or preprocessing coverage, so these numbers should not be generalized into end-to-end deployment speed.
Cross-dataset results on SBU-TimeLapse process frames independently and demonstrate qualitative temporal consistency, without additional cross-dataset BER values. In the ShadowFormer removal user study, separate masks with object-region refinement receive 80.7% preference versus 19.3% for a joint mask. Participant counts and confidence intervals are not reported in the main text; preference is not an objective reconstruction improvement.

Highlights & Insights¶

The physical meaning of a shadow becomes a learned feedback constraint rather than merely a reason to expand the receptive field. A local dot-product rule supplies interpretable directional guidance, while the network compensates for its incomplete visibility and appearance modeling.
Supervising total extent separately from fine-grained type preserves the usefulness of unified shadow detection without allowing a correct union to hide confused subclasses. This hierarchical supervision pattern is potentially useful for other segmentation tasks with parent and child classes.
Undefined labels isolate uncertain ownership instead of presenting partial annotation as complete ground truth. The practical annotation advantage should always be reported together with its restricted evaluation scope.

Limitations & Future Work¶

The authors acknowledge that normal errors propagate into shadow predictions, especially when directions are flipped. Confidence-gated geometric priors or joint normal refinement are possible future directions, not validated components of this method.
One scene-wide directional light does not adequately represent multi-light interiors, soft shadows from area sources, or strongly indirect nighttime illumination. Local back-facing orientation is not a complete occlusion test; extending to multiple lights would require revisiting the prior and its supervision.
The dataset contains only 1458 images, with type annotation concentrated on foreground objects. Full-image qualitative generalization does not provide quantitative guarantees on unlabeled background regions. More diverse lighting, complete scene-level type labels, and cross-dataset quantitative evaluation are needed.
Inputs, type labels, and direction supervision differ between this method and the fine-tuned baselines, so the entire improvement cannot be attributed to the loop. No-normal and non-iterative ablations offer partial evidence, but broader matched-input, matched-supervision comparisons and random-seed variance are still missing.
The differentiable treatment of the hard prior, complete numerical construction of heuristic lighting targets, and exact unit-norm loss implementation cannot be reliably recovered from the damaged cached equations. Reproduction requires consulting the original equations and implementation rather than filling in those details speculatively.

Compared with SILT: SILT iteratively adjusts labels to handle noisy shadow annotations. This paper retains its detection backbone but iterates between predicted shadows and lighting conditions. The shared word "iterative" does not mean the training mechanisms are the same.
Compared with FDRNet and SDCM: FDRNet decomposes and reweights features to reduce intensity bias; SDCM exploits complementary shadow and non-shadow branches. This method adds normals and explicit light direction to distinguish formation mechanisms, rather than relying solely on how appearance features are organized.
Compared with earlier joint geometry and illumination inference: Panagopoulos and colleagues already studied joint inference, so using shadows to reason about lighting is not itself new. The advance here is a learned dual-module loop combined with explicit attached-shadow annotation and evaluation.
Implications for shadow removal: The ShadowFormer study suggests that dark regions with different causes may benefit from different processing. However, the experiment also changes object-region refinement, so preference differences cannot be isolated as the pure effect of predicting an extra class.

Rating¶

Novelty: 4/5. Fine-grained shadow types and geometry-light feedback fit together well, although the physical idea of joint inference has precedents.
Experimental Thoroughness: 3/5. Main comparisons and geometry, iteration, and loss ablations are useful; complex lighting, statistical stability, and strictly matched supervision remain underexplored.
Writing Quality: 4/5. The problem, physical intuition, and module relationships are clear, but some reproduction details cannot be verified from the current main-text cache.
Value: 4/5. Type annotations and an interpretable baseline are valuable for shadow understanding and editing, with applicability limited by normal quality and inference cost.