From Phase to Phenomenon: Self-Supervised Learning of Subsurface Scattering with Minimal Phase-shift Inputs¶

Conference: ECCV2026
arXiv: 2606.29461
Code: Yes (GitHub, link in paper)
Area: Subsurface Scattering / Self-Supervised Learning
Keywords: Subsurface Scattering, Self-Supervised Learning, Relighting, Phase-Shift Profilometry, SimSiam

TL;DR¶

This work proposes a self-supervised pre-training framework that learns general subsurface scattering (SSS) representations using a projector-camera system from only 8 high-frequency phase-shift profilometry (PSP) images. It then predicts anisotropic scattering footprints via a decoder to achieve zero-shot relighting, reducing the data acquisition requirements for new objects from thousands of images to just 8.

Background & Motivation¶

Subsurface scattering (SSS) is the physical phenomenon where light penetrates the surface of a translucent material, undergoes multiple scattering events internally, and exits from another point. SSS determines the visual appearance of materials such as skin, fruit, and wax. Accurately reconstructing SSS is crucial for applications like film special effects, digital twins, and virtual try-ons. However, fully measuring the SSS response (i.e., the point spread function) at every point on an object's surface is extremely expensive. Point-by-point laser scanning methods (e.g., DISCO) require millions of measurements, and even more efficient schemes (e.g., dot-matrix projection) require thousands of images. This data acquisition bottleneck hinders the scalability of SSS modeling to new objects.

From another perspective, high-frequency Phase-Shift Profilometry (PSP) is widely used in high-precision 3D scanning to reconstruct object geometry. Interestingly, the degree of stripe blurring and local contrast in PSP images directly reflects the local SSS intensity—stripes on strongly scattering objects (e.g., apples) blur severely, while those on weakly scattering objects (e.g., pears) remain sharp. However, existing methods have historically treated SSS as a disturbance to be suppressed in 3D reconstruction, rather than actively utilizing the scattering information inherent in PSP images.

The key insight of this paper is that since PSP images naturally encode SSS information, scattering representations can be learned directly from them, thereby bypassing expensive point-by-point illumination acquisition. Core Idea: Utilizing only 8 high-frequency phase-shift strip projection (PSP) images, the method learns general subsurface scattering representations through SimSiam non-contrastive self-supervised pre-training, and then employs a decoder to predict anisotropic scattering footprints, achieving zero-shot relighting.

Method¶

Overall Architecture¶

The overall framework consists of a two-stage training process and an inference phase. Stage 1 (Self-supervised pre-training): An encoder is pre-trained under the SimSiam non-contrastive learning framework using a projector-camera system. Eight PSP images (4 vertical + 4 horizontal sinusoidal stripes) are captured for the training objects, and a \(90 \times 90\) image patch is extracted for each surface pixel. PSP-specific weak/strong augmentation pairs are applied. Stage 2 (Supervised decoder training): The encoder parameters are frozen, and a progressive upsampling decoder is trained to map the compact representations from the encoder back to \(90 \times 90 \times 3\) SSS scattering footprints. This process is supervised using a self-balancing hybrid loss (\(\mathcal{L}_{\text{SBH}}\)) against the captured ground-truth footprints. Inference phase: For a new object, only 8 PSP images are required. The scattering footprints are predicted pixel-by-pixel via the encoder-decoder pipeline, followed by scaling and accumulation synthesis to achieve relighting under arbitrary illumination patterns.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["8 PSP Images<br/>(4 Vertical + 4 Horizontal)"] --> B["Pixel-wise Extraction of<br/>90x90 PSP Patches"]
    B --> C["SimSiam Self-Supervised Pre-training<br/>with PSP-specific Weak/Strong Augmentation"]
    C --> D["Encoder Frozen"]
    D --> E["Decoder Progressive Upsampling<br/>to Predict SSS Scattering Footprints"]
    E --> F["Self-Balancing Hybrid Loss<br/>\mathcal{L}_{SBH}"]
    F --> G["Relighting: Scale Footprints<br/>× Virtual Illumination → Synthesis"]

Key Designs¶

1. PSP-Specific Data Augmentation and SimSiam Self-Supervised Pre-training

The success of self-supervised learning on non-natural images depends heavily on data augmentation—the augmentation strategy determines what invariances the encoder learns. Standard ImageNet-style augmentations (e.g., random color jitter, Gaussian blur) perform poorly on PSP images. This is because the phase structure in PSP images contains critical geometric and scattering information; weak augmentations may lose effective information, while strong augmentations can destroy the structural integrity of the fringe patterns. To address this, a weak-strong asymmetric two-branch augmentation pipeline designed specifically for PSP images is proposed. The weak augmentation branch only performs random cropping and slight color jitter to preserve the primary phase structure. The strong augmentation branch applies salt-and-pepper noise, Gaussian blur, random flipping, random intensity scaling, random phase jitter (overlaying phase-shift-like noise onto the RGB channels), and random shuffling of the PSP image order (to break the fixed \(2\pi/N\) phase-shifting symmetry). This design forces the encoder to learn scattering-related material invariances while maintaining geometric phase info. The encoder adopts a Pre-activation ResNet architecture to replace standard U-Net downsampling blocks, avoiding dimensional collapse. Training is stabilized by minimizing the negative cosine similarity between the outputs of the weak and strong branch encodings within the SimSiam framework.

2. Progressive Upsampling Decoder with Attention-Gated Skip Fusion

Reconstructing the \(90 \times 90 \times 3\) SSS scattering footprints from the compact representations learned by the encoder is a fine-grained reconstruction task. The core challenge lies in simultaneously preserving the high-frequency sharpness of the scattering peak and the low-frequency energy of the decaying tail. The decoder employs step-by-step sub-pixel convolution (pixel shuffle) to double the spatial resolution at each stage, combined with attention-gated skip fusion. This selects the most relevant spatial information for scattering reconstruction from the encoder's intermediate layer features and utilizes attention weights to suppress background noise and non-scattering textures. Compared to the checkerboard artifacts of transposed convolutions, sub-pixel upsampling increases resolution more naturally. Compared to direct concatenation without gating, the attention gate effectively suppresses interference from non-scattering regions in the encoder features. The decoder is trained with random sampling across multiple training objects, avoiding catastrophic forgetting caused by per-object training.

3. Self-Balancing Hybrid Loss: Composite Optimization for SSS Footprint Distribution

Physically, the distribution of SSS scattering footprints is highly non-uniform: there is a sharp intensity peak at the incident point, which decays exponentially with radius into an extremely weak long-tail region. Simple \(L_1\)/\(L_2\) losses tend to bias optimization toward the peak while ignoring the tail, leading to rigid reconstructed scattering appearances. This work designs a self-balancing hybrid loss (\(\mathcal{L}_{\text{SBH}}\)) that integrates five complementary components: an \(L_1\) data term as the main supervision; a log-domain \(L_1\) loss (\(\mathcal{L}_{\text{log}}\)) to amplify the relative error in the tail region, improving weak scattering recovery precision; structural similarity (\(\mathcal{L}_{\text{SSIM}}\)) to maintain visual perceptual quality; a gradient-domain \(L_1\) loss (\(\mathcal{L}_{\nabla}\)) to sharpen boundary transitions of the scattering peak and prevent color channel collapse; and a variance-weighted MSE (\(\mathcal{L}_{\text{wMSE}}\)) that adaptively weights pixels based on their variance within the batch—high-variance regions (strong scattering, hard-to-predict areas) receive larger weights, automatically balancing the optimization priorities of the peak and the tail. The weight configuration is set to \(\lambda_1 = 1\), \(\lambda_{\text{wMSE}} = 5\), \(\lambda_{\nabla} = 10\), \(\lambda_{\text{SSIM}} = 0.5\), and \(\lambda_{\text{log}} = 1\).

Loss & Training¶

Self-supervised pre-training employs the SimSiam negative cosine similarity loss. Since the PSP data distribution differs significantly from natural images, directly using the original SimSiam hyperparameters leads to dimensional collapse. Therefore, an alternative strategy is adopted: an initial learning rate of \(\text{lr} = 0.05\), a 20-epoch warmup followed by cosine annealing decay with no restarts. The supervised training of the decoder utilizes the self-balancing hybrid loss \(\mathcal{L}_{\text{SBH}}\). The training dataset is composed of 5 objects (green apple, orange, pear, star toy, and shovel) captured from up to 4 views each, totaling 1,126,827 SSS footprint samples.

Key Experimental Results¶

Main Results¶

Evaluation of \(90 \times 90\) scattering footprint reconstruction quality on unseen test objects compared to real camera captures:

Object	\(\text{MSE} \times 10^{-5}\downarrow\)	PSNR\(\uparrow\)	SSIM\(\uparrow\)	\(\text{LPIPS} \times 10^{-2}\downarrow\)
Apple	4.1	44.67	0.96	8.4
Orange	4.9	43.84	0.94	8.6
Crab	3.5	47.98	0.98	6.5
Hand	3.8	42.51	0.96	6.5

Ablation Study¶

Comparison of the impact of PSP-specific augmentation versus standard ImageNet augmentation on the encoder's kNN classification accuracy (tested in a leave-one-view-out manner):

Object	PSP Augment Accuracy\(\uparrow\)	ImageNet Augment Accuracy\(\uparrow\)
Apple	99.6%	99.04%
Pear	99.99%	98.74%
Orange	99.99%	98.92%
Star	99.95%	99.93%
Shovel	91.43%	84.91%

Key Findings¶

PSP-specific augmentation consistently outperforms ImageNet augmentation across all objects, with the gap reaching up to 6.5 percentage points on the highly anisotropic material (Shovel), indicating that domain-customized data augmentation is crucial for SSL on structured light images.
The encoder learns material-related scattering features: PCA-RGB visualization shows that regions with similar scattering characteristics in spatial locations cluster together in the feature space, and regions with sharp scattering changes (such as leaf veins and material boundaries) display clear color transitions.
Relighting results maintain high fidelity even under complex geometries: scattering inter-reflections between the fingers of a sand rake and the highlight regions of citrus fruits are accurately reconstructed.
The decoder successfully predicts the directional dependence of anisotropic scattering footprints, performing remarkably well on materials with highly directional scattering like the shovel.

Highlights & Insights¶

Perspective Shift from "Interference" to "Signal": This work is the first to exploit the SSS-induced stripe blurring effect, typically suppressed in 3D scanning, as a rich source of scattering info. Using the same set of PSP images, both geometric reconstruction and scattering modeling are completed simultaneously, significantly reducing acquisition costs.
Successful Paradigm of SSL on Industrial/Structured Light Vision Data: Domain-customized augmentation is key to transferring SSL from natural to non-natural images. Standard augmentations destroy the structural info of PSP, leading to performance degradation. This suggests that future SSL applications must design tailored augmentation strategies customized to their specific data distributions.
Transferable Design of Self-Balancing Hybrid Loss: The variance-weighted MSE enables the model to automatically focus on difficult regions without manually tuning region-specific weights, which can be extended to other reconstruction tasks with non-uniform distributions (e.g., light fields, MRI).

Limitations & Future Work¶

The scattering footprint is limited to a local \(90 \times 90\) image patch (corresponding to roughly a 15mm radius in physical space), which fails to capture long-range, large-scale light transport (such as deep subsurface scattering inside large translucent objects).
The method is trained for a fixed projector-camera configuration. Generalization across different camera or projector poses has not been validated, which may limit practical deployment flexibility.
It relies on standard dynamic range (SDR) images; noise and dynamic range limitations in weak-signal areas might affect the reconstruction accuracy of the far-tail region, making HDR acquisition a natural direction for future improvement.
The current model relies on deterministic regression. Introducing generative or stochastic modeling (e.g., diffusion models, VAEs) could better represent highly anisotropic, complex scattering behaviors.

vs. DISCO [Goesele et al., 2004]: DISCO uses point-by-point laser scanning to measure SSS, requiring millions of images. In contrast, this method performs inference using only 8 PSP images, achieving an order-of-magnitude reduction in data requirements.
vs. Vicini et al. (Learned SSS): This prior work uses a conditional VAE + MLP to learn efficient BSSRDF sampling, which requires integration into a renderer. In contrast, this approach is a purely data-driven, image-based SSS modeling method that directly predicts pixel-wise scattering footprints.
vs. Majumdar et al. (U-Net baseline): This baseline uses a U-Net to learn the mapping from PSP to footprints end-to-end, which requires 3,000 captured images and offers a non-reusable encoder. In contrast, this work introduces SSL pre-training to reduce the required acquisitions to 8, and the encoder can be independently reused for downstream tasks such as classification.

Rating¶

Novelty: ⭐⭐⭐⭐ The work is the first to introduce self-supervised learning to subsurface scattering modeling, cleverly redefining SSS-induced PSP blurring from a disturbance into a signal source.
Experimental Thoroughness: ⭐⭐⭐⭐ Standard qualitative and quantitative evaluations are conducted on multiple unseen objects, including kNN representation analysis, PCA visualization, and multi-view relighting comparisons. However, the ablation study mainly compares augmentation strategies, while the analysis of the contribution of each individual loss component is somewhat insufficient.
Writing Quality: ⭐⭐⭐⭐ The motivation is clearly articulated, the method's logical chain is coherent, and the figures and tables are highly intuitive. The appendix provides reproducible architectural details.
Value: ⭐⭐⭐⭐ Reducing the SSS acquisition cost from thousands of images to 8 is a substantial advancement. The SSL design methodology can be generalized to other structured-light vision tasks (e.g., transparent object reconstruction, fluorescence imaging).