Bridge-UniPS: Bridging Calibrated Photometric Stereo toward Universal Photometric Stereo¶
Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: Universal Photometric Stereo / Calibrated Photometric Stereo / Surface Normal Estimation / Dynamic Neural Query / Physics-aligned Intermediate Representation
TL;DR¶
Bridge-UniPS introduces a bridging paradigm that connects universal photometric stereo (UniPS) to calibrated photometric stereo (CPS) via a task-driven lighting adapter that translates unconstrained multi-illumination observations into CPS-compatible bridge image–illumination pairs without intermediate supervision, achieving state-of-the-art normal estimation accuracy by leveraging the physical priors frozen in pretrained CPS networks.
Background & Motivation¶
Photometric stereo (PS) aims to recover fine-grained surface normal maps from multi-illumination images captured from a fixed viewpoint. Classical calibrated photometric stereo (CPS) and uncalibrated photometric stereo (UPS) have achieved remarkable precision under idealized distant point-light or directional lighting assumptions. However, their strict reliance on calibrated illumination sources or specific parametric lighting models confines their deployment to controlled darkroom laboratory setups, hindering practical 3D surface reconstruction in uncontrolled natural environments. Universal photometric stereo (UniPS) emerged to tackle this challenge by directly reconstructing surface normals from images captured under arbitrary, unknown, and unconstrained illumination without predefined lighting configurations.
Despite promising advances, existing UniPS methods predominantly rely on fully end-to-end learning paradigms based on global pooling encoders or latent token transformers. These data-driven black-box formulations present two fundamental bottlenecks. First, the lack of an explicit, physically grounded intermediate representation forces illumination characteristics and surface geometry to be learned in an intricately coupled manner, making models susceptible to out-of-distribution shifts and prone to geometric ambiguities when handling complex real-world light interactions. Second, summarizing illumination via global statistical features or abstract tokens suppresses local photometric nuances—such as fine cast shadows and specular gradients—that are indispensable for resolving high-frequency geometry.
The core insight of this work is that modern pretrained CPS networks inherently encode rich, physically grounded light–reflection–geometry priors within their parameters. Rather than learning unconstrained photometric stereo from scratch, one can transform unconstrained observations into a standardized set of virtual "bridge image–illumination pairs" that conform to the standard inputs expected by calibrated models. Core idea: train a task-driven lighting adapter without direct intermediate supervision to dynamically query and synthesize CPS-compatible bridge image–illumination pairs from unconstrained observations, guided solely by task-level normal estimation gradients back-propagated through a frozen pretrained CPS network.
Method¶
Overall Architecture¶
The Bridge-UniPS framework operates across two distinct stages: a trainable task-driven lighting transformation stage and a frozen calibrated photometric stereo (CPS) stage.
In the first stage, the input consists of an unconstrained image sequence \(\mathbf{I} \in \mathbb{R}^{N_{in} \times 3 \times H \times W}\) acquired under unknown illumination. A ConvNeXt backbone extracts multi-scale visual features. Simultaneously, a predefined set of target light directions is sampled equiareally over the upper hemisphere to yield light embedding features. The Dynamic Neural Query (DNQ) module takes these target light embeddings as queries, assesses the physical compatibility between each input image and target direction, retrieves informative pixel-level features, and filters out noisy observations via Top-K confidence scoring. An attention-guided decoder equipped with multi-scale skip connections then reconstructs \(N_{out}\) light-conditioned bridge images paired with their associated unit light vectors \((\mathbf{B}, \mathbf{L})\). In the second stage, these bridge image–illumination pairs are directly forwarded to a completely frozen pretrained CPS network (Norm-PSN) to predict the final surface normal map \(\mathbf{N} \in \mathbb{R}^{3 \times H \times W}\). Gradients originating from the surface normal objective are propagated backward through the frozen CPS network to optimize the lighting adapter.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Unconstrained Image Sequence<br/>N_in × 3 × H × W"] --> B["ConvNeXt Multi-Scale Feature Extraction<br/>Feature F & Global Confidence G"]
B --> C["Target Light Prior & Light-Image Compatibility<br/>Upper-hemisphere Equiareal Sampling & Similarity S"]
C --> D["Dynamic Neural Query & Attention-Guided Skip Connections<br/>Cross-Attention Retrieval & Top-K Filtering"]
D --> E["Bridge Image-Illumination Pairs<br/>N_out Bridge Images B & Directions L"]
E --> F["Frozen Calibrated Photometric Stereo Stage<br/>Pretrained Norm-PSN Network"]
F --> G["Predicted Surface Normals N<br/>Cosine & L1 Loss to Optimize Adapter"]
Key Designs¶
1. Target Light Prior and Light–Image Compatibility: establishing directional anchors and filtering unreliable observations
To bridge unconstrained capture with calibrated photometric stereo, the adapter must establish a canonical set of target illumination directions. Leveraging the physical observation that over 95% of active light sources in classical photometric stereo datasets reside in the forward hemisphere (\(z > 0.6\)), the authors perform equiareal uniform sampling to generate \(N_{light} = 64\) target light directions, which are mapped through an MLP to construct the light feature tensor \(\mathbf{F}_l \in \mathbb{R}^{N_{light} \times C \times L}\). In parallel, to quantify the global quality and reliability of each input image under varying illuminations, the feature representations of each input image are mapped via MLPs along both channel and spatial dimensions, yielding a global confidence matrix \(\mathbf{G} \in \mathbb{R}^{N_{light} \times N_{in}}\).
To evaluate how suitable each raw input image is for explaining the photometric behavior under a specified target light direction, a refined key representation \(\mathbf{K}\) is derived from the feature maps via channel-wise MLPs and spatial convolutions. Computing its spatial average produces a global key \(\bar{\mathbf{K}} \in \mathbb{R}^{N_{in} \times C}\). The light–image compatibility matrix \(\mathbf{S}\) is then calculated by combining feature correlation with image reliability: $\(\mathbf{S}_{j,i} = \text{Einsum}(\mathbf{F}_{l,j}, \bar{\mathbf{K}}_i) + \mathbf{G}_{j,i}\)$ The score \(\mathbf{S}_{j,i}\) indicates the degree of physical consistency between input observation \(i\) and target illumination \(j\), providing an explicit foundation for discarding irrelevant shadows or specular outliers.
2. Dynamic Neural Query and Attention-Guided Skip Connections: target-light-conditioned retrieval and micro-geometry preservation
When generating bridge images under a specific illumination vector, simple global feature averaging flattens fine structural variations, while naively concatenating all observations introduces conflicting shading patterns. The Dynamic Neural Query module treats the spatial expansion of target light feature \(\mathbf{F}_l\) as query tensor \(\mathbf{Q} \in \mathbb{R}^{L \times N_{light} \times C}\), while the refined key \(\mathbf{K}\) and the original feature tensor \(\mathbf{F}\) serve as key and value respectively in a multi-head cross-attention mechanism \(\mathbf{T} = \text{Attn}(\mathbf{Q}, \mathbf{K}, \mathbf{V})\). To prevent degraded observations from corrupting the synthesized representations, the compatibility scores \(\mathbf{S}\) are thresholded (\(\mathbf{S} > 0.8\)) and the top-ranked candidates are retained subject to minimum and maximum count bounds, creating a binary selection mask that isolates the most informative observations.
To prevent the decoder from losing high-frequency spatial boundaries and subtle surface relief, the decoder symmetrically mirrors the ConvNeXt encoder with transposed upsampling stages and employs illumination-aware skip connections. At each decoding stage \(s\), the cross-attention weights \(\mathbf{A} \in \mathbb{R}^{L \times N_{out} \times N_{in}}\) are resized to match the spatial resolution \(L_s = H_s W_s\), forming \(\mathbf{A}^{(s)}\). The multi-scale skip features \(\mathbf{F}^{(s)}\) from the encoder are aggregated via spatial element-wise weighting: $\(\mathbf{F}_{\text{skip}}^{(s)} = \sum_{i=1}^{N_{in}} \mathbf{A}^{(s)}_{:,:,i,:} \odot \mathbf{F}^{(s)}_{:,i,:,:}\)$ This mechanism guarantees that each synthesized bridge image preserves fine-grained local textures and edge contours that correspond strictly to the queried illumination angle.
3. Task-Driven Bridging without Intermediate Supervision: physics-aligned guidance from frozen CPS networks
A conventional approach to relighting might involve synthetic supervised training using ground-truth directional-light renderings. However, synthetic renderings suffer from domain gaps in BRDF realism and ambient light complexity. Bridge-UniPS circumvents this need by training the lighting adapter completely without intermediate supervision. No ground-truth bridge images or intermediate illumination labels are provided. Instead, the synthesized bridge image–illumination pairs \((\mathbf{B}, \mathbf{L})\) are directly passed into a completely frozen pretrained CPS network \(\mathcal{P}\) (Norm-PSN) to estimate surface normals: $\(\mathbf{N} = \mathcal{P}(\mathbf{B}, \mathbf{L})\)$
The entire optimization objective is defined strictly on the final predicted surface normals against the ground-truth normals \(\mathbf{N}_{gt}\): $\(\mathcal{L} = \gamma \mathcal{L}_{\cos}(\mathbf{N}, \mathbf{N}_{gt}) + \|\mathbf{N} - \mathbf{N}_{gt}\|_1\)$ Because the parameters of \(\mathcal{P}\) are completely frozen, the network acts as a rigorous, physics-aligned differential operator. Gradients can only back-propagate through the physical mapping pathways encoded inside \(\mathcal{P}\), forcing the lighting adapter to produce bridge images that strictly respect directional lighting physics, exhibit normalized contrast, and minimize non-Lambertian disturbances.
Loss & Training¶
The network is trained using a composite loss function penalizing both angular error and absolute coordinate deviation between the predicted normal map \(\mathbf{N}\) and ground truth \(\mathbf{N}_{gt}\): $\(\mathcal{L} = \gamma (1 - \cos(\mathbf{N}, \mathbf{N}_{gt})) + \|\mathbf{N} - \mathbf{N}_{gt}\|_1\)$ where the first term enforces angular alignment and the second term regularizes coordinate-wise \(L_1\) distances.
The model is primarily trained on the Uni-MS-PS training split, augmented with roughly 10,000 samples from the synthetic point-light datasets PS-Sculpture and PS-Blobby (constituting ~10% of the training pool) to preserve foundational point-light fidelity. The network is optimized using AdamW with an initial learning rate of \(1 \times 10^{-5}\), scaled by a factor of 0.8 every 20 epochs, and a batch size of 12. During training, the number of input images \(N_{in}\) per batch is randomly sampled between 6 and 24 to promote robustness to variable input cardinality. Training is carried out over 140 epochs across four NVIDIA A100 GPUs, taking approximately 9 days.
Key Experimental Results¶
Main Results¶
The method was evaluated on both the controlled point-light DiLiGenT benchmark and the unconstrained near-field LUCES benchmark using the Mean Angular Error (MAE, in degrees \(^\circ\), lower is better).
On the classical DiLiGenT benchmark, Bridge-UniPS achieves state-of-the-art accuracy across all categories even when restricted to only 16 input images (\(4.5^\circ\)), outperforming calibrated and uncalibrated baselines that use the full set of 96 images. With all 96 images, the average MAE drops to \(4.3^\circ\).
| Method | Task | Input Count | Buddha | Goblet | Harvest | Reading | Average MAE (↓) |
|---|---|---|---|---|---|---|---|
| JS22 (Norm-PSN) | Calibrated | 96 | 7.1 | 7.5 | 12.3 | 9.9 | 6.8 |
| LL22a | Calibrated | 96 | 8.0 | 6.7 | 14.9 | 8.8 | 6.5 |
| JS23 (GR-PSN) | Uncalibrated | 96 | 6.7 | 6.8 | 12.0 | 9.7 | 6.6 |
| LL22b | Uncalibrated | 96 | 9.3 | 7.1 | 14.6 | 10.5 | 7.1 |
| IK22 (SDM-UniPS) | Universal | 96 | 19.4 | 24.2 | 25.2 | 18.8 | 14.7 |
| IK23 | Universal | 96 | 7.5 | 8.5 | 10.2 | 8.2 | 5.8 |
| LC25 (LINO-UniPS) | Universal | 96 | 6.2 | 5.2 | 8.6 | 6.7 | 4.7 |
| Ours (Bridge-UniPS) | Universal | 96 | 5.2 | 6.2 | 6.5 | 5.2 | 4.3 |
| Ours (K=16) | Universal | 16 | 5.3 | 6.5 | 6.7 | 5.5 | 4.5 |
| Ours (K=4) | Universal | 4 | 7.2 | 8.8 | 9.1 | 7.6 | 6.0 |
On the LUCES benchmark, which features unconstrained near-field lighting and intricate non-Lambertian surfaces across 14 diverse objects, Bridge-UniPS consistently outperforms all competing universal methods:
| Method | Task | Bell (Metallic) | Bowl | Cup (Specular) | House (Complex) | Squirrel | Average MAE (↓) |
|---|---|---|---|---|---|---|---|
| IK22 | Universal | 24.12 | 23.84 | 28.64 | 35.93 | 25.36 | 23.77 |
| IK23 | Universal | 12.76 | 8.44 | 19.67 | 26.07 | 16.01 | 13.50 |
| HQ24 (Uni-MS-PS) | Universal | 10.52 | 6.98 | 13.68 | 25.29 | 11.36 | 11.21 |
| LC25 (LINO-UniPS) | Universal | 8.78 | 6.96 | 8.15 | 22.91 | 10.25 | 9.43 |
| Ours (Bridge-UniPS) | Universal | 4.78 | 3.97 | 4.92 | 16.08 | 7.43 | 7.11 |
Ablation Study¶
The paper systematically analyzes downstream photometric stereo paradigms, lighting adapter components, and essential control configurations (Table 4):
| Config / Variant | Investigated Dimension | DiLiGenT MAE (↓) | LUCES MAE (↓) | Note |
|---|---|---|---|---|
| Norm-PSN (Full model) | Full Model | 4.3 | 7.1 | Learning-based CPS with complete adapter |
| Woodham (1980) | Downstream PS Paradigm | 9.2 | 14.1 | Traditional CPS; strict Lambertian model causes artifacts |
| SCPS-NIR | Downstream PS Paradigm | 6.5 | 12.3 | Learning-based UPS baseline |
| GR-PSN | Downstream PS Paradigm | 5.8 | 10.8 | Advanced learning-based UPS baseline |
| CNN-PS | Downstream PS Paradigm | 5.4 | 9.7 | Early convolutional learning-based CPS |
| w/o dynamic query | Module Design | 6.3 | 12.5 | Replaces DNQ with standard non-adaptive attention |
| w/o top-K selection | Module Design | 5.6 | 9.4 | Disables confidence/similarity candidate filtering |
| w/o pixel feat | Module Design | 4.8 | 7.9 | Removes sequence-level pixel feature branch |
| w/o global conf | Module Design | 4.7 | 8.1 | Removes global image reliability assessment |
| w/o point-light data | Control Ablation | 4.4 | 7.1 | Removes synthetic sculpture/blobby data from training |
| learnable latent tokens | Control Ablation | 4.7 | 8.4 | Replaces explicit light directions with learned latent tokens |
| unfrozen CPS | Control Ablation | 5.4 | 7.9 | End-to-end CPS fine-tuning triggers co-adaptation |
| shuffled bridge pairing | Control Ablation | 44.5 | 43.8 | Randomly shuffles bridge images against lighting directions |
Key Findings¶
- Dynamic Neural Query (DNQ) is the structural cornerstone: Disabling DNQ causes error spikes from 4.3 to 6.3 on DiLiGenT and 7.1 to 12.5 on LUCES. Qualitative results show that without DNQ, synthesized bridge images lose their directional illumination properties, yielding noisy, unphysical normal predictions.
- Synthesized bridge pairs reflect genuine physical geometry rather than generic image enhancements: Randomly shuffling the pairings between bridge images and their target light vectors results in catastrophic failure (\(44.5^\circ\) and \(43.8^\circ\) MAE). This confirms that bridge images maintain strict physical correspondence with their associated lighting directions.
- Freezing the CPS network outperforms joint fine-tuning: Unfreezing the downstream CPS model degrades MAE to 5.4 on DiLiGenT and 7.9 on LUCES due to adapter–CPS co-adaptation, which erodes the rigid physical prior encoded in the original weights.
- Illumination contrast stabilization mechanism: An analysis of cross-object contrast statistics on DiLiGenT revealed that raw inputs exhibit massive intensity variance (e.g., highly specular Goblet vs. diffuse Pot2), whereas bridge images exhibit tightly grouped mean and median contrast values between 0.31 and 0.37 across all objects. The adapter effectively normalizes reflectance variations into canonical point-light shading.
Highlights & Insights¶
- Task-driven unsupervised physical bridging: The paper demonstrates that a complex domain adaptation problem (mapping unconstrained ambient illumination into directional point lighting) can be solved entirely without direct intermediate supervision by allowing gradients to back-propagate through a frozen physical model.
- Attention-guided skip connections for geometric preservation: Recycling cross-attention maps as spatial gates for multi-scale skip connections elegantly couples directional lighting queries with the high-resolution geometric detail of raw inputs.
- High data efficiency: Achieving SOTA performance with as few as 16 raw inputs dramatically reduces the capture requirements for high-precision photometric reconstruction.
Limitations & Future Work¶
- Performance in weak photometric variation scenarios: When objects are illuminated under nearly uniform diffuse ambient lighting with negligible shadow/shading variation, the adapter struggles to extract sufficient photometric cues, leading to over-smoothed normal predictions.
- Computational overhead during training: Generating, caching, and propagating multiple multi-scale bridge images through the frozen CPS network requires considerable GPU memory and a 9-day training duration on four A100 GPUs.
- Future directions: Developing adaptive sensitivity mechanisms for subtle shading gradients and extending the physics-bridged paradigm to multi-view photometric stereo in wild scenes.
Related Work & Insights¶
- vs. LINO-UniPS (LC25): LINO-UniPS uses light register tokens and interleaved attention inside a pure end-to-end transformer; Bridge-UniPS decouples lighting and geometry by generating explicit bridge pairs, reducing LUCES MAE from 9.43 to 7.11.
- vs. Uni-MS-PS (HQ24): Uni-MS-PS aggregates global context across scales; Bridge-UniPS adds directional neural querying and Top-K filtering, eliminating inconsistent shadow noise and preserving fine surface relief.
- vs. Norm-PSN (JS22): Norm-PSN cannot directly handle unconstrained illumination. Bridge-UniPS serves as a front-end adapter that lowers Norm-PSN's error on raw DiLiGenT from 6.8 to 4.3 while expanding its utility to unconstrained environments.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering unsupervised physics-bridging paradigm that connects unconstrained inputs to frozen calibrated solvers]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive evaluations across controlled, near-field, and real-world scenes with insightful contrast stabilization and pairing shuffle ablations]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural formulation, rigorous motivation, and strong empirical validation]
- Value: ⭐⭐⭐⭐⭐ [High practical value for deploying laboratory-grade photometric stereo models into unconstrained real-world capture pipelines]