---

title: >- [Paper Note] Physically Grounded Monocular Depth via Nanophotonic Wavefront Encoding description: >- [ECCV 2026][3d_vision][depth estimation] Physically grounding depth foundation models (Depth Anything V2) with nanophotonic birefringent metalenses that passively encode metric depth into polarization-multiplexed rotating PSFs. tags: - ECCV 2026 - 3d_vision - depth estimation - metasurface - computational photography date: 2026-09-19 content_hash: da1dc8b3107983fe

Physically Grounded Monocular Depth via Nanophotonic Wavefront Encoding

Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: monocular depth estimation, metalens, depth foundation models, computational photography, polarization multiplexing

TL;DR

By engineering a birefringent metalens to passively encode metric depth into depth-dependent rotational shifts across orthogonal polarization channels in a single monocular capture, combined with a zero-overhead pseudo-RGB input adaptation strategy, this paper directly fine-tunes depth foundation models (Depth Anything V2) to deliver highly accurate metric monocular depth without active LiDAR sensors.

Background & Motivation

Depth foundation models (DFMs) have substantially advanced 3D visual perception by learning extensive monocular structural priors and robust semantic features from web-scale RGB imagery. However, single-view intensity capture intrinsically lacks absolute physical scale references. This missing physical grounding leaves monocular metric depth estimation severely ill-posed, suffering from pervasive scale and shift ambiguities that hamper reliable deployment in robotics navigation, AR/VR tracking, and precision 3D reconstruction.

To break scale ambiguity, current mainstream approaches predominantly rely on fusing auxiliary active sensors such as LiDAR (e.g., PromptDA) or performing intensive test-time optimization over defocus blur cues. Yet, active LiDAR systems introduce substantial physical bulk, energy consumption, and hardware complexity, breaking the pure monocular form factor. Conversely, depth-from-defocus (DfD) test-time optimization is computationally prohibitive (often demanding several minutes per frame) while conventional symmetric defocus blur inherently degrades sharp high-frequency details, preventing effective synergy with the representation capacity of modern vision backbones.

This work addresses this fundamental tension through passive nanophotonic wavefront engineering using a planar metasurface. By patterning anisotropic subwavelength structures, incoming light is decomposed into orthogonal polarization channels that directly modulate depth into rotational wavefront displacements in a single exposure. Core idea: passively encode metric depth into opposite depth-dependent PSF rotations via a birefringent metalens, inject the resulting polarization pairs into a pretrained depth foundation model through an architecture-preserving input adaptation strategy, and bridge the sim-to-real gap using a disocclusion-aware optical forward simulator.

Method

Overall Architecture

The integrated hardware-software pipeline comprises three key pillars: an ultra-compact birefringent metalens that encodes scene depth into polarization-resolved point spread function (PSF) shifts, an input adaptation layer mapping the two polarization observations into a representation compatible with depth foundation models, and a physics-based, disocclusion-aware optical simulator coupled with polarization augmentations to close the domain gap.

At capture time, incoming optical wavefronts pass through a 3-mm-aperture birefringent metalens, which splits the beam along vertical directions into \(x\)- and \(y\)-polarized wavefronts focused onto distinct halves of a single monochrome CMOS sensor. The captured polarization image pair is adapted into a pseudo-RGB representation and forwarded into a fine-tuned Depth Anything V2 (DPT architecture) backbone, which outputs dense metric depth maps directly.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Monocular Incident Light Wave"] --> B["Birefringent Metalens Wavefront Encoding<br/>700nm TiO2 nanopillars impart polarization phases"]
    B --> C["Single-Sensor Snapshot Imaging<br/>Orthogonal polarization pair (Ix, Iy)"]
    C --> D["Input Adaptation & Prior Alignment<br/>Pseudo-RGB mapping: (Ix, Iy, (Ix+Iy)/2)"]
    D --> E["Depth Foundation Model Dense Prediction<br/>Depth Anything V2 (DPT) fine-tuning"]
    E --> F["Metrically Grounded Dense Depth Output"]

Key Designs

1. Birefringent Metalens & Polarization-Multiplexed Rotating PSFs: Encoding Metric Depth as Rotational Disparity in a Single Optical Path

To circumvent the loss of high frequencies caused by conventional symmetric defocus blur, the authors design and fabricate a 3-mm-diameter visible-light (\(\lambda = 590\text{ nm}\)) metalens composed of 700-nm-tall anisotropic \(\text{TiO}_2\) nanopillars on a 500-\(\mu\)m glass substrate. The phase profile for polarization \(k \in \{x, y\}\) decomposes into focusing power and depth-encoding rotation: $\(\psi_k = \psi_{f,k} + \psi_{r,k}\)$ The focusing component \(\psi_{f,k}\) imparts equal and opposite vertical deflection angles to the orthogonal states, separating the \(x\)- and \(y\)-polarized channels onto the upper and lower halves of a single sensor chip without viewpoint parallax. The rotation phase \(\psi_{r,k}\) partitions the pupil into \(N=8\) concentric annular zones with topological charges \(n=1,\dots,N\), causing the single-lobed PSF to rotate around the optical axis by an angle \(\Delta \phi_i(z)\) inversely proportional to object distance \(z\): $\(\Delta \phi_i(z) = \frac{\pi R^2}{N \lambda} \left(\frac{1}{z} - \frac{1}{z_f}\right)\)$ Because the \(x\)- and \(y\)-polarized phase profiles are spatially rotated by \(180^\circ\), the two resulting PSFs rotate in opposite orientations. This converts absolute metric depth into a clean, monotonic spatial disparity vector between the conjugate image channels along a single optical axis.

2. Architecture-Preserving Input Adaptation: Seamless Alignment with Pretrained Natural Image Priors

Existing computational imaging frameworks frequently discard pretrained vision backbones, opting to train multi-stream CNNs or U-Nets from scratch. This forfeits the geometric and semantic priors acquired by foundation models through web-scale pretraining. To feed two polarization channels \((I_x, I_y)\) into a standard three-channel DPT backbone without altering its architecture or adding auxiliary fusion branches, the method maps the observation to a pseudo-RGB tuple: $\((I_x, I_y) \Rightarrow \left(I_x, \; I_y, \; \frac{I_x + I_y}{2}\right)\)$ The third channel provides an unpolarized irradiance average that maintains standard visual semantics, while the first two channels retain the precise differential PSF shifts. Centered Kernel Alignment (CKA) feature similarity analysis across ViT transformer blocks confirms that this remapping yields representations with CKA scores exceeding 0.95 in deep layers relative to standard RGB inputs. This permits direct parameter fine-tuning that retains rich pretraining priors while grounding metric scale.

3. Disocclusion-Aware Forward Simulator & Polarization Augmentations: Resolving Boundary Artifacts for Sim-to-Real Transfer

Collecting large-scale real scenes with dense, sub-centimeter metric depth ground truth is physically impractical. Consequently, training requires synthesizing polarization pairs from synthetic RGB-D datasets via layered PSF convolutions. However, standard linear convolution models break down around steep depth discontinuities when using asymmetric, rotating PSFs: overlapping PSF trajectories produce artificial bright halos at occluding edges, while diverging shifts leave dark disocclusion gaps. To address this, the simulator explicitly detects depth boundaries, extrapolates occluded background regions, and applies an alpha-correction normalization to eliminate undersampling fringes. Furthermore, a suite of polarization-aware data augmentations—simulating spatial lighting imbalances, sensor Poisson-Gaussian noise, and fabrication blur—prevents the model from overfitting to absolute illumination intensity, forcing it to decode depth strictly from positional wavefront shifts.

Loss & Training

The network is fine-tuned on the Hypersim dataset with synthetic polarization pairs generated by the proposed forward simulator, with ground truth mapped to the 0.2–1.2 m range. The loss function combines \(L_1\) depth error and multi-scale depth gradient loss: $\(\mathcal{L} = \mathcal{L}_1 + 0.5 \mathcal{L}_{\text{grad}}\)$ To bridge remaining photometric domain shifts, 5 real captured frames with approximate planar object annotations are mixed into the training batch with a probability of 0.05 (Few-Shot Real Adaptation). The model is optimized for 80k steps using AdamW with a base learning rate of \(4 \times 10^{-6}\).

Key Experimental Results

Main Results

The proposed approach was evaluated on the standard indoor NYU Depth V2 benchmark and the zero-shot geometry-focused MIT-CGH-4k synthetic dataset against leading monocular metric depth estimators (using optimal per-image scale-shift post-alignment) and dual-sensor LiDAR-guided PromptDA (see Table 1 of the original paper):

Method	Extra Inputs / Fine-tuning	NYU MAE↓	NYU RMSE↓	NYU AbsRel↓	NYU \(\delta_{0.5}\)↑	MIT-CGH MAE↓	MIT-CGH RMSE↓	MIT-CGH AbsRel↓	MIT-CGH \(\delta_{0.5}\)↑
Ours-Large	Passive Monocular	0.023	0.040	0.039	0.951	0.067	0.126	0.105	0.764
Ours-Base	Passive Monocular	0.022	0.039	0.036	0.957	0.068	0.129	0.102	0.772
Ours-Small	Passive Monocular	0.025	0.043	0.043	0.936	0.076	0.137	0.125	0.724
PromptDA	w/ Simulated LiDAR (Dual Sensor)	0.021	0.042	0.036	0.955	0.058	0.113	0.099	0.802
MoGe v2	Monocular (Post-Aligned)	0.034	0.058	0.063	0.867	0.133	0.169	0.272	0.346
UniDepth v2	Monocular (Post-Aligned)	0.034	0.059	0.063	0.865	0.127	0.164	0.259	0.360
DepthAny. v3	Monocular (Post-Aligned)	0.037	0.062	0.069	0.846	0.134	0.172	0.276	0.349
DepthAny. v2	Monocular (Post-Aligned)	0.043	0.067	0.079	0.805	0.151	0.190	0.308	0.300
DepthAny. v2*	Fine-tuned w/o Physical Encoding	0.128	0.148	0.267	0.341	0.301	0.371	0.410	0.100

On 42 physical test captures featuring 25 diverse objects (Table 2 of the original paper), Ours-Base achieves an MAE of 0.032 m, RMSE of 0.089 m, and \(\delta_{0.5}\) of 0.895, substantially outperforming all unassisted monocular models (e.g., DepthAny. v2 MAE 0.135 m) and matching the LiDAR-aided PromptDA (MAE 0.030 m).

Ablation Study

Ablations on FlyingThings3D isolate the relative performance gains across optical design, network architecture, and pretraining priors (Table 4 of the original paper):

Configuration	Metalens (Meta)	ViT Backbone	Pretrained Prior	MAE↓	RMSE↓	\(\log_{10}\)↓	\(\delta_1\)↑
1 (Ours-Small)	✓	✓	✓	0.022	0.132	0.005	0.995
2 (DeepDfD Optics + Prior)	✗	✓	✓	0.035	0.133	0.009	0.994
3 (Meta + Scratch ViT)	✓	✓	✗	0.061	0.268	0.014	0.984
4 (Meta + Scratch U-Net)	✓	✗	✗	0.063	0.254	0.014	0.983
5 (DeepDfD End-to-End Baseline)	✗	✗	✗	0.089	0.191	0.034	0.941

Additionally, input adaptation ablation (Table 5) confirms that direct pseudo-RGB conversion achieves 0.032 m MAE versus 0.063 m MAE when using decoder-side cross-attention fusion branches. Disocclusion handling ablations (Table 6) show that omitting disocclusion handling increases MAE from 0.032 m to 0.046 m under real fine-tuning, and causes severe degradation from 0.107 m to 0.121 m in zero-shot synthetic evaluation.

Key Findings

• Pretrained Priors Drive Majority of Accuracy: Removing the pretrained prior triples the metric error (MAE jumps from 0.022 m to 0.061 m). Meanwhile, replacing conventional defocus optics with rotating PSF metalens optics yields a further 37% error reduction (0.035 m to 0.022 m) on the same pretrained model.
• Pseudo-RGB Adaptation Outperforms Custom Fusion: Mapping \((I_x, I_y)\) to \((I_x, I_y, (I_x+I_y)/2)\) leverages the full capacity of pretrained first-layer patch projection filters without introducing optimization instability or extra latency.
• Disocclusion Modeling is Indispensable for Sim-to-Real Transfer: Asymmetric PSFs create non-reciprocal occlusion and disocclusion artifacts at object boundaries that naive linear models cannot reproduce, making accurate geometric boundary modeling critical for zero-shot physical transfer.

Highlights & Insights

• Nanophotonic-Foundation Model Synergy: Harnesses subwavelength birefringent metasurfaces to encode physical metric depth into orthogonal polarization wavefronts, providing passive optical grounding that resolves fundamental scale ambiguity in single-view depth estimation.
• Zero-Overhead Input Repurposing: Injects dual polarization views directly into the default three-channel RGB interface of Depth Anything V2, preserving pretrained feature statistics without architectural surgery.
• Accurate Forward Modeling for Asymmetric Optics: Formalizes the optical formation of non-symmetric rotating PSFs at depth discontinuities, establishing edge extrapolation and alpha correction techniques that bridge the simulation-to-real gap.

Limitations & Future Work

• Optical Throughput & Narrow Bandwidth: The fabricated prototype features a 3-mm aperture (\(f/11.3\)) and relies on a 10-nm bandpass filter centered at 590 nm, resulting in low photon efficiency that limits operation in low-light environments.
• Effective Resolution & Field-of-View Trade-off: Splitting the orthogonal polarization channels across the top and bottom halves of a single sensor halves the effective vertical sensor area and restricts the operational field of view.
• Sensitivity to High-Frequency Polarized Reflections: While augmentations handle diffuse lighting discrepancies, strong specular glints and highly polarized reflective materials can disrupt the phase-encoded PSF shifts.

Related Work & Insights

• vs PromptDA (CVPR 2024): PromptDA relies on an external active LiDAR module to inject sparse metric depth hints; this paper operates in a purely passive optical regime using a single metalens, matching LiDAR-assisted depth accuracy while eliminating active power and bulky sensor packages.
• vs DeepDfD (ICCP 2021) / Split-Aperture (CVPR 2024): Traditional depth-from-defocus techniques rely on phase masks and shallow U-Nets trained from scratch, suffering from severe defocus blur and weak generalization; this approach combines high-sharpness rotating PSFs with a vision foundation model, cutting MAE by over 75% on FlyingThings3D.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Pioneering integration of birefringent nanophotonic wavefront encoding with pretrained depth foundation models.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluations across NYU Depth V2, MIT-CGH-4k, FlyingThings3D, and an optical-bench physical prototype with 42 real-world captures.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulations, clean narrative flow, and well-structured ablation insights.
• Value: ⭐⭐⭐⭐⭐ Offers an inspiring, ultra-compact hardware-software paradigm for sub-centimeter passive 3D sensing in miniature robotics and biomedical endoscopy.

Related Papers

• [CVPR 2026] PhyGaP: Physically-Grounded Gaussians with Polarization Cues
• [CVPR 2026] Artiverse: A Diverse and Physically Grounded Dataset for Articulated Objects
• [CVPR 2026] SPE-MVS: Spatial Position Encoding Enhanced Multi-View Stereo with Monocular Depth Priors
• [ECCV 2026] Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch
• [CVPR 2026] Recovering Physically Plausible Human-Object Interactions from Monocular Videos