SynLF: Zero-Shot Metric Depth from Light Field Cameras via Physics-Grounded Synthesis

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Area: 3D Vision
Keywords: Light Field Depth Estimation, Single-lens Metric Depth, Physics-Grounded Synthesis, Zero-Shot Generalization, Occlusion Reasoning

TL;DR

To overcome data scarcity and poor real-world robustness in light field depth estimation, SynLF synthesizes supervisory light fields on-the-fly from large-scale RGB-D data using a physics-grounded pipeline (PG-LF) modeling defocus and non-Lambertian perturbations, coupled with VisDepth, an iterative estimator with explicit visibility reasoning that achieves zero-shot metric depth (MAE 30.05 mm, AbsRel 1.38%) on real captures without fine-tuning.

Background & Motivation

Accurate single-lens metric depth estimation is essential for embodied robotics, autonomous driving, and computational photography. However, conventional monocular cameras suffer from intrinsic scale ambiguity: infinitely many geometrically scaled 3D scenes can project into identical 2D images, causing even large-scale foundation models to experience severe scale drift under unseen environments. In contrast, light field (LF) cameras provide a principled single-lens optical configuration: by capturing spatial-angular rays simultaneously via micro-lens arrays within a single exposure, micro-baseline disparities directly encode absolute metric geometry without active range sensors.

Despite this fundamental advantage, learning-based light field depth estimation has been critically bottlenecked by training data scarcity. Rendering multi-view 4D light fields requires substantial graphics computation, leaving current supervised benchmarks (e.g., HCI and UrbanLF) orders of magnitude smaller than modern monocular RGB-D corpora. This data shortage forces networks to overfit idealized, clean Lambertian appearances, resulting in catastrophic failure when deployed to real-world captures: real plenoptic imaging exhibits pronounced depth-dependent defocus blur, severe specular reflections, transparent surfaces, and extensive textureless regions. Furthermore, conventional architectures typically fuse multi-view information through implicit, learned attention weights, lacking explicit geometric visibility constraints to handle partial occlusions and optical mismatches.

To dismantle the data acquisition ceiling and eliminate domain degradation, this work introduces a paradigm shift: instead of physically capturing or exhaustively rendering massive 4D datasets, it exploits abundant commodity single-view RGB-D corpora to synthesize physically faithful light fields dynamically. Core idea: develop a physics-grounded light field synthesis pipeline (PG-LF) that synthesizes multi-view sub-aperture views from RGB-D on-the-fly via occlusion-aware forward splatting, depth-coupled defocus blur, and non-Lambertian disparity perturbations; and couple it with VisDepth, a prior-initialized iterative network that shares the forward projection operator to perform explicit multi-view visibility reasoning, achieving robust zero-shot metric depth estimation on real captures.

Method

Overall Architecture

SynLF consists of two closely coupled modules: the physics-grounded light field synthesis pipeline (PG-LF Synthesis) used during training, and the visibility-aware depth estimation network (VisDepth). During training, PG-LF consumes indoor RGB-D frames from Hypersim, maps metric depth to signed light field disparities, and generates physically grounded sub-aperture views by modeling forward splatting occlusions, disparity-dependent optical defocus blur, and selective non-Lambertian perturbations. During inference, PG-LF is completely bypassed: real multi-view light field captures from a physical camera are directly fed into VisDepth, which outputs center-view disparity and converts it to metric depth via pre-calibrated camera mapping without any fine-tuning.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Data<br/>RGB-D depth map and center view"] --> B["Geometry-Consistent View Synthesis<br/>Depth-to-disparity + forward splatting z-buffer"]
    B --> C["Modeling Non-Ideal Imaging Effects<br/>Disparity-sliced defocus + non-Lambertian perturbation"]
    C --> D["Synthesized Light Field Views"]
    D --> E["Prior-Guided Disparity Initialization<br/>DAv2 monocular prior + 3D cost aggregation"]
    E --> F["Explicit Visibility-Guided Iterative Estimation<br/>Depth-aware projection + visibility-masked aggregation"]
    F --> G["Multi-Scale ConvGRU Residual Updates<br/>Local signed disparity sampling and refinement"]
    G --> H["Output Disparity and Metric Depth"]

Key Designs

1. Geometry-Consistent View Synthesis: Depth-Aware Forward Splatting and Stochastic Hole Filling Existing single-image light field hallucination methods rely on generative networks that often distort strict epipolar geometry and multi-view disparity correspondence. In contrast, PG-LF enforces rigorous micro-baseline ray geometry. Given the central view depth \(Z_c\), it maps depth to light field disparity via: $\(d = \frac{\alpha}{Z_c} + \beta\)$ where \(\alpha\) scales the disparity baseline and \(\beta\) fixes the zero-disparity focal plane. For an angular sub-aperture coordinate \((u_i, v_i)\), central-view pixel \((x_c, y_c)\) projects to target coordinate \((x_i, y_i) = (x_c + u_i d, y_c + v_i d)\). To eliminate ordering errors in backward warping around foreground-background boundaries, PG-LF employs depth-aware forward splatting with a z-buffer mechanism, resolving projection collisions by retaining the closest depth sample: $\(\mathbf{q}^* = \arg\min_{\mathbf{q} \in \mathcal{S}_i(\mathbf{p})} Z_c(\mathbf{q}), \quad I_i(\mathbf{p}) = I_c(\mathbf{q}^*)\)$ Retained samples are bilinearly splatted across grid coordinates to suppress discrete sampling artifacts, and newly exposed disocclusion holes are filled with randomized strategies to prevent the network from learning visual shortcuts.

2. Modeling Non-Ideal Imaging Effects: Disparity-Conditioned Defocus and Non-Lambertian Perturbations Synthetic RGB-D benchmarks are universally rendered under pinhole, all-in-focus assumptions, whereas real light field cameras feature finite apertures with strong depth-dependent defocus blur. Under thin-lens optics, circle of confusion (CoC) diameter obeys \(c(z) \propto |\frac{1}{z} - \frac{1}{z_f}| \propto |d|\), proving blur size is directly proportional to absolute disparity. PG-LF implements defocus via disparity slicing: the disparity range is discretized into bins, assigned corresponding Gaussian blur kernels, and composited in back-to-front order with alpha blending to prevent foreground textures from bleeding into background regions. Furthermore, real specular highlights and glass transmissions violate Lambertian constancy, making apparent optical depth diverge from geometric surface depth. PG-LF identifies locally planar, textured regions and applies targeted spatial disparity perturbations exclusively to the synthesized sub-aperture views while keeping the central view untouched. This intentionally breaks trivial photometric matching, forcing the model to rely on robust global geometric structure.

3. Prior-Guided Disparity Initialization: Structural Prior Injection and 3D Cost Regularization Initializing iterative optimization directly from raw multi-view correlation is notoriously unstable in untextured walls and reflective regions. VisDepth introduces a prior-guided initialization module: a shared-weight full-resolution CNN extracts multi-view feature maps to construct the raw correlation volume \(C_{\text{raw}} = \text{Corr}(f_c, \{f_v\})\). Simultaneously, a frozen monocular foundation model (Depth Anything V2-Small) extracts dense semantic geometric features \(F_{\text{mono}}\) from the central view. A lightweight 3D CNN regularization operator \(\Phi\) fuses the scale-free monocular prior with multi-view matching costs: $\(C_{\text{reg}} = \Phi(C_{\text{raw}}, F_{\text{mono}})\)$ The regularized volume is normalized via softmax along the disparity dimension, and an initial global disparity map \(d_0\) is regressed through soft-argmin, establishing an accurate and scale-consistent baseline.

4. Explicit Visibility-Guided Iterative Estimation: Reusable Geometric Projection and Signed Disparity Updates Starting from \(d_0\), VisDepth dynamically computes per-view physical visibility at each iteration \(k\), abandoning black-box attention mechanisms. Crucially, the network reuses the exact depth-aware forward projection operator from PG-LF: given current disparity estimate \(d_k\), it calculates binary/continuous visibility masks \(V_k^{(v)} \in [0, 1]\) across all source views. The raw correlation volume is then aggregated via masked normalization: $\(C_{\text{vis}}^k = \frac{\sum_v C_{\text{raw}}^{(v)} \odot V_k^{(v)}}{\sum_v V_k^{(v)}}\)$ Occluded sightlines are explicitly suppressed, preventing contradictory photometric costs from polluting geometric optimization. The dynamically filtered cue \(C_{\text{vis}}^k\) is concatenated with the foundational volume \(C_{\text{reg}}\) and processed by 3D convolutional block \(\mathcal{F}_{\text{fuse}}\) into volume \(G_k\). Taking advantage of the compact micro-baseline disparity range (tens of levels), the model indexes a localized 1D neighborhood along the signed disparity dimension around \(d_k\). These local matching features, along with central contextual embeddings, feed into a multi-scale ConvGRU to iteratively update hidden states and predict residual \(\Delta d_k\), progressively refining depth boundaries as visibility masks converge.

Loss & Training

The network is supervised end-to-end on both initial disparity \(d_0\) and the iterative sequence \(\{d_k\}_{k=1}^K\) over \(K=24\) ConvGRU steps, combining L1 disparity loss and multi-scale scale-and-shift invariant gradient regularization: $\(\mathcal{L} = \mathcal{L}_{\text{init}} + \mathcal{L}_{\text{gru}} + \lambda \mathcal{L}_{\text{grad}}\)$ $\(\mathcal{L}_{\text{init}} = \|d_0 - d\|_1, \quad \mathcal{L}_{\text{gru}} = \sum_{k=1}^K \gamma^{K-k} \|d_k - d\|_1\)$ $\(\mathcal{L}_{\text{grad}} = \sum_{l=1}^L \left( \|\nabla_x^l (\hat{d}_K) - \nabla_x^l (\hat{d})\|_1 + \|\nabla_y^l (\hat{d}_K) - \nabla_y^l (\hat{d})\|_1 \right)\)$ where \(d\) is ground-truth disparity, \(\hat{d}_K\) and \(\hat{d}\) denote aligned disparity maps under scale-and-shift normalization, and \(\gamma\) is a decaying factor weighting later iterations. Training is conducted on 74,013 Hypersim indoor RGB-D scenes with AdamW optimizer, warming up linearly to \(5 \times 10^{-4}\) and converging within only 8K steps (batch size 16, crop size \(768 \times 768\)).

Key Experimental Results

Main Results

Quantitative zero-shot evaluation is performed on the custom real-world light field dataset captured by an industrial camera with a \(9 \times 9\) micro-lens array and synchronized structured-light ground truth across 173 indoor scenes (operating range 0.5–2.5 m):

Dataset / Setting	Method	Training Data	MAE (mm) ↓	RMSE (mm) ↓	AbsRel (%) ↓	\(\delta_1\) (%) ↑
Real-World LF Dataset	OAVC [14]	None (Optimization)	146.99	291.90	7.02	88.75
Real-World LF Dataset	OACC-Net [30]	HCI (Synthetic LF)	113.66	200.27	5.99	93.33
Real-World LF Dataset	OPAL [17]	Unsupervised LF	141.77	202.35	7.39	93.47
Real-World LF Dataset	PlaneNet [7]	HCI (Synthetic LF)	91.31	164.91	4.48	96.03
Real-World LF Dataset	ESMNet [6]	HCI (Synthetic LF)	110.86	208.99	5.36	92.20
Real-World LF Dataset	Ours (HCI-trained)	HCI (Synthetic LF)	52.95	148.40	3.01	95.01
Real-World LF Dataset	Ours w/o Prior	PG-LF (Ours Synth)	35.75	158.50	1.63	98.50
Real-World LF Dataset	Ours w/o Vis	PG-LF (Ours Synth)	39.03	160.34	1.90	98.04
Real-World LF Dataset	SynLF (Full Model)	PG-LF (Ours Synth)	30.05	144.34	1.38	98.77

Cross-dataset evaluations on public synthetic benchmarks (HCI 4D and Inria Dense) highlight that models trained exclusively on HCI degrade substantially on Inria (PlaneNet MSE: \(0.69 \rightarrow 1.97\), OACC-Net: \(0.92 \rightarrow 2.74\)), whereas SynLF maintains consistent cross-domain accuracy (HCI MSE×100: 2.08, BP0.07: 9.36; Inria MSE×100: 2.35, BP0.07: 6.45).

Ablation Study

Ablations on real-world captures isolate the impacts of synthetic optical degradation and architectural choices:

Configuration	Defocus Blur	Non-Lambertian Pert.	MAE (mm) ↓	AbsRel (%) ↓	Note
Plain Splatting	\(\times\)	\(\times\)	50.01	3.41	Pure geometric splatting without optical degradation
+ Non-Lambertian	\(\times\)	\(\checkmark\)	43.25	2.40	Discourages shortcut photometric matching
+ Defocus Blur	\(\checkmark\)	\(\times\)	46.28	2.51	Alleviates pinhole vs. real aperture mismatch
Full PG-LF Pipeline	\(\checkmark\)	\(\checkmark\)	30.05	1.38	Joint optical modeling yields compounding gains

Reciprocal control experiments (Data vs. Architecture) demonstrate orthogonal contributions: - PlaneNet trained on HCI achieves 91.31 mm MAE, which drops to 53.11 mm when trained on PG-LF (41.8% reduction). - VisDepth trained on HCI yields 52.95 mm MAE, which drops to 30.05 mm under PG-LF (43.2% reduction). - VisDepth outperforms PlaneNet by 23.06 mm under identical PG-LF supervision.

Material-stratified evaluation on 143 cleanly separable scenes validates metric fidelity across challenging surfaces: Diffuse (78 scenes, 59.8% valid pixels) achieves 8.83 mm MAE; Specular (31 scenes, 54.8% valid pixels) achieves 21.76 mm MAE; and Transparent (34 scenes, 37.2% valid pixels) achieves 39.81 mm MAE.

Key Findings

• Supervision domain mismatch is the primary bottleneck: Cross-training experiments prove that real-world failure in light field depth estimation is caused by synthetic-to-real optical distribution divergence rather than network capacity. Training an existing architecture (PlaneNet) with PG-LF cuts error by over 41%.
• Super-additive synergy between defocus and non-Lambertian modeling: Adding defocus alone cuts MAE by 3.73 mm and non-Lambertian perturbation alone cuts MAE by 6.76 mm; combining both slashes MAE by 19.96 mm (from 50.01 to 30.05 mm), proving that real cameras corrupt disparity correspondences through coupled blur and specularity.
• Explicit visibility eliminates boundary leakage: Disabling explicit visibility masking (Ours w/o Vis) increases MAE from 30.05 mm to 39.03 mm, with qualitative visualizations confirming severe disparity bleeding across occluded object silhouettes.

Highlights & Insights

• Task-grounded physical synthesis over generic photorealism: Instead of computationally prohibitive NeRF or path-tracing pipelines, SynLF selectively models the specific physical phenomena that break micro-baseline stereo correspondence (defocus blur, occlusion, specularity), achieving extraordinary zero-shot transfer at minimal compute.
• Symmetric reuse of the geometric projection operator: The depth-aware forward splatting operator used to synthesize multi-view training data in PG-LF is identically reused inside VisDepth during inference to project visibility masks, establishing mathematical consistency between generative and discriminative stages.
• Complementary decoupling of monocular scale-free priors and multi-view disparity: Using a frozen foundation model (DAv2) for structural initialization prevents iterative optimization from falling into textureless local minima, while micro-baseline correlation strictly handles metric scale recovery.

Limitations & Future Work

• Extreme specular and transparent surface degradation: Under pure transparent glass and mirror reflection, micro-baseline disparity physically reflects background or mirrored objects, resulting in higher MAE (39.81 mm) on transparent surfaces.
• Higher-order optical aberrations unmodeled: The synthesis pipeline assumes an idealized thin-lens model with radial defocus, ignoring real-world vignetting, lens distortion, field curvature, and coma near sensor boundaries.
• Future directions: Integrating differentiable wave optics and ray-tracing into on-the-fly synthesis, extending PG-LF to dynamic video light fields, and exploring physics-grounded synthesis for 6-DoF plenoptic pose tracking and novel view synthesis.

Related Work & Insights

• vs PlaneNet [7] / ESMNet [6]: Conventional state-of-the-art light field networks rely on small pre-rendered datasets (HCI) and implicit attention aggregation; SynLF leverages on-the-fly PG-LF synthesis over 74k RGB-D scenes and explicit visibility masking, cutting MAE on real captures from 91.31 mm / 110.86 mm down to 30.05 mm.
• vs Monocular Metric Depth (Metric3D v2 [16], Depth Pro [4]): Monocular models infer metric scale from visual semantic priors and are prone to gross scale collapse under unseen focal lengths or camera tilts; SynLF computes physical metric depth from unambiguous micro-baseline disparity triangulation within a single exposure.
• vs Neural Single-Image Light Field Synthesis [12, 23]: Purely neural synthesis hallucinates plausible textures but lacks rigid multi-view geometric consistency; PG-LF enforces explicit depth projection and epipolar consistency, serving as high-fidelity geometric supervision.

Rating

• Novelty: ⭐⭐⭐⭐ [Physics-grounded on-the-fly light field synthesis paired with bidirectional operator reuse for explicit visibility reasoning]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Custom 9x9 plenoptic and structured-light rig, reciprocal cross-training, material-stratified metrics, and outdoor qualitative checks]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical derivation, transparent motivation, and well-structured empirical analysis]
• Value: ⭐⭐⭐⭐⭐ [Provides a scalable, storage-efficient blueprint for robust real-world plenoptic sensing and robotic metric depth perception]