FoundDP: Revisiting Weak Disparity Observability in Dual-Pixel Depth Estimation¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/EchoLighting/FoundDP
Area: 3D Vision
Keywords: dual-pixel imaging, metric depth estimation, depth foundation model, disparity observability, feature alignment

TL;DR¶

Addressing the structural collapse caused by weak disparity observability under dual-pixel (DP) narrow baselines in textureless and downsampled regions, FoundDP bridges physically grounded DP metric depth with global structural priors from a monocular foundation model (DAV2), leveraging defocus-aware feature alignment and adaptive spatial gating to deliver high-fidelity, metric-accurate dense depth estimation.

Background & Motivation¶

Dual-pixel (DP) sensors split each photodiode into left and right sub-pixels beneath a shared micro-lens, enabling single-exposure capture of sub-aperture disparity. By turning consumer camera sensors into implicit stereo systems without extra hardware overhead, DP imaging has emerged as an attractive hardware primitive for single-shot metric depth estimation. However, the effective baseline between DP sub-apertures is extremely narrow—only a tiny fraction of the aperture diameter. This physical constraint fundamentally restricts disparity observability: in textureless flat surfaces, low-contrast regions, distant backgrounds, or downsampled frames, the sub-pixel disparity signal weakens below the sensor noise floor. As a result, conventional learning-based DP estimators (e.g., DPNet, DDDNet, SFBDNet) that rely strictly on local correspondence matching suffer severe structural degradation, boundary tearing, and extensive depth holes.

Conversely, recent monocular depth estimation foundation models based on Vision Transformers (ViT), such as Depth Anything V2 (DAV2) and MoGe, demonstrate remarkable zero-shot structural reasoning and boundary sharpness. By modeling long-range contextual dependencies across large-scale pretraining datasets, they reconstruct visually coherent scene geometry even across untextured surfaces. Nonetheless, monocular depth estimation is an intrinsically ill-posed inverse problem; its predictions are statistically plausible but lack physical constraints, remaining ambiguous up to an unknown affine scale and shift. This exposes a clear complementary dilemma: DP imaging provides physically grounded metric scale but suffers from weak local observability, whereas foundation models excel at global structural consistency but cannot provide absolute metric dimensions.

Bridging foundation models directly with DP inputs is further hindered by an overlooked physical phenomenon: DP imaging inherently suffers from substantial optical defocus blur in out-of-focus regions. Defocus induces severe high-frequency attenuation and domain shifts relative to the sharp natural images used for pretraining, disrupting ViT patch self-attention and triggering unstable, hallucinated geometry during depth guidance. The core idea of FoundDP is to anchor absolute metric scale via DP disparity, restore global geometric topology using monocular depth foundation priors, and reconcile the two through defocus-aware ViT feature alignment and adaptive metric-conditioned spatial gating.

Method¶

Overall Architecture¶

FoundDP follows a cascaded, progressive guidance pipeline comprising three main stages: the Dual-Pixel Depth Estimation Module (DDE), the Structure Refinement Module (SR), and the Depth Guidance Module (DG). Given a DP sub-aperture image pair \(I_L, I_R\), the framework first infers an initial physically scaled metric depth \(D_{\text{dp}}\) via DDE. Next, SR combines the input images with \(D_{\text{dp}}\) to predict a residual correction field, patching local geometric artifacts and yielding refined metric depth \(D_{\text{metric}}\). In parallel, DP defocus blur is mitigated by aligning intermediate ViT representations. Finally, DG uses the aligned ViT structural features modulated by \(D_{\text{metric}}\) through an adaptive spatial gating mechanism to generate the final high-precision metric depth \(D_{\text{guide}}\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Dual-Pixel Sub-Aperture Pair (IL, IR)"] --> DDE["Symmetric Cost Volume & DDE Metric Regression<br/>Establishes absolute physical scale anchor from micro sub-aperture disparity"]
    DDE --> SR["Structure Refinement Module<br/>Multi-scale residual attention repairs local holes to yield D_metric"]
    In --> ViTAlign["Defocus-Aware ViT Feature Alignment<br/>Cross-domain feature distance constraints eliminate blur-induced attention bias"]
    SR --> DG["Adaptive Gated Metric Conditioning<br/>Spatial gating dynamically balances local metric evidence and global structural priors"]
    ViTAlign --> DG
    DG --> Out["High-Fidelity Metric Depth Prediction D_guide"]

Key Designs¶

1. Symmetric Cost Volume and DDE Metric Depth Regression: Eliminating Optical Asymmetry Bias

Standard binocular stereo frameworks typically search disparity in a single lateral direction. In DP sensors, however, the left and right sub-apertures share an identical optical axis and exhibit symmetric point spread functions across the focal plane. Single-sided search ranges induce systematic regression bias across in-focus and out-of-focus transitions. DDE addresses this by employing a shared multi-scale convolutional backbone (enhanced with dilated convolutions and pooling branches) to extract \(1/4\)-resolution feature maps \(F_L, F_R \in \mathbb{R}^{C \times H' \times W'}\). It constructs an explicit symmetric disparity hypothesis space centered at zero disparity, \(d \in [-D/2, D/2]\), concatenating matching features along the channel axis into a 3D cost volume of size \(2C \times D \times H' \times W'\). After 3D CNN regularization, continuous expectation regression via Softmin calculates initial metric depth:

\[P(d \mid x, y) = \frac{\exp(-C(x, y, d))}{\sum_{d'} \exp(-C(x, y, d'))}, \quad D_{\text{dp}}(x, y) = \sum_{d \in [-D/2, D/2]} d \cdot P(d \mid x, y)\]

Continuous expectation avoids discrete quantization artifacts and provides smooth, physically grounded depth \(D_{\text{dp}}\) even under sub-pixel shifts.

2. Structure Refinement Module: Multi-Scale Residual Attention for Local Void Infilling

While \(D_{\text{dp}}\) anchors physical scale, low disparity observability in planar and textureless areas leaves residual noise and local depth tears. To fix these local artifacts without distorting the physical scale, the SR module formulates depth refinement as residual correction rather than full prediction. The network adopts an encoder-decoder architecture taking the channel-wise concatenation of \(I_L, I_R\), and \(D_{\text{dp}}\) as input. High-resolution stages integrate efficient channel attention (ECA) and 2D relative positional encodings to emphasize salient geometric edges while suppressing noisy responses. The module predicts a residual correction field \(\Delta D\):

\[D_{\text{metric}} = D_{\text{dp}} + \mathcal{F}_{\text{SR}}(I_L, I_R, D_{\text{dp}})\]

Restricting the target to a residual manifold preserves the underlying physical distance scale while smoothing out noise and reconstructing sharp depth transitions.

3. Defocus-Aware ViT Feature Alignment: Eliminating Optical Blur Representation Shift

When monocular foundation encoders pretrained on sharp natural images are directly applied to DP inputs, large-aperture optical defocus blur severely degrades high-frequency details. This degradation shifts the input distribution, flattening shallow patch embeddings and dispersing ViT self-attention across blurred boundaries. To stabilize structural priors, FoundDP introduces an explicit feature alignment objective during training. Feeding paired sharp RGB images \(I_{\text{clear}}\) and their DP defocus counterparts \(I_{\text{blur}}\) into a shared ViT backbone, intermediate features \(\Phi_l\) are constrained via multi-layer \(L_1\) supervision:

\[\mathcal{L}_{\text{align}} = \sum_{l \in \{3, 6, 9, 12\}} \|\Phi_l(I_{\text{clear}}) - \Phi_l(I_{\text{blur}})\|_1\]

This constraint forces the encoder to extract defocus-invariant geometric representations, boosting the cosine feature similarity between blurred and sharp inputs from 0.8737 to 0.9454 and ensuring reliable conditioning for downstream guidance.

4. Adaptive Gated Metric Conditioning: Harmonizing Physical Anchors with Foundation Priors

The Depth Guidance module (DG) deploys a DPT-style progressive decoder over multi-scale ViT features \(\{F_1, F_2, F_3, F_4\}\). Rather than simple feature summation or concatenation, DG adopts an adaptive spatial gating mechanism. At each decoding scale with intermediate feature \(X\), the aligned and normalized metric depth \(\hat{D}_{\text{metric}}\) is projected into a spatial modulation signal \(\psi(\hat{D})\) and a continuous gating map \(g(\hat{D}) \in [0, 1]\):

\[X' = X + \phi(X) + g(\hat{D}) \odot \psi(\hat{D})\]

Here \(\phi(X)\) denotes residual feature transformation. In high-observability regions with strong texture and unambiguous disparity, \(g(\hat{D}) \to 1\), firmly tying predictions to the physical metric condition \(\psi(\hat{D})\). In weak-observability zones (e.g., textureless surfaces or severe downsampling), the gate smoothly attenuates, allowing the foundation model's globally consistent topological prior \(\phi(X)\) to govern depth reconstruction and prevent structural collapse.

Loss & Training¶

Loss Function: Unlike monocular depth tasks that employ scale-invariant losses (e.g., SiLog), DP depth contains physical scale. FoundDP supervises all three stages (\(D_{\text{dp}}\), \(D_{\text{metric}}\), \(D_{\text{guide}}\)) using a unified Smooth L1 loss in logarithmic depth space:

\[\mathcal{L}_{\text{depth}} = \frac{1}{N} \sum_{i=1}^N \text{SmoothL1}\left(\log D^{(i)} - \log D_{\text{gt}}^{(i)}\right)\]

Stage-Wise Training Strategy: 1. Stage 1 (DDE Training): DDE is trained independently to regress reliable physical disparity and metric scale; 2. Stage 2 (SR Training): Freezing DDE, SR is optimized to refine residual geometry conditioned on image appearance; 3. Stage 3 (ViT Alignment): The pretrained DAV2 encoder is fine-tuned under \(\mathcal{L}_{\text{align}}\) to bridge the DP defocus domain gap; 4. Stage 4 (DG Optimization): With DDE, SR, and the ViT encoder frozen, only the DPT decoder and spatial gating modules are trained, preventing destruction of the physical metric anchor while integrating foundation priors.

Key Experimental Results¶

Main Results¶

FoundDP was evaluated on synthetic NYUData (generated via a ray-tracing DP simulator) and three real-world benchmarks (DP2020, DP5K, DP2019) across the physical range \([1.0\text{m}, 10.0\text{m}]\). Baselines include DPNet, SFBDNet, DDDNet, and CADSNet (CVPR 2024). Metrics include Affine-Invariant errors \(AI(1)\) and \(AI(2)\), rank correlation \(1 - |\rho_s|\), and threshold accuracies Acc-1 (\(\delta < 1.25\)) and Acc-2 (\(\delta < 1.25^2\)).

Dataset	Method	AI(1) ↓	AI(2) ↓	1 - \|\(\rho_s\)\| ↓	Acc-1 ↑	Acc-2 ↑
NYUData (Synthetic)	DPNet	0.9271	1.2896	0.2179	0.2925	0.5777
	SFBDNet	0.5330	0.8576	0.1489	0.6579	0.9174
	DDDNet	0.2271	0.3333	0.0299	0.9520	0.9953
	CADSNet (CVPR'24)	0.2018	0.2931	0.0248	0.9271	0.9933
	Ours (FoundDP)	0.1475	0.2368	0.0178	0.9661	0.9988
DP2020 (Real)	DPNet	0.3509	0.5629	0.1649	0.7521	0.8180
	SFBDNet	0.0633	0.2095	0.0241	0.9963	0.9987
	DDDNet	0.0851	0.2564	0.1138	0.6934	0.9757
	CADSNet	0.0162	0.0424	0.0467	0.8153	0.9997
	Ours (FoundDP)	0.0110	0.0280	0.0232	0.9998	1.0000
DP5K (Real)	CADSNet	0.2680	0.7854	0.0977	0.6716	0.8773
	Ours (FoundDP)	0.2579	0.7562	0.0989	0.7976	0.9251
DP2019 (Real)	CADSNet	0.1313	0.2807	0.3491	0.7335	0.9277
	Ours (FoundDP)	0.1168	0.2622	0.3224	0.8571	0.9404

Ablation Study¶

Ablations on NYUData evaluate the progressive inclusion of core components, while experiments on DP5K validate the necessity of ViT feature alignment.

Config ID	Component Setup (DDE / SR / DG)	AI(1) ↓	AI(2) ↓	1 - \|\(\rho_s\)\| ↓	Acc-1 ↑	Acc-2 ↑	Note
A1	DDE Baseline Only	0.2463	0.3764	0.0332	0.9297	0.9610	Relies purely on weak disparity; noisy surfaces
A2	DDE + SR	0.2312	0.3486	0.0298	0.9491	0.9715	Residual refinement sharpens local discontinuities
A3	DDE + DG (w/o SR)	0.1865	0.2871	0.0220	0.9348	0.9883	Injects global priors; substantial structure gain
A4	Full (DDE + SR + DG)	0.1475	0.2368	0.0178	0.9661	0.9988	Harmonizes local continuity and global prior

Ablation on ViT feature alignment on DP5K: without alignment, the base model achieves \(AI(1) = 0.2741\) and \(\text{Acc-1} = 0.7839\); incorporating feature alignment improves \(AI(1)\) to \(0.2579\) and \(\text{Acc-1}\) to \(0.7976\). Cosine similarity against clear features improves from 0.8737 to 0.9454, proving that compensating for optical blur is critical for stable guidance.

Key Findings¶

Unrivaled Gains in Weak Regions and Spatial Downsampling: In weak-disparity masks extracted via Sobel gradient thresholds on NYUData, FoundDP slashes \(AI(1)\) from 0.1949 (CADSNet) to 0.1280. On the custom DPDown70 dataset (where spatial downsampling degrades sub-pixel disparity precision), FoundDP sustains an Acc-1 of 0.7035, while SFBDNet, DDDNet, and CADSNet degrade drastically to 0.4113, 0.3524, and 0.5047, respectively.
Histogram Alignment: Ground-truth depth histograms show that pure monocular foundation models suffer from lateral peak shifts (scale ambiguity), whereas pure DP models preserve mean scale but suffer from dispersion noise. FoundDP aligns tightly with the ground-truth distribution peak, confirming the synergy between metric conditioning and structural regularisation.

Highlights & Insights¶

Reconciling Hardware Observability Limits with Foundation Models: Identifies the fundamental trade-off between physical baseline observability in computational imaging and learned structural priors, avoiding the dead-end of solely increasing DP stereo matching network depth.
Defocus-Aware Feature Harmonization: Diagnoses how camera optical blur degrades Vision Transformer patch representations and resolves it via explicit feature alignment, offering a transferable design for computational photography pipelines adopting vision foundation models.

Limitations & Future Work¶

Breakdown under Extreme Sensor Noise: The system relies on DP depth as an absolute metric anchor. Under near-zero illumination or extreme sensor overexposure where sub-pixel gradients collapse completely, erroneous metric anchors can degrade the downstream spatial gating.
Inference Latency of ViT Architectures: Deploying large Vision Transformer encoders and multi-scale DPT decoders incurs significant computational overhead compared to lightweight CNNs, posing deployment challenges for on-device real-time mobile pipelines.

vs DPNet / DDDNet / SFBDNet: Previous DP networks treat depth purely as a sub-pixel disparity matching problem, inevitably failing in untextured regions; FoundDP breaks this ceiling by introducing monocular foundation priors to reconstruct topology where disparity is unobservable.
vs Depth Anything (DAV2) / MoGe: Monocular foundation models produce rich relative geometry but lack metric scale; FoundDP uses DP sub-apertures as an intrinsic optical ruler, achieving zero-drift absolute metric depth without external LIDAR or bulky stereo rigs.

Rating¶

Novelty: ⭐⭐⭐⭐☆ [Compelling synergy between DP physical baseline limits and foundation models; principled defocus alignment]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive synthetic and real-world evaluations, weak-region slicing, and dedicated downsampling benchmark]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous physical explanations, and structured problem formulation]
Value: ⭐⭐⭐⭐⭐ [Provides a practical and principled blueprint for metric computational photography in consumer cameras]