Zero-Shot Depth from Defocus: Zero-Shot Metric Depth Estimation from Focus Stacks¶

Conference: ECCV 2026
arXiv: 2603.26658
Code: https://github.com/princeton-vl/FOSSA
Area: 3D Vision
Keywords: depth from defocus, zero-shot generalization, Transformer, focus stack attention, depth benchmark

TL;DR¶

This paper proposes FOSSA, a Transformer-based zero-shot depth from defocus (DfD) estimation network. By efficiently aggregating defocus cues in focus stacks through a novel stack attention layer and training on large-scale synthetic data, it achieves strong zero-shot generalization across datasets and scenes. Additionally, the authors release ZEDD—a high-quality real-world depth from defocus benchmark featuring 100 scenes, 4K resolution, and LiDAR ground truth.

Background & Motivation¶

The task of Depth from Defocus (DfD) is to reconstruct per-pixel metric depth from a set of images captured from the same viewpoint but at different focus distances (i.e., a focus stack). As the camera scans the focal plane from near to far, objects at different depths in the scene sequentially transition from sharp to blurry. This blur pattern varying with focus distance naturally encodes geometric depth information, making DfD uniquely valuable for downstream applications such as post-capture defocus control, view synthesis, and relighting. However, almost all existing DfD methods are limited to in-domain settings: they are trained and tested on some small-scale dataset, and their performance drops drastically when scenes, camera parameters, or lighting conditions change, falling far short of practical deployment requirements for zero-shot generalization.

Two factors contribute to this dilemma. First is the lack of high-quality real-world DfD benchmarks. The existing DDFF dataset only contains 12 scenes, and the images are synthesized from light field cameras (with extremely small apertures, making defocus effects barely visible), while the depth ground truth comes from structured light sensors (which suffer from low resolution, high noise, and a range limit of only 3.5m). Second, the design capacity of existing DfD networks is insufficient, mostly relying on handcrafted domain-specific layers (such as differential focus volumes and variational solvers), making it difficult to scale and benefit from large-scale pre-trained general vision models. These two issues are intertwined: without a large and high-quality benchmark, it is difficult to develop a robust network; with weak network capabilities, it is impossible to learn effectively from large-scale data.

This paper breaks through this bottleneck from both the benchmark and model sides. On the benchmark side, the authors build ZEDD—using a 4K DSLR, various aperture sizes (up to F/1.4), and a high-end Ouster LiDAR to capture 100 diverse indoor and outdoor scenes. The number of scenes is 8.3 times that of DDFF, with real and visible defocus effects and high-precision, high-density depth ground truth. On the model side, they propose FOSSA (FOCus Stack Attention Transformer), a lightweight ViT-based architecture tailored for the DfD task, packaged with a domain-randomized synthetic data pipeline that enables the model to work zero-shot on ZEDD and various cross-domain datasets. Core Idea: By inserting a one-dimensional stack attention layer between ViT layers and encoding the geometric position of each frame with focus distance embeddings, the network can efficiently exchange defocus cues along the stack dimension. This decoupled design of "per-frame spatial encoding + cross-frame stack interaction" both preserves the pre-trained weights of large models and precisely captures the defocus patterns unique to DfD that vary with focus distance, achieving up to a 55.7% error reduction under zero-shot scenarios.

Method¶

Overall Architecture¶

The input to FOSSA is a focus stack $\mathbf{I} = (\mathbf{I}_1, \dots, \mathbf{I}_M)$ ($M$ pixel-aligned RGB images) and their corresponding focus distances $\mathbf{d}=(d_1,\dots,d_M)$ (in meters), and the output is a dense metric depth map $\hat{\mathbf{D}} \in \mathbb{R}^{H \times W}$. The entire forward process is divided into two stages.

The first stage is per-frame feature extraction. Each image independently passes through $L_1$ feature extractor layers, each consisting of two sub-modules: a weight-shared ViT Block (performing spatial self-attention on each frame) and a novel stack attention layer (interacting across the $M$ frames). The second stage is feature fusion and refinement. The $M$ per-frame feature maps are average-pooled along the stack dimension to collapse into a single global feature map, which is then further refined through $L_2$ ViT Blocks. Finally, a DPT decoding head regresses the dense depth map.

The key strategy of this design is to exchange cross-frame information in the early stage of feature extraction, rather than processing each frame independently and fusing them only at the end. The former allows defocus cues to reinforce each other within the global understanding, while the latter is prone to losing subtle cross-frame differences.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Focus Stack<br/>M images + focus distances"] --> B["ViT Block<br/>Per-frame spatial self-attention"]
    B --> C["Stack Attention Layer<br/>Focus distance embedding +<br/>cross-frame attention along stack dimension"]
    C --> D{"Reached<br/>layer L1?"}
    D -->|No| B
    D -->|Yes| E["Average Pooling<br/>M frames → 1 global feature map"]
    E --> F["L2 ViT Blocks<br/>Global feature refinement"]
    F --> G["DPT decoder head"]
    G --> H["Dense depth map"]

Key Designs¶

1. Stack Attention Layer: Focus-distance-guided cross-frame information exchange

The core signal of DfD does not reside in any single image, but in the transition pattern of a spatial location from blurry to sharp and back to blurry across different focus distances. To capture this pattern, the network needs to explicitly pass information across images taken at different focus distances. The most straightforward way is to perform full attention across all feature maps of the $M$ frames, but this incurs a huge computational cost of $O(MHW)$. The key insight of this paper is that cross-frame interaction only needs to occur among different focus versions at the same spatial location—since defocus signals vary independently for each pixel location.

Based on this insight, the mechanism of the stack attention layer operates in three steps. First, the focus distance $d_i$ of each image is encoded into a $C$-dimensional vector (focus distance embedding) via a two-layer MLP, which is added to all image tokens of that frame. Then, the feature maps of the $M$ frames are concatenated along the stack dimension into a 4D tensor $\mathbf{F}' \in \mathbb{R}^{M \times C \times (H/p \times W/p)}$. Finally, independently for each spatial location, standard self-attention is performed along the stack dimension (of size $M$)—meaning that the $M$ feature vectors at the same patch position interact with each other to perceive "which is the sharpest, which is the blurriest, and what their respective focus distances are."

The elegance of this design lies in its computational efficiency: the attention complexity $O(M^2)$ only depends on the stack size $M$ (typically $M=5$ in experiments), which is much smaller than the number of spatial tokens $HW/p^2$ (typically $>10^3$). In other words, it enables critical multi-frame information exchange with minimal extra overhead, while its focus distance embedding serves as a geometric ruler—telling the model at what depth level each feature vector was captured, allowing the model to decode absolute depth from the ordered changes in blurriness.

2. Synthetic Data and Domain Randomization Pipeline: Enabling zero-shot generalization

Large-scale real DfD datasets do not exist—collecting real focus stacks requires precise hardware synchronization and high-precision depth ground truth, which is highly costly. Therefore, FOSSA is trained entirely on synthetic focus stacks. The synthetic pipeline builds on existing RGBD datasets (Hypersim with 66K indoor samples + TartanAir with 307K outdoor samples) and utilizes a physical defocus model to generate realistic focus stacks.

The core formula is the Circle of Confusion (CoC): $$\text{CoC} = \frac{|\mathbf{D} - d|}{\mathbf{D}} \cdot \frac{f^2}{N(d-f)}$$

where $d$ is the focus distance, $f$ is focal length, and $N$ is the f-number (aperture). From CoC, a per-pixel blur kernel (Point Spread Function, PSF) can be generated, which is convolved with the original image to obtain the defocused image. However, the PSF shape of real cameras is not fixed: it is Gaussian-like when dominated by diffraction/small-apertures and approaches a uniform disc shape when dominated by geometric optics/large-apertures. To bridge the synthetic-to-real domain gap, they propose a generalized PSF formula: $$\mathcal{F}^{\text{Generalized}}(u,v,p) = \frac{1}{c^2} \exp\!\left(-2\left(\frac{u^2+v^2}{c^2}\right)^{\frac{p}{2}}\right)$$

When the shape parameter $p=2$, it is a Gaussian PSF; when $p \to +\infty$, it is a uniform disc PSF. During training, $p \sim 2^{U(1,5)}$ is randomly sampled to cover the complete continuous spectrum from Gaussian to super-disc. At the same time, the f-number $N$ is randomly selected from $[1.0, 1.4, 2.0, 2.8, 4.0]$, and the focus distances are mixed-sampled using two modes (photographer's manual designation vs. automatic presets). This comprehensive domain randomization allows the model to remain robust to various optical setups of real cameras after being trained purely on synthetic data.

3. Early Fusion Strategy: Collapsing the stack early in feature extraction

Most DfD methods choose to process all $M$ frames independently through the entire encoder before merging features. However, FOSSA employs the opposite approach—collapsing the $M$-frame feature maps into a single global feature map via average pooling after only $L_1=4$ extraction layers, and then using the subsequent $L_2=8$ ViT Blocks to perform single-image depth reasoning.

The motivation for this strategy is that defocus cues are most critical in shallow features—subtle differences in blurriness are most prominent in low-level spatial details (texture, edges), whereas high-level semantic features lose defocus contrast after abstracting. Therefore, collapsing early and using a stronger downstream network to unify reasoning not only avoids redundant computation from $M$-way parallel encoding (the subsequent $L_2$ layers only need to process a single feature map) but also allows the refinement phase to comprehend depth relationships within a complete global context. Intuitively, this decouples the DfD task into two phases: "shallow defocus signal extraction + deep monocular depth reasoning," leveraging the strengths of ViT in both.

Loss & Training¶

The loss function is a weighted sum of the SiLog loss ($\lambda=0.5$ balancing metric and relative depth) and the gradient matching loss: $$L(\hat{\mathbf{D}}, \mathbf{D}) = \text{SiLog}(\hat{\mathbf{D}}, \mathbf{D}) + 0.1 \cdot \text{GradientMatching}(\hat{\mathbf{D}}, \mathbf{D})$$

The ViT Blocks are initialized with indoor metric depth pre-trained weights from DepthAnythingv2 (crucial for zero-shot inference), and the MLPs in the stack attention layers are zero-initialized. Training runs for 40 epochs with a batch size of 8, taking about 2 days on 4× L40 GPUs. The learning rate is exponentially decayed, and the learning rate scale factor for the pre-trained backbone is set to 0.5.

Key Experimental Results¶

Main Results¶

Dataset	Metric	Ours (ViT-B)	Prev. Best (DepthPro)	Gain
ZEDD (test)	AbsRel ↓	0.089	0.201	-55.7%
ZEDD (test)	δ₁.₂₅ ↑	0.918	0.665	+38.0%
Infinigen Defocus	AbsRel ↓	0.091	0.176 (DepthPro)	-48.3%
DDFF	MSE ↓	— (finetuned)	—	-40.4% vs DualFocus

Ablation Study¶

Configuration	ZEDD AbsRel	Description
Ours (ViT-B)	0.089	Full model
Ours (ViT-S)	0.098	Lightweight model still significantly outperforms all baselines
DFF-DFV (retrained w/ same data)	—	Quantitative gap remains after retraining with the same data pipeline, proving architectural contribution
Monocular Baseline DepthPro	0.201	State-of-the-art monocular method, still significantly outperformed
HybridDepth	0.754	Traditional DfD method nearly fails under zero-shot settings

Key Findings¶

Zero-shot generalization gap: All traditional DfD methods (HybridDepth, DFF, DEReD) perform poorly on ZEDD (δ₁.₂₅ < 0.58), even worse than monocular methods, confirming they totally lack cross-domain capabilities. In contrast, FOSSA stands out (δ₁.₂₅ = 0.918), demonstrating that the holistic design of "architecture + data pipeline" is effective.
Ablation Attribution: Even when DFF-DFV is provided with the exact same training data, FOSSA still outperforms it across the board, demonstrating that FOSSA's architectural design (stack attention + early fusion) itself contributes the core gains, rather than it being a pure data advantage.
Excellent performance on transparent objects (HAMMER): On the HAMMER dataset (containing transparent/reflective objects), FOSSA achieves δ₁.₂₅=0.999, AbsRel=0.017, which is nearly perfect. This is likely because the stack attention layer, through focus distance embeddings, learns to leverage the defocus features of transparent objects (while transparent objects are extremely difficult for standard depth sensors to capture, their defocus patterns are still clearly discernible).
Strong Robustness: When evaluated over a range of aperture sizes from F/1.4 to F/5.6 and stack sizes from 3 to 9 images, FOSSA exhibits significantly smaller performance drops than baseline methods, showing excellent environmental adaptability.

Highlights & Insights¶

Computational efficiency design of stack attention: By limiting attention to the stack dimension ($M$ is typically only 5) instead of full spatial cross-attention, it exchanges cross-frame defocus information while keeping computational overhead extremely low. This is a structural prior unique to the DfD task: "information transfer among different focuses at the same location."
Early fusion over late fusion: While most methods independently encode all $M$ frames to the end and then fuse them, FOSSA’s early fusion approach is surprising yet highly effective—suggesting that defocus cues are essentially low-level vision signals; deep semantics do not help. Collapsing features early allows subsequent depth inference to be more focused.
Focus distance as a learning anchor: Traditional DfD methods (like DDFS) require embedding full camera parameters (focal length, aperture, pixel size, etc.). FOSSA uses only focus distance as semantic embeddings, letting other parameters be learned implicitly through the highly diverse randomization during training, which simplifies the model while achieving better generalization.
Physics-inspired domain randomization of PSF shapes: A shape parameter $p$ is utilized to unify Gaussian and disc-like PSFs into a continuous spectrum, sweeping across various optical degradation modes during training. This elegantly bridges the simulation gap between synthetic data and real-world cameras.

Limitations & Future Work¶

The current model only supports static scenes—if the camera or the objects move during capture, the focus stack will be misaligned. Future work could introduce optical flow alignment or temporal modeling to handle handheld capture or dynamic environments.
Although the synthetic data pipeline utilizes domain randomization, simple PSF convolution still exhibits a gap from the complex optical aberrations (e.g., lens distortion, vignetting) of real cameras. Employing more realistic learning-based defocus rendering (such as BokehMe) represents a promising next step.
The stack size is fixed at 5 images. In ZEDD testings, the model's performance almost saturates at 9 inputs. Future directions could explore adaptive stack sizes or inference strategies with variable-length stacks.

vs. Traditional DfD Methods (HybridDepth/DFV/DefocusNet): These rely on handcrafted domain-specific network layers (e.g., differential focus volumes, multi-stage alignment, variational solvers), which are effective in-domain but fail to generalize zero-shot. FOSSA shifts to a ViT backbone + stack attention, utilizing large-scale pre-training to break through this limitation.
vs. Monocular Depth Estimation (DepthPro/DepthAnythingv2): Monocular methods infer depth from a single all-in-focus image. While they have strong zero-shot capacity, it is difficult to obtain precise metric depth (due to scale ambiguity). FOSSA leverages the multi-view blur variation of focus stacks to disambiguate scale, showing natural advantages in metric accuracy.
vs. ZEDD and DDFF: The acquisition quality of ZEDD (4K resolution, large F/1.4 aperture, LiDAR ground truth) vastly outperforms DDFF (low resolution, light-field simulated defocus barely visible, structured light depth limited to 3.5m), offering a more challenging and practical benchmark for testing DfD.

Rating¶

Novelty: ⭐⭐⭐⭐☆ FOSSA applies an elegant adaptation of standard ViT + stack attention to DfD rather than hard-coding network structures, which is novel; the ZEDD benchmark contribution is significant though leans towards engineering.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensively evaluated across 6 datasets (encompassing synthetic, real, and cross-domain setups). The ablation study clearly separates the contributions of the architecture and data pipeline, coupled with solid robustness analysis.
Writing Quality: ⭐⭐⭐⭐⭐ The motivation is clear (solid explanation of how the two problems are intertwined), the method description is detailed with adequate schematic diagrams, and the appendix contains plenty of implementation details for reproducibility.
Value: ⭐⭐⭐⭐⭐ Simultaneously delivers model, benchmark, and data pipeline resources, effectively driving the DfD field forward from "in-domain overfitting" to "zero-shot generalization."