Zero-Shot Depth from Defocus: Zero-Shot Metric Depth Estimation from Focus Stacks¶

Conference: ECCV 2026
arXiv: 2603.26658
Code: https://github.com/princeton-vl/FOSSA
Area: 3D Vision
Keywords: Depth from defocus, zero-shot generalization, Transformer, focus stack attention, depth benchmark

TL;DR¶

This paper proposes FOSSA, a Transformer-based zero-shot depth-from-defocus network that efficiently aggregates defocus cues from a focus stack via a novel stack attention layer. Trained on large-scale synthetic data, it achieves strong zero-shot generalization across datasets and scenes. Additionally, this work releases ZEDD, a high-quality real-world depth-from-defocus benchmark containing 100 scenes, 4K resolution, and LiDAR ground truth.

Background & Motivation¶

The task of Depth from Defocus (DfD) is to reconstruct per-pixel metric depth from a set of images captured from the same viewpoint but at different focus distances (i.e., a focus stack). As the camera scans the focal plane from near to far, objects at different depths in the scene successively transition into focus and then blur again. This focus-dependent blur pattern naturally encodes geometric depth information, giving DfD unique value in downstream applications such as post-capture defocus control, view synthesis, and relighting. However, almost all existing DfD methods are restricted to in-domain settings, where they are trained and tested on small-scale datasets. Once scenes, camera parameters, or lighting conditions change, their performance drops sharply, falling far short of the requirements for practical zero-shot deployment.

Two main factors contribute to this bottleneck. First, there is a lack of high-quality real-world DfD benchmarks. The existing DDFF dataset contains only 12 scenes, with images synthesized from a light field camera (having an extremely small aperture where defocus effects are almost invisible) and depth ground truth obtained from structural light sensors (low resolution, noisy, and limited range up to 3.5m). Second, existing DfD networks are limited in capability, mostly relying on hand-crafted specialized layers (such as differential focus volumes or variational solvers) that are difficult to scale and benefit from large-scale pre-trained general vision models. These two problems are intertwined: without a large and high-quality benchmark, it is difficult to develop robust networks; and with weak network designs, models cannot learn effectively from large-scale data.

This paper tackles both issues simultaneously. On the benchmark side, the authors construct ZEDD, which uses a 4K DSLR with multiple apertures (up to F/1.4) and a high-end Ouster LiDAR to collect 100 diverse indoor and outdoor scenes (8.3 times more scenes than DDFF), featuring visually distinct defocus effects and high-precision, high-density depth ground truth. On the model side, FOSSA (FOCus Stack Attention Transformer) is proposed, a lightweight ViT-based architecture tailored for the DfD task, combined with a domain-randomized synthetic data pipeline that enables zero-shot generalization on ZEDD and other cross-domain datasets. Core Idea: A 1D stack attention layer is inserted between ViT layers, using focus distance embeddings to encode the geometric position of each frame. This allows the network to efficiently exchange defocus cues along the stack dimension. This decoupled design of "per-frame spatial encoding + cross-frame stack interaction" both preserves the pre-trained weights of large models and precisely captures the focus-distance-dependent blur pattern unique to DfD, achieving an error reduction of up to 55.7% in zero-shot scenarios.

Method¶

Overall Architecture¶

The input to FOSSA is a focus stack $\mathbf{I} = (\mathbf{I}_1, \dots, \mathbf{I}_M)$ ($M$ pixel-aligned RGB images) and their corresponding focus distances $\mathbf{d}=(d_1,\dots,d_M)$ (in meters). The output is a dense metric depth map $\hat{\mathbf{D}} \in \mathbb{R}^{H \times W}$. The entire feedforward process consists of two stages.

The first stage is per-frame feature extraction. Each image is processed independently through $L_1$ blocks of the feature extractor. Each block contains two sub-modules: a weight-shared ViT Block (performing spatial self-attention on each frame) and a novel stack attention layer (exchanging information across the $M$ frames). The second stage is feature fusion and refinement. The $M$ per-frame feature maps are average-pooled along the stack dimension to collapse into a single global feature map. This map is further refined through $L_2$ ViT Blocks, and finally, a dense depth map is regressed using a DPT decoder head.

The key strategy of this design is to exchange cross-frame information in the early stages of feature extraction instead of processing frames independently and merging them only at the end. The former allows defocus cues to reinforce each other during global representation learning, whereas the latter is prone to discarding subtle cross-frame differences.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Focus Stack<br/>M images + focus distances"] --> B["ViT Block<br/>Per-frame spatial self-attention"]
    B --> C["Stack Attention Layer<br/>Focus distance embedding +<br/>Cross-frame attention along stack dim"]
    C --> D{"Reached<br/>Layer L1?"}
    D -->|No| B
    D -->|Yes| E["Average Pooling<br/>M frames → 1 global feature map"]
    E --> F["L2 ViT Blocks<br/>Global feature refinement"]
    F --> G["DPT Decoder Head"]
    G --> H["Dense Depth Map"]

Key Designs¶

1. Stack Attention Layer: Guiding cross-frame information exchange with focus distance

The core cue of DfD does not reside in any single image, but in the transition pattern of a spatial location from blurry to sharp and back to blurry under different focus distances. To capture this pattern, the network must explicitly transfer information across images of various focus distances. The most straightforward way is to perform full self-attention over all $M$ frames concurrently, but this incurs a massive computational cost of $O(MHW)$. The key insight of this work is that cross-frame interaction only needs to occur between different focus versions of the same spatial location—since the defocus signal varies independently per pixel location.

Based on this insight, the mechanism of the stack attention layer is divided into three steps. First, the focus distance $d_i$ of each image is encoded into a $C$-dimensional vector (focus distance embedding) via a two-layer MLP, which is then added to all image tokens of that frame. Next, the feature maps of $M$ frames are concatenated along the stack dimension to form a 4D tensor $\mathbf{F}' \in \mathbb{R}^{M \times C \times (H/p \times W/p)}$. Finally, a standard self-attention is performed independently for each spatial location along the stack dimension (of size $M$)—meaning the $M$ feature vectors at the same patch location interact with each other to perceive "which is the sharpest, which is the blurriest, and what their respective focus distances are."

The elegance of this design lies in its computational efficiency: the complexity of the attention $O(M^2)$ depends solely on the stack size $M$ ($M=5$ in experiments), which is far smaller than the number of spatial tokens $HW/p^2$ (typically $>10^3$). In other words, it enables critical multi-frame information exchange at a minimal extra cost. Meanwhile, the focus distance embedding acts like a geometric ruler—informing the model of the depth level at which each feature vector was acquired, allowing the model to decode absolute depth from the ordered variation in blur level.

2. Synthetic Data and Domain Randomization Pipeline: Enabling Zero-Shot Generalization

Large-scale DfD datasets do not exist, as collecting real focus stacks requires precise hardware synchronization and high-precision depth ground truth, which is extremely costly. Therefore, FOSSA is trained entirely on synthetic focus stacks. The synthesis pipeline builds upon existing RGBD datasets (66K indoor samples from Hypersim + 307K outdoor samples from TartanAir) and utilizes a physical defocus model to generate realistic focus stacks.

The core formula is the Circle of Confusion (CoC): $$\text{CoC} = \frac{|\mathbf{D} - d|}{\mathbf{D}} \cdot \frac{f^2}{N(d-f)}$$

where $d$ is the focus distance, $f$ is the focal length, and $N$ is the f-number (aperture). Based on the CoC, a per-pixel blur kernel (Point Spread Function, PSF) can be generated and convolved with the original image to obtain the defocused image. However, the PSF shape of real cameras is not fixed: it is Gaussian-like when dominated by diffraction/small apertures, and approaches a uniform disc under geometric optics/large apertures. To bridge the gap from synthetic to real domains, this paper proposes a generalized PSF formula: $$\mathcal{F}^{\text{Generalized}}(u,v,p) = \frac{1}{c^2} \exp\!\left(-2\left(\frac{u^2+v^2}{c^2}\right)^{\frac{p}{2}}\right)$$

When the shape parameter $p=2$, it corresponds to a Gaussian PSF; as $p \to +\infty$, it approaches a uniform disc PSF. During training, $p \sim 2^{U(1,5)}$ is randomly sampled to cover the complete continuum from Gaussian to super-disc. Meanwhile, the f-number $N$ is randomly selected from $[1.0, 1.4, 2.0, 2.8, 4.0]$, and the focus distance is also hybrid-sampled using two modes (manually specified by photographer vs. automatic presets). This extensive domain randomization allows the model to work robustly under various optical settings of real cameras after being trained purely on synthetic data.

3. Early-Fusion Strategy: Stack collapse in the early stages of feature extraction

Most DfD methods independently process all $M$ frames through the entire encoder before fusing features. In contrast, FOSSA adopts the opposite approach—collapsing the feature maps of the $M$ frames into a single global feature map via average pooling after only $L_1=4$ layers of extraction. Subsequently, $L_2=8$ layers of ViT Blocks perform single-map depth reasoning.

The motivation behind this strategy is that defocus cues are most critical in shallow features. Subtle differences in blur levels are most pronounced in high-frequency spatial details (textures, edges) at early layers, whereas deep semantic features tend to lose defocus contrast after high abstraction. Therefore, collapsing early and leveraging a strong downstream network for unified reasoning avoids the redundant computation of $M$-way parallel encoding (the subsequent $L_2$ layers only process a single feature map), and allows the subsequent refinement stage to understand depth relationships within a complete global context. Intuitively, this strategy decomposes the DfD task into "shallow defocus signal extraction + deep monocular depth reasoning", fully exploiting the strengths of ViT in both phases.

Loss & Training¶

The loss function is a weighted sum of the SiLog loss ($\lambda=0.5$ balancing metric and relative depth) and the gradient matching loss: $$L(\hat{\mathbf{D}}, \mathbf{D}) = \text{SiLog}(\hat{\mathbf{D}}, \mathbf{D}) + 0.1 \cdot \text{GradientMatching}(\hat{\mathbf{D}}, \mathbf{D})$$

The ViT Blocks are initialized with the pre-trained weights for indoor metric depth from DepthAnythingv2 (crucial for zero-shot inference), while the MLPs in the stack attention layers are zero-initialized. Training runs for 40 epochs with a batch size of 8, taking 2 days on 4×L40 GPUs. An exponential learning rate decay is adopted, with a learning rate scale factor of 0.5 for the pre-trained backbone.

Key Experimental Results¶

Main Results¶

Dataset	Metric	Ours (ViT-B)	Prev. SOTA (DepthPro)	Gain
ZEDD (test)	AbsRel ↓	0.089	0.201	-55.7%
ZEDD (test)	δ₁.₂₅ ↑	0.918	0.665	+38.0%
Infinigen Defocus	AbsRel ↓	0.091	0.176 (DepthPro)	-48.3%
DDFF	MSE ↓	— (finetuned)	—	-40.4% vs DualFocus

Ablation Study¶

Configuration	ZEDD AbsRel	Description
Ours (ViT-B)	0.089	Full model
Ours (ViT-S)	0.098	The small model also substantially outperforms all baselines
DFF-DFV (retrained w/ same data)	—	Still shows a gap even after retraining with the same data pipeline, proving architectural contribution
Monocular Baseline DepthPro	0.201	Leading monocular method, yet still significantly outperformed
HybridDepth	0.754	Traditional DfD methods almost entirely fail under zero-shot settings

Key Findings¶

Zero-shot generalization gap: All traditional DfD methods (HybridDepth, DFF, DEReD) perform extremely poorly on ZEDD ($\delta_{1.25} < 0.58$), even worse than monocular methods, proving their complete lack of cross-domain capability. FOSSA stands out ($\delta_{1.25} = 0.918$), indicating that the integrated design of architecture and data pipeline is effective.
Ablation attribution: Even when DFF-DFV is provided with the exact same training data, FOSSA still leads comprehensively, indicating that FOSSA's architectural design (stack attention + early fusion) contributes the core gains rather than pure data advantage.
Outstanding performance on transparent objects (HAMMER): On the HAMMER dataset (transparent/reflective objects), FOSSA achieves $\delta_{1.25}=0.999$ and $\text{AbsRel}=0.017$, which is almost perfect. This likely benefits from the stack attention layer learning to leverage defocus features of transparent objects via focus distance embeddings (while transparent objects are difficult to capture with conventional depth sensors, their defocus patterns remain identifiable).
Strong robustness: Under evaluations with f-numbers ranging from F/1.4 to F/5.6 and focus stack sizes from 3 to 9 images, FOSSA's performance degradation is significantly smaller than that of baseline methods, demonstrating excellent environmental adaptability.

Highlights & Insights¶

Computationally efficient design of stack attention: By limiting attention to the stack dimension ($M$ is typically only 5) instead of fully connecting across spatial dimensions, the model exchanges cross-frame defocus information while keeping computational overhead extremely low. This is a structural prior unique to the DfD task where "information is transferred between different focal levels at the same location."
Early fusion instead of late fusion: While most methods process $M$ channels independently to the end before fusing them, FOSSA's early-fusion approach is unconventional yet effective. This shows that defocus cues are essentially low-level vision signals, while deep semantics do not assist much; collapsing them early allows subsequent depth reasoning to be more focused.
Using focus distance as a learning anchor: Traditional DfD methods (such as DDFS) require embedding complete camera parameters (focal length, aperture, pixel size, etc.). In contrast, FOSSA only uses focus distance for semantic embedding, leaving other parameters to be learned implicitly via diverse dataset randomization during training. This simplifies the model while dramatically improving generalization.
Physics-inspired domain randomization of PSF shapes: A shape parameter $p$ is used to unify Gaussian and disc-like PSFs into a continuous spectrum. Randomly sweeping through various optical degradation styles during training ingeniously bridges the simulation gap between synthetic data and real-world cameras.

Limitations & Future Work¶

Currently, the model only supports static scenes. If the camera or subject moves during capturing, the focus stack will be misaligned. Future work could introduce optical flow alignment or temporal modeling to handle handheld shooting or dynamic scenes.
Although the synthetic data pipeline employs domain randomization, simple PSF convolutions still differ from the complex optical aberrations and lens distortions of real cameras. Implementing more photorealistic learned defocus rendering (such as BokehMe) could be a direction for further improvement.
The stack size is fixed to 5 frames. In ZEDD evaluations, the model's performance approaches saturation with 9 inputs. Future work could explore adaptive stack sizes or variable-length stack inference strategies.

vs. Traditional DfD Methods (HybridDepth/DFV/DefocusNet): These approaches utilize hand-crafted network layers (differential focus volume, multi-stage alignment, variational solvers), which are effective within the training domain but lack zero-shot transferability. FOSSA replaces these with a ViT-based architecture and stack attention, leveraging large-scale pre-training to overcome this limitation.
vs. Monocular Depth Estimation (DepthPro/DepthAnythingv2): Monocular methods infer depth from a single all-in-focus image. Although possessing strong zero-shot capabilities, they struggle to obtain precise metric depth due to scale ambiguity. FOSSA resolves the scale ambiguity by leveraging multi-focus blur variations in the focus stack, showcasing a natural advantage in metric accuracy.
vs. ZEDD and DDFF: The acquisition quality of ZEDD (4K resolution, F/1.4 wide aperture, LiDAR ground truth) far exceeds DDFF (low resolution, light-field synthesized defocus almost invisible, structural light depth limited to 3.5m), providing a more challenging and practical benchmark for evaluating DfD models.

Rating¶

Novelty: ⭐⭐⭐⭐☆ FOSSA applies an elegant adaptation of standard ViT and stack attention to DfD rather than a hand-crafted network, representing an innovative concept; the ZEDD benchmark contribution is significant but leans towards engineering.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations were conducted on six datasets (including synthetic, real-world, and cross-domain). Ablation studies clearly separate the contributions of the architecture and data pipeline, and robustness analyses are thorough.
Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation (solid explanation of the intertwined nature of the two problems), detailed methodology with intuitive diagrams, and the appendix contains ample implementation details for reproducibility.
Value: ⭐⭐⭐⭐⭐ Concurrently provides model, benchmark, and data pipeline resources, practically driving the DfD field forward from "in-domain fitting" to "zero-shot generalization."