Efficient and High-Quality Depth Estimation via Pixel-Space Diffusion with Linear Attention

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/VISION-SJTU/Lapis
Area: 3D Vision
Keywords: monocular depth estimation, diffusion models, linear attention, pixel-space diffusion, high-resolution perception

TL;DR

Addressing the prohibitive quadratic complexity and multi-step sampling latency in high-resolution generative monocular depth estimation, Lapis presents the first linear-attention pixel-space diffusion framework with single-step x-prediction, bridging the fidelity gap via patch-level consistency and pixel-level refinement modules to deliver state-of-the-art accuracy, sharp boundary recovery, and up to 10.9x speedups.

Background & Motivation

Monocular depth estimation (MDE) serves as an essential pillar of 3D computer vision, underpinning core downstream systems across autonomous navigation, robotics manipulation, and 3D/4D neural reconstruction. Seeking fine-grained geometric fidelity, the research frontier has steadily shifted from discriminative regression models toward generative diffusion frameworks. While discriminative methods tend to produce over-smoothed depth surfaces that blur intricate details, generative approaches excel at recovering complex, high-frequency structures. Nevertheless, as real-world applications scale toward high resolutions to resolve delicate geometric boundaries, current generative frameworks suffer from an acute trade-off between fidelity and computational efficiency.

Most existing generative frameworks adapt pre-trained latent diffusion models (LDMs) built on U-Net backbones, which are intrinsically bottlenecked by the irreversible information loss of VAE downsampling and struggle to preserve long-range coherence across high-resolution scenes. Conversely, recent works that turn to pixel-space Diffusion Transformers (DiTs) successfully avoid the latent bottleneck, yet their standard multi-head self-attention scales quadratically as \(O(N^2)\), incurring prohibitive latency at high resolutions. Coupled with the iterative multi-step sampling schedule, existing pixel-space models become unusable for real-time deployment. Although replacing full self-attention with linear attention and deploying single-step sampling appear conceptually straightforward, naive linearization triggers severe patch-level layout misalignment, pixel-level detail dilution, and acute noise artifacts under standard velocity-prediction trajectories.

To overcome these intertwined bottlenecks, the authors investigate how to achieve high-fidelity single-pass depth reconstruction within linear time complexity. Core idea: propose Lapis, the first linear-attention pixel-space generative depth framework, combining single-step direct x-prediction on the clean data manifold with a coarse-to-fine hierarchy of semantic patch-level consistency and pixel-level skip refinement.

Method

Overall Architecture

Lapis operates directly in pixel-space without any VAE compression. Given a conditioning RGB image \(c\) and a noisy depth map \(x_t\), the model predicts the clean depth map in a single forward pass. The underlying architecture builds upon a linearized Diffusion Transformer (DiT) backbone, substituting standard Softmax self-attention with linear attention kernels to scale computational complexity at \(O(Nd^2)\) instead of \(O(N^2d)\). To eliminate the fidelity gap induced by linear low-rank approximations, Lapis incorporates a coarse-to-fine dual-scale design: a Patch-level Consistency Module (PCM) operating across token representations, followed by a Pixel-level Refinement Module (PRM) operating at the native image resolution.

The end-to-end forward flow begins with the concatenated noisy depth and conditioning RGB input. Macro-level semantic descriptors from a frozen vision encoder guide the global layout within the linearized DiT blocks, local depth-wise convolutions enforce inter-patch continuity, and long-range skip connections feed high-frequency spatial cues directly into the pixel-level decoder to reconstruct sharp boundaries.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input RGB and single-step noisy depth"] --> B["Single-step pixel-space diffusion<br/>Direct x-prediction on clean manifold"]
    B --> C["Patch-level Consistency Module<br/>DINOv2 semantic prior + ConvFFN bias"]
    C --> D["Linearized Transformer backbone<br/>Non-negative kernel linear attention"]
    D --> E["Pixel-level Refinement Module<br/>Long-range skip connection + PixelShuffle"]
    E --> F["High-fidelity continuous depth output"]

Key Designs

1. Single-step pixel-space diffusion: direct manifold x-prediction Multi-step iterative denoising creates massive computational overhead, while latent-space x-prediction is hindered by the isotropic Gaussian regularization of VAEs that distorts the low-dimensional manifold of natural 3D geometry. Grounded in the low-dimensional manifold hypothesis and flow-matching mechanics, Lapis abandons volatile single-step velocity-space prediction (v-prediction) in favor of direct clean data prediction (x-prediction). Given the linear flow-matching schedule \(x_t = (1-t)x_0 + t\varepsilon\) with \(t \in [\tau, 1]\), the network \(x_\theta(x_t, t, c)\) directly targets the clean depth \(x_0\). Because clean depth maps reside on a structured continuous manifold whereas noise artifacts do not, direct x-prediction stabilizes single-pass generation and effectively suppresses pixel-space sampling noise.

2. Patch-level Consistency Module: global semantic alignment and local continuity Standard self-attention relies on the exponential Softmax function for sharp attention focalization across long sequences. In contrast, linear attention computes similarity via a non-negative feature map \(\phi(x) = \text{elu}(x)+1\) and exploits matrix multiplication associativity:

\[\text{Attn}_{\text{Linear}}(Q_i, K, V) = \frac{\phi(Q_i) \sum_{j=1}^N \phi(K_j)^T V_j}{\phi(Q_i) \sum_{j=1}^N \phi(K_j)^T}\]

This linear approximation sacrifices sharp focalization, leaving the network vulnerable to patch-level structural collapse. To anchor the macro-level geometry, the global PCM extracts high-level semantic descriptors from a frozen DINOv2 encoder, refines them with a channel-gating mechanism, and integrates them with the diffusion timestep embedding via an extended AdaLN-Zero scheme to modulate both the linear attention and feed-forward layers. Concurrently, to eliminate localized depth discontinuities and grid artifacts between adjacent patches, the local PCM integrates a \(3 \times 3\) residual depth-wise convolution into the feed-forward layer (termed ConvFFN), injecting strong local inductive bias that guarantees smooth spatial transitions.

3. Pixel-level Refinement Module: long-range skip correspondence and pixel decoding Deep cascades of linear attention induce an iterative averaging effect that progressively washes out high-frequency structural boundaries. To recover delicate geometry when mapping discretized tokens back to pixel-space, the PRM introduces a long-range skip connection that carries the raw concatenated input with learnable patch-wise positional embeddings into a high-resolution feature tensor \(y' \in \mathbb{R}^{D_{\text{pixel}} \times H \times W}\). In parallel, the latent tokens output by the DiT backbone are reshaped and upsampled to the original spatial resolution via PixelShuffle to produce \(s_{\text{pixel}}\). A cascade of Pixel Refiner blocks, modulated by \(s_{\text{pixel}}\) through AdaLN-Zero ConvFFN layers, progressively fuses structural guidance with the skip features, followed by a linear projection head that reconstructs razor-sharp depth boundaries.

Loss & Training

Lapis is trained end-to-end via a composite objective combining velocity-based flow supervision and multi-scale geometric gradient matching:

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{velocity}} + \lambda_{\text{grad}} \mathcal{L}_{\text{grad}}\]

The velocity loss \(\mathcal{L}_{\text{velocity}}\) computes the Mean Squared Error between the predicted velocity \(\hat{v}_t = (x_t - \hat{x}_0) / t\) and the ground-truth velocity \(v_t = \varepsilon - x_0\). To penalize blurred occlusions, the multi-scale gradient matching loss computes spatial depth gradients across different resolution scales \(S\):

\[\mathcal{L}_{\text{grad}} = \mathbb{E}_{x_0, \varepsilon, t} \left[ \sum_{s \in S} w_s \left\lVert \nabla_s x_0 - \nabla_s \hat{x}_0 \right\rVert \right]\]

The balancing weight is set to \(\lambda_{\text{grad}} = 1.5\). The 520M-parameter model (following DiT-Large configurations) is trained for 400K steps on 4 NVIDIA RTX Pro 6000 Blackwell GPUs using the AdamW optimizer with a constant learning rate of \(1 \times 10^{-4}\) and bfloat16 mixed precision. Exponential Moving Average (EMA) with a decay of 0.9999 is maintained for all evaluations.

Key Experimental Results

Main Results

Evaluated across five standard unseen real-world benchmarks (NYUv2, KITTI, ETH3D, ScanNet, DIODE) under native resolutions, Lapis achieves state-of-the-art zero-shot accuracy, surpassing prominent discriminative and generative baselines (Table 1 summary):

Dataset	Metric	Lapis (Ours)	Pixel-Perfect Depth (Prev. SOTA)	Depth Anything V2	MoGe-2
NYUv2	AbsRel↓ / \(\delta_1\)↑	3.8 / 98.2	3.7 / 98.1	4.5 / 97.9	3.5 / 98.0
KITTI	AbsRel↓ / \(\delta_1\)↑	5.1 / 97.7	5.4 / 97.1	7.9 / 93.6	5.9 / 97.1
ETH3D	AbsRel↓ / \(\delta_1\)↑	3.2 / 99.2	3.1 / 99.2	7.4 / 97.7	4.5 / 96.7
ScanNet	AbsRel↓ / \(\delta_1\)↑	3.8 / 98.2	3.8 / 98.2	4.2 / 97.8	3.4 / 98.3
DIODE	AbsRel↓ / \(\delta_1\)↑	4.8 / 97.9	4.9 / 97.5	10.0 / 94.9	6.7 / 95.5
Overall Average	AbsRel↓ / \(\delta_1\)↑	4.1 / 98.2	4.2 / 98.0	6.8 / 96.4	4.8 / 97.1

Under test-time resolution scaling at 1080P and 1440P (Table 2) along with runtime measurements on an NVIDIA RTX Pro 6000 Blackwell GPU (Table 4), Lapis achieves breakthrough speedups while maintaining superior accuracy:

Test Resolution	Metric	Lapis (Ours)	Pixel-Perfect Depth	Marigold v1.1	MoGe-2
1080P Average	AbsRel↓ / \(\delta_1\)↑	5.1 / 97.1	5.7 / 97.1	13.0 / 83.8	6.3 / 95.3
1440P Average	AbsRel↓ / \(\delta_1\)↑	5.2 / 96.9	6.5 / 96.6	18.3 / 76.2	6.4 / 95.1
1080P Runtime	Latency↓ / Peak Mem↓	119.0 ms / 3.6 GB	905.5 ms / 3.5 GB	758.1 ms / 11.7 GB	115.3 ms / 1.7 GB
1440P Runtime	Latency↓ / Peak Mem↓	224.7 ms / 3.9 GB	2457.0 ms / 3.8 GB	1479.0 ms / 16.9 GB	245.5 ms / 2.1 GB

Ablation Study

Component-wise ablation under the lightweight DiT-Base configuration across five benchmark datasets (Table 5):

Config	Mean AbsRel↓	Mean \(\delta_1\)↑	Note
Full Model	4.8	97.7	full model with coherent layout and sharp thin structures
w/o Global PCM	19.8	70.1	Disabling DINOv2 semantic priors destroys global geometry layout
w/o Local PCM	5.1	97.5	Removing ConvFFN causes prominent patch grid discontinuities
w/o PRM	4.9	97.6	Boundary sharpness drops noticeably due to signal dilution

Key Findings

• Global PCM anchors linearized attention: Without Global PCM guidance, mean AbsRel collapses drastically from 4.8 to 19.8. This proves that linear attention cannot independently coordinate macro-level scene geometry due to its low-rank kernel approximation, necessitating pre-trained semantic priors for spatial grounding.
• Order-of-magnitude high-resolution acceleration: At 1440P resolution, Lapis cuts inference latency from 2457.0 ms (Pixel-Perfect Depth) to 224.7 ms (a 10.9x speedup) with comparable peak memory (3.9 GB vs 3.8 GB), matching the throughput of pure regression architectures while achieving a higher boundary F1-score (27.0 vs 22.3).
• Single-step direct x-prediction superiority: Comparing multi-step sampling and v-prediction schedules confirms that targeting clean data via x-prediction in a single step yields optimal efficiency and clean spatial coherence, validating the low-dimensional manifold hypothesis for dense continuous geometry.

Highlights & Insights

• Decoupled macro-layout and micro-detail modeling: Delegating global context to semantically conditioned linear DiT blocks while delegating high-frequency edge restoration to skip-connected pixel refiners elegantly reconciles linear scalability with geometric sharpness.
• Dissolving the generative vs. discriminative dichotomy: By matching discriminative inference latency (119 ms at 1080P) while preserving the superior boundary fidelity characteristic of generative frameworks, Lapis sets a new paradigm for high-resolution visual perception.
• Generalizability of single-step manifold projection: The success of direct x-prediction on pixel-space continuous manifolds offers clear design blueprints for other dense prediction tasks such as surface normal estimation, optical flow, and 3D Gaussian Splatting initialization.

Limitations & Future Work

• Dependency on semantic encoder robustness: Global consistency relies heavily on pre-trained DINOv2 representations; extreme visual degradations such as dense fog, severe motion blur, or lens glare could mislead the global depth layout.
• Synthetic training domain shift: Training exclusively on 122K synthetic pairs (Hypersim, TartanAir, VKITTI2, etc.) leaves room for further adaptation in challenging optical domains containing complex specular highlights, mirrors, and transparent surfaces.
• Future directions: Integrating lightweight end-to-end learnable semantic backbones and extending the framework to temporal multi-view video depth and dynamic 4D reconstruction.

Related Work & Insights

• vs. Pixel-Perfect Depth: Both use pixel-space DiT to prevent VAE information loss, but Pixel-Perfect Depth requires quadratic attention and multi-step sampling, taking nearly 2.5 seconds at 1440P. Lapis achieves a 10.9x speedup via linear attention and single-step x-prediction while retaining sharper boundaries.
• vs. Depth Anything V2 / MoGe-2: Discriminative regression baselines achieve high inference speeds but suffer from over-smoothed boundaries around thin structures like power lines and railings. Lapis delivers comparable latency while substantially improving boundary sharpness F1-scores (27.2 vs 23.7 at 1080P).
• vs. Marigold / Lotus-2: Latent-space diffusion models fine-tune image generators but remain hindered by VAE bottleneck artifacts and multi-step sampling latency. Lapis demonstrates that native pixel-space diffusion with appropriate structural rectification is both faster and geometrically more precise.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First framework to successfully unite linear-attention DiT with single-step pixel-space diffusion for monocular depth estimation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 12 real-world and synthetic benchmarks measuring accuracy, boundary F1, latency, and memory scaling.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous motivation, coherent coarse-to-fine hierarchy, and clear architectural diagrams.
• Value: ⭐⭐⭐⭐⭐ Resolves the computational bottleneck of high-resolution generative depth perception with immediate practical utility for real-time 3D vision.