Scaling Dense Prediction with Latent Decoding

Conference: ECCV 2026
Paper: ECCV Official
Code: https://pointcept.org/l-dpt
Area: Model Compression
Keywords: dense prediction, latent decoding, scalable architecture, ultra-high resolution, lightweight decoder

TL;DR

Addressing the severe efficiency wall and visual degradation of conventional pyramidal pixel decoders at ultra-high resolutions, L-DPT decouples dense prediction modeling capacity from pixel rendering by performing attention-based reasoning entirely on a compact token grid in latent space, reading out to pixels via a lightweight PixelShuffle operator to achieve stable, detail-preserving inference on photographs up to 100 megapixels.

Background & Motivation

Modern dense predictors (such as DPT, Mask2Former, and U-Net) have long dominated monocular depth estimation, semantic segmentation, and surface normal estimation. Their foundational design follows a pyramidal pixel-decoding paradigm: low-resolution representations from visual encoders are progressively lifted through cascades of upsampling stages, multi-scale lateral connections, and pixel-space convolutions or attention operations to produce high-resolution output fields. However, as autonomous driving, embodied robotics, and professional computational photography demand dense perception at extreme resolutions, tightly coupling representation capacity directly with the output pixel grid leads modern systems directly into two severe bottlenecks.

The first is a system-level "efficiency wall": pyramidal decoders force the runtime engine to materialize and fuse massive pixel-aligned activation maps across multiple intermediate stages. Peak GPU memory escalates quadratically with image resolution, and latency suffers catastrophic regime shifts. For example, a standard DPT decoder on a frozen DINOv3 ViT-7B encoder immediately hits an out-of-memory (OOM) crash on 32-megapixel inputs even on a 192GB NVIDIA B200 GPU, while suffering multi-minute latencies. The second and even more critical barrier is an algorithmic "capability wall": simply feeding higher-resolution images into traditional pyramidal decoders fails to yield crisper geometric boundaries. Instead, repeated multi-stage upsamplings and spatial interpolation artifacts wash away fine structures such as leaves, thin branches, and slender utility poles. The system ends up paying an exorbitant computational tax only to produce a denser yet blurrier prediction.

The root cause of this failure is the entanglement between representation capacity and the physical pixel grid. Dense visual perception inherently demands global semantic reasoning and cross-layer geometric synthesis, which can be executed much more effectively inside a compact, low-dimensional latent space rather than repeatedly across tens of millions of raw spatial pixels. Core idea: decouple dense prediction into high-capacity reasoning in a compact latent space and lightweight rendering to pixels, concentrating all decoding computation off the pixel grid and delegating spatial resolution strictly to a minimal readout operator.

Method

Overall Architecture

L-DPT refactors dense prediction into a streamlined two-stage pipeline: "reasoning in latents" followed by "rendering to pixels." Given an input image \(x \in \mathbb{R}^{H \times W \times 3}\), a frozen Vision Transformer encoder (e.g., DINOv3 ViT-7B) partitions it into patches of size \(P \times P\), producing a token sequence of length \(N = \frac{H}{P} \times \frac{W}{P}\) and yielding intermediate feature representations \(\{F^{(s)}\}_{s=1}^S\) from \(S\) selected layers.

The entire decoding trajectory operates strictly on a single-scale latent state grid \(Z \in \mathbb{R}^{N \times D}\), initialized as all zeros \(Z^{(0)}\). Across decoding stages, a stage-wise adaptive selection module dynamically projects the concatenated multi-layer encoder feature pool into stage-specific condition vectors, which are injected into the latent stream via lightweight addition. The latent state is then iteratively refined through self-attention blocks equipped with 2D continuous-coordinate RoPE and feed-forward networks. Only after all latent reasoning stages are complete is the final latent state mapped to high-resolution output pixels via a single linear projection and a parameter-free PixelShuffle operation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image (H × W × 3)<br/>Ultra-high resolution photography"] --> B["Frozen ViT Encoder<br/>Extract S intermediate feature layers"]
    B --> C["Stage-wise Adaptive Selection<br/>Channel concatenation + linear projection"]
    C --> D["Minimal Latent Injection<br/>Lightweight additive conditioning onto Z"]
    D --> E["Continuous RoPE Attention<br/>Latent self-attention and FFN refinement"]
    E --> F["Lightweight Pixel Readout<br/>Linear projection + PixelShuffle"]
    F --> G["Dense Output Field (H × W × K)<br/>High-fidelity depth or segmentation map"]

Key Designs

1. Stage-wise Adaptive Selection: Decoupling channel compression from multi-level features

Conventional pyramidal decoders enforce rigid stage bindings, hard-wiring deep encoder features to coarse early stages and shallow encoder features to fine late stages. L-DPT removes these hand-crafted schedules entirely. It concatenates the intermediate features from all \(S\) selected encoder layers along the channel dimension into a unified global feature pool \(\bar{F} = [F^{(1)} \parallel F^{(2)} \parallel \cdots \parallel F^{(S)}] \in \mathbb{R}^{N \times \sum C_s}\). At decoding stage \(\ell\), a lightweight learnable linear projection \(W_{\text{sel}}^{(\ell)}\) adaptively extracts the task-specific feature mixture required for that specific abstraction level:

\[U^{(\ell)} = \bar{F} W_{\text{sel}}^{(\ell)}\]

This design shifts multi-scale feature integration from spatial pyramid resampling into purely channel-space selection. Every decoding stage accesses the complete encoder hierarchy without spatial distortion, allowing the network to freely combine fine texture cues and high-level semantics based on learned requirements rather than rigid structural heuristics.

2. Minimal Latent Injection: Lightweight additive conditioning

After selecting stage-specific conditioning features \(U^{(\ell)}\), integrating them into the latent state \(Z^{(\ell)}\) must avoid introducing heavy cross-attention mechanisms or dense spatial convolutions. L-DPT treats injection purely as an information exposure mechanism. In its default configuration, it uses simple element-wise addition:

\[Z^{(\ell)} \leftarrow Z^{(\ell)} + U^{(\ell)}\]

This operation requires strictly \(O(N \cdot D)\) compute, keeping the latent backbone uncluttered while providing sufficient conditioning capacity for accurate dense geometry and semantic synthesis. For heterogeneous outputs or multi-task settings, the framework seamlessly accommodates cross-attention or adaptive layer normalization (AdaLN) as drop-in alternatives.

3. Continuous RoPE Attention: Resolution extrapolation without pixel pyramids

Reasoning across a single-scale latent token grid requires the model to remain spatially robust under massive changes in input resolution. L-DPT incorporates 2D Rotary Position Embeddings (2D RoPE) using continuous coordinates normalized to \([0, 1]\) into all self-attention layers.

To enable robust zero-shot resolution extrapolation, the coordinates are strongly augmented during training with random coordinate scaling (sampled from \(\times 1/8\) to \(\times 8\)) and random coordinate jitter (intensity 1.2) before computing RoPE. This ensures that even when trained exclusively on fixed \(768 \times 768\) image crops, the attention geometry naturally generalizes to hundreds of thousands of tokens, enabling seamless inference on ultra-high-resolution images up to 100 megapixels without any structural modification or post-hoc tuning.

4. Lightweight Pixel Readout: Letting pixels be the readout

To prevent the output stage from devolving into another heavy pixel-space decoder, L-DPT restricts spatial upsampling strictly to the final readout layer. The renderer consists solely of a linear projection that maps latent channels to \(P^2 \times K\) (where \(P\) is the patch size and \(K\) is the output channel count), followed by standard PixelShuffle.

The operator contains zero learnable spatial filters and zero iterative refinement stages. Consequently, expanding image resolution introduces only trivial memory-view rearrangements rather than escalating decoder computation, fully realizing the principle that representation capacity lives in latents while pixels serve purely as a rendering surface.

Loss & Training

L-DPT follows a unified resolution-oriented training recipe across all dense prediction tasks. The pre-trained DINOv3 ViT-7B encoder remains completely frozen, isolating optimization exclusively to the lightweight L-DPT task head.

Training employs aggressive scale augmentation: each image is randomly resized with a scale factor sampled from \([0.5, 2.0]\) before being cropped to a fixed \(768 \times 768\) patch. Combined with the coordinate scaling (\(\times 1/8 \sim \times 8\)) and coordinate jitter (1.2) in RoPE, the model learns to handle massive sampling density variations under a constant training memory footprint. Relative depth estimation is trained with an affine-invariant scale-and-shift loss on synthetic datasets, while semantic segmentation uses standard cross-entropy loss with auxiliary supervision, optimized via AdamW.

Key Experimental Results

Main Results

Evaluation spans zero-shot relative depth estimation (trained solely on synthetic datasets and tested on real-world benchmarks) and semantic segmentation. All DINOv3 entries share the exact same frozen ViT-7B backbone, ensuring that performance differences directly reflect decoder architecture.

The table below shows zero-shot relative depth estimation across NYUv2, KITTI, ETH3D, and the challenging out-of-domain benchmark DIODE:

Method	Encoder	Decoder Params	Trainable Params	NYUv2 (ARel ↓ / \(\delta_1\) ↑)	KITTI (ARel ↓ / \(\delta_1\) ↑)	ETH3D (ARel ↓ / \(\delta_1\) ↑)	DIODE (ARel ↓ / \(\delta_1\) ↑)
MiDaS	87M	19M	106M	11.1 / 88.5	23.6 / 63.0	18.4 / 75.2	33.2 / 71.5
LeReS	89M	44M	133M	9.0 / 91.6	14.9 / 78.4	17.1 / 77.7	27.1 / 76.6
Marigold	-	2.3B	2.3B	5.5 / 96.4	9.9 / 91.6	6.5 / 96.0	30.8 / 77.3
DPT (Depth Anything V2)	1.1B	200M	1.3B	4.4 / 97.9	7.5 / 94.7	13.1 / 86.5	- / -
DPT (DINOv3 Baseline)	7B (frozen)	230M	230M	4.3 / 98.0	7.3 / 96.7	5.4 / 97.5	25.6 / 82.2
L-DPT S (Ours)	7B (frozen)	23M	23M	5.0 / 98.4	8.6 / 94.8	6.8 / 96.4	22.6 / 84.4
L-DPT B (Ours)	7B (frozen)	98M	98M	4.9 / 98.4	8.2 / 95.5	6.0 / 97.0	22.2 / 84.9
L-DPT L (Ours)	7B (frozen)	269M	269M	4.3 / 98.2	7.4 / 96.5	5.2 / 97.6	22.0 / 84.5

The table below reports semantic segmentation performance on COCO-Stuff 164k and PASCAL VOC 2012 in terms of mIoU under simple single-scale evaluation and lightweight horizontal flip test-time augmentation (TTA):

Method	Encoder	Decoder Params	Trainable Params	COCO-Stuff (Simple / TTA)	PASCAL VOC 2012 (Simple / TTA)
Previous Best	-	-	1.5B/0.5B	53.7 / 53.7	- / 90.0
DINOv3 Segmentor (Mask2Former)	7B (frozen)	927M	927M	53.8 / 54.0	90.1 / 90.4
L-DPT S (Ours)	7B (frozen)	23M	23M	53.5 / 53.6	93.4 / 93.6
L-DPT B (Ours)	7B (frozen)	98M	98M	53.8 / 54.0	93.8 / 93.8
L-DPT L (Ours)	7B (frozen)	269M	269M	54.0 / 54.2	94.0 / 94.1

Ablation Study

1. Feature injection and selection mechanism (Hypersim relative depth validation split)

Mechanism	ARel ↓	\(\delta_1\) ↑ (%)	Note
Fixed-Schedule Fusion	9.67	91.77	Simulates traditional progressive stage-bound pyramid fusion
Shared Selection	9.24	92.06	Reuses a single channel-projection matrix across all stages
Stage-wise Adaptive Selection	8.89	92.76	Independent learned projections per stage (Ours)

2. Ultra-high resolution efficiency comparison against DPT and continuous field InfiniDepth

Setup / Model	Input Resolution	Hardware	Peak Memory	Latency	Status / Behavior
DINOv3 + DPT	32MP	NVIDIA B200 (192G)	OOM	-	Memory explosion failure
DPT Decoder Sweep	63MP limit	Single GPU	59.4 GB	394.0 s	Severe latency escalation
InfiniDepth (DINOv3-L)	Arbitrary continuous field	NVIDIA H800	10.2 GB	3.38 s	Implicit neural coordinate query
L-DPT L (DINOv3-L)	Matched setting	NVIDIA H800	5.7 GB	1.74 s	44% less memory, ~2× faster
L-DPT Decoder Sweep	63MP limit	Single GPU	15.3 GB	5.3 s	Smooth linear scaling, stable runtime

Key Findings

• Superior out-of-domain transfer robustness: On the challenging DIODE benchmark, the lightweight L-DPT S (23M parameters) achieves 22.6 ARel and 84.4% \(\delta_1\), substantially surpassing the 230M-parameter DINOv3 DPT baseline (25.6 ARel, 82.2% \(\delta_1\)). This demonstrates that progressive pixel-space convolutions tend to overfit local domain priors, whereas latent-space attention captures more invariant geometric fundamentals.
• Necessity of stage-wise adaptive selection: Replacing fixed pyramid schedules with stage-wise adaptive selection cuts ARel by 0.78 on Hypersim (9.67 \(\rightarrow\) 8.89). Independent projection matrices consistently outperform a shared selector, verifying that different decoding depths benefit from distinct semantic mixtures.
• Matching heavy segmenters with 10× fewer parameters: On COCO-Stuff, L-DPT B (98M parameters) matches the 927M-parameter ViT-Adapter + Mask2Former baseline at 53.8 mIoU, while outperforming it by 3.7 mIoU on PASCAL VOC 2012 (93.8 vs. 90.1), demonstrating that heavy pixel-space multi-scale heads are unnecessary when latent decoding is used.

Highlights & Insights

• Paradigm shift from pixel pyramids to latent reasoning: "Reason in latents, render to pixels." By relocating all heavy representational modeling to compact token grids, image resolution transforms into a lightweight rendering choice rather than a decoder cost multiplier.
• Robust zero-shot resolution extrapolation: Pairing continuous-coordinate 2D RoPE with aggressive scale augmentation during training allows a model trained solely on \(768 \times 768\) inputs to evaluate seamlessly at 100 megapixels, preserving intricate structures such as leaves and branches without visual smoothing.
• A general blueprint for scalable vision decoders: L-DPT establishes that a single-scale latent attention stream coupled with minimal linear transformations can outperform complex feature pyramids, providing an efficient foundation for high-resolution perception across robotic and photographic applications.

Limitations & Future Work

• Quadratic scaling of dense attention at extreme token lengths: While L-DPT compresses spatial dimension to ViT patch tokens (\(P=14\) or \(P=16\)), a 100MP image still yields approximately 400,000 tokens. Full self-attention across such lengths incurs high computational demand, motivating the integration of sparse queries or linear attention mechanisms in future work.
• Absence of native full-resolution end-to-end training: The current model relies on synthetic coordinate jitter to simulate resolution shifts. Exploring native ultra-high-resolution training through resolution-aware dynamic batching remains an open research avenue.
• Deterministic output versus generative uncertainty modeling: L-DPT currently generates deterministic outputs. Incorporating latent diffusion or flow-matching formulations within the compact latent grid could enable probabilistic modeling of depth uncertainty and complex occlusions.

Related Work & Insights

• vs DPT (Vision Transformers for Dense Prediction): DPT reassembles ViT tokens into multi-scale pyramids (\(\frac{H}{8}, \frac{H}{4}, \frac{H}{2}\)) and fuses them via progressive convolutional blocks. L-DPT eliminates multi-scale spatial pyramids entirely, reducing memory by over 70% and preventing the high-frequency blurring caused by repetitive upsampling.
• vs Mask2Former / ViT-Adapter: These heavy segmentation architectures rely on multi-scale pixel decoders and dense cross-attention mask queries with up to 1B parameters. L-DPT achieves superior segmentation accuracy using one-tenth of the parameter count through adaptive channel selection and compact self-attention.
• vs InfiniDepth: InfiniDepth relies on implicit neural representations to query depth continuously at arbitrary coordinates. L-DPT targets the decoder-side computation bottleneck directly, achieving lower peak memory (5.7GB vs. 10.2GB) and faster latency (1.74s vs. 3.38s) with matching or better accuracy on NYUv2 and DIODE.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Decouples dense prediction from pixel-grid pyramids, establishing a scalable "reason in latents, render to pixels" paradigm.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-domain validation across relative depth and semantic segmentation benchmarks, accompanied by rigorous resolution scaling benchmarks up to 100MP.
• Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear framing of the "efficiency wall" and "capability wall", supported by consistent mathematical formulations and clean empirical comparisons.
• Value: ⭐⭐⭐⭐⭐ Delivers an efficient, high-performance architectural standard for ultra-high-resolution dense visual perception in edge and server deployments.