Last-Layer-Centric Feature Recombination: Unleashing 3D Geometric Knowledge in DINOv3 for Monocular Depth Estimation¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/book-book24/LFR
Area: 3D Vision
Keywords: monocular depth estimation, DINOv3, last-layer-centric, feature recombination, minimal-similarity selection

TL;DR¶

Addressing the flaw in conventional monocular depth estimation pipelines that uniformly sample ViT intermediate layers and ignore geometric heterogeneity, this paper reveals that deeper layers in DINOv3 possess stronger 3D predictive capacity, and proposes a Last-Layer-Centric Feature Recombination (LFR) module that adaptively fuses complementary intermediate layers based on minimal-similarity selection to achieve state-of-the-art results across standard and zero-shot benchmarks.

Background & Motivation¶

Monocular depth estimation (MDE) aims to infer pixel-wise metric depth from a single RGB image, serving as a cornerstone for autonomous driving, robotics, augmented reality, and 3D reconstruction. Because a 2D image physically maps to infinitely many 3D spatial configurations, MDE is inherently ill-posed. Early methods sought to resolve this geometric ambiguity by introducing handcrafted priors—such as piecewise planarity, local surface normal consistency, or depth continuity constraints. However, these human-engineered priors struggle in complex, unconstrained physical environments and restrict the model's expressive capacity. Recently, the advent of large-scale vision foundation models (VFMs), particularly self-supervised vision transformers such as the DINO family, has drastically shifted the field, serving as the core feature extractor in prevailing models like Depth Anything and VGGT.

Despite impressive progress, current architectures bridging ViT backbones and dense prediction decoders follow a rigid, empirical recipe: they uniformly sample intermediate layers (e.g., layers 6, 12, 18, and 24 in a 24-layer ViT-L) to construct multi-scale feature pyramids inside a DPT-style decoder. This common practice implicitly presumes that 3D geometric cues are homogeneously distributed across all transformer layers. Yet, because ViTs lack convolutional inductive biases, feature representations shift drastically across depth. Rigidly sampling intermediate layers without understanding their internal representations introduces substantial semantic redundancy and can dilute the critical 3D geometric signals concentrated near the top of the network.

Through a rigorous layer-wise statistical audit on DINOv3-L (7B parameters pre-trained on 17B images), the authors show that middle-to-deep layers tend to collapse toward a shared representation across samples, while deeper layers exhibit strictly higher linear depth predictability (\(R^2\)) and capture inter-sample 3D geometric variations far more effectively. The last layer alone possesses dense, fine-grained geometric localization power. The core tension is that while intermediate layers can contribute multi-scale contextual diversity, naive multi-layer aggregation disrupts the superior geometric anchoring of the deepest layers. The core idea is to treat the last-layer representations of DINOv3 as the dominant geometric anchor, adaptively select complementary intermediate layers via a minimal-similarity criterion, and recombine them through compact gated residual adapters, unleashing the foundation model's 3D geometric potential with minimal overhead.

Method¶

Overall Architecture¶

The end-to-end framework consists of four sequential stages: backbone token extraction, adaptive auxiliary layer selection, last-layer-centric feature recombination (LFR), and confidence-weighted multi-level depth prediction. Given an RGB input image \(I \in \mathbb{R}^{H \times W \times 3}\), a patch projection layer produces a sequence of patch tokens \(T_0 \in \mathbb{R}^{N \times C}\) where \(N = \frac{H}{16} \times \frac{W}{16}\), which are subsequently processed across \(L\) transformer layers. Operating as a plug-and-play intermediary between the backbone and the DPT decoder, the LFR module establishes the final layer \((F_L, \text{Cls}_L)\) as the primary anchor, selects \(K=4\) preceding auxiliary layers with minimal cosine similarity, fuses them via lightweight bottleneck adapters with learnable gated residuals to form \(K\) recomposed representations, and aggregates per-level DPT depth predictions weighted by the recomposed class tokens.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Input RGB Image<br/>H × W × 3"] --> B["DINOv3 Backbone Encoding<br/>Extract L Layers of Patch & Class Tokens"]
    B --> C["Minimal-Similarity Feature Selection<br/>Select K Complementary Layers w.r.t. Anchor"]
    C --> D["Last-Layer-Centric Feature Recombination<br/>Channel Concat + Bottleneck Adapter + tanh Gate"]
    D --> E["Multi-Level DPT Decoder Regression<br/>Independent Recomposed Feature Pyramids"]
    E --> F["Confidence Adaptive Weighted Aggregation<br/>Predict Level Weights from Recomposed Cls Tokens"]
    F --> G["Final High-Accuracy Depth Map"]

Key Designs¶

1. Minimal-similarity feature selection: maximizing knowledge complementarity while cutting layer redundancy

To resolve the feature redundancy caused by uniform layer sampling, this design anchors the selection directly on the final layer. Let \(F_L \in \mathbb{R}^{N \times C}\) and \(\text{Cls}_L \in \mathbb{R}^{1 \times C}\) denote the patch and class tokens of layer \(L\). The algorithm computes the average cosine similarity between \(\text{Cls}_L\) and the patch tokens of each preceding layer \(l \in \{1, \dots, L-1\}\). Rather than picking the most similar layers, it strictly selects the \(K\) layers with the lowest similarity scores (\(K=4\)), denoting their representations as \(\{F_{ak}, \text{Cls}_{ak}\}_{k=1}^K\). Empirical distributions reveal that these selected auxiliary layers predominantly fall into layers 17–22. This strategy effectively pairs the individualized geometric representations of the final layer with the global shared representations of the middle-to-deep layers, maximizing semantic complementarity and avoiding feature dilution.

2. Bottleneck residual gated adapter: last-layer-dominant feature recombination

To integrate auxiliary information without overriding the strong geometric localization of the final layer, the recombination module uses a last-layer-dominant residual formulation. For each selected auxiliary layer \(a_k\), the auxiliary features and the anchor features are concatenated along the channel dimension, preserving their native representation spaces. The concatenated vector is passed into a compact bottleneck adapter \(A^k\), which comprises a linear down-projection, a non-linear activation, and a linear up-projection. A learnable scalar parameter \(g^k\), constrained to \([-1, 1]\) via a \(\tanh\) activation, dynamically modulates the injection magnitude, before adding the result residually back to the anchor features:

\[F_{rk} = \tanh(g^k) \cdot A^k([F_{ak}; F_L]) + F_L\]

\[\text{Cls}_{rk} = \tanh(g^k) \cdot A^k([\text{Cls}_{ak}; \text{Cls}_L]) + \text{Cls}_L\]

This ensures that the recomposed features \(\{F_{rk}, \text{Cls}_{rk}\}_{k=1}^K\) remain strictly anchored to the last layer while absorbing targeted contextual refinements from complementary intermediate representations.

3. Confidence-driven multi-level weighted prediction: dynamically aggregating multi-scale branches

Instead of compressing all multi-scale features into a single fused pyramid as done in standard DPT decoders, the architecture constructs \(K\) independent feature pyramids from the recomposed features \(\{F_{rk}\}_{k=1}^K\) and performs per-level depth regression to produce \(K\) candidate depth maps \(\{\widetilde{D}_k\}_{k=1}^K\). To determine the global reliability of each branch, the recomposed class tokens \(\text{Cls}_{rk}\) are projected through a linear layer and normalized via Softmax to obtain level-wise confidence weights:

\[w^k = \frac{\exp(\text{Linear}(\text{Cls}_{rk}))}{\sum_{j=1}^K \exp(\text{Linear}(\text{Cls}_{rj}))}\]

The final depth prediction is the weighted sum \(\widetilde{D} = \sum_{k=1}^K w^k \cdot \widetilde{D}_k\). This enables the network to adaptively shift trust across different complementary branches depending on scene distance and structural layout.

Loss & Training¶

The model is supervised end-to-end using two complementary loss functions. The first is the scale-invariant logarithmic loss (\(\mathcal{L}_{s\text{log}}\)):

\[\mathcal{L}_{s\text{log}} = \sqrt{\frac{1}{N}\sum_{n=1}^N g_n^2 - \frac{\lambda}{N^2}\left(\sum_{n=1}^N g_n\right)^2}\]

where \(g_n = \log \widetilde{d}_n - \log d_n\) is the log-depth residual at pixel \(n\), \(N\) is the number of valid pixels, and \(\lambda = 0.85\). The second is the hierarchical normalization loss (\(\mathcal{L}_{HN}\)), which normalizes predicted and ground-truth depth within multi-scale local patches \(M \in \{1, 4, 8\}\) to enforce both broad topological consistency and crisp local boundaries:

\[\mathcal{L}_{HN} = \frac{1}{N}\sum_{n=1}^N \frac{1}{|U_n|}\sum_{u \in U_n} \left| \mathcal{N}_u(d_n) - \mathcal{N}_u(\widetilde{d}_n) \right|\]

The total training objective is \(\mathcal{L} = \mathcal{L}_{s\text{log}} + \mathcal{L}_{HN}\). Training is conducted on 4 NVIDIA A100 GPUs using AdamW for 30 epochs with batch size 8. The backbone learning rate is set to \(10^{-5}\) and the head/adapter learning rate to \(10^{-4}\) with a 10% linear warmup followed by cosine annealing. The backbone is frozen for the final 10 epochs. A single forward pass on a \(640 \times 480\) image requires approximately 680 GFLOPs.

Key Experimental Results¶

Main Results¶

The proposed method was benchmarked on indoor NYU Depth v2 (0–10m range) and outdoor KITTI (Eigen split, 0–80m range) against leading discriminative, generative, and data-driven methods:

Dataset	Method	Param	AbsRel ↓	RMSE ↓	SqRel ↓	\(\delta_1\) ↑
NYU Depth v2	DPT (ViT-Hybrid)	343M	0.110	0.357	-	0.904
NYU Depth v2	NeWCRFs	270M	0.095	0.334	0.045	0.922
NYU Depth v2	VPD (Diffusion)	600M	0.069	0.254	0.027	0.964
NYU Depth v2	DAR-L (Autoregressive)	2.0B	0.056	0.205	0.011	0.982
NYU Depth v2	Depth Anything v2	343M	0.056	0.206	-	0.984
NYU Depth v2	DINOv3-L (Baseline)	342M	0.060	0.212	0.026	0.981
NYU Depth v2	DINOv3-L + LFR (Ours)	350M	0.057	0.206	0.017	0.984
KITTI	NeWCRFs	270M	0.052	2.129	0.155	0.974
KITTI	VPD (Diffusion)	207M	0.050	2.072	0.148	0.975
KITTI	DAR-L (Autoregressive)	2.0B	0.044	1.799	0.110	0.986
KITTI	Depth Anything v2	343M	0.045	1.861	-	0.983
KITTI	DINOv3-L (Baseline)	342M	0.046	1.794	0.107	0.986
KITTI	DINOv3-L + LFR (Ours)	350M	0.043	1.711	0.099	0.988

Ablation Study¶

A component-wise ablation on the KITTI benchmark tracks the contribution of each architectural element starting from the baseline DPT with uniform sampling (Table 5 in original paper):

Config	Last-layer Anchor	LFR Recombination	HN Loss	Multi-level Weighting	AbsRel ↓	SqRel ↓	RMSE ↓	\(\delta_1\) ↑
Baseline (uniform sampling)	\(\times\)	\(\times\)	\(\times\)	\(\times\)	0.046	0.107	1.794	0.986
+ Last-layer only (replicated 4x)	\(\checkmark\)	\(\times\)	\(\times\)	\(\times\)	0.045	0.105	1.743	0.987
+ LFR Recombination adapter	\(\checkmark\)	\(\checkmark\)	\(\times\)	\(\times\)	0.043	0.099	1.721	0.987
+ HN Loss supervision	\(\checkmark\)	\(\checkmark\)	\(\checkmark\)	\(\times\)	0.043	0.099	1.719	0.987
+ Multi-level weighted prediction (Full)	\(\checkmark\)	\(\checkmark\)	\(\checkmark\)	\(\checkmark\)	0.043	0.099	1.711	0.988

Comparison of auxiliary layer selection strategies on KITTI (Table 6 in original paper):

Strategy	Mechanism	AbsRel ↓	SqRel ↓	RMSE ↓	\(\delta_1\) ↑
Scores	Score intermediate layers via an auxiliary projection on class tokens	0.044	0.101	1.738	0.987
Node degree	Select layers with smallest average similarity between class and patch tokens	0.044	0.100	1.721	0.987
Maximal-similarity	Select layers with highest average similarity to the last-layer class token	0.044	0.100	1.723	0.987
Minimal-similarity (Ours)	Select layers with lowest average similarity to last-layer class token	0.043	0.099	1.711	0.988

Key Findings¶

Crucial role of the final layer as geometric anchor: Simply repeating the final layer 4 times into DPT drops KITTI RMSE from 1.794 to 1.743, empirically validating that the deepest features of DINOv3 encode sufficient multi-scale geometric structure, while naive intermediate layer sampling dilutes this signal.
Superiority of complementarity over similarity: Selecting auxiliary layers by minimal similarity outperforms maximal similarity (RMSE 1.711 vs. 1.723), proving that intermediate layers benefit depth estimation when they provide diverse, non-redundant contextual information rather than duplicating last-layer representations.
Superior zero-shot cross-dataset generalization: In indoor transfer from NYUv2 to SUN RGB-D without fine-tuning, the proposed method achieves AbsRel 0.099 and \(\delta_1\) 0.907, outperforming Depth Anything v1 (0.119 / 0.864) and DAR-L (0.112 / 0.885). In outdoor transfer from KITTI to Argoverse, AbsRel reaches 0.204 and \(\delta_1\) reaches 0.730, surpassing vanilla DINOv3-L (0.219 / 0.677).

Highlights & Insights¶

From empirical heuristic to representation-driven architecture: Moving away from the long-standing convention of uniform layer sampling in ViT-based dense prediction, this paper quantitatively demonstrates through representational dissimilarity matrices and linear regression probing that 3D geometric information heavily concentrates in the deepest layers of self-supervised vision foundation models.
Last-layer-dominant residual fusion: Adding only 8M parameters, the bottleneck adapter preserves last-layer geometric fidelity by treating intermediate features as a gated residual perturbation, preventing early-layer noise from corrupting precise boundary cues.
Single-forward efficiency versus generative complexity: Matching or exceeding 2B-parameter autoregressive models (DAR-L) and diffusion approaches (VPD, ECoDepth), LFR retains the speed of standard deterministic forward passes (680 GFLOPs at \(640 \times 480\)), offering high practical utility.

Limitations & Future Work¶

Backbone fine-tuning computational requirements: While the LFR adapter adds minimal parameters, the backbone is unfreezed during the first 20 epochs. Exploring parameter-efficient fine-tuning (PEFT) to keep the 7B/300M+ ViT entirely frozen would improve training accessibility.
Exploration in broader 3D tasks: The current study evaluates only monocular metric depth estimation. Extending the last-layer-centric recombination paradigm to surface normal estimation, 3D object detection, and feed-forward Gaussian Splatting remains an open direction.
Indoor vs. outdoor layer selection dynamics: Auxiliary layers selected by minimal similarity are tightly concentrated in outdoor driving scenes (layers 20–22 on KITTI), but exhibit broader variance in indoor environments (layers 18–21 on NYUv2). Further theoretical modeling is needed to understand how spatial complexity governs layer-wise visual features.

vs DPT (Vision Transformers for Dense Prediction): DPT relies on uniform intermediate layer sampling to mimic CNN feature pyramids; this work shows that such uniform assumptions fail for self-supervised ViTs, and that anchoring on the final layer while selecting complementary layers eliminates redundancy.
vs Depth Anything (v1/v2): Depth Anything relies on massive pseudo-labeling (62M+ unlabeled images) and complex student-teacher distillation; this work shows that analyzing and recombining the internal layer dynamics of DINOv3 yields state-of-the-art transfer without massive external data pipelines.
vs DAR / VPD (Generative and Autoregressive MDE): Generative methods treat depth estimation as multi-step diffusion or progressive scale autoregression requiring multi-billion parameter footprints; this work achieves superior accuracy within a single deterministic forward pass.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Uncovers the non-uniform distribution of 3D geometry across DINOv3 layers and proposes an elegant minimal-similarity last-layer recombination module.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation on NYUv2 and KITTI, extensive layer-selection ablations, attention maps, and zero-shot evaluations on SUN RGB-D and Argoverse.
Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative driven by statistical probing, clean mathematical formulas, and intuitive visual explanations.
Value: ⭐⭐⭐⭐⭐ Provides practical guidance for transferring large-scale visual foundation models to downstream 3D dense prediction tasks with minimal overhead.