Geometric Distillation from Rectified Stereo: Leveraging Epipolar Cues for Monocular Depth Estimation

Conference: ECCV 2026
Paper: ECCV Official
Code: None
Area: 3D Vision
Keywords: Epipolar Geometry, Knowledge Distillation, Monocular Depth Estimation, Stereo Vision, Scale Ambiguity

TL;DR

To prevent multi-view foundation models from suffering architectural attention collapse and losing scale priors during single-view inference, EpiDistill transfers multi-view epipolar geometry and metric scale priors into a monocular Transformer using depth-guided epipolar attention and rectified stereo tokens, significantly advancing zero-shot metric depth estimation.

Background & Motivation

Monocular depth foundation models, pretrained on massive and diverse datasets, have exhibited remarkable zero-shot generalization in predicting relative scene geometry. However, accurately estimating absolute metric depth in unconstrained real-world environments remains a persistent fundamental challenge. Due to the inherent projective scale ambiguity of single-image inference, monocular networks can faithfully predict relative spatial variations but frequently suffer from severe scale drift and magnitude misalignments. While early techniques relied on ground-truth camera intrinsics to disentangle focal length from physical depth, and recent approaches have explored implicitly regressing intrinsics directly from 2D appearance features, these single-view estimators remain fragile when faced with extreme domain shifts and varying fields of view.

Concurrently, multi-view foundation models have demonstrated superior capabilities by processing image sequences through cross-view global attention, successfully establishing dense geometric correspondences and inferring globally consistent metric scales in a single feed-forward pass. Nevertheless, a critical architectural breakdown occurs when these multi-view models are constrained to single-view inference: lacking auxiliary observations, their cross-view global attention degenerates into ordinary intra-frame self-attention. This structural mismatch between training and inference causes the scale-aware geometric priors acquired during multi-view training to instantly vanish, resulting in severe scale ambiguity and disjointed point cloud accumulations when evaluated sequentially frame-by-frame.

To bridge this fundamental divide between single-view scale ambiguity and multi-view inference collapse, this paper investigates knowledge distillation: since multi-view geometric reasoning fundamentally relies on epipolar constraints, can we construct a synthetic stereo setup that allows a monocular model to preserve continuous epipolar attention pathways during inference? Core idea: propose EpiDistill, a geometric distillation framework that utilizes spatial Gaussian biases derived from ground-truth depth to form Depth-Guided Epipolar Attention, while introducing learnable Rectified Stereo Tokens as persistent structural anchors to emulate a horizontal rectified search space, successfully distilling multi-view metric scale priors into single-view inference.

Method

Overall Architecture

EpiDistill is designed to transfer epipolar correspondence reasoning and metric scale priors from a multi-view teacher mechanism into any frozen ViT-based monocular depth foundation model, such as UniDepthV2 and DepthPro. During training, the multi-view branch takes overlapping frame triplets and extracts spatial features alongside learnable scale tokens using a frozen ViT encoder; these features interact across viewpoints through Depth-Guided Epipolar Attention under explicit geometric supervision. The single-view branch takes only the reference frame and introduces a set of learnable Rectified Stereo Tokens to act as proxies for the missing source views. By querying the reference features via cross-attention, these tokens form reference-guided embeddings that attend to the reference tokens along horizontal epipolar scanlines. Both feature representations and attention distributions are distilled into the single-view pipeline. Finally, a decoupled prediction design separately estimates dense structural offsets, global scale factors, and camera intrinsic residuals.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input Images<br/>Multi-view Triplet / Single-view Reference"] --> Ext["Frozen ViT Encoder<br/>Extract Spatial Tokens + Scale Tokens"]
    Ext --> DGEA["Depth-Guided Epipolar Attention<br/>GT Projection + Gaussian Spatial Bias"]
    Ext --> RST["Rectified Stereo Tokens<br/>Simulate Horizontal Scanlines + Feature Distillation"]
    DGEA -.->|Knowledge Distillation| RST
    RST --> Dec["Decoupled Prediction Heads<br/>DPT Shift Offset + MLP Metric Scale & Intrinsics"]
    Dec --> Out["Final Outputs<br/>Scale-Aligned Metric Depth + Adjusted Focal Length"]

Key Designs

1. Depth-Guided Epipolar Attention: Enforcing Physical Constraints via Gaussian Spatial Biases Conventional cross-view attention computes unconstrained correlations across full feature maps, which often causes attention dispersion in textureless or repetitive regions and lacks explicit geometric supervision. To force the multi-view attention mechanism to focus precisely on true 3D physical correspondences, this work introduces Depth-Guided Epipolar Attention. For any query pixel in the reference view, the fundamental matrix \(F\) is calculated from the relative camera pose \([R|t]\) and intrinsic parameters, defining an epipolar sampling line in the source image. Using the ground-truth depth \(D_{gt}\) of the reference pixel, the exact target correspondence point \(p^*_{src}\) is computed via back-projection and re-projection onto the source plane.

To inject this physical correspondence into the attention distribution, a spatial Gaussian bias \(B_{ij} = -\gamma \|p_{src,ij} - p^*_{src,i}\|\) (with \(\gamma=50\)) is applied to each candidate coordinate \(p_{src,ij}\) sampled along the epipolar line. This bias is added directly to the raw query-key dot-product logits, yielding a depth-guided target attention map \(A^*_{ij} = \text{Softmax}_j(E_{ij} + B_{ij})\). Detaching \(A^*\) from the gradient flow, a cross-entropy loss \(\mathcal{L}_{attn}\) explicitly penalizes deviations in the unguided predicted attention distribution \(\hat{A}_{ij}\):

\[A^*_{ij} = \text{Softmax}_j\left(\frac{q_{ref,i}^\top k_{src,j}}{\sqrt{d}} + B_{ij}\right), \quad \mathcal{L}_{attn} = -\frac{1}{N}\sum_{i=1}^N \sum_{j=1}^M A^*_{ij} \log(\hat{A}_{ij})\]

This formulation enforces strict physical alignment along the epipolar sampling trajectory, providing a reliable, hallucination-free geometric teacher for subsequent single-view distillation.

2. Rectified Stereo Tokens: Synthetic Epipolar Anchors for Single-View Inference When a multi-view model is deployed for single-view inference, the absence of source frames causes cross-view attention layers to collapse into intra-frame self-attention, discarding all multi-view scale priors. To overcome this limitation, the authors exploit the baseline-agnostic nature of rectified stereo geometry. In a calibrated rectified stereo pair, the relative camera translation is strictly horizontal (\(t = [b, 0, 0]^\top\)) and rotation is an identity matrix (\(R = I\)). Under this configuration, the fundamental matrix simplifies such that the baseline constant \(b\) cancels out, restricting the epipolar search line for any reference point \([x_0, y_0, 1]^\top\) strictly to the identical horizontal scanline \(y = y_0\).

Leveraging this invariant, the single-view architecture incorporates a learnable grid of Rectified Stereo Tokens \(T_{rst} \in \mathbb{R}^{H \times W \times d}\). During monocular inference, \(T_{rst}\) first attends to the encoded reference features \(F_{ref}\) via cross-attention, incorporating scene-specific semantics and structural context to form reference-guided tokens \(T_{rst-ref}\). Subsequently, the model executes rectified epipolar attention between \(F_{ref}\) and \(T_{rst-ref}\) strictly along horizontal rows. By reducing the correspondence search from a complex 2D space to a 1D horizontal line, \(T_{rst}\) provides a stable geometric anchor that maintains continuous cross-view attention pathways without requiring actual multi-view inputs.

3. Decoupled Structure and Scale Heads: Preserving Relative Representations while Resolving Scale Entanglement Regressing metric depth directly from single-view features frequently compromises the rich relative depth representations already learned by foundation models and entangles local geometric details with global scale factors. The proposed framework explicitly decouples metric depth prediction into scale-invariant base geometry, dense structural offsets, and a global metric scale. The raw depth prediction from the frozen backbone \(D_{orig}\) is normalized by its spatial mean to produce a scale-invariant base map \(D_{base} = D_{orig} / (\frac{1}{N}\sum_i D_{orig,i})\).

The attention-refined spatial tokens are then passed to a lightweight Dense Prediction Transformer (DPT) head to regress a structural shift offset \(\Delta d\), which sharpens depth boundaries and refines occluded geometry. Concurrently, the scale token, enriched by both frame and epipolar attention, is processed by an MLP scale head to predict the scalar metric scale factor \(\hat{s}\). The final metric depth map is synthesized as:

\[\mathbf{D}_{\text{adj}} = \hat{s} \cdot (\mathbf{D}_{\text{base}} + \Delta d)\]

Additionally, an auxiliary camera head regresses normalized focal length residuals \((\delta_x, \delta_y)\) from the scale token to compute the adjusted focal length \(f_{adj} = f_{base} + (\delta_x W, \delta_y H)\), successfully decoupling depth scale from camera focal length.

Loss & Training

The overall training objective is a multi-task loss combining structural depth accuracy, global scale consistency, camera geometry, and token distillation:

\[\mathcal{L}_{total} = \mathcal{L}_{rel} + \mathcal{L}_{scale} + \lambda_{ray}\mathcal{L}_{ray} + \lambda_{distill}\mathcal{L}_{distill} + \lambda_{attn}\mathcal{L}_{attn}\]

where hyper-parameters are set to \(\lambda_{ray}=0.3, \lambda_{distill}=1.0\), and \(\lambda_{attn}=0.1\). - Relative Depth Loss \(\mathcal{L}_{rel}\): Defined as the sum of a log-space L1 loss \(\mathcal{L}_{depth}\) between the mean-normalized relative depth \(\tilde{D}_{rel} = D_{base} + \Delta d\) and normalized ground-truth depth, combined with an edge-aware Sobel gradient loss \(\mathcal{L}_{edge}\) to maintain sharp surface discontinuities. - Scale Loss \(\mathcal{L}_{scale}\): Measures the log-space L1 penalty between the predicted metric scale \(\hat{s}\) and the mean of ground-truth depth over the valid region. - Ray/Intrinsic Loss \(\mathcal{L}_{ray}\): Applies an L1 penalty on the predicted focal length offsets \((\delta_x, \delta_y)\) relative to ground-truth camera intrinsics normalized by image dimensions. - Distillation Loss \(\mathcal{L}_{distill}\): Aligns single-view tokens \(T_s\) with multi-view epipolar-aggregated features \(T_m\) over valid epipolar masks using combined cosine similarity direction alignment and \(L_2\) magnitude penalties. - Attention Loss \(\mathcal{L}_{attn}\): Aligns predicted epipolar attention logits with Gaussian depth-guided target distributions via cross-entropy.

Training is performed on a composite mixture of 7 diverse indoor, outdoor, and synthetic datasets (Hypersim, Cityscapes, Eden, ScanNet, ScanNet++, Waymo, nuScenes) with \(\ge 20\%\) inter-frame visual overlap. Models are trained for 100K iterations across 8 NVIDIA H100 GPUs using AdamW with a learning rate of \(1 \times 10^{-4}\).

Key Experimental Results

Main Results

On challenging zero-shot outdoor and mixed benchmarks characterized by extreme scale variation and expansive scene depths (Tab. 1), EpiDistill consistently improves baseline performance. Combining UniDepthV2 with EpiDistill achieves the top overall rank (3.1 across 9 models), outperforming baseline models and matching or surpassing methods that require ground-truth camera intrinsics at inference time.

Dataset	Metric	UniDepthV2	UniDepthV2+EpiDistill	DepthPro	DepthPro+EpiDistill	Metric3DV2 (w/ GT Intrinsics)
KITTI	A.Rel ↓ / \(\delta_1\) ↑	0.080 / 0.945	0.074 / 0.951	0.141 / 0.838	0.123 / 0.875	0.054 / 0.975
DDAD	A.Rel ↓ / \(\delta_1\) ↑	0.144 / 0.859	0.132 / 0.854	0.391 / 0.233	0.348 / 0.353	0.122 / 0.858
DIODE	A.Rel ↓ / \(\delta_1\) ↑	0.716 / 0.549	0.396 / 0.641	0.386 / 0.419	0.357 / 0.462	0.536 / 0.077
ETH3D	A.Rel ↓ / \(\delta_1\) ↑	0.176 / 0.752	0.137 / 0.779	0.355 / 0.433	0.354 / 0.393	0.175 / 0.724
Overall Rank	Rank ↓ (across 9 models)	4.7	3.1	8.8	6.4	4.5

On indoor benchmarks (NYU, Bonn, Booster, IBims-1), EpiDistill delivers stable gains across both backbones. Notably, on the specular and transparent Booster benchmark, UniDepthV2+EpiDistill reduces A.Rel from 0.179 to 0.171, achieving an average indoor rank of 2.4, tied with the GT-intrinsic-dependent DepthAnything3.

Ablation Study

Ablations on UniDepthV2 evaluated on averaged KITTI and ETH3D benchmarks demonstrate the complementary nature of DGEA and RST (Tab. 4, benchmarked on a single NVIDIA RTX A6000 GPU):

Config	DGEA	RST	GPU Memory	Latency	A.Rel ↓	RMS ↓	\(\delta_1\) ↑	Note
Fine-tuned UniDepthV2	-	-	57.87M	131ms	0.124	2.721	0.840	Decoder fine-tuning baseline
+ Global Attention	-	-	154.65M	180ms	0.119	2.649	0.858	Standard multi-view global cross-attention
+ DGEA (Epipolar Only)	✓	-	155.65M	190ms	0.113	2.419	0.866	Guided epipolar attention without stereo tokens
EpiDistill Full Framework	✓	✓	180.98M	218ms	0.101	2.206	0.870	Combined DGEA and RST; RMSE drops 16.7%
Multi-view Model Bound	-	-	180.98M	396ms	0.100	2.102	0.880	Upper bound with 3-frame multi-view input

Key Findings

• Rectified Stereo Tokens prevent structural collapse: Introducing DGEA alone yields modest improvements because single-view inference still lacks source view tokens, causing cross-attention to collapse into self-attention. Combining DGEA with RST unlocks the full potential, dropping A.Rel from 0.113 to 0.101 and reducing RMSE by 16.7% compared to the baseline.
• Approaching multi-view performance without multi-view cost: The full single-view EpiDistill model achieves an A.Rel of 0.101, nearly matching the multi-view upper bound (0.100), while cutting inference latency from 396ms to 218ms (a 45% reduction).
• Extreme scale robustness under Dolly Zoom perturbations: On the DepthPerturb dataset, camera translation coupled with continuous focal length changes keeps 2D object sizes constant, thoroughly confounding naive baselines. UniDepthV2+EpiDistill reduces A.Rel from 0.241 to 0.167 and boosts focal length correlation \(|PCorr|\) to 0.965, verifying genuine disentanglement between focal length and physical distance.

Highlights & Insights

• Geometric dimensional reduction via rectified stereo: Abstracting the baseline-invariant property of rectified stereo pairs (\(y=y_0\)) into learnable token grids allows single-view transformers to simulate binocular disparity matching along 1D scanlines.
• Ground-truth Gaussian bias avoids attention dispersion: Modulating cross-attention logits with distance-based Gaussian biases creates a smooth, physically grounded target distribution that guides multi-view training without discrete matching artifacts.
• Model-agnostic modular distillation: EpiDistill seamlessly integrates into frozen ViT backbones (UniDepthV2, DepthPro) via lightweight heads and epipolar layers, achieving double-digit metric error reductions without extensive backbone retraining.

Limitations & Future Work

• Increased computational overhead: Incorporating 4 epipolar attention layers and a \(37 \times 37\) stereo token grid increases memory from 57.87M to 180.98M and latency from 131ms to 218ms, posing challenges for resource-constrained edge deployments.
• Dependency on calibrated 6-DoF poses and accurate depth during training: The computation of epipolar lines and ground-truth 3D projections requires precise extrinsic poses and dense depth; pose noise or extreme depth sparsity could misdirect the Gaussian bias supervision.
• Future directions: Exploring dynamic or sparse token sampling to reduce attention overhead, and extending rectified geometric distillation to wide-angle or omnidirectional non-pinhole camera models.

Related Work & Insights

• vs DepthAnything3 / VGGT: Multi-view foundation models rely on multi-frame cross-view attention to resolve metric scale, which collapses into intra-frame attention when applied to a single image. EpiDistill introduces RST to preserve epipolar pathways, transferring multi-view robustness directly into single-view inference.
• vs Metric3DV2: Metric3DV2 strictly requires ground-truth camera intrinsics during inference to scale canonical depth predictions. In contrast, EpiDistill predicts metric scale and intrinsic residuals simultaneously from RGB inputs alone, outperforming Metric3DV2 on diverse zero-shot benchmarks.
• vs UniDepthV2 / DepthPro: While these baselines directly regress depth and intrinsics from single images, they remain vulnerable to scale drift in out-of-distribution scenes. EpiDistill serves as an effective plug-and-play enhancer that anchors their predictions to rectified epipolar geometry.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant exploitation of rectified stereo invariance via learnable tokens to bridge the single-to-multi-view architectural divide.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 7 training domains, 8 zero-shot evaluation benchmarks, and specialized Dolly Zoom scale tests.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical derivation of epipolar constraints coupled with clear architectural illustrations.
• Value: ⭐⭐⭐⭐⭐ Establishes a practical distillation paradigm for infusing multi-view geometric scale priors into monocular depth foundation models.

Related Papers

• [ECCV 2026] Boosting 6D Object Pose Estimation via Monocular Depth Cues
• [ECCV 2026] EventVGGT: Exploring Cross-Modal Distillation for Consistent Event-based Depth Estimation
• [ECCV 2026] One Scene, Two Depths: Probing Geometric Ambiguity in Monocular Foundation Models
• [CVPR 2026] MD2E: Modeling Depth-to-Edge Cues for Monocular Metric Depth Estimation
• [AAAI 2026] Geometry Meets Light: Leveraging Geometric Priors for Universal Photometric Stereo under Limited Multi-Illumination Cues