ZipDepth: Bringing Lightweight Zero-Shot Monocular Depth Anywhere, on Any Device

Conference: ECCV 2026
Paper: ECCV page
Area: Model Compression
Keywords: Knowledge Distillation, Lightweight Monocular Depth, Structural Reparameterization, Cross-Domain Generalization, Hardware-Adaptive Upsampling

TL;DR

ZipDepth distills the multi-domain depth knowledge of Depth Anything v2-Large into a convolutional student of about 6.1M parameters, pairing a half-resolution detail path with hardware-adaptive upsampling to preserve boundaries; on a Jetson Orin NX at 15 W in FP32 it reaches 34.4 FPS at 396.6 mJ per frame, though accuracy still trails the teacher and the output remains relative depth only.

Background & Motivation

Monocular depth foundation models already transfer across indoor scenes, roads, and internet photos, but the cost of their large backbones blocks mobile deployment — the authors measure Depth Anything v2-Large at just 0.3 FPS on a Jetson Orin NX at 15 W in FP32. Whether a device can run a model continuously depends not only on parameter count, but also on per-frame energy, input resolution, and runtime support for specific operators.

Small depth networks are usually trained on limited scenes, so low latency does not mean they can handle unfamiliar environments. This paper tackles the knowledge source and the deployment structure at the same time: first expose the student to sufficiently diverse teacher supervision, then concentrate the limited compute on low-resolution semantic extraction and high-resolution boundary recovery. Swapping in a smaller backbone alone, or merely adding training data, does not automatically solve the other half of the problem.

Core idea: keep the expensive cross-domain knowledge on the offline teacher side, let the student carry semantic extraction with fusible convolutions and recover details through a shallow bypass, and pick upsampling operators per hardware backend that execute efficiently.

Method

Overall Architecture

The input is a single RGB image and the output is an inverse-depth map at the same resolution; it carries scale-and-shift ambiguity and cannot be read directly as distance in meters. On the training side, the student is supervised by multi-domain teacher distillation; at inference it flows through reparameterizable context encoding, half-resolution progressive decoding, and hardware-adaptive upsampling. The teacher takes no part in deployment, and the upsampling path is selected per target backend before training rather than switched automatically per frame.

The encoder produces features at strides 4, 8, 16, and 32 while retaining the stem's stride-2 shallow branch. The decoder fuses from coarse to fine, finally reconnects to the shallow branch, first predicts half-resolution inverse depth, and then upsamples to full resolution. Most semantic computation happens at low resolution; shallow detail is brought in only when contours need to be recovered.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    T["Multi-domain teacher distillation<br/>offline pseudo-labels"]
    I["RGB image"] --> E["Reparameterizable context encoding"]
    E --> D["Half-resolution progressive decoding"]
    D --> U["Hardware-adaptive upsampling"]
    U --> O["Full-resolution inverse depth"]
    T -.->|training-only supervision| O

Key Designs

1. Multi-domain teacher distillation: unified teacher supervision instead of per-domain ground-truth depth collection

The authors build a training pool of about 14.1M images from 17 image sources and generate pseudo-labels offline with Depth Anything v2-Large. The sources include Objects365, ADE20K, COCO, SA-1B, and OpenImages, plus MegaDepth, Google Landmarks, Mapillary, Cityscapes, BDD100K, DrivingStereo, Gated2Depth, Trans10K, Flickr1024, HRWSI, HoloPix50K, and ACDC. They contribute diverse objects and capture conditions rather than ground-truth depth under a shared calibration protocol; the teacher's outputs form the unified relative-geometry supervision.

Training uses temperature-based domain-balanced sampling to keep large sources from dominating every update; the main text does not give the temperature or the sampling quotas. Distillation happens at the output level, matching the teacher with a scale-and-shift-invariant loss, and no per-layer feature distillation is proposed. The student learns a convolutional representation from scratch and does not inherit the teacher's Transformer architecture. "Zero-shot" here means the student is never trained on the five target depth benchmarks, not that the system requires no training.

2. Reparameterizable context encoding: multi-branch during training, folded into plain convolutions for deployment

The encoder uses RepVGG-style units: during training it keeps 3×3 and 1×1 convolutions plus an identity branch (whenever channel counts allow) in parallel, with batch normalization and ReLU; at inference the linear branches and batch normalization are fused into a single 3×3 convolution with bias. This is not pruning or quantization but a conversion from the training structure to an equivalent deployment structure. Parameters go from 6.79M during training to 6.14M after fusion, rounded to 6.1M in the main table. About 89.6% of the deployment parameters sit in Stages 3 and 4, keeping expensive wide-channel computation away from high resolutions.

A small convolutional backbone still needs global context. Stage 2 adds depthwise convolution branches with dilation rates 1 and 2 plus horizontal and vertical strip pooling to extend the spatial reach, gated by a gate that can suppress or amplify features; Stage 3 adds SE and a GCBlock, re-weighting channels and summarizing scene information through spatially weighted pooling. After Stage 4, an SPPF cascades three 5×5 max-poolings on the bottleneck channels, and grouped 1×1 convolutions then exchange information between Stages 3 and 4. Together these components compensate for the limited context of lightweight local convolutions, and they should not be split into multiple independent inventions.

3. Half-resolution progressive decoding: leave contour localization to a shallow bypass

Two stride-2 stem convolutions drop the spatial resolution to a quarter. ZipDepth retains the half-resolution feature after the first downsampling and reconnects it in the last decoding stage, instead of keeping the entire backbone at high resolution. Deep layers decide foreground-background relations while the shallow branch supplies contour positions; once edge positions have been mixed away in early downsampling, final interpolation alone usually cannot recover them.

The decoder starts at stride 32, upsamples level by level, and fuses with encoder skips at matching scales. Projections use grouped 1×1 convolutions, with the group count set to the largest value up to 4 that divides both input and output channels; channel counts shrink as resolution rises, and the FPN fusion totals about 150K parameters. After the stride-4 fusion, the stride-2 stem branch is fused in to yield 32-channel half-resolution features, and a 3×3 prediction head outputs inverse depth. The ablation shows this path improves accuracy but also clearly raises energy cost, so it cannot be called free detail recovery.

4. Hardware-adaptive upsampling: the same boundary-quality goal, realized with different operators per backend

The GPU/TensorRT path borrows the convex upsampling from RAFT: from the half-resolution features, a head predicts, for every output sub-pixel, nine weights over the 3×3 neighborhood, normalizes them with a temperature softmax, and blends the neighborhood depths before rearranging them to twice the resolution. Neighbors are gathered via replicate padding and unfold, and a ReLU is applied to the output. The weights are non-negative and sum to one, so the model can favor the depth on one side of a boundary and reduce the blur of directly averaging foreground with background. The paper does not provide a reliably readable temperature value.

Mobile NPUs and DSPs are not necessarily good at running unfold, softmax, and PixelShuffle, so an alternative uses only standard convolutions and interpolation: the half-resolution features pass through a 1×1 reduction, a 5×5 depthwise convolution, and a 1×1 projection to produce a sigmoid gate that blends nearest-neighbor and bilinear upsampling. It keeps sharp steps at edges and stays smooth in flat regions; it is less flexible than nine free neighbor weights, but its operators are easier for mobile backbones to support. This is a structural choice, not a loss-free format conversion of the GPU weights.

The gated mixture works as follows, with both interpolations acting on the same half-resolution inverse depth:

\[ \hat d=\operatorname{ReLU}\left(\alpha\operatorname{NN}(d)+(1-\alpha)\operatorname{Bilinear}(d)\right),\quad 0\leq\alpha\leq1. \]

Loss & Training

Predictions and teacher labels are normalized per image over valid pixels with median-and-MAD normalization, removing scale and shift ambiguity; the losses then combine the normalized mean absolute error with a multi-scale gradient loss. The former constrains relative geometry values, the latter spatial variation. The loss weights as given in the paper:

\[ \mathcal L=\mathcal L_{\mathrm{SSI}}+2\mathcal L_{\mathrm{grad}}. \]

Normalization at training time uses only teacher labels; least-squares alignment to ground truth at test time is a separate matter, and nothing here means a deployed model recovers true scale. Part of the formulas in the cached text is corrupted, so the MAD implementation details and the gradient-scale list are not reconstructed by guesswork.

Training runs in three stages: 256×256 for 10 epochs with batch 192 per GPU and peak learning rate 0.002; 384×384 for 5 epochs with batch 128 and learning rate 0.0005; 512×512 for 3 epochs with batch 96 and learning rate 0.00025. Each stage inherits the previous weights, warms up linearly over half an epoch, then cosine-decays to 1% of the peak. Images are resized on the short side and then randomly square-cropped, with horizontal flipping at probability 0.5.

The authors report about three days on two RTX 3090 GPUs, but teacher pretraining, offline pseudo-labeling, and the images actually sampled per epoch are not fully disclosed, so three days should not be read as the total cost. Default inference preserves the aspect ratio with the short side at 384, while the efficiency profiling fixes the input at 384×384 — the two workloads differ in compute.

Key Experimental Results

Main Results

Evaluation covers 654 NYUv2 images, 800 ScanNet samples, 652 KITTI Eigen-split images, 454 ETH3D samples, and 325 indoor plus 446 outdoor DIODE samples. Predictions are least-squares aligned to ground truth following the Marigold protocol. AbsRel is the mean absolute relative error over valid pixels; δ1 is the fraction of pixels where the larger of the prediction/GT ratio in either direction stays below 1.25.

Each dataset cell below is AR↓/δ1↑, keeping the original table's percentage convention — AR 8.4 means AbsRel 0.084. Efficiency conditions: Jetson Orin NX, 15 W, FP32, fixed input 384×384, though each model internally resizes to its native configuration. † marks lightweight baselines retrained by the authors on the same multi-domain data; ZipDepth uses the GPU upsampling path.

Model	Params M / MACs G	FPS↑ / mJ per frame↓	NYUv2	KITTI	ETH3D	ScanNet	DIODE
DA-V2-Large (teacher)	335 / 652.7	0.3 / 54457.1	5.1 / 97.1	7.7 / 94.9	5.1 / 97.8	4.2 / 98.0	21.0 / 76.7
DA-V2-Small	24.8 / 57.8	2.2 / 6740.3	6.1 / 96.2	8.4 / 93.4	6.3 / 96.7	5.2 / 97.2	21.4 / 75.9
Lite-Mono †	8.3 / 13.5	7.3 / 1883.5	8.7 / 92.7	10.9 / 89.5	8.7 / 93.6	9.6 / 90.2	22.4 / 72.9
PyDNet †	1.9 / 5.5	34.8 / 398.0	11.4 / 87.1	12.9 / 84.9	10.4 / 90.6	11.7 / 85.8	23.4 / 70.8
ZipDepth	6.1 / 3.0	34.4 / 396.6	8.4 / 93.3	12.3 / 86.4	10.0 / 92.2	8.8 / 92.1	22.6 / 73.9

ZipDepth does not lead on every cell: Lite-Mono is better on both KITTI and ETH3D metrics and on DIODE AR, but its MACs are 4.5× higher. Against the similar-throughput PyDNet, ZipDepth is better on both metrics in all five domains. Relative to the teacher it uses about 55× fewer parameters and 218× fewer MACs, but with a clear accuracy loss — it should be read as an efficiency trade-off.

Ablation Study

Structural ablations use the same data, training schedule, and Jetson 15 W, 384×384 conditions. AR values below keep the percentage convention.

Config	NYUv2 AR↓	KITTI AR↓	ETH3D AR↓	ScanNet AR↓	DIODE AR↓	FPS↑	mJ per frame↓
Full model	8.4	12.3	10.0	8.8	22.6	34.4	396.6
w/o SE + GCBlock	8.7	12.6	10.3	9.2	22.8	36.2	377.2
w/o SPPF + Cross-Scale	9.0	13.1	10.6	9.0	23.0	36.5	375.8
w/o Half-Res Path	9.1	13.8	11.2	10.3	23.8	48.9	271.8

Removing the half-resolution path raises KITTI and ScanNet AR by 1.5 points each while cutting energy by about 31.5%. The detail branch therefore carries a clear quality benefit and a real runtime cost. The other ablations remove whole groups, so credit cannot be attributed to any single module within a group.

The upsampling experiment measures the scale-invariant boundary F1 (SI-BF1) on 1,000 synthetic images from UnrealStereo4K. It scores how well depth contours match rather than whole-image error; the cache does not spell out the matching tolerance. The original column header carries a percent sign but the values are decimals, so the values below are kept as reported, without conversion.

Upsampling	NYUv2 AR / δ1	KITTI AR / δ1	SI-BF1 (as reported)↑	CPU FPS↑	GPU FPS↑	NPU FPS↑
Bilinear	8.3 / 93.2	12.2 / 86.5	0.088	47.4	40.3	415
NPU path	8.3 / 93.2	12.2 / 86.5	0.105	44.6	34.1	375
GPU path	8.4 / 93.3	12.3 / 86.4	0.120	25.6	34.8	240

The CPU is an i7-11800H at 45 W with ONNX Runtime; the GPU is the Jetson Orin NX at 15 W with PyTorch Fused FP32; the NPU is an iPhone 12 ANE at 6 W with CoreML — the different columns must not be read as one same-condition hardware ranking. The GPU path lifts boundary F1 by about 36.4% over bilinear while the standard depth metrics barely move. The 34.8 FPS in this table and the 34.4 FPS in the main table come from different profiling tables, not a single measurement.

Key Findings

• Data gains are not monotone everywhere. From 1% to the full set, NYUv2 AR drops from 12.2 to 8.4 and ScanNet from 13.2 to 8.8; KITTI reads 12.1, 12.1, 12.4, 12.6, 12.3 across the five fractions, so more data does not guarantee better accuracy in every domain.
• Backend and precision set the throughput claim. Fused PyTorch FP32 on the Jetson 15 W reaches about 34 FPS, and TensorRT FP16 reaches 77 FPS; that 77 FPS cannot be combined with the main table's FP32 396.6 mJ per frame into a single result.
• The title does not make every device fast. The mobile path on an iPhone 12 ANE runs at 375 FPS, while the low-end Poco X3 NFC reaches 16 FPS via TFLite GPU FP16. The cross-device table uses median forward latency after warm-up and excludes capture, transmission, and display.

Highlights & Insights

• Training and deployment structures are kept separate. Reparameterization solves the deployment graph and distillation solves the knowledge source; the two are complementary rather than interchangeable.
• Edge provenance and interpolation work together. The shallow bypass preserves contour locations while adaptive upsampling decides how to use them — judge the pair jointly through the structural ablation and the boundary metric.
• Operators, more than parameter counts, sit closest to hardware constraints. unfold costs different amounts on different backends, and standard convolutions plus interpolation can instead be the better fit for mobile accelerators.

Limitations & Future Work

• Accuracy and temporal consistency remain limited. The model trails foundation models, and frame-by-frame video prediction flickers; temporal modules, metric depth, and point maps are the authors' future directions, not existing features.
• Relative depth cannot measure distance directly. Evaluation relies on ground-truth alignment; deploying it for robot control still requires scale recovery and downstream validation.
• Extreme scenes are qualitative evidence only. The DA-2K adverse-weather, aerial, reflective, and underwater examples come without ground-truth depth, so they cannot demonstrate quantitative reliability.
• Total cost and reproduction details are incomplete. The cost of offline pseudo-labeling, the domain sampling quotas, and the energy-measurement details are not fully reported, and there are no confidence intervals across random seeds.

Related Work & Insights

• vs Depth Anything v2: the teacher supplies pseudo-labels while this paper retrains a convolutional student — not layer-wise pruning of the teacher.
• vs Lite-Mono / PyDNet: retraining on the same data helps separate architectural from data gains, though those models still hold advantages in single-domain accuracy or parameter count.
• vs RepVGG / RAFT: this paper borrows structural reparameterization and convex upsampling from them respectively; the increment is the lightweight cross-domain depth combination and the mobile operator substitutes.
• Resources: the project page is taken from the paper's first page; its release status has not been verified online, so it is not presented as a code repository.

Rating

• Novelty: 3/5. A systematic combination of mature components, focused on cross-domain lightweight deployment.
• Experimental Thoroughness: 4/5. Five domains, same-data baselines, ablations, and multi-device profiling are fairly complete, but end-to-end cost and statistics remain thin.
• Writing Quality: 4/5. The data-to-hardware argument is coherent, though some details depend on a supplement not included in the cache.
• Value: 4/5. Provides a reference budget for low-power relative depth; it cannot replace high-accuracy foundation models or direct distance measurement.