Zero-Shot Novel Depth Synthesis Using Foundation Models Scene Representations¶

Conference: ECCV 2026
Paper: ECCV 2026 page
Area: 3D Vision
Keywords: novel-view depth synthesis, 3D foundation models, latent diffusion, VGGT, occlusion completion

TL;DR¶

Z3D generates target-view representations in the scene-token space of frozen VGGT/WorldMirror, conditioned on source images and target camera poses, then decodes them with a frozen depth head; against the authors' pixel-space depth-diffusion baselines it is more accurate in-domain and on DTU-like settings, but not uniformly better on NRGBD.

Background & Motivation¶

Given one or two source images and a new camera position, predicting the depth of that novel view requires more than reprojecting already-seen surfaces — it also means inferring structure that is occluded or lies outside the source field of view. Novel-view RGB synthesis supplies appearance, but nothing guarantees the generated image strictly respects the target camera's geometry; under sparse observations, conventional reconstruction has no direct constraint on unobserved regions either. Monocular depth estimation mainly handles the input view, so it cannot directly substitute for a geometry prediction at the target camera placement.

3D foundation models such as VGGT and WorldMirror have learned scene representations from massive data, yet their standard interfaces still mostly emit input-view depth and point maps. The authors first probe whether usable information about hidden surfaces is present, using a layered depth image probe: freeze the backbone and train a lightweight decoding head to predict multiple surfaces along each sightline. The result beats a mean baseline, supporting the claim that these representations contain cues useful for occlusion inference — but it does not prove they have completely stored the invisible geometry of any scene.

Core idea: let the pretrained 3D model own scene representation and depth decoding, and let a conditional generative model learn only how to complete the intermediate representation for a target pose — instead of jointly learning geometric representation and generative ability from scratch.

Method¶

Overall Architecture¶

The input is source images plus the relative poses of target cameras, and the output is target-view depth. The frozen representation codec supplies scene tokens and the depth head; target single-layer latent generation restricts what gets generated; dual-path geometric conditioning injects source features and poses into the DiT; flow matching with progressive training handles learning and sampling. At deployment, generation starts from noise and produces the target tokens, which the frozen DPT depth head finally decodes.

During training, target images serve only to extract the supervision representation; at test time, target images are unavailable. Source tokens stay clean as conditions, and only the target tokens are noised. Although the backbone and the decoding head are frozen, the conditioning encoders and the DiT still require dedicated training; "zero-shot" mainly refers to transfer to unseen target datasets, not to a generative system that needs no training.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Source images and target poses"] --> B["Frozen representation codec<br/>source image encoding"]
    B --> C["Target single-layer latent generation"]
    C --> D["Dual-path geometric conditioning"]
    D --> E["Flow matching and progressive training<br/>50-step inference sampling"]
    E --> F["Frozen DPT decoding<br/>target-view depth"]

Key Designs¶

1. Frozen representation codec: learn a generative mapping between existing geometric representations

The same 3D foundation model extracts patch tokens from both source images and training-target images, while the pretrained DPT head decodes generated tokens into depth. In terms of roles this resembles the encoder and decoder of a latent generative model, but it is not a retrained VAE. The backbone has already learned cross-view scene relations, so the generator only needs to model the conditional distribution of target features given source features and camera pose.

The paper's layered depth image (LDI) experiment supplies the premise evidence: the depth head is expanded into a four-layer LDI with confidence prediction, everything else stays frozen, and a lightweight probe is trained. An LDI records multiple surfaces along the same pixel ray, with later layers corresponding to geometry occluded by front surfaces; both VGGT and WorldMirror features beat the mean baseline. This experiment shows hidden geometry can be predicted from the representations — it does not mean every prediction corresponds to a surface that actually exists. Handing the completion task to a generative model still involves unavoidable prior-driven inference.

2. Target single-layer latent generation: compress what is generated while keeping richer source conditioning

VGGT/WorldMirror depth prediction uses high-dimensional tokens from four intermediate layers, each token about 2048 dimensions. Directly generating the concatenated multi-layer features is expensive; based on a layer-contribution analysis, the authors restrict the diffusion object for the target view to layer-17 tokens, while the source side keeps the aggregated four-layer features to provide geometric conditions via cross-attention. This asymmetric arrangement reduces the variables being generated rather than cutting conditioning information by the same proportion.

The implementation ambiguity in the original text must be preserved: the first half of §3.2 writes "only tokens from layer 17 are used for conditioning," while the later half explicitly writes that target layer 17 is the generation object and the source's four layers serve as conditions. This note follows the later, concrete description — the two sentences cannot both be presented as consistent facts. How the single-layer generation output concretely fits the original multi-layer DPT interface is not fully developed in the main text; reproduction requires checking the code or supplementary material, and one should not assume copy, zero-padding, or layer-mapping rules on one's own.

3. Dual-path geometric conditioning: scene information enters via attention, poses via the time embedding

Every DiT block adds cross-attention that retrieves target-view-relevant content from the clean source tokens. The other path encodes each target view's relative camera pose as a 12-dimensional vector, Fourier-encodes it and maps it through an MLP, adds the result to the diffusion timestep embedding, and thereby influences denoising through adaptive layer normalization. All poses are expressed relative to the first view of the source set, so camera relations share a unified reference instead of treating absolute world coordinates directly as transferable semantics.

The two conditioning paths have distinct roles: source features answer what structure the scene contains, while the pose specifies where to observe from. Multi-target configurations must jointly generate several target representations, and per-image plausibility does not certify geometric consistency. The evidence that the model supports multi-view input and output mainly comes from the one-source-one-target and two-source-four-target experiments; it cannot further guarantee equal stability at arbitrary view counts.

4. Flow matching and progressive training: adapting the generation schedule to high-dimensional scene representations

The implementation section uses flow-matching velocity prediction, with FlowMatchEulerDiscreteScheduler as the training scheduler, 1000 training timesteps, and 50 sampling steps at inference. The authors adjust the timestep shift according to the effective representation dimension, giving a scaling factor equal to the square root of the ratio between the effective dimension and a reference dimension of 4096. This is the schedule adaptation adopted by this paper, not a universal stability guarantee for all high-dimensional representations; how the effective dimension is instantiated also depends on the implementation.

Training starts from one-source-one-target, and the two-source-four-target version is initialized from those weights. The authors observed that training the latter directly was less stable and converged more slowly, hence the curriculum-style initialization. The main text, however, contains no full training-curve ablation, so this cannot be called the only viable option. What is generated is a geometric hypothesis drawn from the conditional distribution — repeated samples may differ, and 50 steps is far from real-time operation.

Loss & Training¶

Training data come from MegaDepth, Hypersim, Taskonomy, Replica, and Habitat HM3D, organized by camera frustum overlap into view sequences of 2–6 images per group. Stage one uses an effective batch size of 128 for 98k steps; stage two uses batch size 32 for 156k steps. Both stages use a learning rate of 0.0002, AdamW with β1/β2 of 0.9/0.95, and no weight decay; the first 10% of steps are linear warmup, and the learning rate decays to zero starting from 30% of training progress.

The original §3.2 also writes the forward noising with DDPM-style notation, while §3.3 explicitly adopts a flow-matching velocity objective. The two formulations cannot be merged into one precise implementation formula without a caveat, so this note follows the scheduler and objective reported in the implementation section and does not patch in a vague noise-coefficient relation. The backbone and DPT are frozen, and what is trained still includes the DiT and the conditioning encoders — not merely a linear head.

Key Experimental Results¶

Main Results¶

The authors construct two baseline families. LVSM first synthesizes the target RGB and then a 3DFM estimates depth from it; DD (Depth Diffusion) uses the same generative architecture and training pipeline but generates directly in the predicted depth-map space rather than in scene tokens. There is no direct quantitative comparison with MVGD, because its code was unavailable when the paper was written — one cannot claim the method comprehensively surpasses all existing approaches.

Evaluation first applies affine alignment and then computes AbsRel and δ1. AbsRel is the mean absolute relative error over valid pixels; δ1 is the fraction of pixels where the larger of the two-way predicted/GT ratios stays below 1.25. The aligned results measure relative geometry and do not mean metric distances can be recovered directly in deployment. Point-cloud Accuracy and Completion measure the nearest-neighbor distance from prediction to ground truth and from ground truth to prediction, respectively — both are lower-is-better, and neither is a percentage precision or recall.

The table below excerpts the original Table 3, one source → one target; cells are AbsRel↓/δ1↑. In-domain is the test subset of the training sources, and the other three columns are cross-domain evaluations.

Method	In-domain	DTU	7-Scenes	NRGBD
LVSM+VGGT	—	0.468 / 0.024	0.342 / 0.745	0.214 / 0.666
LVSM+WM	—	0.468 / 0.025	0.328 / 0.749	0.210 / 0.672
VGGT-DD	0.098 / 0.914	0.022 / 0.998	0.274 / 0.963	0.192 / 0.743
WM-DD	0.108 / 0.902	0.036 / 0.997	0.272 / 0.967	0.127 / 0.837
Z3D-VGGT	0.076 / 0.935	0.012 / 0.999	0.260 / 0.963	0.260 / 0.676
Z3D-WM	0.075 / 0.939	0.021 / 0.999	0.266 / 0.967	0.133 / 0.826

Z3D-VGGT lowers DTU AbsRel from VGGT-DD's 0.022 to 0.012, but on NRGBD it rises from 0.192 to 0.260; Z3D-WM is also worse than WM-DD on NRGBD. The authors' "consistently outperforms" summary is therefore stronger than the table supports.

Point clouds involve the same trade-off: in the DTU one-source-one-target setting, Z3D-VGGT's Accuracy mean/median is 2.725/1.782, better than VGGT-DD's 6.337/4.386, yet its Completion mean is 2.938, worse than the baseline's 1.644. More accurate geometry does not mean more complete coverage, and the two distances must not be merged into a single "point-cloud accuracy."

Ablation Study¶

The table below is the original Table 5, the two-source-four-target representation-space comparison. DD shares the architecture and training pipeline with Z3D, making it the main evidence for the choice of generation space; it is not a pure per-module-removal ablation.

Method	In-domain AbsRel / δ1	DTU AbsRel / δ1	7-Scenes AbsRel / δ1	NRGBD AbsRel / δ1
LVSM+WM	—	0.475 / 0.006	0.346 / 0.935	0.121 / 0.842
VGGT-DD	0.248 / 0.667	0.055 / 0.986	0.372 / 0.959	0.225 / 0.686
WM-DD	0.215 / 0.735	0.059 / 0.979	0.383 / 0.943	0.147 / 0.812
Z3D-VGGT	0.112 / 0.891	0.024 / 0.995	0.342 / 0.965	0.223 / 0.715
Z3D-WM	0.118 / 0.889	0.036 / 0.987	0.362 / 0.962	0.131 / 0.830

In the VGGT family, in-domain AbsRel drops from 0.248 to 0.112 and DTU from 0.055 to 0.024, supporting scene latents over these direct depth-generation baselines. On NRGBD, however, LVSM+WM still beats Z3D-WM.

The premise probe results come from the original Table 1:

Layered depth image prediction	AbsRel↓	δ1↑
Mean baseline	0.319	0.564
VGGT-LDI	0.197	0.717
WM-LDI	0.167	0.789

Relative to the mean baseline, VGGT reduces AbsRel by about 38.2% and WorldMirror by about 47.6%. This is evidence of decodable geometric cues, not a guarantee that unobserved surfaces are real — and not the result of the frozen original depth head directly outputting LDIs, since the probe head requires training.

Key Findings¶

The representation space is more favorable than raw depth maps for these generative settings. The in-domain and DTU comparisons consistently support this, but it should not be generalized into "latents always beat pixel space in every task."
The outdoor results are an in-domain split. MegaDepth is already among the training sources; the separately reported two-source-four-target AbsRel is Z3D-VGGT 0.039 versus VGGT-DD 0.102, which cannot be called zero-shot results on an unseen outdoor domain.
Swapping the backbone still requires retraining the generator. The VGGT-Ω one-source-one-target version needed no algorithmic change, but it was retrained; this cannot be read as inheriting all gains training-free by swapping backbone weights.
One original table has a suspicious duplicate column. The NRGBD Completion values in point-cloud Table 2 duplicate the depth-table numbers and disagree with Table 7. This note does not compute coverage improvements from that column; the original version needs re-checking.

Highlights & Insights¶

Treating geometric features as the generation space. The foundation model does more than provide conditioning: its internal representation is the target variable, so pretrained knowledge participates at both the encoding and decoding ends.
Lean on the generation side, information-preserving on the conditioning side. Generating only the target key layer while keeping multi-layer source features is a design worth borrowing, though the concrete interface still needs implementation-level verification.
Probe first, then build the generator. The LDI experiment first checks whether the representations carry task-relevant cues, avoiding back-inferring the mechanism from final performance alone.

Limitations & Future Work¶

Low overlap and first-frame sensitivity. When source-target coverage is insufficient, fine-grained geometry and point-cloud consistency degrade; using the first source view as the reference also inherits the foundation models' sensitivity to the choice of reference view.
No guarantee of true scale or of unseen geometry. Affine-aligned evaluation is not direct metric ranging, and generating beyond occlusions calls for additional uncertainty estimates or downstream verification.
Incomplete reproduction details. Layer 17's conditioning role is inconsistent across passages, the interface from single-layer output to the multi-layer depth head is underspecified, and the diffusion versus flow-matching notation needs checking.
Cost and stochasticity are under-reported. 50-step sampling plus foundation-model forward passes is not cheap, and inference latency and multi-sample statistics are missing.
Limited comparison scope. The baselines are mostly author-constructed, and NRGBD provides a clear counterexample — one cannot claim on this basis that all novel-view depth methods are surpassed.

vs MVGD: it also involves geometry generation, but this paper reuses frozen representations; without a unified protocol there is no direct quantitative comparison.
vs LVSM+3DFM: synthesizing RGB first and estimating depth afterwards can propagate pose inconsistency, which this paper avoids by skipping the intermediate image — a specific baseline observation, not proof that photometric generation necessarily breaks geometry.
vs Marigold: monocular depth diffusion mainly predicts the input view, whereas this paper generates the target view for a new camera pose — different task conditioning.
vs CUT3R: persistent state with virtual-view querying offers another route; this paper focuses on conditional generation within frozen foundation-model features, and architecture differences alone cannot settle which is better.

Rating¶

Novelty: ⭐⭐⭐⭐ (4/5) — using foundation-model scene representations as the generation space, with a probe validating the premise, is a targeted combination.
Experimental Thoroughness: ⭐⭐⭐ (3/5) — multi-domain, multi-view, and representation-space comparisons are valuable; direct comparison with the closest prior method and cost statistics are missing.
Writing Quality: ⭐⭐⭐ (3/5) — the argument line is clear, but layer selection, training notation, and some table columns need clarification.
Value: ⭐⭐⭐⭐ (4/5) — a reference recipe for sparse-observation geometric completion; uncertainty and low-overlap scenes must be checked before deployment.