Repurposing Geometric Foundation Models for Multi-view Diffusion

Conference: ECCV 2026
Paper: ECCV 2026
Area: 3D Vision
Keywords: novel view synthesis, geometric foundation models, latent diffusion, cross-view consistency, cascaded generation

TL;DR

GLD repurposes frozen Depth Anything 3 features as a multi-view diffusion space, combining joint boundary-level generation, frozen deeper-feature propagation, and a shallow cascade to achieve Re10K PSNR 16.362 without text-to-image generative pretraining while directly reusing geometry decoding for depth and ray maps.

Background & Motivation

Novel view synthesis is not the independent production of attractive images: it completes the appearance of one scene from sparse observations and specified target cameras. Occluded regions may admit multiple plausible completions, but walls, object boundaries, and camera motion must remain mutually compatible. Conventional VAE latents primarily support single-image compression and reconstruction, without explicitly prescribing correspondence between views. A multi-view diffusion model must therefore learn generation and rediscover cross-view geometry from its training data. Methods such as MVGenMaster add depth estimation and explicit warping as conditions, but this does not turn the underlying generation space into a geometric representation.

RAE demonstrates that frozen visual representations can support image diffusion, so a VAE is not the only possible entry point. However, features suited to semantic recognition need not preserve geometry across viewpoint changes, while deep geometric features may discard colors and textures. Geometric foundation models such as DA3 offer a more task-aligned starting point: cross-image attention jointly reasons about views and supplies multi-level features to a geometric decoder. The question becomes which features support both faithful appearance and reliable correspondence, and how to avoid separately modeling an entire high-dimensional hierarchy. Answering this requires examining reconstruction, cross-view matching, and actual generation quality together, rather than choosing a level from one representation probe.

The paper first trains an RGB decoder, then selects a boundary for explicit synthesis, and finally addresses misalignment between independently generated feature levels. It does not require predicted depth as an explicit conditioning input; instead, diffusion operates directly in a representation with existing geometric priors. Here, training “from scratch” means the diffusion model inherits no text-to-image generative weights; the DA3 encoder and its original geometric decoder remain pretrained. Core idea: generate joint multi-view features at an intermediate level that preserves both correspondence and appearance, use the frozen backbone for deeper features and a cascade for shallow details, and decode images and geometry from the same latent hierarchy.

Method

Overall Architecture

Inputs are \(N\) source images with camera poses and \(M\) target camera poses; outputs are novel views at those target cameras. The main backbone is DA3-Base, using four intermediate feature levels numbered \(0,1,2,3\). These level indices are not the original Transformer's specific block indices; the cached main paper does not provide that block-level mapping. Source images first pass through the frozen encoder on their own, producing the conditions available at inference time. Conditioned on those features and camera information, diffusion generates joint level-1 features, which propagate through the frozen backbone to produce levels 2 and 3. A second, cascaded diffusion model generates level 0 from the generated level 1, and the complete hierarchy feeds an RGB decoder and the original geometric decoder.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Source images and cameras<br/>Target cameras"] --> B["Joint Boundary Generation<br/>and Frozen Propagation"]
    B --> C["Shallow Cascaded Generation"]
    B -->|Levels 1, 2, 3| D["Multi-level Dual Decoding"]
    C -->|Level 0| D
    D --> E["Target RGB<br/>Depth and ray maps"]

Key Designs

1. Joint Boundary Generation and Frozen Propagation: explicitly synthesize only the necessary levels

DA3's cross-view attention mixes information across images, so source features encoded alone differ from the source portion obtained by jointly encoding source and target views. The former can serve as inference conditions, whereas the latter belongs to the joint representation used during training. Generating only target features while retaining independently encoded source features would give downstream propagation a joint representation assembled from incompatible encoding processes. GLD therefore generates features for both source and target views; this is not a requirement to reinvent known source images at the image level. Page 5 expresses the joint representation as concatenation along the view dimension:

\[ \widetilde{\mathbf F}_l=[\widetilde{\mathbf F}^{\mathrm{src}}_l,\widetilde{\mathbf F}^{\mathrm{tgt}}_l]. \]

Each joint feature level contains \(N+M\) views, with spatial tokens and channels for each view. Multi-view diffusion adopts the DiTDH architecture used by RAE and adds 3D self-attention, PRoPE, and Plücker ray embeddings to encode cross-view and camera-related geometry. Source-only conditioning features are concatenated with noisy latents along the channel dimension, allowing the model to predict the complete joint features. These camera conditions are not target depth maps and do not require explicitly warping source RGB into target viewpoints.

To determine how far explicit synthesis must extend, the authors first train four independent diffusion models, one for each level from 0 to 3. For a candidate boundary \(k\), corresponding models generate levels up to \(k\), and frozen subsequent DA3 layers derive deeper features from the boundary representation. This is a diagnostic experiment for boundary selection, not a requirement to invoke four independent generators during final inference. Table 2 favors \(k=1\): generating only level 0 is insufficient, while independently generating levels 2 and 3 adds no benefit. The final system therefore retains the level-1 boundary generator and a subsequent cascaded generator for level 0. Frozen propagation lets deeper features evolve from the same boundary representation through the backbone's existing computation, avoiding additional deep-level diffusion distributions.

Why not simply choose the most geometric level? In Table 5's ScanNet correspondence probe, level 2 reaches PCK 40.70, above level 1's 35.98. PCK is the fraction of correctly matched points within a distance threshold; the main paper references the Probe3D protocol without specifying that threshold, so no value is inferred here. Yet Table 6 reports single-level reconstruction PSNR of 14.01 for level 2, compared with 25.36 for level 1. Level 0 achieves reconstruction PSNR 28.01 but only PCK 22.25, showing that abundant shallow appearance information does not imply sufficient cross-view structure. Level 1 is useful because it retains both types of information, not because it wins every individual probe. The loss of photometric detail in deeper levels is the authors' interpretation of these experiments, not a universal law established for every geometric backbone.

2. Shallow Cascaded Generation: attach texture features to the generated geometric representation

After choosing the boundary, level 0 remains necessary because RGB decoding needs shallow appearance information. Independently generating levels 0 and 1 can nevertheless sample different scene interpretations, even under the same source-image conditions. GLD replaces the independent level-0 model with a cascaded model \(\mathcal M_{1\rightarrow0}\) that generates shallow features with level 1 as an additional condition. This dependency anchors shallow details to the intermediate-level scene structure already selected, rather than asking the RGB decoder to repair conflicts afterward. The cascade uses the independent level-0 model's architecture and training configuration; the key change is cross-level conditioning, not an entirely different generator.

During training, the additional condition is a noisy version of ground-truth level-1 features; during inference, it is the first diffusion model's actual output. Noise exposes the cascade to imperfect representations, reducing the mismatch that would arise if it only learned from ideal conditions. This does not mean training reproduces every upstream model error, and the main paper offers no such guarantee. Table 8 compares independent and cascaded generation with the same four-source-view setting and improves both image and geometry metrics, supporting the importance of cross-level alignment. Because the second model depends on the first model's output, generation includes sequential stages; faster training convergence alone does not establish lower sampling latency.

3. Multi-level Dual Decoding: establish reconstructability and preserve the geometry interface

Features capable of estimating depth are not automatically capable of recovering detailed RGB, so the authors first train a ViT-based RGB decoder to test this property. The encoder remains frozen while the decoder reconstructs images from the complete four-level hierarchy. Randomly masking feature levels during training makes the decoder usable with incomplete inputs; this level-wise dropout also enables the later single-level reconstruction analysis. On 4,000 samples from the Re10K test set, Table 1 reports PSNR 35.41, SSIM 0.960, and LPIPS 0.019. These numbers measure reconstruction from real-image features, not novel-view generation from sparse observations.

At inference time, the RGB decoder converts generated target-view features into images. The original DPT-based geometric decoder uses features from the same hierarchy to predict depth and ray maps, without training another geometry head for this output. Page 5 gives the two decoding relationships:

\[ \widetilde{\mathbf I}^{\mathrm{tgt}}=\mathcal D_{\mathrm{rgb}}(\{\widetilde{\mathbf F}^{\mathrm{tgt}}_l\}_{l=0}^{L-1}),\qquad \widetilde{\mathbf G}^{\mathrm{tgt}}=\mathcal D_{\mathrm{geo}}(\{\widetilde{\mathbf F}^{\mathrm{tgt}}_l\}_{l=0}^{L-1}). \]

Here \(L=4\), and geometry outputs allow synthesized pixels to be unprojected into a point cloud. The point of dual decoding is not to add another postprocessor, but to keep generated representations interpretable by the original geometric model. Zero-shot geometry output means reusing the existing decoder without additional fine-tuning, not that the system has never received geometric pretraining.

Loss & Training

The diffusion models use a flow-matching objective; the main paper does not provide its complete formula, so no noise path, loss weights, or solver details are supplied here. The RGB decoder uses a weighted sum of L1, LPIPS, and adversarial losses, with Re10K and DL3DV training data and AdamW optimization. Its schedule uses cosine learning-rate decay, 1 epoch of warmup, global batch size 128, and EMA decay 0.9978, training for approximately 170k steps on 8 B200 GPUs. Diffusion training instead combines Re10K, DL3DV, HyperSim, and TartanAir; each sample contains 8 views, with 1 to 4 randomly selected as sources. Its AdamW learning rate is fixed at \(5\times10^{-5}\), batch size is 48, and EMA decay is 0.9995, with 175k iterations on 8 B200 GPUs. Camera embeddings receive 10% dropout during training, and inference uses classifier-free guidance with a CFG scale of 1.5. Both training procedures use multiple resolutions: \(504\times504\), \(504\times378\), \(504\times336\), and \(504\times280\). Real target images provide supervision features during training, but inference conditions come only from source images and cameras; target images must not leak into those conditions. VAE and DINO diffusion baselines use the same architecture and iteration count, testing latent-space choices rather than treating pretrained diffusion and random initialization as identical controls. Figure 1 reports more than 4.4-fold faster training convergence; its horizontal axis is training steps, so the same factor cannot be claimed for end-to-end wall-clock time or inference speed.

Key Experimental Results

Main Results

Table 3, page 8: the main evaluation uses 2 source images and 200 samples per dataset; Re10K and DL3DV are in-domain, while Mip-NeRF 360 is out-of-domain. The excerpt below includes latent-space controls and MVGenMaster; higher PSNR and SSIM and lower LPIPS and ATE are better. ATE is computed after external VGGT estimates cameras from generated images, testing adherence to target pose conditions rather than directly measuring GLD's built-in geometry head.

Dataset	Method	PSNR	SSIM	LPIPS	ATE
Re10K	VAE	15.656	0.606	0.456	0.278
Re10K	DINO	15.638	0.601	0.448	0.345
Re10K	MVGenMaster	15.226	0.588	0.456	0.282
Re10K	GLD	16.362	0.630	0.431	0.211
DL3DV	VAE	14.725	0.446	0.476	0.589
DL3DV	DINO	14.345	0.411	0.471	0.546
DL3DV	MVGenMaster	14.565	0.442	0.460	0.281
DL3DV	GLD	15.499	0.468	0.438	0.209
Mip-NeRF 360	VAE	13.942	0.274	0.548	1.221
Mip-NeRF 360	DINO	13.718	0.267	0.542	0.949
Mip-NeRF 360	MVGenMaster	14.170	0.304	0.511	0.320
Mip-NeRF 360	GLD	14.542	0.288	0.504	0.589

GLD beats identically trained VAE and DINO models on image metrics across all three datasets, but it does not beat generatively pretrained methods on every out-of-domain metric. For example, MVGenMaster's Mip-NeRF 360 SSIM of 0.304 and ATE of 0.320 outperform GLD's 0.288 and 0.589. The complete Table 3 also reports Re10K MEt3R of 0.319 for DINO, better than GLD's 0.328; MEt3R measures multi-view consistency through projected feature similarity. Pretrained NVS methods do not all use identical training data, and CAT3D is a CAMEO-provided reproduction, so their ranking is not a causal comparison controlling every variable.

Ablation Study

Table 2, page 7: the explicit synthesis boundary changes while frozen backbone propagation supplies deeper features; independently trained level models are used for this boundary-selection experiment. The main text does not separately specify this table's dataset and source-view count, so its values are not directly subtracted from the final main results. AbsRel measures relative depth error and RMSE measures root mean squared depth error; lower is better for both.

Boundary	Explicitly Generated Levels	PSNR	SSIM	LPIPS	AbsRel	RMSE
\(k=0\)	0	12.55	0.323	0.579	0.267	0.400
\(k=1\)	0, 1	13.61	0.366	0.555	0.191	0.311
\(k=2\)	0, 1, 2	13.35	0.355	0.566	0.254	0.393
\(k=3\)	0, 1, 2, 3	13.35	0.355	0.567	0.260	0.402

Table 8, page 13: independent versus cascaded generation on Re10K with 4 source images; RPEr and RPEt denote relative rotation and translation errors. Values retain Table 8's original precision rather than substituting the slightly different SSIM, reprojection, or MEt3R entries from Table 4's four-source-view row.

Config	PSNR	SSIM	LPIPS	ATE	RPEr	RPEt	Reprojection Error	MEt3R
Independent	18.81	0.692	0.335	0.197	7.179	0.430	0.666	0.335
Cascaded	19.00	0.695	0.327	0.182	6.694	0.397	0.652	0.326

Key Findings

• Generating more levels is not always better: \(k=1\) beats both shallower and deeper boundaries, consistent with the appearance and geometry probes.
• Cascade gains are modest but directionally consistent, supporting cross-level coordination as a requirement distinct from single-level generation quality.
• In Table 9's ETH3D depth evaluation on page 13, GLD achieves AbsRel 0.160 versus Matrix3D's 0.197; corresponding RGB PSNR values are 14.80 and 14.13.
• Table 4 on page 11 reports single-source DL3DV ATE of 0.237 for GLD versus 0.880 for VAE, showing that geometric priors remain useful with sparse inputs.

Highlights & Insights

• Geometry enters the generation representation rather than only an auxiliary loss or condition map. This lets the same generated latents remain interpretable by the original geometric decoder.
• Representation selection is supported by three complementary tests: RGB reconstructability, cross-view matching, and improved final NVS. Depth prediction or reconstruction alone would not establish the entire claim.
• Frozen propagation and the shallow cascade address different problems: the former reduces explicitly modeled levels, while the latter resolves inconsistency between the separately generated shallow and boundary levels. They are not interchangeable acceleration tricks.

Limitations & Future Work

• Out-of-domain pose accuracy still trails some generatively pretrained methods, so geometric latents do not eliminate distribution shift. Broader scene coverage should be assessed under controlled data and compute budgets.
• The system depends on pretrained DA3 representations and trains an RGB decoder plus multiple diffusion stages; no T2I pretraining does not mean no pretraining cost or inexpensive training.
• Main evaluation uses only 200 samples per dataset, and the main paper provides no repeated-training variance or confidence intervals; small differences should not be treated as robust statistical advantages.
• The local full text ends with references on page 19 and omits the cited Appendices A/C; conditioning-noise strength, sampling steps, exact loss weights, and the VGGT alternative-backbone details cannot be verified from it.
• The paper has no dedicated limitations section; the boundaries above are reading-based judgments from the experiments and visible implementation details, not purported verbatim author admissions.

Related Work & Insights

• Versus VAE and RAE/DINO: GLD retains the frozen-encoder plus trainable-RGB-decoder approach but replaces single-image semantic features with joint multi-view geometric features and addresses hierarchical generation.
• Versus MVGenMaster: that method uses external depth and warped source-view conditions, whereas GLD does not need these explicit depth conditions; it still requires cameras and geometric pretraining.
• Versus Matrix3D: both provide images and geometry, but GLD obtains geometry through the existing DA3 decoder rather than training an additional joint RGB/depth generation objective.
• Versus DA3 and VGGT: geometric foundation models define the generation space rather than merely evaluating generated images afterward; VGGT also serves as the external pose estimator in the main paper, distinct from its alternative-backbone role in the appendix.

Rating

• Novelty: 4/5. Makes geometric multi-level representations usable for diffusion through boundary selection and cascaded alignment.
• Experimental Thoroughness: 4/5. Covers main comparisons, representation probes, boundaries, cascades, and depth, but lacks visible uncertainty estimates and complete cost accounting.
• Writing Quality: 4/5. The method is logically organized, although some protocols depend on appendices absent from the cache.
• Value: 4/5. Provides experimentally supported task-specific latent design for novel view synthesis, with comparisons that also expose its limits.