Video Generative Models as Geometry Learner

Conference: ECCV2026
Paper: ECCV Paper
Area: 3D Vision
Keywords: monocular geometry estimation, joint depth-normal prediction, video diffusion priors, image reconstruction, zero-shot generalization

TL;DR

GeoNeXt treats depth and normals from one RGB image as subsequent video-model "frames" and jointly reconstructs appearance and geometry, reducing KITTI depth AbsRel from unified generative baseline GeoWizard's 14.4 to 8.2 with only 59K synthetic training samples, although it does not outperform specialist models on every metric.

Background & Motivation

Recovering geometry from a photograph requires both the relative distances between objects and the orientations of local surfaces. Depth describes scene layout along viewing directions, whereas normals describe surface orientation at each location; these outputs are related but not interchangeable. Discriminative foundation models improve cross-scene generalization by expanding training data and pseudo-label coverage, but fine boundaries can still be lost and data and computation costs remain substantial. An alternative treats geometry maps as specialized images and fine-tunes pretrained image diffusion models with limited geometric supervision. Methods such as Marigold and Lotus show the value of generative priors for dense prediction, but maintaining separate depth and normal models misses opportunities for joint modeling.

A unified model needs more than parameter sharing: substantial changes to an image-generation backbone for cross-task communication can widen the gap from pretraining. GeoWizard and Orchid provide the joint-generation context discussed by the authors, who identify additional architecture or broader retraining as important sources of transfer cost. GeoNeXt therefore seeks a pretrained architecture that already supports interactions across output slots instead of adding another specialized geometry decoder. Video diffusion models have learned to propagate content and structure across frames, allowing their temporal attention to be repurposed to connect RGB, depth, and normals. "Time" here denotes a reusable architectural relationship, not three observations of the same scene at successive physical instants.

The authors further argue that RGB conditioning alone is insufficient: the network should also reconstruct RGB along the same denoising trajectory. Image details then become content that the model must preserve in its joint output space, rather than merely external hints. This is a design hypothesis tested by removing image reconstruction, not a rigorously established geometric law. Core Idea: organize RGB, depth, and normals as a jointly denoised sequence, reusing cross-frame video priors so that image reconstruction helps geometry prediction retain details and cross-modal consistency.

Method

Overall Architecture

GeoNeXt takes one RGB image and outputs its depth map and surface normal map. Training additionally requires paired depth and normal ground truth; testing requires neither those targets nor an input video. The system uses "Geometry Serialization," "Latent Conditioning Adaptation," and "Joint Image-Geometry Reconstruction" in that order. Its base model is Stable Video Diffusion (SVD), with a frozen image VAE and only the denoising U-Net fine-tuned. A fixed RGB latent supplies conditioning, while a separate noisy image latent participates in reconstruction alongside depth and normals.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Single RGB image"] --> SEQUENCE["Geometry Serialization"]
    SEQUENCE --> ADAPT["Latent Conditioning<br/>Adaptation"]
    ADAPT --> JOINT["Joint Image-Geometry<br/>Reconstruction"]
    TRUTH["Paired RGB / depth / normals"] -.->|Training supervision only| JOINT
    JOINT --> OUTPUT["Decode depth and normals<br/>Discard generated image"]

The diagram orders data organization and computation, not independently trained stages. During training, the frozen VAE encodes ground truth, which is corrupted to create denoising inputs and also supplies reconstruction targets; inference instead starts from random noise. Next-frame prediction does not mean fully generating depth before passing it to a normal model: the sequence is denoised jointly.

Key Designs

1. Geometry Serialization: use existing frame slots for related geometric modalities

Conventional single-task diffusion depth estimation learns to generate depth given an image, with normals requiring another model. GeoNeXt arranges the clean targets into image, depth, and normal slots so that one denoiser can access their related structures. The first slot reconstructs the input image and subsequent slots carry depth and normals, retaining the sequence organization of image-to-video generation. This arrangement preserves the video model's existing cross-frame communication paths instead of assigning geometric targets to disconnected generators. Temporal attention becomes cross-modal information exchange, allowing object boundaries, surface orientations, and distance layouts to influence shared representations. These constraints arise through learned representations and joint objectives, not an explicit analytical relationship between depth gradients and normals.

Slot order should not be mistaken for physical causality, and the paper explicitly swaps the depth and normal positions. The two orders perform similarly overall in Table 4; for example, ETH3D AbsRel is 5.5 versus 5.6, rather than all values being identical. NYUv2 depth AbsRel is 5.7 versus 5.3, showing that "flexible order" still permits measurable metric changes. A cautious interpretation is that the network does not strongly depend on one depth-normal order, while training and inference must still follow the chosen configuration. This does not establish support for arbitrary slot counts or adding new modalities without further training and evaluation. Single-task ablations additionally test whether sharing helps, rather than assuming that a three-slot design necessarily improves multi-task performance.

2. Latent Conditioning Adaptation: retain spatially aligned RGB conditioning and remove CLIP

RGB, depth, and normals are encoded by a frozen image VAE, and joint diffusion operates in compressed latent space rather than pixels. Single-channel depth is replicated across three channels to fit the RGB encoder, and the three decoded output channels are averaged afterward. The text describes normalizing depth values and applying a linear map to match the VAE input range. Writing a normalized scalar as \(u\), this transformation is:

\[ u_{\mathrm{VAE}}=2u-1,\qquad u\in[0,1]. \]

Normals also enter the VAE as images, but the cache does not clearly specify every coordinate convention and normalization detail, so it cannot establish a complete reproduction protocol. The shared encoding facilitates generator reuse, while requiring geometric details to pass through a representation originally trained for images.

For conditioning, the clean input RGB latent is replicated across slots and concatenated channel-wise with the latents being denoised. It remains fixed throughout denoising and supplies spatially corresponding appearance structure rather than only a global semantic hint. Original SVD also supports CLIP image embeddings through cross-attention; GeoNeXt disables that conditioning path. The authors argue that resizing for CLIP input can distort geometric structure and hinder detailed depth and normal prediction. The gain in Table 3 is small, so the evidence supports removing CLIP in this configuration, not the general claim that semantic conditioning harms geometry estimation. Temporal attention remains responsible for cross-slot interactions: removing CLIP conditioning does not remove all attention mechanisms.

3. Joint Image-Geometry Reconstruction: denoise appearance and geometry along a shared trajectory

If the first slot only supplies fixed conditioning, the network can generate geometry without directly reconstructing input details. GeoNeXt also corrupts the image output slot and has the U-Net predict its clean latent together with depth and normals. The system therefore contains both a fixed RGB condition and an RGB output to restore; their different roles must not be conflated. Figure 2 depicts reconstruction errors for image, depth, and normal latents, with all three outputs sharing the generation process. Training uses SVD's EDM-preconditioned denoising rather than introducing a separately trained geometry-consistency discriminator. Because several cached equations are damaged, the complete EDM weighting, preconditioning coefficients, and exact total loss are not reconstructed here.

At test time, all three generated latents, image, depth, and normals, are initialized from standard Gaussian noise. The input RGB image is encoded once as fixed conditioning, while each denoising step updates all three output slots until final latents are obtained. The generated image is then discarded and only geometry is decoded and returned; image reconstruction is an auxiliary prediction aid, not a new photograph to deliver. The default configuration uses 5 denoising steps and ensembles predictions from 5 different random seeds. Random-seed ensembling makes this more expensive than one forward pass, and the paper's single-step results belong to a different operating configuration. Removing image reconstruction worsens both NYUv2 depth and normals, providing the most direct ablation evidence for this joint-generation design.

A Worked Example

Consider an indoor RGB photograph containing a chair and a wall; this illustrates the method and is not an additional experiment. During training, the image, ground-truth depth, and ground-truth normals are encoded separately into three noisy target slots. The clean latent of the same RGB image is replicated as conditioning, helping locate chair-back boundaries and wall regions. The network must recover color structure in those regions while restoring distances and surface directions in the geometric slots. At test time, geometry ground truth is unavailable, all three output slots start from random noise, and fixed RGB conditioning still comes from the original image. After 5 joint denoising steps and ensembling predictions from 5 random seeds, the model delivers depth and normals, not the generated image. This example explains why the auxiliary image slot remains during inference instead of being removed after training.

Loss & Training

Only the denoising U-Net is updated; the VAE is frozen, Adam uses a learning rate of \(5\times10^{-6}\), and augmentation uses random horizontal flips. Hypersim retains approximately 39K complete samples from about 54K official training examples, with inputs resized to \(576\times768\). Virtual KITTI 2 contributes about 20K samples from four of its five urban scenes, cropped to \(352\times1216\), with an 80 m far-plane depth limit. The combined set contains about 59K samples, but training batches select the datasets at 9:1, with 90% probability for Hypersim and 10% for Virtual KITTI. This differs from the raw sample-count ratio and biases the training distribution toward indoor data. The depth implementation operates in disparity space, using the relationship stated in the text:

\[ d=1/d'. \]

Here \(d\) is predicted disparity and \(d'\) is the corresponding depth; evaluation concerns affine-invariant depth, not uncalibrated absolute metric depth. The text writes noise sampling as \(p(\sigma)=\mathcal{N}(0.7,1.6)\) without clearly stating whether this denotes a log-noise variable, so it should not be treated as an executable sampling specification. The main paper does not clearly provide total training steps, batch size, or all geometric preprocessing details; full reproduction still requires implementation or supplementary material.

Key Experimental Results

Main Results

None of the evaluation datasets participates in the paper's geometric fine-tuning; "zero-shot" means cross-dataset evaluation, not training without geometric supervision. The following excerpt comes from Table 1, page 9, evaluating monocular affine-invariant depth. AbsRel is mean absolute relative error, retained on the table's percentage scale, with lower being better; \(\delta_1\) is the fraction of pixels whose prediction-to-ground-truth ratio error is below \(1.25\), with higher being better. Each cell lists AbsRel / \(\delta_1\), both as percentages; the GeoWizard asterisk denotes reproduction by the authors.

Dataset	GeoWizard*, 208K	Lotus-G*, 59K	GeoNeXt, 59K
NYUv2	5.6 / 96.3	5.4 / 96.6	5.3 / 96.8
KITTI	14.4 / 82.0	8.5 / 92.2	8.2 / 92.6
ETH3D	6.8 / 95.8	5.9 / 97.0	5.6 / 97.2
ScanNet	6.4 / 95.2	5.9 / 95.6	5.9 / 95.8
DIODE	33.0 / 73.5	23.0 / 73.0	22.6 / 74.3

The second table excerpts Table 2, page 12, for zero-shot surface normal estimation. Each cell lists mean angular error in degrees, lower is better, and the percentage of pixels with angular error below \(11.25^\circ\), higher is better. Lotus-G here is a normal-specific model, not the depth model in the previous table; two specialist checkpoints must not be treated as one joint model.

Dataset	GeoWizard*, 208K	Lotus-G*, 59K	GeoNeXt, 59K
NYUv2	18.9 / 50.1	16.6 / 59.4	16.7 / 60.0
ScanNet	17.2 / 53.8	15.1 / 63.8	16.0 / 62.8
iBims-1	19.4 / 62.8	17.2 / 66.3	16.4 / 69.2
Sintel	40.3 / 12.8	33.6 / 21.0	33.0 / 21.5
OASIS	25.0 / 23.7	22.9 / 29.3	22.8 / 30.8

Ablation Study

The following excerpt comes from Table 3, page 13; depth columns report AbsRel in percent and normal columns report mean angular error in degrees, both lower is better. Unified configurations predict both geometric targets, whereas single-task configurations train only the corresponding target; the original table's iBims-N label is retained.

Config	NYUv2 depth	ETH3D depth	NYUv2 normals	iBims-N normals
Without image reconstruction	6.5	6.7	17.9	18.5
With CLIP embedding	5.6	5.8	16.9	16.5
Full model	5.3	5.6	16.7	16.4
Depth only	5.9	6.4	Not applicable	Not applicable
Normals only	Not applicable	Not applicable	17.2	17.3

Key Findings

Relative to GeoWizard, GeoNeXt reduces KITTI AbsRel by 6.2 percentage points; \(\delta_1\) rises from 82.0% to 92.6%, an actual difference of 10.6 percentage points. The prose on page 11 calls the latter improvement 9.4, inconsistent with Table 1; the table is retained and the discrepancy explicitly noted here. Removing image reconstruction raises NYUv2 AbsRel from 5.3 to 6.5 and normal mean angular error from 16.7 to 17.9, affecting both targets. CLIP and single-task ablations support the chosen configuration, but do not independently control pretraining scale, so not all gains can be attributed to temporal attention. Table 5, page 14, reports GeoNeXt on one NVIDIA A5000 with \(768\times768\) input: the \(1\times1\) configuration takes 1.0 s and \(5\times5\) takes 10.0 s, both with 1.5B parameters. Their ETH3D AbsRel values are 5.8 and 5.6, respectively; extra sampling improves accuracy, but main-result quality must not be advertised together with single-step latency. NFEs are defined as denoising steps multiplied by ensemble size; reported runtimes exclude I/O overhead for switching independent models.

Highlights & Insights

Reusing frames as modality containers keeps the adaptation closer to video pretraining than introducing separate geometry branches; the transfer concerns input-output organization. RGB reconstruction places appearance inside the joint prediction objective, and Table 3 provides supporting evidence beyond visual examples. Keeping an auxiliary slot during inference illustrates that an ultimately discarded output can still participate in intermediate computation needed by other predictions. Joint training has implications for both accuracy and checkpoint management, but sharing one model does not automatically make inference fastest.

Limitations & Future Work

Data efficiency concerns geometric fine-tuning only: 59K excludes the large image and video corpora already consumed by SVD pretraining, so it does not establish low total training cost. Table 1 lists 62.6M samples for DA/DA-V2, whereas page 10 says 63.5M, and "nearly 100 times" does not match the direct ratio of these values to 59K; this note does not adopt that factor claim. Normal estimation is not uniformly superior: ScanNet mean error is 15.1 degrees for Lotus-G versus 16.0 for GeoNeXt, and E2E-FT reaches 14.7 degrees in Table 2. NYUv2 normal threshold accuracy is 60.0% in Table 2 but 60.2% for the full configuration in Tables 3 and 4; the paper does not explain the difference, and source-specific values are retained. Several cached mathematical expressions are damaged, and the cited WAN/DiT supplementary experiments are absent, preventing verification of cross-backbone claims and full noise parameterization. Readers could test whether explicit depth-normal consistency constraints help, but that is a future research direction, not an implemented component of this paper. Absolute scale recovery, robustness to different camera intrinsics, and fine-structure retention at lower sampling budgets require further evaluation.

Related Work & Insights

Marigold / Lotus: transfer image diffusion priors to geometric prediction; GeoNeXt differs by jointly using video frame slots and including RGB in its denoising targets. GeoWizard / Orchid: also address joint geometry or appearance-geometry generation; this paper emphasizes minimal architectural changes, but its main tables do not directly compare against Orchid. DepthCrafter / Geo4D: use video generation priors for temporal consistency or geometric reconstruction; GeoNeXt mainly evaluates single-image cross-modal prediction, not superior long-video consistency. Depth Anything / DSINE: represent discriminative depth or normal models; comparisons locate accuracy, but training scales, task settings, and pretraining sources differ. A transferable lesson is to identify an existing information-exchange dimension in a pretrained model, map related dense targets into it, and ablate whether sharing genuinely helps. By its actual task, this paper fits 3D Vision better than Image Generation: the generator is the means, while depth and normals are the primary deliverables.

Rating

• Novelty: 4/5. Repurposing video slots for monocular geometry while retaining joint image reconstruction provides a clear task reformulation.
• Experimental Thoroughness: 4/5. Two tasks, multiple cross-domain datasets, and key ablations are covered, but pretraining-scale attribution and supplementary results remain verification gaps.
• Writing Quality: 3/5. The overall pipeline is understandable, but metric gains, training data counts, and some parameter descriptions are inconsistent.
• Value: 4/5. A reusable design for unified dense prediction with limited geometric supervision, with deployment still requiring a sampling-cost trade-off.