content_hash: 6fda3d63595a3a02

MetaView: Monocular Novel View Synthesis with Scale-Aware Implicit Geometry Priors¶

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Area: 3D Vision
Keywords: Monocular Novel View Synthesis, Scale-Aware RoPE, Implicit Geometry Priors, Dense Matching Distance, Diffusion Transformer

TL;DR¶

MetaView establishes a monocular novel view synthesis framework under large viewpoint changes without explicit 3D reconstruction, by injecting intermediate feed-forward geometry priors into a frozen MM-DiT via parallel attention and embedding metric depth with camera poses into scale-aware RoPE, effectively resolving both scale drifting and reconstruction degradation.

Background & Motivation¶

Data-driven visual generation models based on diffusion backbones have emerged as a prominent paradigm for novel view synthesis (NVS). However, synthesizing novel views from a single source image (monocular NVS) under large viewpoint changes remains an unresolved challenge, where existing methods struggle between explicit geometric constraints and purely implicit generation. On one hand, two-stage reconstruction-based pipelines—such as ViewCrafter, Gen3C, Voyager, and PE-Field—rely on explicit sparse point clouds or depth reprojection/warping to provide geometric conditions. This setup essentially restricts the diffusion model to localized image inpainting rather than spatial reasoning, suffering from catastrophic visual degradation, holes, and blurriness when explicit reconstructions fail under extreme viewpoint shifts.

On the other hand, interactive world generation models, including HY-World and Lingbot-World, explore purely implicit scene modeling by directly conditioning the network on target camera poses. While bypassing the fragile assumptions of explicit 3D reconstruction, these purely implicit architectures rely entirely on 2D pixel-level supervision and lack calibrated 3D geometric perception. Consequently, their internal scene representations suffer from severe scale ambiguity, resulting in noticeable scale drifting and spatial inconsistency during camera translation and view trajectory control.

This paper argues that advancing monocular large-viewpoint synthesis requires transcending this false dichotomy. The key lies in decoupling geometric perception from image generation: incorporating feed-forward geometric priors to regularize relative scene structures without imposing brittle reconstruction steps, while injecting minimal yet essential explicit metric scale cues to ground the spatial coordinate frame. Core idea: MetaView freezes a pretrained multi-modal diffusion transformer (MM-DiT) backbone, integrates hierarchical implicit geometry priors extracted from a feed-forward foundation model via non-invasive parallel attention layers, and allocates an extra metric depth subspace alongside camera poses within Rotary Positional Encoding (RoPE), achieving precise camera controllability and high-fidelity generalization under extreme viewpoint variations.

Method¶

Overall Architecture¶

Given a single source image $X_{\text{src}}$, camera intrinsics $K$, and a relative target camera pose $T=(R, t) \in \text{SE}(3)$ with 6 degrees of freedom, MetaView aims to synthesize the scene from the target perspective $X_{\text{gen}}$. The architecture is built upon Qwen-Image-Edit, a pretrained multi-modal diffusion transformer (MM-DiT) operating under the flow matching formulation. All backbone parameters are frozen to prevent catastrophic forgetting. The framework extracts multi-level intermediate features from the feed-forward geometry model DepthAnything3 to form geometry tokens $G$, injects camera pose matrices and discrete metric depth $(i, j, z)$ into attention via a modified 3D RoPE, and incorporates these geometric cues through lightweight parallel image-geometry attention layers across transformer blocks.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Monocular Input X_src<br/>& Target Camera Pose T"] --> B["Feed-forward Geometry Extraction & Tokenization<br/>DepthAnything3 Implicit Geometry Prior G"]
    A --> C["Metric Depth Estimation & Scale Injection<br/>Discrete Sparse Depth z' Extraction"]
    B --> D["Scale-Aware 3D RoPE Injection<br/>Camera Pose Projection + (i, j, z) Rotary Encoding"]
    C --> D
    D --> E["Parallel Dual-Stream Attention Fusion<br/>Text-Image Attention + Geometry-Image Attention"]
    E --> F["Flow Matching Denoising<br/>Backbone Weights Fully Frozen"]
    F --> G["High-Fidelity Novel View Image X_gen"]

Key Designs¶

1. Scale-Aware 3D RoPE Injection: Mitigating Scale Drifting in Implicit Generation
To address the severe translation offset and scale drifting that plague purely implicit generative models lacking physical ground truth, MetaView avoids handcrafted depth-lifting or explicit voxel volumes. Instead, it directly grounds metric scale within the positional encoding. The method leverages an off-the-shelf DepthAnything3-Metric network to predict a metric depth map $z$, which is downsampled to $z'$ to match the spatial token resolution of the latent representations. Following PRoPE, the rotary positional encoding is decomposed into a frustum projection subspace and a grid coordinate subspace: $$D_t^{\text{RoPE}} = \begin{bmatrix} D_t^{\text{Proj}} & \mathbf{0} \\ \mathbf{0} & D_t^{\text{Grid}} \end{bmatrix}$$ The projection component $D_t^{\text{Proj}} = \mathbf{I}_{d/8} \otimes \tilde{P} \in \mathbb{R}^{\frac{d}{2} \times \frac{d}{2}}$ encodes camera intrinsics $K$ and extrinsics $T$, while the coordinate component $D_t^{\text{Grid}}$ expands the conventional 2D spatial grid $(i_t, j_t)$ with an allocated $d/6$-dimensional subspace for the metric depth $z_t$: $$D_t^{\text{Grid}} = \begin{bmatrix} \text{RoPE}_{d/6}(i_t) & \mathbf{0} & \mathbf{0} \\ \mathbf{0} & \text{RoPE}_{d/6}(j_t) & \mathbf{0} \\ \mathbf{0} & \mathbf{0} & \text{RoPE}_{d/6}(z_t) \end{bmatrix} \in \mathbb{R}^{\frac{d}{2} \times \frac{d}{2}}$$ Source image tokens $X_{\text{src}}$ and geometry tokens $G$ are anchored at origin coordinates $T^o = \mathbf{I}$ and bound to the estimated sparse metric depth $z'$. In contrast, target generation tokens $X_{\text{gen}}$ are assigned the relative target pose $T$ with their $z$-coordinate zeroed out ($\mathbf{0}$), avoiding overly rigid constraints on unobserved regions while ensuring cross-view attention interactions align with consistent physical metric scale.

2. Implicit Geometry Prior Tokenization: Preserving Relative Structures with Strong Generalization
Explicit 3D representations often suffer from structural tears, severe occlusion holes, and blurriness when novel viewpoints differ sharply from the input perspective. To preserve fine-grained topological relations without restrictive geometric lifting, MetaView harnesses the rich structural representations learned by large feed-forward 3D perception foundation models. Specifically, intermediate representations $\{f^\phi\}_{\phi \in \text{decode}}$ from the deep layers of DepthAnything3—originally fed into its 3D prediction decoder—are extracted, concatenated along the channel dimension, and projected to dimension $d$ using a learnable linear layer $W_G$: $$G = W_G([\ \{f^\phi\}_{\phi \in \text{decode}}\ ]) \in \mathbb{R}^{L \times d}$$ These geometry tokens $G$ preserve fine-grained structural topology and relative spatial layouts. Generated view tokens, conditioned on relative camera pose $T$, attend to $G$ defined under $T^o$, effectively suppressing contour distortions, warped text signboards, and depth inversion artifacts while retaining the open-world inpainting and outpainting capabilities of generative diffusion models.

3. Non-Invasive Parallel Attention Adaptation: Retaining Pretrained Semantic Knowledge
To prevent catastrophic forgetting and preserve the broad semantic priors acquired during massive text-to-image pretraining, MetaView strictly freezes the entire pretrained MM-DiT backbone (Qwen-Image-Edit). Inside each multi-modal attention block, the original text-image self-attention pathway remains untouched, and a parallel image-geometry attention layer is introduced to process interactions between image tokens $X$ and geometry tokens $G$: $$\text{Out} = \text{Attention}([X; C]) + \text{Attention}([X; G])$$ where $X = [X_{\text{src}}; X_{\text{gen}}]$ and $C$ denotes text prompt tokens. The newly added parallel branch introduces dedicated query, key, value, and output projection matrices ($W_X^Q, W_X^K, W_X^V, W_X^O$ and $W_G^Q, W_G^K, W_G^V, W_G^O$). Geometry tokens share the existing frozen feed-forward network (FFN) with image tokens to minimize parameter overhead. Only these projection weights and the mapping layer $W_G$ are updated during training.

4. Dense Matching Distance (DMD) Metric: Faithful Evaluation for Extrapolative Synthesis
Traditional low-level metrics such as PSNR and SSIM are dominated by low-frequency structural matching, frequently assigning higher scores to blurry, over-smoothed images while penalizing sharp, visually faithful generations under large viewpoint changes. To evaluate spatial alignment under extreme camera variations, the authors propose Dense Matching Distance (DMD). Using the unified optical flow matching model UFM, DMD identifies co-visible dense correspondences $P_{\text{src}}$ and $P_{\text{gen}}$ between the ground truth image $X_{\text{GT}}$ and the source and generated images, respectively: $$\text{DMD}(X_{\text{src}}, X_{\text{gen}}, X_{\text{GT}}) = \frac{1}{|P_{\text{src}}|} \sum_{p \in P_{\text{src}}} \begin{cases} \|\text{dist}(p)\|, & p \in P_{\text{gen}} \\ \sigma, & p \notin P_{\text{gen}} \end{cases}$$ where $\text{dist}(p)$ denotes the optical flow displacement, and unassigned points incur a maximum penalty $\sigma$. Scaled from $0$ to $100$, lower DMD values indicate superior spatial alignment and accurate camera pose control. Unlike PSNR/SSIM, DMD demonstrates strict monotonic degradation as viewpoint disparity widens.

Loss & Training¶

The newly introduced parameters are optimized end-to-end under the standard flow matching objective: $$\mathcal{L}_{\text{FM}}(\theta) = \mathbb{E}_{t, x_0, x_1, C, G} \|v_\theta(x_t, t, C, G) - (x_1 - x_0)\|_2^2$$ All original MM-DiT backbone weights remain frozen. Text instructions are fixed to the generic prompt "Turn to the target view". Optimization is performed using AdamW with a learning rate of $5 \times 10^{-5}$ across 8 NVIDIA H800 GPUs with a total batch size of 32. At inference time, the model executes 40 flow matching sampling steps with a classifier-free guidance (CFG) scale of 4.

Key Experimental Results¶

Main Results¶

Quantitative evaluations are conducted on DL3DV (split into Easy: overlap $>80\%$, Medium: $50\%\sim80\%$, and Hard: $30\%\sim50\%$), RealEstate10K, and Sekai-Real-Walking-HQ datasets, benchmarking against reconstruction-based approaches (ViewCrafter, Gen3C, Voyager, PE-Field) and implicit world models (HY-World-1.5, Lingbot-World).

Table 1: Quantitative comparisons across difficulty splits on DL3DV (Easy / Medium / Hard)

Method	DL3DV-Easy PSNR↑	Easy SSIM↑	Easy LPIPS↓	Easy DMD↓	DL3DV-Med PSNR↑	Med SSIM↑	Med LPIPS↓	Med DMD↓	DL3DV-Hard PSNR↑	Hard SSIM↑	Hard LPIPS↓	Hard DMD↓
ViewCrafter	15.45	0.4454	0.2037	8.84	13.19	0.3679	0.2733	24.35	11.68	0.3449	0.3223	48.83
Gen3C	15.32	0.4339	0.1964	6.52	14.14	0.4092	0.2556	12.43	11.87	0.3424	0.3089	41.07
Voyager	15.19	0.4171	0.2530	12.21	13.45	0.3610	0.2886	23.89	11.24	0.3302	0.3462	54.26
PE-Field	15.01	0.3964	0.2129	8.91	13.18	0.3435	0.2749	15.41	11.97	0.3422	0.3278	41.46
HY-World-1.5	12.27	0.3323	0.2906	14.58	11.34	0.2948	0.3304	18.50	10.71	0.3043	0.3520	34.45
Lingbot-World	11.82	0.3191	0.3073	29.20	11.99	0.2912	0.3141	21.37	11.13	0.3319	0.3447	55.39
MetaView (Ours)	17.94	0.5542	0.1397	2.56	15.05	0.4225	0.2132	7.57	12.54	0.3802	0.2881	20.74

Table 2: Cross-dataset quantitative comparisons across diverse domains

Method	DL3DV PSNR↑	DL3DV SSIM↑	DL3DV LPIPS↓	DL3DV DMD↓	RE10K PSNR↑	RE10K SSIM↑	RE10K LPIPS↓	RE10K DMD↓	Sekai PSNR↑	Sekai SSIM↑	Sekai LPIPS↓	Sekai DMD↓
ViewCrafter	13.44	0.3861	0.2664	27.34	13.16	0.4994	0.2597	10.92	15.73	0.4388	0.3068	17.93
Gen3C	13.78	0.3952	0.2536	20.01	14.07	0.5162	0.2306	9.01	16.56	0.4659	0.2568	8.72
Voyager	13.29	0.3694	0.2959	30.12	14.55	0.5170	0.2519	12.95	16.24	0.4402	0.3205	19.07
PE-Field	13.39	0.3607	0.2719	21.93	14.51	0.5191	0.2354	14.81	15.74	0.4418	0.2900	19.35
HY-World-1.5	11.44	0.3105	0.3243	22.51	12.34	0.4466	0.3040	13.91	14.62	0.4088	0.3240	15.04
Lingbot-World	11.65	0.3141	0.3220	35.32	12.26	0.4554	0.3026	24.51	14.52	0.3973	0.3258	18.71
MetaView (Ours)	15.18	0.4456	0.2137	10.29	15.27	0.5705	0.1980	6.44	17.72	0.5047	0.2333	6.69

Ablation Study¶

Ablations on DL3DV-Medium systematically evaluate the contribution of geometry tokens, the metric $z$-axis in RoPE, and the underlying diffusion backbone.

Table 3: Ablation study on DL3DV-Medium

Config	PSNR↑	SSIM↑	LPIPS↓	DMD↓	Note
Ours (Full model)	14.21	0.3765	0.2353	9.33	Full model with Qwen-Image-Edit backbone
Ours w/o geometry	13.53	0.3538	0.2530	11.61	Drops geometry tokens; fine structural relations degrade
Ours w/o z-axis	12.49	0.3204	0.2900	14.45	Drops metric depth in RoPE; suffers from scale drifting
Ours w/ FLUX.1-Kontext	13.90	0.3594	0.2571	10.22	Replaces backbone with FLUX.1-Kontext; retains superiority
PE-Field (FLUX.1-Kontext)	13.18	0.3435	0.2749	15.41	Baseline using same FLUX.1 backbone

Key Findings¶

Metric Depth ($z$-axis) is Essential to Prevent Scale Drifting: Ablating the $z$-axis subspace in RoPE drops PSNR from 14.21 to 12.49 and worsens DMD from 9.33 to 14.45. Visual inspection confirms that while camera rotation is roughly maintained, camera translation suffers from substantial scale drift without metric anchors.
Implicit Geometry Priors Preserve Fine Structural Consistency: Removing geometry tokens (w/o geometry) causes severe degradation in object contours, relative object arrangements, and fine-grained text signboards, confirming that feed-forward 3D representations supply critical spatial inductive biases beyond 2D pixel supervision.
DMD Distinguishes Geometric Alignment Where Pixel Metrics Fail: On DL3DV-Hard, PSNR values cluster tightly between 10.71 and 12.54 dB across all methods, whereas DMD clearly highlights the performance divide, with MetaView achieving 20.74 compared to 34.45–55.39 for baseline competitors.

Highlights & Insights¶

Decoupled Perception and Generation: Rather than forcing the diffusion model to perform 3D inpainting over noisy point clouds, MetaView freezes the generative foundation model and channels 3D spatial priors via parallel attention, retaining open-domain image synthesis quality.
Minimalist Scale Grounding via RoPE: Instead of computing expensive 3D lifting operations or volumetric grids, injecting a single downsampled metric depth coordinate $z'$ into Rotary Position Embeddings cleanly anchors the internal spatial representation to real-world metric dimensions.
Principled Evaluation Metric for Large View Changes: By tracking co-visible dense optical flow correspondences with UFM, the proposed DMD metric provides a monotonic, human-aligned measure of camera control accuracy under extreme view rotations.

Limitations & Future Work¶

Static Scene Restriction: The model assumes rigid, static scenes and does not model 4D dynamic motion or temporal evolution.
Hallucinations in Extreme Occlusion Blindspots: When view rotations approach 180 degrees where zero visual overlap exists with the source frame, generated content relies primarily on diffusion priors, introducing plausible hallucinations lacking physical verification.
Future Directions: Extending the scale-aware implicit geometry prior to 4D dynamic video diffusion and real-time interactive world simulators.

vs ViewCrafter / Gen3C / Voyager / PE-Field: Explicit reconstruction methods rely on point clouds or depth warping before diffusion inpainting; when viewpoint variations are severe, sparse geometry collapses, leading to blurry artifacts. MetaView operates entirely implicitly in latent space, achieving vastly superior robustness under large view changes.
vs HY-World / Lingbot-World: Purely implicit world models conditioned on camera poses lack metric scale awareness, causing pervasive scale drifting during translation; MetaView resolves this by anchoring scene scale with metric depth RoPE.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Elegant integration of feed-forward implicit geometry priors and scale-aware RoPE that bypasses explicit reconstruction bottlenecks.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across multiple benchmarks and difficulty splits, supported by a novel, well-validated DMD metric.
Writing Quality: ⭐⭐⭐⭐⭐ Clearly written with coherent technical narrative, well-formulated equations, and intuitive figures.
Value: ⭐⭐⭐⭐⭐ Highly impactful for monocular view synthesis, 3D perception foundation models, and controllable world generation.