NEOMAP: Novel-View Synthesis via Noise Initialization by Manifold Alternating Projection¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/vLAR-group/NeoMap
Area: 3D Vision
Keywords: novel-view synthesis, video generation model, flow matching, manifold alternating projection, training-free

TL;DR¶

To tackle view misalignment and generation artifacts in single-image and monocular video novel-view synthesis, NeoMap discovers high-fidelity view-consistent solutions directly within the native video data manifold of pre-trained models by optimizing the initial Gaussian noise via manifold alternating projection iterations without task-specific training or online guidance.

Background & Motivation¶

Synthesizing photorealistic novel views from a single image or unconstrained monocular video (Novel View Synthesis, NVS) is a foundational challenge in computer vision with extensive applications across AR/VR, 3D/4D asset creation, and robotic spatial perception. However, monocular inputs intrinsically suffer from severe visual occlusions, incomplete scene geometry, and inherent depth ambiguities. Conventional 3D reconstruction paradigms, such as NeRF and 3D Gaussian Splatting, struggle drastically under sparse or single-view inputs, frequently producing severe floaters, hollow cavities, and geometric distortion. Recently, large-scale video generation foundation models trained on massive real-world video corpora have demonstrated an exceptional understanding of spatial-temporal physical continuity, opening up a compelling avenue for addressing monocular NVS.

Nonetheless, current methods built on video generation models predominantly operate under a shared assumption: pre-trained video models lack native novel view synthesis capabilities and therefore require explicit restructuring of the generative mapping relation. Under this assumption, prior efforts diverge into two paradigms: the first fine-tunes diffusion models with explicit camera parameters or trajectory conditions (e.g., TrajectoryAttention, ReCamMaster), which inevitably overfits to the narrow camera motion distributions and depth estimation noise present in training sets, leading to poor in-the-wild generalization; the second adopts training-free warping-and-inpainting heuristics (e.g., NVS-Solver), step-by-step blending warped geometry priors into the denoising latents. Crucially, such aggressive stepwise latent manipulation breaks the native, continuous generation dynamics of the pre-trained model, inducing incomplete structures, warped geometry, and semantic collapse.

This paper proposes an orthogonal perspective that challenges this prevailing mindset: foundational video generation models trained on diverse real-world scenes already possess rich representations of 3D geometry and natural camera trajectories within their native output data manifold. Valid, geometrically consistent novel-view solutions already exist within this learned manifold, and the core challenge is simply locating the optimal solution. Because modern diffusion and flow-matching models constitute a deterministic Ordinary Differential Equation (ODE) mapping from standard Gaussian noise space to the video data manifold, discovering a specific video solution is mathematically equivalent to locating its corresponding initial noise latent. Core idea: formulate monocular novel-view synthesis as an initial noise optimization problem, leveraging convergent manifold alternating projection iterations between semantic manifold completion and geometric pixel constraints to guide the initial noise toward the intersection of the natural video manifold and the 3D geometric constraint subspace.

Method¶

Overall Architecture¶

NeoMap takes as input a single reference image or monocular video along with a user-specified target camera trajectory, and outputs a temporally consistent, photorealistic video synthesized from the novel views. The pipeline first uses visual foundation models (VGGT, VIPE) to jointly estimate per-frame depth maps and camera extrinsics, then performs 3D backward warping to project reference pixels onto the target trajectory, constructing a partial novel-view prior accompanied by a binary visibility mask. Next, the pipeline enters its core manifold alternating projection loop, iteratively refining the noise latent between a near-noise timestep and the clean data manifold. Finally, the optimized noise is decoded via a standard ODE solver equipped with an auxiliary trajectory re-anchoring mechanism during early denoising steps.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Monocular Reference Input<br/>Image/Video + Target Trajectory"] --> B["Novel View Prior Module<br/>Depth/Pose Estimation and 3D Warping"]
    B --> C["Anchored Manifold Projection<br/>One-Step Euler Reverse Flow & Similarity Blending"]
    C --> D["Pixel-Constrained Projection<br/>Pixel-Space Masked Overwrite & VAE Re-encoding"]
    D -->|Alternating Projections over K Iterations| E["Optimal Initial Noise<br/>Converged to Data Manifold & Geometry Intersection"]
    E --> F["Auxiliary Trajectory Re-anchoring<br/>Early Denoising Latent Calibration to Counter Drift"]
    F --> G["Target Novel-View Video<br/>High Fidelity & Strict View Consistency"]

Key Designs¶

1. Novel View Prior Module: Establishing Initial Noisy Latents from Warped Geometric Priors

To anchor the unconstrained generative process to the reference observation, the framework first employs pre-trained geometric foundation models to recover dense depth maps and camera parameters from the source view. Using the specified target camera trajectory, a 3D forward-backward warping operator projects visible source pixels onto each target viewpoint, yielding a partial novel-view prior $\boldsymbol{X}_c$ and an aligned binary visibility mask $\boldsymbol{M}_c$. Passing this video prior through the pre-trained VAE encoder produces the latent prior $\boldsymbol{z}_c = \text{Enc}(\boldsymbol{X}_c)$. Directly utilizing forward-diffused $\boldsymbol{z}_c$ as initial noise yields severe generation degradation because depth inaccuracies cause out-of-distribution artifacts in visible areas, while unobserved regions remain empty and unconstrained. Consequently, the framework maps $\boldsymbol{z}_c$ to an early near-noise timestep $\tau = 1 - \Delta t$ via forward flow, defining the initial state $\boldsymbol{z}_\tau^0$ for alternating projection optimization.

2. Anchored Manifold Projection: Hallucinating Unseen Semantics via One-Step Reverse Flow and Latent Blending

The Anchored Manifold Projection (AMP) module pushes the noisy state toward the natural video data manifold $\mathcal{M}$ to plausibly complete unobserved regions while rectifying visible artifacts using pre-trained video generation priors. To avoid the computational bottleneck of multi-step ODE integration during each iteration, AMP adopts a large-stride one-step linear Euler approximation of the generative reverse flow: $$\boldsymbol{z}_\mathcal{M}^k = \Phi_{\tau \to 0}(\boldsymbol{z}_\tau^k) \approx \boldsymbol{z}_\tau^k - \tau \boldsymbol{v}_\theta(\boldsymbol{z}_\tau^k, \tau)$$ Although the spatial-temporal attention inside the velocity field $\boldsymbol{v}_\theta$ broadcasts contextual information to unobserved holes, it also inadvertently diffuses noisy, unconstrained signals back into reliable visible regions. To strictly safeguard source geometry, AMP measures the feature-level cosine similarity between the manifold prediction $\boldsymbol{z}_\mathcal{M}^k$ and the original prior $\boldsymbol{z}_c$, deriving an adaptive similarity mask $\boldsymbol{S}_\lambda$: $$\boldsymbol{S}_\lambda = \mathbb{I}(\text{CosSim}(\boldsymbol{z}_\mathcal{M}^k, \boldsymbol{z}_c) > \lambda)$$ $$\hat{\boldsymbol{z}}_\mathcal{M}^k = \boldsymbol{S}_\lambda \odot \boldsymbol{z}_c + (1 - \boldsymbol{S}_\lambda) \odot \boldsymbol{z}_\mathcal{M}^k$$ This dynamic feature blending retains the model's coherent hallucinations across artifact-prone regions while adaptively enforcing the geometric prior on high-confidence features. The harmonized latent is subsequently mapped back to timestep $\tau$ via forward noising.

3. Pixel-Constrained Projection: Enforcing Rigid Geometry Boundaries against Latent Bleeding

While AMP successfully injects semantic realism, the similarity mask in early iterations can be overly sparse, and the lossy compression of the VAE latent space inherently causes spatial bleeding and boundary blurring during latent blending, pushing the code off the geometric constraint subspace $\mathcal{V}$. The Pixel-Constrained Projection (PCP) module rectifies this by decoding $\hat{\boldsymbol{z}}_\mathcal{M}^k$ back into the pixel space, strictly enforcing the source observation using the exact visibility mask $\boldsymbol{M}_c$, and re-encoding back to latent space: $$\tilde{\boldsymbol{z}}_\mathcal{M}^k = \text{Enc}\big(\boldsymbol{M}_c \odot \boldsymbol{X}_c + (1 - \boldsymbol{M}_c) \odot \text{Dec}(\hat{\boldsymbol{z}}_\mathcal{M}^k)\big)$$ From a theoretical perspective, AMP acts as a projection operator onto the unconstrained natural video manifold $\mathcal{M}$, whereas PCP acts as a projection operator onto the strict geometric constraint subspace $\mathcal{V}$. Neither projection satisfies both properties in isolation, but by iteratively alternating between them ($\boldsymbol{z}_\tau^k \to \hat{\boldsymbol{z}}_\mathcal{M}^k \to \tilde{\boldsymbol{z}}_\mathcal{M}^k \to \boldsymbol{z}_\tau^{k+1}$), the classical principle of alternating projections onto sets mathematically guarantees monotonic convergence toward the optimal intersection $\mathcal{M} \cap \mathcal{V}$.

4. Auxiliary Trajectory Re-anchoring: Countering Integration Drift During Early Generation Timesteps

Although alternating projection produces an optimized initial noise latent $\boldsymbol{z}_\tau^*$, the linear Euler approximation employed during projection inevitably incurs integration errors. Given the high non-linearity of the learned vector field $\boldsymbol{v}_\theta$, standard ODE solving starting from $\boldsymbol{z}_\tau^*$ can gradually drift away from the geometric constraints over time. Recognizing that diffusion and flow models establish global scene layouts, camera trajectories, and low-frequency structures predominantly in their earliest generation steps, NeoMap executes a lightweight AMP calibration step at intermediate timesteps ($t = \tau' < \tau$) during the first 3 denoising steps. This re-anchors the trajectory back to the NVS constraints without interfering with high-frequency appearance synthesis in later steps.

Key Experimental Results¶

Main Results¶

The framework is implemented on the open-source video foundation model Wan2.2-I2V-A14B using a vanilla UniPC solver with only 15 total sampling steps. Evaluations cover static single-image NVS across Tanks-and-Temples (large baseline camera trajectories) and LLFF (complex forward-facing trajectories), as well as dynamic monocular video NVS on the DAVIS benchmark.

Table 1: Quantitative comparison on Tanks-and-Temples and LLFF benchmarks (Static Single-Image NVS)

Dataset	Method	PSNR ↑	SSIM ↑	LPIPS ↓	FID-2048 ↓	CLIP-S ↑	ATE ↓
Tanks and Temples	TrajectoryAttention	13.064	0.577	0.522	151.418	0.899	0.061
	ReCamMaster	11.549	0.533	0.594	139.487	0.903	0.099
	ViewCrafter	13.675	0.590	0.564	133.219	0.902	0.010
	FlexWorld	14.655	0.601	0.531	92.686	0.918	0.005
	NVS-Solver	12.356	0.547	0.548	131.179	0.892	0.079
	LanPaint	13.299	0.544	0.575	196.341	0.834	0.006
	Reversed Noise	10.715	0.435	0.520	160.994	0.902	0.029
	Flow Warped Noise	11.353	0.510	0.552	130.484	0.909	0.080
	NeoMap (Ours)	15.250	0.596	0.486	83.341	0.935	0.008
LLFF	TrajectoryAttention	12.557	0.356	0.498	123.126	0.923	1.644
	ReCamMaster	10.928	0.312	0.561	173.737	0.885	1.300
	ViewCrafter	11.573	0.344	0.594	135.944	0.907	0.864
	FlexWorld	13.119	0.378	0.538	97.500	0.930	0.618
	NVS-Solver	11.418	0.304	0.531	150.368	0.895	1.925
	LanPaint	11.946	0.334	0.506	196.799	0.862	0.543
	Reversed Noise	10.866	0.287	0.505	148.187	0.923	1.142
	Flow Warped Noise	10.678	0.261	0.538	130.105	0.941	2.011
	NeoMap (Ours)	13.385	0.360	0.432	80.888	0.953	0.551

On the dynamic DAVIS benchmark, NeoMap achieves an FVD of 234.634, FID-2048 of 68.864, and ATE of 0.534, markedly outperforming previous state-of-the-art methods including FlexWorld (FVD 243.930, ATE 0.805) and NVS-Solver (FVD 341.127, ATE 0.996).

Ablation Study¶

Ablations on Tanks-and-Temples rigorously analyze iteration count $K$, threshold $\lambda$, core modules, and re-anchoring horizons.

Table 2: Ablation study of core components and hyperparameters on Tanks-and-Temples

| Config # | Iterations $K$ | Threshold $\lambda$ | AMP | PCP | Re-anchor Steps | FID-192 ↓ | FID-2048 ↓ | CLIP-S ↑ | ATE ↓ | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | 1 | - | - | ✗ | ✗ | - | 17.275 | 160.994 | 0.902 | 0.029 | | 2 | 5 | 0.8 | ✓ | ✓ | 3 | 4.864 | 96.359 | 0.923 | 0.010 | | 3 | 10 | 0.8 | ✓ | ✓ | 3 | 4.774 | 88.310 | 0.930 | 0.008 | | 4 | 15 | 0.7 | ✓ | ✓ | 3 | 5.240 | 92.013 | 0.926 | 0.010 | | 5 | 15 | 0.9 | ✓ | ✓ | 3 | 3.976 | 88.237 | 0.933 | 0.022 | | 6 | 15 | 0.8 | ✗ | ✓ | 3 | 4.943 | 91.215 | 0.929 | 0.012 | | 7 | 15 | 0.8 | ✓ | ✗ | 3 | 5.839 | 115.214 | 0.925 | 0.089 | | 8 | 15 | 0.8 | ✓ | Latent | 3 | 5.661 | 93.343 | 0.928 | 0.011 | | 9 | 15 | 0.8 | ✓ | ✓ | 0 | 2.773 | 83.750 | 0.940 | 0.017 | | 10 | 15 | 0.8 | ✓ | ✓ | 5 | 4.674 | 86.026 | 0.932 | 0.008 | | 11 | 15 | 0.8 | ✓ | ✓ | 8 | 6.326 | 96.721 | 0.926 | 0.007 | | 12 (PCP) | 15 | 0.8 | ✓ | ✓ | 3 w/ PCP | 6.893 | 107.733 | 0.909 | 0.006 | | Full | 15 | 0.8 | ✓ | ✓ | 3 | 3.712 | 83.341 | 0.935 | 0.008 |

Key Findings¶

PCP is essential for geometric alignment: Completely omitting PCP (Config 7) causes the absolute trajectory error (ATE) to surge from 0.008 to 0.089 (more than an 11-fold degradation) and degrades FID-2048 to 115.214. Applying PCP strictly in latent space via interpolated masks (Config 8) likewise underperforms pixel-space enforcement, confirming that latent compression leads to spatial bleeding and geometric decoupling.
Similarity threshold $\lambda$ balances fidelity and geometry: Setting $\lambda = 0.7$ is too permissive and incorporates noisy warping artifacts, degrading FID; conversely, $\lambda = 0.9$ discards valid geometric priors, causing ATE to jump to 0.022. The balanced choice $\lambda = 0.8$ achieves optimal perceptual quality and view consistency.
Trajectory re-anchoring trades off generation freedom and camera drift: Omitting re-anchoring completely (Config 9) preserves unconstrained generation freedom (lowest FID-192 of 2.773) but exhibits noticeable trajectory drift (ATE 0.017). Extending re-anchoring beyond 3 steps (Configs 10 and 11) progressively suppresses natural details and degrades perceptual metrics without further trajectory gains. Applying re-anchoring across the first 3 steps strikes the optimal compromise.

Highlights & Insights¶

Paradigm shift from mapping manipulation to noise discovery: Dismantles the standard assumption that foundational video models need fine-tuning or intrusive online guidance, demonstrating that valid NVS solutions already reside within the pre-trained data manifold and can be pinpointed through noise initialization.
Theoretical convergence via alternating projections: Formulates semantic inpainting and geometric constraints as alternating projections between the data manifold $\mathcal{M}$ and constraint subspace $\mathcal{V}$, providing formal convergence guarantees toward optimal intersections.
Unified zero-shot capability across tasks: By treating future frames as unobserved occluded regions, NeoMap seamlessly generalizes to zero-shot video-to-video future extrapolation without requiring any architectural modifications.

Limitations & Future Work¶

Dependency on upstream monocular geometry foundation models: The initial geometric prior relies directly on depth and pose estimations from models like VGGT or VIPE; substantial scale drift or depth distortions in reference inputs can propagate through the projection cycles.
Approximation error from single-step Euler projections: To ensure computational feasibility, AMP adopts single-step Euler approximations, which introduce truncation errors across non-linear velocity regions that require early-step trajectory re-anchoring to stabilize.
Future directions: Investigating adaptive multi-step manifold projection schemes and expanding alternating projections to multi-modal condition spaces for interactive embodied virtual environment simulation.

vs ViewCrafter / FlexWorld: These methods rely on point cloud lifting and task-specific fine-tuning of video diffusion models, making them vulnerable to out-of-distribution camera motions; NeoMap is completely training-free and directly harnesses foundational video generative priors.
vs NVS-Solver: NVS-Solver modifies the score function at every denoising step with warped priors, corrupting the native temporal flow of the generator; NeoMap solely optimizes the initial noise, executing uninterrupted, natural video decoding.
vs Go-With-The-Flow / Reversed Noise: Prior noise-warping techniques only address visible regions via optical flow and lack semantic hallucination mechanisms for occluded areas; NeoMap leverages AMP to hallucinate unobserved regions while adaptively anchoring visible geometry.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formulates NVS as an initial noise search problem and introduces manifold alternating projection with theoretical convergence.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across three canonical static and dynamic benchmarks with extensive ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulations, rigorous motivation, and logically coherent prose.
Value: ⭐⭐⭐⭐⭐ Unlocks strong NVS and temporal extrapolation capabilities from foundation video models without any fine-tuning.