GeoNVS: Geometry Grounded Video Diffusion for Novel View Synthesis

Conference: ECCV 2026
arXiv: 2603.14965
Code: Yes (Project page, see paper Project page)
Area: Video Generation / Novel View Synthesis / Diffusion Models
Keywords: Novel View Synthesis, Video Diffusion, 3D Gaussian, Geometric Prior, Feature Modulation

TL;DR

GeoNVS proposes a plug-and-play GS-Adapter. It "lifts" the diffusion features of reference views into 3D Gaussians and splats them back to the target view, injecting geometric priors in the feature space instead of at the image input level. Consequently, it significantly outperforms SEVA / CameraCtrl in camera controllability and geometric consistency, reducing translation error by up to 2x and Chamfer distance by up to 7x.

Background & Motivation

Novel View Synthesis (NVS) aims to render photorealistic images from arbitrary camera poses while maintaining geometric consistency across scenes. Classic scene-specific optimization schemes like NeRF and 3D Gaussian Splatting (3D-GS) yield stunning results under dense observations, but severely overfit under sparse inputs or when the target view is far from the input views. Two alternative paths have emerged: first, feed-forward geometric methods (MVSplat, DepthSplat, VGGT, Pi3, etc.) regress explicit 3D representations from sparse images in a single forward pass, showing strong structural consistency in observable regions but losing fine-grained appearance in low-overlap scenarios; second, generative methods feed camera poses or Plücker coordinates into video diffusion models (CameraCtrl, SEVA, etc.) to generate semantically rich, dense frames even with low input overlap, but at the cost of multi-view inconsistency, hallucinating structures that deviate from the inputs, and poor camera controllability—where the generated videos often fail to follow the specified camera trajectories.

A natural remedy is to inject explicit geometric priors from feed-forward geometric models into generative models to align the generated results with the input view structure. Several works (ReconFusion, GenFusion, Difix3D, MVSplat360, etc.) have followed this direction, but they almost exclusively perform fusion at the input layer: refining by treating the noisy target view images (or features) rendered by 3D-GS as inputs to the diffusion model. The authors observe that this methodology actually introduces geometric distortion and hallucinated structures, because view-dependent color noise in the rendered images overwhelms the structural signals, misguiding the diffusion model. Furthermore, such methods typically bind to a specific geometric model, requiring retraining if the geometric backbone is changed.

The Key Challenge of this paper is: the structural information from geometric priors is useful, but its associated view-dependent color noise is harmful, and input-level injection cannot decouple them. The Key Insight of GeoNVS is to move the coupling point from the image input level to the feature space: utilizing only the structural information encoded in 3D Gaussians to modulate internal diffusion features, bypassing the noisy color. Core Idea: Use a plug-and-play GS-Adapter to lift reference-view diffusion features into 3D Gaussians, splat them according to the target camera to obtain geometrically-constrained novel-view features, and then inject them back into the diffusion features using confidence-aware adaptive fusion. This allows the geometric prior to anchor the 3D structure throughout the entire denoising process without introducing color noise from rendered images.

Method

Overall Architecture

The inputs to GeoNVS are reference images \(I_{ref}\) alongside their camera poses \(\pi_{ref}\), and the target camera pose \(\pi_{tgt}\). The output is the target view image \(I_{tgt}\). The entire system is built upon a frozen video diffusion backbone (SEVA or CameraCtrl), training only the LoRA layers inserted into attention blocks, the multi-scale fusion module, and the GS-Adapter.

The pipeline operates on three levels: Preparing the geometric prior—Establishes point clouds from reference images using a feed-forward geometric model (such as VGGT, Pi3, DepthSplat). For models predicting both poses and point clouds (VGGT, Pi3), the predicted camera scale is aligned to the target camera, and then InstantSplat is used to fit the 3D Gaussians \(G\). Feature modulation at each denoising step—The GS-Adapter takes the current diffusion features \(F_t = [F_{ref}; F_{tar}]\) (composed of both reference and target views) and the geometric prior \(G\), outputting geometrically enhanced features \(\hat{F}_t = \Phi(F_t, G)\) via three internal steps: feature lifting, feature refinement, and feature fusion. Integration into the diffusion backbone—Since diffusion features have varying spatial resolutions at different U-Net layers, a DPT-style multi-scale aggregation is used to upsample and hierarchically merge features from the encoder scales into a unified feature, which is then fed to the GS-Adapter. The geometrically-corrected features are then downsampled and added back to the decoder via skip connections. This modulation runs through every denoising step.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Reference Images + Poses<br/>Target Camera Poses"] --> B["Feed-forward Geometric Model<br/>Estimate Point Cloud → Fit 3D Gaussians G"]
    A --> C["Video Diffusion Backbone<br/>SEVA / CameraCtrl (Frozen)"]
    C --> D["Multi-scale Feature Aggregation<br/>DPT-style Unified Feature F"]
    B --> E["Feature Lifting<br/>Ref. Features → 3D Gaussians"]
    D --> E
    E -->|Splat to Target View| F["Feature Refinement<br/>GS-PE + RefineNet for Details"]
    F --> G["Adaptive Feature Fusion<br/>Confidence-weighted Injection Back to Diffusion"]
    D --> G
    G -->|Loop per Denoising Step| C
    G --> H["Target View Image I_tgt"]

Key Designs

1. Three-step GS-Adapter: Moving Geometric Priors into the Feature Space

This is the core of the paper, directly tackling the issue of "input-level prior injection introducing color noise". It operates on diffusion features rather than rendered RGBs, proceeding in three stages. The first stage is Feature Lifting: since diffusion features vary dynamically at each timestamp, scene-specific optimization for feature embedding is impractical. Inspired by the feed-forward concepts of Dr.Splat and LUDVIG, the authors project 2D reference-view features onto each Gaussian via a weighted average based on rendering weights. For the \(i\)-th Gaussian, its feature is the weighted normalized sum across all contributing view-pixel pairs \((d,p)\):

\[\hat{f}_i = \frac{f_i}{\lVert f_i \rVert_2}, \qquad f_i = \sum_{(d,p) \in \mathcal{I}_i} \frac{w_i(d,p)}{\sum_{(d',p') \in \mathcal{I}_i} w_i(d',p')} F_{ref}(d,p)\]

Here, soft assignment (accumulating all Gaussians along a ray, similar to alpha-blending) is deliberately used instead of only selecting the Gaussian with the maximum rendering weight (hard assignment, which is equivalent to projecting features onto point clouds). Supplementary experiments show that as the number of sampled Gaussians \(k\) per ray increases from 1 to 255, the PSNR rises monotonically, proving that soft assignment captures richer structural signals. This feed-forward lifting allows 3D Gaussians to explicitly participate in each denoising step and supports end-to-end joint training with the diffusion model. The second stage yields target-view geometric features \(G_{tar}\) (rendered via Gaussian feature splatting), and the third stage fuses these back into the diffusion features. All three stages are fully differentiable and processed per-timestamp, which is the key distinction from scene-specific optimization methods.

2. Feature Refinement: Recovering Details Lost in Lifting via GS-PE + RefineNet

Simply fusing the rendered geometric features directly with diffusion features fails to recover fine-grained details due to information loss inherent in lifting and rendering. To compensate, two operations are performed on the rendered features \(G\). The first is adding Gaussian Positional Encoding (GS-PE): for each pixel, the most contributing Gaussian \(i^*(d,p)=\arg\max_i w_i(d,p)\) is identified. Its 3D location, normalized within the scene bounding box, along with its rendering weight, are encoded sinusoidally and concatenated with the rendered features to provide explicit 3D structural cues to the downstream network:

\[G'(d,p) = G(d,p) + \left[\gamma(\tilde{x}) \,\|\, \gamma(\tilde{y}) \,\|\, \gamma(\tilde{z}) \,\|\, \gamma(w^*)\right]\]

Second, the features are passed through a lightweight ResNet refinement network \(R\) to obtain \(\hat{G}=R(G')\). During training, since reference-view features \(F_{ref}\) serve as reliable anchors, a cosine similarity loss is used to align the refined reference features \(\hat{G}_{ref}\) with \(F_{ref}\): \(\mathcal{L}_{feat} = \lVert 1 - \cos(\hat{G}_{ref}, F_{ref}) \rVert_2\). Ablation studies indicate that both GS-PE and \(\mathcal{L}_{feat}\) are essential; removing RefineNet results in performance drops across all benchmarks, proving that the gains from refinement are independent of Gaussian splatting itself.

3. Adaptive Fusion: Deciding Trust Levels Based on Local Reliability of Geometric Priors

Given refined geometric features \(\hat{G}_{tar}\), how to fuse them with diffusion features \(F_{tar}\) is critical. A naive approach (Naïve Fusion) directly concatenates them and projects them back using MLPs, i.e., \(\hat{F}_{tar}=\text{MLPs}(F_{tar} \oplus \hat{G}_{tar})\), which strictly relies on the geometric prior and fails entirely if the prior contains errors (e.g., in specular regions). Instead, the authors propose Adaptive Fusion: treating the diffusion features \(F_{tar}\) as queries and geometric features \(\hat{G}_{tar}\) as keys/values in cross-attention to produce geometry-attended features \(F_{tar}^A\). A gated MLP then predicts pixel-wise confidence weights \(W_{tar}\in[-1,1]\), which steer the weighted injection:

\[W_{tar} = \text{Tanh}\!\left(\text{MLP}(F_{tar} \oplus F_{tar}^A)\right), \qquad \hat{F}_{tar} = F_{tar} + W_{tar} \cdot F_{tar}^A\]

The authors verify that \(W_{tar}\) indeed dynamically learns what to trust: measuring the Pearson correlation between \(|W_{tar}|\) and 3D-GS uncertainty on 3 reflective scenes reveals a consistent negative correlation (\(r\approx-0.30\)), which becomes more pronounced in specular regions as denoising progresses. This indicates that in areas where the geometric prior is unreliable, adaptive fusion actively down-weights the geometric guidance. This step yields the most value in ablation studies: replacing the prior with Pi3 under Naïve Fusion on DTU drops the score to 14.83 (lower than the SEVA baseline of 17.03), whereas Adaptive Fusion consistently stabilizes at 19.02–19.18 regardless of the geometric prior chosen, realizing a true plug-and-play capability independent of the geometric backbone.

Loss & Training

LoRA layers are injected into the attention modules of the frozen video diffusion model (setting rank/\(\alpha\) to 16 for SEVA and 4 for CameraCtrl), and are trained alongside the multi-scale fusion module and the GS-Adapter. The total loss is the latent alignment loss combined with the feature alignment loss, the latter weighted at 0.05:

\[\mathcal{L} = \mathcal{L}_{latent} + 0.05\,\mathcal{L}_{feat}\]

The training data is from DL3DV-10K, utilizing VGGT + InstantSplat to construct 3D Gaussians for 150k video clips. The reference/novel view distribution \((P,Q)\) varies within \(\{(1,20),(3,18),(6,15),(9,12),(12,9)\}\), with the total views fixed at 21. The SEVA version uses a resolution of \(384\times384\), while the CameraCtrl version uses \(320\times576\). LoRA fine-tuning is performed on 8 A6000 GPUs (48GB) for approximately 80 hours with a total batch size of 8.

Key Experimental Results

Main Results

Evaluation is performed across 9 scenes and 18 settings on the SEVA benchmark, covering small-angle, large-angle, and long-trajectory categories. Generative methods and "generative + geometry prior" methods employ the optimal feed-forward geometric prior for each dataset respectively.

Setting	Metrics	GeoNVS (Ours)	SEVA Baseline	Key Comparisons
Small-angle Set NVS (Mean of 9 Datasets)	PSNR ↑	19.09	17.15	Difix3D 16.89 / GenFusion 15.30
Small-angle Set NVS (Mean)	SSIM ↑	0.622	0.560	Difix3D 0.545
Small-angle Set NVS (Mean)	LPIPS ↓	0.253	0.293	Difix3D 0.339
Large-angle Set NVS (Low overlap, Mean)	PSNR ↑	14.20	12.64	DepthSplat prior 10.46
Long-trajectory NVS (Mip360/DL3DV/T&T)	PSNR ↑	Outperforms SEVA across the board	Baseline	Gain increases with more reference views

In terms of geometric fidelity (under the Trajectory NVS setting), GeoNVS yields a comprehensive reduction in translation error \(T_{err}\), rotation error \(R_{err}\), and Chamfer distance (CD) compared to SEVA. The overall gains reported in the abstract are +11.3% / +14.9% over SEVA / CameraCtrl respectively, with \(T_{err}\) reduced by up to 2x and CD by up to 7x. In contrast, competitors using input-level injection (Difix3D, SEVA + input-level injection) suffer catastrophic degradation in controllability—for instance, on Mip360, \(T_{err}\) for SEVA + input-level injection surges to 233 and \(R_{err}\) to 99.79, far worse than the pure generative baseline SEVA (18.76 / 3.29), whereas this work achieves 16.24 / 2.98.

On the unobserved region (regions invisible relative to reference views) PSNR\(_U\), GeoNVS shows improvements over SEVA on Mip360 (+1.06), DL3DV (+2.08), and WRGBD-Sh (+1.72), demonstrating that geometric guidance assists in geometrically consistent extrapolation into unobserved regions.

Ablation Study

Ablating individual components of the GS-Adapter using VGGT as the geometric prior (DL3DV 6-view PSNR):

Configuration	DL3DV PSNR	Description
SEVA Baseline	15.84	Pure generative
+ Input-level injection	16.73	Gain only when geometric prior is nearly perfect
+ LoRA only	17.35	Fine-tuning only, no geometry
+ GS-Adapter (Naïve Fusion)	17.88	Concatenation fusion
+ GS-Adapter (Adaptive Fusion)	17.95	Confidence-weighted
− RefineNet	17.71	Performance drops without refinement network
Full GeoNVS	18.11	Multi-scale + Refinement + Adaptive

Geometric prior compatibility (SEVA backbone, 3 reference views, DTU/CO3D): Naïve Fusion is sensitive to prior quality—VGGT yields +3.37dB on T&T, but Pi3 drops to 14.83 on DTU, falling below the baseline of 17.03. On the other hand, Adaptive Fusion consistently stabilizes DTU at 19.02–19.18 and CO3D at 19.22–19.26, regardless of the geometric prior utilized.

Key Findings

• Adaptive fusion is the key to robustness: Naïve Fusion can be dragged down by poor geometric priors. Adaptive Fusion actively down-weights geometric guidance in unreliable regions via confidence weights \(W_{tar}\), ensuring no performance drop regardless of the geometric backbone chosen—a prerequisite for plug-and-play capability.
• Feature space vs. input space is the most significant empirical evidence: Input-level injection of geometric priors only provides gains when the prior is nearly perfect, otherwise collapsing (with camera controllability dropping below even the pure generative baseline). In contrast, feature-level modulation consistently improves PSNR regardless of the quality of the prior.
• Soft assignment and refinement jointly recover details: More Gaussians sampled during lifting lead to higher PSNR (soft > hard assignment). RefineNet yields a +1.37dB gain in the 9-view setting, with larger gains as more reference views are provided.
• Controllable computational overhead: The bottleneck lies in the number of Gaussians generated by the geometric prior (inference at 4.19 s/frame, peak memory of 18.03GB). Utilizing voxel-based Gaussian pruning achieves a 2.22x speedup to 1.89 s/frame under a 96.2% pruning rate, with memory dropping to 12.93GB (comparable to SEVA's 12.81GB), while PSNR remains virtually unchanged.

Highlights & Insights

• The "Input Layer vs. Feature Layer" distinction is extraordinarily sharp: The paper clearly pinpoints the common pitfall of prior geometry-guided NVS methods in a single insight—namely, that the view-dependent color noise in rendered images overwhelms structural signals. By shifting the coupling point from image inputs to the feature space, the method leverages structure without importing color noise. This insight can be transferred to any generative task seeking to integrate geometric/structural priors while avoiding modal noise.
• Confidence gating \(W_{tar}\) validated for explainability via uncertainty: Rather than merely claiming "adaptiveness", the authors empirically establish that \(|W_{tar}|\) correlates negatively with 3D-GS uncertainty, effectively quantifying the 'degree of trust'. This paradigm of predicting prior-reliability weights for adaptive fusion is highly reusable.
• Genuine plug-and-play capability: GS-Adapter exhibits zero-shot compatibility across 4 geometric priors and 2 diffusion backbones without retraining, removing the limitation of prior methods that bind to a specific geometric model. Additionally, soft-assignment feature lifting allows 3D Gaussians to explicitly participate in each denoising step, which is the technical prerequisite enabling end-to-end joint training.

Limitations & Future Work

• The authors acknowledge: Generation quality drops in regions far from the input views, which is attributed to increasing uncertainty in GS-Adapter feature rendering as distance grows. Occasionally, blurry textures emerge in detailed regions (strong structure but compromised perceptual sharpness), caused by information loss during feature lifting (Fig. 10).
• Identified by reader: Inference overhead heavily depends on the number of Gaussians produced by the geometric prior; although pruning mitigates this, it remains slower than pure SEVA (1.89 vs. 1.34 s/frame). Performance is strongly tied to the quality of the geometric prior; while adaptive fusion alleviates sensitivity, the model remains helpless in regions where both the geometric prior and the generative baseline fail.
• Future Directions: Designing stronger completion/refinement mechanisms for uncertain feature rendering under distant views, or introducing generative compensation to restore high-frequency details in blurry textures.

Related Work & Insights

• vs. Feed-forward Geometric Methods (MVSplat / DepthSplat / VGGT / Pi3): These methods output explicit 3D structures in a single forward pass, showing strong structural consistency but poor appearance in low-overlap settings. This work does not replace them; instead, it treats them as plug-and-play sources of geometric priors, leveraging generative models to supplement appearance and density.
• vs. Input-Level Geometry-Guided Generation (GenFusion / Difix3D / MVSplat360 / ReconFusion): Operating by feeding noisy 3D-GS rendered images/features into diffusion models for refinement, these introduce color noise that leads to geometric distortion, and are typically constrained to a specific geometric model. In contrast, our method modulates strictly with structural information in the feature space, yielding significantly superior geometric consistency and controllability while remaining plug-and-play for diverse geometric and diffusion backbones.
• vs. 3D-GS Feature Field Distillation (Dr.Splat / LUDVIG): These distill static semantic features into Gaussians for discrete tasks like segmentation. In our case, the diffusion features change dynamically at each denoising step, and NVS is highly sensitive to feature distortion. Thus, we adopt feed-forward lifting + RefineNet refinement to preserve pixel-level feature fidelity, rather than the coarse-grained discrimination typical of segmentation.

Rating

• Novelty: ⭐⭐⭐⭐⭐ The transition of geometric prior injection from the "input layer to the feature space" is clear and powerful, with an elegantly designed adaptive confidence gate backed by explainability validation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 9 scenes and 18 settings, utilizing 2 backbones and 4 geometric priors. Ablation studies cover every individual component, complemented by pose-free, pruning, and CFG sensitivity analysis.
• Writing Quality: ⭐⭐⭐⭐⭐ Figures 2 and 4 intuitively convey why input-level injection fails and why adaptive fusion is necessary, presenting a clearly structured three-step methodology.
• Value: ⭐⭐⭐⭐⭐ Truly plug-and-play, offering zero-shot compatibility across various geometric and diffusion backbones. It yields substantial gains in geometric consistency and camera controllability, serving as a reusable paradigm for geometry-guided generation.

Related Papers

• [ICCV 2025] FVGen: Accelerating Novel-View Synthesis with Adversarial Video Diffusion Distillation
• [ECCV 2024] SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion
• [CVPR 2025] StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models
• [CVPR 2025] Geometry-guided Online 3D Video Synthesis with Multi-View Temporal Consistency
• [ECCV 2026] LatSearch: Latent Reward-Guided Search for Faster Inference-Time Scaling in Video Diffusion