UniGeo: Unifying Geometric Constraints for Camera-Controllable Image Editing via Video Priors¶

Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://mo230761.github.io/UniGeo.github.io/
Area: 3D Vision
Keywords: view synthesis / unified geometric guidance / video diffusion models / camera control / point cloud injection

TL;DR¶

Addressing structural collapse and cross-view drift caused by fragmented geometric guidance in camera-controllable view synthesis, UniGeo systematically integrates unified geometric guidance across representation, architecture, and loss function levels within a video diffusion foundation model, achieving state-of-the-art geometric consistency under continuous extensive camera trajectories.

Background & Motivation¶

Camera-controllable view synthesis aims to generate realistic scene transformations from novel viewpoints given specified camera trajectories, while strictly preserving cross-view 3D geometric consistency. This capability is pivotal for downstream applications like visual effects post-production, virtual reality scene exploration, and 3D perception in embodied AI. Conventional image diffusion models treat view synthesis as pairwise mappings between discrete viewpoints; lacking continuous 3D temporal trajectories, they suffer severe structural flicker and geometric degradation under extended viewpoint shifts.

To incorporate smooth camera evolution, recent works adapt pretrained video diffusion models, exploiting their continuous spatio-temporal priors. However, these video-prior methods invariably struggle with fragmented geometric guidance. In existing frameworks, geometric injection is confined to an isolated stage—most notably at the representation level where noisy point clouds or depth maps are concatenated along channel dimensions. Because the internal network architecture and optimization objectives lack explicit geometric correspondence, structural information fails to propagate coherently across the generative pipeline, leading to broken geometry, object duplication, and perspective collapse.

The fundamental insight of this work is that generative synthesis is jointly governed by representation, architecture, and loss formulation. Enforcing strict geometric consistency requires a unified, multi-level guidance design. The core idea is to systematically inject unified geometric guidance into all three constitutive tiers of video diffusion models: frame-decoupled point cloud injection at the representation level, geometric anchor attention at the architecture level, and trajectory-endpoint geometric supervision at the loss level.

Method¶

Overall Architecture¶

Given a single reference image and user-specified camera motion instructions, UniGeo synthesizes high-fidelity novel views that follow the continuous trajectory while preserving underlying 3D structures. The framework builds upon the Wan2.2-TI2V-5B video generative foundation model, trained using rectified flow matching and parameter-efficient LoRA tuning.

The pipeline operates across three cooperative tiers: First, a feed-forward visual geometry transformer (VGGT) reconstructs an initial 3D point cloud and camera trajectory from the input frame, producing a sequence of trajectory-aligned point cloud renderings. At the representation level, these geometric latents are concatenated with target video tokens along the frame dimension. Within the DiT architecture, geometric anchor attention binds features across all timesteps to the structural anchors of the initial frame. At the loss level, a time-varying quadratic weighting schedule and endpoint temporal extension explicitly enforce geometric fidelity at target viewpoints.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Motion Prompt"] --> B["Point Cloud Geometry Construction<br/>VGGT reconstruction & trajectory rendering"]
    B --> C["Frame-Decoupled Geometry Injection<br/>Frame-dimension concatenation of video & point cloud latents"]
    C --> D["Geometric Anchor Attention<br/>Align cross-view features using first-frame structural anchor"]
    D --> E["Trajectory-Endpoint Geometric Supervision<br/>Quadratic loss weighting & target-view temporal extension"]
    E --> F["High-Fidelity Geometrically Consistent Target View"]

Key Designs¶

1. Frame-decoupled point cloud injection: decoupling representations to prevent missing geometry contamination Conventional methods inject geometric context via channel-wise concatenation of point cloud renderings and image latents. This rigid coupling forces hard pixel-to-pixel correspondence; any depth estimation error, boundary hole, or missing region in the sparse reconstructed point cloud directly corrupts pixel-level texture synthesis. UniGeo resolves this bottleneck by decoupling the geometric guidance along the temporal frame dimension. Given input frame $I_0$, the model reconstructs point cloud $P_0$ and trajectory $C=\{C_0, \dots, C_{N-1}\}$ via VGGT, generating rendered sequence $R_f = \pi(P_0, C_f)$ (with $R_0 = I_0$ explicitly substituted). The target latent $z_t$ and point cloud latent $z_s$ are patchified and concatenated along the frame dimension: $$x_i = [x_t, x_s]_{\text{frame-dim}} \in \mathbb{R}^{b \times 2f \times l \times d}$$ By treating rendered point clouds as independent contextual reference frames, the DiT backbone flexibly queries geometric priors through cross-token attention, retaining global 3D spatial guidance without propagating point cloud artifacts into generated pixels.

2. Geometric anchor attention: parameter-efficient structural alignment across viewpoints Standard video self-attention primarily captures smooth temporal appearance transitions, leaving wide-baseline viewpoint transformations susceptible to progressive geometric drift. UniGeo introduces Geometric Anchor Attention (GAA) into the DiT blocks. The first frame's intermediate feature $X_0$ is designated as the geometric anchor, projecting into keys $K_0 = X_0 W_K$ and values $V_0 = X_0 W_V$ with frozen pretrained weights. For any subsequent timestep $i \in \{1, \dots, N-1\}$ with feature $X_i$, a trainable projection matrix $W_Q'$ extracts geometric query $(Q_i)' = X_i W_Q'$. The anchor attention output is residually blended with the standard self-attention: $$X_i^{\text{out}} = \text{Attention}(Q_i, K_i, V_i) W_O + \alpha \cdot \text{Attention}((Q_i)', K_0, V_0) W_O'$$ Here $W_O'$ is initialized to zero to preserve initial generative priors, and scalar $\alpha$ modulates geometric guidance intensity. Introducing merely two low-overhead matrices ($W_Q', W_O'$), GAA continuously pulls intermediate viewpoint features toward the reliable reference structure, preventing multi-view divergence.

3. Trajectory-endpoint geometric supervision: focusing optimization on terminal viewpoint fidelity In camera-controllable view synthesis, the primary objective is high structural fidelity at the final target viewpoint, whereas intermediate views serve largely as trajectory transitions. Uniform temporal loss weighting over-allocates optimization capacity to intermediate frames. UniGeo addresses this through Trajectory-Endpoint Geometric Supervision (TEGS). The pipeline sparsifies the trajectory to key viewpoints (sampling 29 frames from 81), and assigns a quadratic penalty increasing toward the trajectory endpoints: $$w_{\text{loss}}(i) = 1 + \gamma \left(\frac{2i}{N-1}\right)^2, \quad i \in \{1, \dots, N-1\}$$ where $\gamma$ governs endpoint supervision strength, yielding weighted loss $\mathcal{L}_{\text{weighted}} = \sum_{i=1}^{N-1} w_{\text{loss}}(i) \mathcal{L}_i$. Furthermore, a temporal extension strategy duplicates the terminal target view across the final 4 consecutive timesteps during denoising, enforcing sustained structural supervision during late-stage diffusion convergence.

Loss & Training¶

UniGeo builds upon Wan2.2-TI2V-5B fine-tuned via LoRA (rank 256). Training utilizes 15,000 curated video samples from DL3DV, MannequinChallenge, and RE10K, resized to $704 \times 1248$ across 29 frames. Optimization proceeds on 4 GPUs for 10,000 iterations with learning rate $1 \times 10^{-4}$ and total batch size 4. Hyperparameters are set to $\alpha = 1.0$ and $\gamma = 0.01$.

Key Experimental Results¶

Main Results¶

Evaluation spans RE10K, Tanks and Temples, and DL3DV benchmarks. Rather than splitting test sets arbitrarily by frame index, the authors categorize trajectories by the proportion of newly revealed scene area in point cloud renderings: mask ratio $> 35\%$ denotes extensive camera motion, and $\le 35\%$ denotes limited camera motion.

Main Result 1: Extensive Camera Motion Setting (Original Paper Table 1)

Dataset	Metric	Ours (UniGeo)	FlexWorld [18]	ViewCrafter [89]	PE-Field [7]	Gain
RE10K	FID ↓	66.67	90.43	97.57	105.34	-23.76 reduction
RE10K	SSIM ↑	0.6522	0.6430	0.5905	0.6210	+0.0092 improvement
RE10K	LPIPS ↓	0.2377	0.3008	0.3668	0.3768	-21.0% error drop
RE10K	PSNR ↑	14.9723	14.3408	14.3176	13.1684	+0.6315 dB gain
DL3DV	FID ↓	113.11	125.27	146.81	131.93	-12.16 improvement
DL3DV	LPIPS ↓	0.3248	0.3726	0.4556	0.4329	Superior perceptual fidelity
DL3DV	PSNR ↑	13.6067	13.3029	12.6128	12.7060	Consistent lead
Tanks	LPIPS ↓	0.2633	0.3395	0.4047	0.3957	-22.4% error drop
Tanks	PSNR ↑	14.4537	13.8118	13.3314	13.1063	+0.6419 dB gain

Main Result 2: Limited Camera Motion Setting (Original Paper Table 2)

Dataset	Metric	Ours (UniGeo)	FlexWorld [18]	ViewCrafter [89]	CameraCtrl [30]	Comparison
RE10K	FID ↓	51.73	73.80	84.45	122.46	Best distribution matching
RE10K	LPIPS ↓	0.1730	0.2573	0.2984	0.3825	Substantial perceptual boost
RE10K	PSNR ↑	17.2989	16.1159	15.5421	12.0042	+1.183 dB over runner-up
Tanks	FID ↓	40.55	54.35	73.97	97.27	Sharpest structural details
Tanks	PSNR ↑	17.8171	16.9580	16.1263	13.0271	Top pixel-level fidelity
DL3DV	PSNR ↑	16.3740	15.4140	15.2119	10.5987	Clear SOTA margin

Ablation Study¶

Ablations on DL3DV isolate the contribution of each core component under extensive and limited motion settings (Original Paper Table 5).

Ablation Study on DL3DV (Original Paper Table 5)

Config	Extensive FID ↓	Extensive SSIM ↑	Extensive LPIPS ↓	Extensive PSNR ↑	Limited PSNR ↑	Limited LPIPS ↓	Note
Full model (Ours)	113.11	0.4830	0.3248	13.6067	16.3740	0.2065	Full unified three-level guidance
w/o FDPCI	121.52	0.4270	0.3491	12.9989	15.7771	0.2218	SSIM drops 0.056; severe distortion
w/o GAA	115.83	0.4683	0.3284	13.3682	15.9162	0.2088	Unaligned cross-view features
w/o TEGS	119.43	0.4588	0.3379	13.0054	15.7987	0.2122	Target geometry diluted (-0.6 dB PSNR)

Hyperparameter tuning demonstrates that setting anchor attention weight $\alpha = 1.0$ strikes the optimal balance ($\alpha = 0.1$ yields FID 115.17 due to under-alignment; $\alpha = 1.5$ yields FID 114.80 due to over-constraint). Similarly, endpoint penalty parameter $\gamma = 0.01$ outperforms $\gamma = 0.001$ (FID 119.58) and $\gamma = 0.1$ (FID 115.60).

Key Findings¶

Frame-decoupled injection is critical: Excluding FDPCI causes SSIM to plummet from 0.4830 to 0.4270 under extensive camera motion. Channel-concatenation or omitting point clouds induces duplicate objects and severe spatial shearing.
Soft attention tolerates imperfect geometry: Although feed-forward VGGT estimates may exhibit localized noise or holes, soft latent-level attention allows the diffusion model to dynamically refine geometry and inpaint occluded textures without transferring point cloud flaws.
Endpoint supervision balances stability and clarity: Eliminating intermediate supervision altogether results in severe blurriness, whereas quadratic weighting ($\gamma=0.01$) alongside 4-frame terminal extension successfully protects smooth trajectory transitions while sharpening endpoint fidelity.

Highlights & Insights¶

Holistic resolution of geometric fragmentation: Identifies isolated representation-level conditioning as the root cause of 3D collapse, establishing a principled paradigm that coordinates representation, architecture, and loss formulation.
Temporal frame decoupling over channel concatenation: Creatively repurposes DiT's sequence-length capacity to accept point clouds as decoupled reference frames, circumventing the hard-correspondence artifacts endemic to prior point-cloud-guided diffusion.
Favorable quality-efficiency trade-off: Geometric anchor attention introduces only two low-rank matrices; total inference latency (75s) is roughly one-third of video-diffusion baseline FlexWorld (240s), while establishing a decisive lead in geometric metrics.

Limitations & Future Work¶

Performance in extreme viewpoint swings: In situations with dramatic perspective changes (e.g., $180^\circ$ reverse angles) or intense occlusions and reflective surfaces, initial point cloud priors become fragmented, leading to texture stretching.
Multi-frame latent compute footprint: Generating 29-frame latent sequences incurs higher memory and compute overhead compared to single-step image diffusion models, pointing toward future acceleration via diffusion distillation or specialized inference kernels.

vs ViewCrafter / FlexWorld: While ViewCrafter and FlexWorld rely on channel-stacked point clouds or progressive scene expansion, UniGeo implements unified multi-level constraints, lowering extensive-motion LPIPS on RE10K from ~0.30–0.36 to 0.2377 and eliminating object duplication artifacts.
vs CameraCtrl / MotionCtrl: Conventional camera-conditioned video models rely on Plücker coordinates or pose embeddings, lacking explicit 3D geometry; UniGeo leverages trajectory-rendered point clouds, slashing cross-view rotation (RotErr) and translation (TransErr) errors on DL3DV.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Pioneering unified geometric guidance across representation, architecture, and loss tiers with frame-decoupled injection]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous evaluation across extensive/limited motion regimes on DL3DV, RE10K, Tanks, and MannequinChallenge with full ablation]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, well-structured multi-tier presentation, and coherent qualitative/quantitative corroboration]
Value: ⭐⭐⭐⭐⭐ [Provides a robust, principled blueprint for camera-controllable view synthesis and 3D-consistent generative editing]