Geometrically Consistent Multi-View Scene Generation from Freehand Sketches

Conference: ECCV 2026
Paper: ECCV Official
Code: 待确认
Area: 3D Vision
Keywords: sketch-based generation / multi-view synthesis / geometric consistency / video diffusion transformer / correspondence supervision

TL;DR

Addressing the severe geometric impoverishment and non-linear spatial distortions of freehand sketches, this paper presents the first single-stage framework for sketch-to-multi-view scene generation, coupling lightweight parallel camera adapters with sparse SfM correspondence supervision to synthesize geometrically consistent 360° novel views without reference photographs or per-scene optimization.

Background & Motivation

Freehand sketching serves as one of the most intuitive and accessible interfaces for humans to convey 3D scene concepts. With only a few strokes—curved outlines of a vehicle, wobbly boundaries of buildings, or stick figures on a pavement—a human observer effortlessly pictures a plausible 3D world, inferring occluded surfaces, spatial depth, and novel perspectives. However, asking a generative model to directly synthesize geometrically consistent novel views from a single freehand scene sketch (Sketch-to-Multi-View, S2MV) represents a notoriously challenging problem. Scene sketches are simultaneously spatially informative yet geometrically unreliable, introducing non-linear distortions, abstract simplifications, and stroke-level noise that actively conflict with rigorous projective geometry.

Existing multi-view diffusion models predominantly condition on high-fidelity photographs that already provide dense textures, shading, and implicit 3D perspective cues. Text-conditioned multi-view models operate under flexible semantic guidance where spatial precision is neither expected nor strictly enforced. Meanwhile, current sketch-to-3D approaches remain largely confined to isolated single-object reconstruction, or require multiple sketch viewpoints alongside expensive per-scene optimization (such as Score Distillation Sampling). Adopting a naive two-stage pipeline—first converting a sketch into a photorealistic front image via a text-to-image generator and then applying novel-view synthesis—inevitably incurs severe error accumulation, view inconsistencies, and heavy computational latency. The underlying bottleneck stems from three compounding barriers: the complete lack of paired sketch-to-multiview benchmarks, the inability of standard generative models to infer 3D camera geometry from distorted 2D strokes, and the failure of attention layers to inherently enforce cross-view spatial correspondences.

The entry point of this paper is to bypass intermediate photo generation entirely, synthesizing all multi-views jointly within a single end-to-end denoising process directly from a sketch. Core idea: Construct a curated benchmark of ~9k sketch-to-multiview samples and augment a pretrained video diffusion transformer (Wan 2.1) with lightweight Parallel Camera-Aware Attention Adapters (CA3) carrying relative projective geometry, supervised by a Sparse Correspondence Supervision Loss (CSL) derived from Structure-from-Motion (SfM) to explicitly teach the model cross-view geometric alignment in the latent space.

Method

Overall Architecture

The framework repurposes a pretrained video diffusion transformer backbone (Wan 2.1, 1.3B parameters) by treating the temporal sequence axis as a camera viewpoint trajectory. Given an input freehand sketch \(S\), a conditioning text prompt, and target camera poses \(\{T_{\text{wc}}, K\}_{n=1}^N\) (\(N=33\) views covering full 360° azimuths across four elevations), the model encodes the inputs into latent space via a VAE encoder. The latent sequence is denoised in a single forward pass by the DiT backbone augmented with parallel camera adapters (CA3) and self-attention LoRA layers via flow matching, and finally decoded into \(N\) photorealistic views by the VAE decoder. During training, sparse 3D correspondence pairs extracted offline via COLMAP SfM directly supervise the adapter query-key projection spaces through an InfoNCE contrastive objective.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Sketch + Text + Virtual Camera Poses"] --> B["Data Generation & Semantic Filtering Pipeline<br/>Multi-seed generation + mIoU alignment filtering"]
    B --> C["Frame Replication Encoding<br/>4x redundancy overcoming VAE temporal compression"]
    C --> D["Parallel Camera Attention Adapters CA3<br/>PRoPE projective geometry injection + LoRA adaptation"]
    D --> E["Sparse Correspondence Supervision Loss CSL<br/>SfM matched pairs + layer-wise InfoNCE contrastive learning"]
    E --> F["Single-stage Flow Matching joint denoising to N views"]

Key Designs

1. Data Generation & Semantic Filtering: Bridging the paired sketch-to-multiview void

Because paired freehand sketches and multi-view 3D scenes do not naturally exist in the physical world, the authors devise an automated generation and multi-stage curation pipeline. Starting with ~10,000 freehand scene sketches from FS-COCO, five photorealistic frontal image candidates are synthesized for each sketch using FLUX.2 dev conditioned on the sketch and caption. Next, semantic segmentation masks are extracted from the sketch using an open-vocabulary sketch encoder, while corresponding object masks are extracted from candidate images using GroundingDINO and SAM3 guided by noun phrases. The candidate image maximizing the mean Intersection-over-Union (mIoU) against the sketch is selected: $\(I^* = \arg\max_{i \in \{1..5\}} \text{mIoU}(\mathcal{M}_s, \mathcal{M}_{g,i})\)$ The selected frontal image is expanded into \(N\) multi-view images across various azimuths and elevations using Qwen Image Edit Angles. A strict weighted mIoU threshold then discards misaligned samples, yielding a final benchmark of 9,222 high-quality sketch-multiview samples.

2. Frame Replication: Preserving viewpoint independence against VAE temporal compression

The 3D VAE within the video diffusion backbone incorporates a 4× temporal downsampling compression, which is highly advantageous for continuous video where adjacent frames share massive visual redundancy. However, in multi-view scene generation where neighboring viewpoints undergo wide-baseline geometric transformations, this compression blurs distinct views and causes catastrophic cross-view feature entanglement. To resolve this, each target view is replicated four times prior to VAE encoding, expanding the sequence to \(4N+1\) frames. Under 4× temporal compression, each compressed latent token sequence corresponds cleanly to exactly one unique viewpoint, fully retaining the spatio-temporal generative priors without compromising view identity.

3. Parallel Camera-Aware Attention Adapters (CA3): Zero-initialized relative projective injection

Standard video DiT self-attention layers treat tokens uniformly without intrinsic awareness of spatial camera translations and rotations. The paper introduces lightweight Camera-Aware Attention Adapters (CA3) as parallel branches alongside backbone self-attention layers. CA3 integrates Projective Rotary Position Encoding (PRoPE), deriving a projective matrix \(P^{(n)}\) from extrinsic \(T^{(n)}\) and shared intrinsic \(K\), partitioning the attention head dimensions such that query-key affinities reflect the relative projective transform \(P^{(n_q)}(P^{(n_k)})^{-1}\). To preserve high-fidelity generative priors, each adapter employs an 8× bottleneck dimension (1536 down to 192) with only 2 attention heads, contributing just 2.7% additional parameters (~35.4M). The output projection is zero-initialized so initial training does not perturb the pretrained backbone. Simultaneously, rank-16 LoRA (~5.9M parameters) is injected into backbone self-attention projections to bridge the domain shift between natural videos and abstract sketches.

4. Sparse Correspondence Supervision Loss (CSL): Guiding cross-view attention onto identical 3D points

Architectural camera bias alone does not guarantee that cross-view attention tracks identical physical coordinates; without supervision, attention remains diffuse. While supervising the full attention matrix across views would consume hundreds of gigabytes per layer, CSL formulates an efficient contrastive InfoNCE objective over CA3 query-key projections at middle-to-deep layers (\(\{10, 15, 20, 25\}\)). Pairwise supervision targets are harvested offline via COLMAP SfM reconstructions, weighted by reprojection confidence \(w_i\): $\(\mathcal{L}_{\text{corr}} = -\frac{1}{M}\sum_{i=1}^M w_i \cdot \log \frac{\exp(q_i^\top k_i^+ / \tau)}{\exp(q_i^\top k_i^+ / \tau) + \sum_{j=1}^{N_{\text{neg}}} \exp(q_i^\top k_j^- / \tau)}\)$ This reduces computational complexity from \(O(T^2)\) to \(O(M \times N_{\text{neg}})\), slashing memory footprint from 14 GB to ~1 MB per layer. A curriculum linearly ramps \(\lambda_{\text{corr}}\) from 0 to 0.01 after warmup, allowing the flow matching loss and LoRA adaptation to stabilize before enforcing tight geometric correspondence.

Loss & Training

The overall training objective blends the standard velocity field flow matching error with the sparse correspondence regularization: $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{flow}} + \lambda_{\text{corr}} \cdot \mathcal{L}_{\text{corr}}\)$ Training follows a two-stage curriculum: during the initial warmup stage, \(\lambda_{\text{corr}}\) is held at zero, allowing LoRA domain adaptation and flow matching to stabilize; \(\lambda_{\text{corr}}\) is subsequently ramped linearly to 0.01. Training is conducted on 8× NVIDIA A100 GPUs using the AdamW optimizer.

Key Experimental Results

Main Results

On the S2MV test set (477 samples, \(N=33\) views each, averaged across 3 random seeds), the proposed single-stage method is compared against two-stage state-of-the-art baselines that first convert the sketch into a photograph via FLUX.2-dev:

Method	Stages	Time / sample	PSNR ↑	SSIM ↑	LPIPS ↓	FID ↓	CLIP-I ↑	Corr-Acc ↑
SEVA	2-stage	~3.1 min	11.310	0.265	0.705	46.34	0.756	0.161
ViewCrafter	2-stage	~35 min	11.148	0.338	0.737	48.22	0.773	0.136
Ours	1-stage	~50 s	12.169	0.302	0.632	18.49	0.828	0.199

Comparison against a strong two-stage FLUX+Qwen pipeline across stages:

Method	Frontal FID ↓	Frontal CLIP-I ↑	Frontal mIoU ↑	Multi-view (N=33) FID ↓	Multi-view CLIP-I ↑	Multi-view Corr-Acc ↑
FLUX+Qwen	47.64	0.776	0.338	51.77	0.804	0.174
Ours	36.33	0.781	0.353	19.12	0.822	0.191

Ablation Study

Stepwise ablation on the S2MV test set across all components (averaged over 3 seeds):

Configuration	PSNR ↑	SSIM ↑	LPIPS ↓	FID ↓	CLIP-I ↑	Corr-Acc ↑	Trainable Params
(a) Full model (CA3 + CSL + LoRA)	12.169	0.302	0.632	18.49	0.828	0.199	~41.3M
(b) w/o CSL (\(\lambda_{\text{corr}}=0\))	12.073	0.287	0.664	20.37	0.817	0.175	~41.3M
(c) w/o CA3 (LoRA only)	5.026	0.266	0.819	266.06	0.632	0.183	~5.9M
(d) w/o LoRA (CA3 only)	12.211	0.304	0.644	19.27	0.823	0.188	~35.4M
(e) w/o frame replication	12.198	0.304	0.652	42.54	0.786	0.174	~41.3M

Key Findings

1. Dramatic leap in visual realism and geometric consistency: The single-stage model slashes distributional FID from 46-48 down to 18.49 (over 60% improvement) and boosts correspondence accuracy Corr-Acc by 23% (0.161 to 0.199), while accelerating inference by 3.7× over SEVA and 42× over ViewCrafter.
2. CA3 is essential for viewpoint disambiguation: Removing CA3 (row c) triggers a catastrophic failure where FID deteriorates to 266.06 and PSNR collapses to 5.026, proving that vanilla video DiTs cannot deduce 3D camera geometry from LoRA alone.
3. CSL transforms latent capacity into sharp correspondence: Attention heatmaps reveal that under CSL, cross-view attention concentrates precisely on corresponding physical regions even across 180° back views, whereas models lacking CSL exhibit diffuse, uninformative attention fields.

Highlights & Insights

• Pioneering freehand sketch-to-3D scene generation: Successfully extends geometrically distorted, abstract freehand scene sketches to full 360° multi-view scene synthesis in a single denoising pass, establishing a practical bridge between human sketches and 3D virtual environments.
• Efficient geometric adapter design: Injects relative projective geometry via PRoPE within zero-initialized parallel adapters adding only 2.7% parameter overhead, avoiding catastrophic forgetting while supplying strong geometric inductive bias.
• Scalable correspondence regularization: Demonstrates that sparse SfM correspondence contrastive loss offers an effective, memory-tractable supervision strategy (~1 MB vs. hundreds of GB) for large-scale video diffusion models.

Limitations & Future Work

• Scale and synthetic bias of training data: The dataset comprises 9,222 samples generated through foundation models, which may propagate mild generation artifacts in complex occlusions.
• Resolution ceiling: Current synthesis is bound to \(480 \times 480\) resolution; integrating progressive super-resolution or direct Gaussian Splatting heads could further expand practical applicability.

Related Work & Insights

• Contrast with CAMEO: While concurrent CAMEO applies dense cross-entropy to UNet self-attention, this paper adopts sparse InfoNCE on parallel adapter projections, preventing appearance-geometry entanglement and overcoming the memory bottleneck in video diffusion architectures.
• Impact on 3D content creation: Providing an end-to-end mapping from rough sketches to consistent multi-view scenes opens streamlined pathways for rapid 3D asset authoring via downstream NeRF or 3DGS reconstruction.

Rating

• Novelty: 4.5 / 5.0
• Experimental Thoroughness: 4.5 / 5.0
• Writing Quality: 4.5 / 5.0
• Value: 4.5 / 5.0