Skip to content

Vista3D: Unravel the 3D Darkside of a Single Image

Conference: ECCV 2024
arXiv: 2409.12193
Code: GitHub
Area: 3D Vision

TL;DR

Vista3D is proposed, which generates diverse and consistent high-fidelity 3D meshes from a single image within 5 minutes. It utilizes a coarse-to-fine two-stage framework (3D Gaussian Splatting \(\rightarrow\) FlexiCubes differentiable isosurface refinement + decoupled texture) combined with viewpoint-aware diffusion prior composition.

Background & Motivation

  • Single-image-to-3D generation faces a dilemma: sparse reconstruction methods lead to blurriness, while pure generative methods relying on 2D priors fail to guarantee 3D consistency.
  • 3D-aware diffusion models like Zero-1-to-3 are trained on synthetic data, leading to oversimplified synthesis for unseen viewpoints.
  • Existing methods (e.g., DreamGaussian, Magic123) are either slow (taking hours) or produce low-quality results.
  • Core Problem: How to balance diversity in generating the "darkside" (unseen views) and global 3D consistency.

Method

Overall Architecture

  1. Coarse Stage: Rapidly generates coarse geometry using 3D Gaussian Splatting (approx. 30 seconds, 500 steps of optimization).
  2. Fine Stage: Extracts SDF from Gaussian Splatting \(\rightarrow\) Refines geometry using FlexiCubes differentiable isosurface representation + decoupled texture learning.
  3. Prior Composition: Integrates a 3D-aware prior (Zero-1-to-3 XL) and a diversity prior (Stable Diffusion) via a viewpoint-aware gradient constraint method.

Key Designs

Top-K Gradient Densification: Retains only the top-K Gaussian points with the highest gradients during each densification. This is more robust than traditional gradient-thresholding strategies and avoids over-densification caused by SDS randomness.

Scale and Transmittance Regularization: - Scale Regularization: \(L_1\) constraint to prevent overly large Gaussians. - Transmittance Regularization: Encourages progressive learning from transparency to solidity, with the threshold \(\tau\) annealing from 0.4 to 0.9.

Gaussian Splatting to SDF Conversion: Extracts the density field from Gaussians via local density queries, extracts the coarse mesh using Marching Cubes, and then queries mesh vertices to initialize the SDF of FlexiCubes.

Decoupled Texture Representation: Utilizes two independent hash encodings combined via an azimuth-related blending ratio \(\eta=(\cos(\Delta\theta)+1)/2\): - H_ref: Hash encoding facing the reference viewpoint. - H_back: Hash encoding facing the back viewpoint. - Solves the slow texture convergence issue at unseen viewpoints caused by overly strong supervision from the reference image.

Viewpoint-Aware Diffusion Prior Composition (Core): - Calculates the gradient ratio \(G\) of the two SDS gradients on the rendered image. - Sets an upper bound \(B_{upper}\) and a lower bound \(B_{lower}\) to constrain the ratio. - For near-reference viewpoints (\(\eta>0.75\)), dampens the upper bound with \((1-\eta)\); for far-away viewpoints (\(\eta<0.5\)), establishes a lower bound to prevent over-smoothing of the 3D prior. - \(B_{upper}\) is annealed from 100 to 10, and \(B_{lower}\) is annealed from 10 to 1.

Loss & Training

Coarse Stage: SDS loss + RGB/Mask reconstruction loss + Scale regularization + Transmittance regularization

Fine Stage: SDS loss + SDF regularization + Normal smoothness loss + RGB/Mask reconstruction loss

Key Experimental Results

Main Results

RealFusion dataset CLIP-Similarity:

Method Type CLIP-Sim↑ Time
DreamGaussian Optimization 0.738 2 min
Magic123 Optimization 0.802 2 h
DreamCraft3D Optimization 0.842 3.5 h
Vista3D-S Optimization 0.831 5 min
Vista3D-L Optimization 0.868 15 min

Quantitative evaluation on the GSO dataset:

Method PSNR↑ SSIM↑ LPIPS↓
DreamGaussian 23.43 0.832 0.092
Magic123 24.89 0.875 0.084
Vista3D-S 25.42 0.912 0.073
Vista3D-L 26.31 0.929 0.062

Ablation Study

User study (scale 1-4, higher is better):

Method View Consistency↑ Overall Quality↑
DreamGaussian 1.78 2.02
Magic123 2.11 1.83
Vista3D-S 2.87 2.81
Vista3D-L 3.24 3.33

Ablation validation: The coarse-to-fine transition is indispensable (pure isosurface easily collapses, while pure Gaussian fails to obtain smooth meshes); decoupled texture effectively reduces back-view artifacts.

Key Findings

  • Vista3D-S outperforms Magic123 (2 hours) within 5 minutes, achieving a 20x speedup.
  • Vista3D-L achieves comprehensive SOTA results on GSO (PSNR 26.31, LPIPS 0.062), leading by a large margin.
  • Viewpoint-aware prior composition enriches textures of unseen viewpoints while maintaining consistency between the front and back.
  • The interval-annealed timestep sampling strategy is more effective than linear annealing, reducing artifacts introduced by large timesteps.

Highlights & Insights

  • Redefines single-image-to-3D as a "generation task" rather than a "reconstruction task", emphasizing the diversity of the darkside.
  • The coarse-to-fine GS \(\rightarrow\) SDF conversion path is highly efficient and practical, taking the best of both stages.
  • The viewpoint-decoupled texture representation elegantly resolves the optimization imbalance caused by the dominance of reference-view supervision.
  • The method of fusing two priors with gradient ratio constraints is more robust and easier to tune than simple weighting.

Limitations & Future Work

  • Relies on SDS optimization, which requires separate optimization for each object.
  • Feed-forward methods (direct 3D prediction) are faster but currently lack sufficient quality.
  • Limited by the generalization capability of Zero-1-to-3 XL, which is trained on synthetic data.

Rating

  • Novelty: ⭐⭐⭐⭐ — Viewpoint prior composition and decoupled texture designs are novel.
  • Effectiveness: ⭐⭐⭐⭐⭐ — Comprehensive SOTA, excellent speed-quality balance.
  • Practicality: ⭐⭐⭐⭐⭐ — 5-minute high-quality 3D.
  • Recommendation: ⭐⭐⭐⭐⭐