MSVS-VAE: Multi-Scale Anchored VecSet for High-Fidelity 3D Reconstruction

Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: 3D VAE, VecSet, 3D Reconstruction, Point-Shuffle Upsampling, Local Attention Convolution

TL;DR

MSVS-VAE tackles the pervasive fidelity and efficiency bottlenecks of set-based 3D representations by hierarchically densifying anchored VecSet latents via point-shuffle upsampling and substituting global cross-attention with localized AVS-Conv, delivering voxel-level reconstruction fidelity with \(\sim\)10\(\times\) higher compactness and up to 14.4\(\times\) faster decoding.

Background & Motivation

Modern native 3D generative modeling is rapidly converging toward the latent diffusion paradigm, where the geometric fidelity and compression capacity of the underlying 3D Variational Autoencoder (VAE) form the ultimate performance ceiling. An ideal 3D VAE must satisfy two inherently contradictory demands: it must remain sufficiently compact to allow computationally tractable diffusion training and sampling, while simultaneously possessing rich spatial capacity to faithfully recover intricate geometric details and thin topologies during decoding.

Existing methodologies predominantly bifurcate into two representation paradigms, both carrying non-trivial compromises. Sparse voxel-based methods (such as SparseFlex, Direct3D-S2, and SparC3D) operate over explicit 3D regular grids; while their localized operators excel at recovering sharp contours, they require preserving tens to hundreds of thousands of active voxels (commonly 50k to 210k tokens), incurring immense memory footprints and rendering generative modeling difficult. Furthermore, because geometry is strictly pinned to discrete grid indices, minor structural mispredictions directly precipitate topological breakdowns such as holes and non-watertight artifacts. Conversely, set-based representations (e.g., VecSet, Lattice) encode shapes into permutation-invariant sets of 1D latent vectors, naturally supporting continuous, arbitrary-resolution queries and clean isosurface extraction; nevertheless, their geometric reconstruction fidelity has persistently lagged behind voxel counterparts.

This fidelity gap stems directly from two structural flaws in standard VecSet architectures. First, decoding relies on global cross-attention between spatial query points and all latent tokens, which acts as a global low-pass filter, imposing excessive global smoothness and eroding localized high-frequency details. Second, to remain tractable for generative modeling, VecSet latents are kept extremely sparse (often around 4k tokens)—dozens of times fewer than sparse voxels—severely restricting the spatial capacity needed for fine-grained geometry.

The core idea is to progressively densify anchored VecSet latents through hierarchical point-shuffle upsampling while replacing costly global cross-attention with a geometry-aware local aggregation operator (AVS-Conv) and multi-scale query decoding, uniting continuous set-based compactness with voxel-level reconstruction fidelity.

Method

Overall Architecture

MSVS-VAE comprises an encoder that extracts a compact anchored latent set and a hierarchical decoder that reconstructs continuous signed distance fields (SDF). In the encoding stage, surface points with normals are processed by a lightweight PointNet and aggregated into voxel anchor queries via AVS-Conv, followed by residual self-attention blocks to construct the base anchored VecSet. In the decoding stage, the compact latents undergo multi-stage Hierarchical Point-Shuffle Upsampling to expand spatial token density, followed by Multi-Scale Query Decoding that queries both coarse and fine latent hierarchies via localized AVS-Conv to predict truncated SDF values.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Surface Points + Anchor Queries"] --> B["AVS-Conv Local Encoding<br/>aggregate point features into anchor queries"]
    B --> C["Self-Attention Refinement<br/>yields base anchored VecSet Z(0)"]
    C --> D["Hierarchical Point-Shuffle Upsampling<br/>split geometric centers + shuffle feature channels"]
    D --> E["Multi-Scale Query Decoding<br/>fuse multi-scale AVS-Conv features"]
    E --> F["Continuous TSDF Output<br/>extract watertight 3D meshes"]

Key Designs

1. Geometry-Aware Local Aggregation (AVS-Conv): Eliminating Global Attention Bottlenecks

Standard VecSet decoding computes dense cross-attention across all \(M\) latents for every query point, which scales at \(O(MN)\) and averages out sharp features across global distances. AVS-Conv replaces global attention with a localized query-to-support operator. For each spatial query point \(q_i\), it retrieves its \(K\) nearest neighbors \(\mathcal{N}(i)\) within the support set under Euclidean distance \(\|p_j - q_i\|_2\). Dynamic aggregation weights are predicted from relative position offsets \(\Delta p_{ij} = p_j - q_i\) via positional embedding \(\gamma(\cdot)\) and an MLP: $\(w_{ij} = \mathrm{MLP}(\gamma(\Delta p_{ij})), \quad j \in \mathcal{N}(i)\)$ Features are aggregated via channel-wise multiplication \(\sum_{j \in \mathcal{N}(i)} w_{ij} \odot f_j\), stabilized by a lightweight residual connection (mean pooling in encoder; identity in decoder), and refined by a Transformer-style Feed-Forward Network with LayerNorm. This bounds computation strictly to local neighborhoods, slashing latency while preserving high-frequency geometric gradients.

2. Hierarchical Point-Shuffle Upsampling: Progressively Expanding Latent Spatial Capacity

To keep the initial representation compact for downstream diffusion while expanding spatial resolution for decoding, the decoder applies \(L\) progressive upsampling stages. Starting from initial latents \(\mathcal{Z}^{(0)}\), each transition from level \(l\) to \(l+1\) expands latent token count by a factor of \(r\). Specifically, a feature shuffle MLP splits each feature \(z_m^{(l)}\) into \(r\) sub-features \(\{\tilde{z}_{m,t}^{(l)}\}_{t=1}^r\) with halved channel dimensions \(C_{l+1} = C_l / 2\). Concurrently, a point split MLP predicts \(r\) spatial coordinate offsets \(\{\Delta c_{m,t}^{(l)}\}_{t=1}^r\) to spawn new anchor centers: $\(c_{m,t}^{(l+1)} = c_m^{(l)} + \Delta c_{m,t}^{(l)}, \quad z_{m,t}^{(l+1)} = \tilde{z}_{m,t}^{(l)}\)$ The resulting densified set \(\mathcal{Z}^{(l+1)}\) is subsequently refined via AVS-Conv self-aggregation over its updated centers and features. This progressively drives the latent tokens closer to the actual physical surface, granting sufficient spatial degrees of freedom to represent complex local geometry without bloating the global latent budget.

3. Multi-Scale Query Decoding: Coarse-to-Fine Fusion to Suppress Local Artifacts

Relying exclusively on the finest densified latent scale restricts the receptive field to an overly localized domain, causing high-frequency noise and bumpy artifacts in low-curvature or flat regions. Multi-Scale Query Decoding addresses this by querying features across all progressive latent scales \(\mathcal{Z}^{(0)}, \dots, \mathcal{Z}^{(L)}\) using separate AVS-Conv operators. The extracted multi-scale feature vectors are concatenated and fed into a lightweight fusion MLP: $\(h_i = \mathrm{Concat}\big(\mathrm{AVS}^{(0)}(q_i; \mathcal{Z}^{(0)}), \dots, \mathrm{AVS}^{(L)}(q_i; \mathcal{Z}^{(L)})\big)\)$ The fused vector \(h_i\) is mapped by an output MLP to the predicted TSDF value \(\hat{s}_i\). Coarse scales anchor structural coherence and global topology, while fine scales inject sharp geometric residuals.

Loss & Training

The framework is supervised using Truncated Signed Distance Function (TSDF) targets, \(\bar{s} = \mathrm{clamp}(s/\tau, -1, 1)\) with truncation distance \(\tau = 1/128\). The total objective combines \(\ell_1\) and squared \(\ell_2\) geometric reconstruction losses with a Kullback-Leibler (KL) divergence regularizer: $\(\mathcal{L}_{\mathrm{total}} = \lambda_{\mathrm{recon}} \left(\|\hat{s} - \bar{s}\|_1 + \|\hat{s} - \bar{s}\|_2^2\right) + \lambda_{\mathrm{KL}} \mathcal{L}_{\mathrm{KL}}\)$ Training follows a two-stage progressive chunk-based protocol on 400k meshes from Objaverse: first, 300k steps of full-asset pre-training with \(M=4096\) tokens to learn global topology; followed by 100k steps of localized random 3D chunk fine-tuning, where the same 4k budget is allocated strictly within cropped local chunks to supercharge high-frequency surface modeling.

Key Experimental Results

Main Results

Reconstruction quality is evaluated across Objaverse, ABO, and in-the-wild datasets using bidirectional Mesh Distance (MD, scaled by \(10^4\)) and F1-score (scaled by \(10^2\)) at thresholds 0.01 and 0.001. Benchmarks compare MSVS-VAE against both leading set-based baselines (Dora, HY3D2.1, Lattice) and sparse voxel-based methods (Direct3D-S2, SparC3D, SparseFlex).

Method	Type	# Latent Tokens	Objaverse (MD↓ / [email protected]↑)	ABO (MD↓ / [email protected]↑)	in-the-wild (MD↓ / [email protected]↑)
Direct3D-S2	Sparse Voxel	\(\sim\)74k	3.301 / 95.100	2.091 / 97.181	2.771 / 97.020
SparC3D	Sparse Voxel	\(\sim\)50k	2.463 / 97.788	1.515 / 99.354	1.947 / 98.137
SparseFlex-512	Sparse Voxel	\(\sim\)50k	3.745 / 93.182	2.017 / 97.449	4.404 / 90.634
SparseFlex-1024	Sparse Voxel	\(\sim\)210k	2.165 / 97.327	1.092 / 99.902	2.014 / 98.944
Dora	Set-based	4k	17.352 / 48.450	7.450 / 73.770	14.641 / 44.110
HY3D2.1	Set-based	4k	11.485 / 63.085	5.384 / 84.036	11.895 / 57.096
Lattice	Set-based	4k	11.502 / 82.365	2.941 / 97.107	10.703 / 74.764
Lattice	Set-based	20k	3.183 / 95.373	2.283 / 99.088	3.737 / 91.872
MSVS-VAE (Ours)	Set-based	4k	5.644 / 86.779	1.885 / 98.902	4.589 / 87.207
MSVS-VAE (Ours)	Set-based	10k	2.422 / 96.458	1.485 / 99.843	2.416 / 95.842
MSVS-VAE (Ours)	Set-based	20k	1.695 / 98.930	1.432 / 99.927	1.642 / 98.775
MSVS-VAE (Ours)	Set-based	40k	1.395 / 99.621	1.216 / 99.924	1.231 / 99.726

On the Dora-Bench Level-3 and Level-4 sharp-geometry benchmarks, MSVS-VAE-40K scores an MD of 1.06 and sharp-edge F-score (S-F) of 97.4% on Level-3 (outperforming SparseFlex's MD 1.74 and S-F 95.6%), confirming superior precision along delicate high-curvature boundaries.

Ablation Study & Efficiency

Table 3 in the original paper verifies the contribution of each module, while Table 4 assesses query latency for 200k spatial points per chunk under different token budgets.

Configuration	Objaverse MD↓	Objaverse [email protected]↑	ABO MD↓	in-the-wild MD↓	Note
Full model (MSQ + PSU)	1.395	99.621	1.216	1.231	Complete hierarchical model
w/o MS-Query (MSQ)	1.690	99.322	1.439	1.626	Single-scale fine latents introduce surface ripple noise
w/o Upsample (PSU)	3.693	93.281	2.818	4.021	Removing upsampling collapses high-frequency geometric fidelity

Query Latency Benchmark (Original Paper Table 4): For 200k query points, Lattice requires 792.21ms, 1419.27ms, and 3068.05ms for 10k, 20k, and 40k tokens respectively. Benefiting from localized KNN attention and cuBQL BVH acceleration, MSVS-VAE requires only 143.75ms, 151.22ms, and 213.29ms, achieving 5.5\(\times\), 9.4\(\times\), and 14.4\(\times\) speedups.

Key Findings

• Hierarchical Point-Shuffle Upsampling (PSU) is the primary driver of fidelity: removing it worsens Objaverse MD from 1.395 to 3.693, proving that spatial token density is essential for recovering delicate geometric features.
• At 20k tokens, MSVS-VAE already achieves an Objaverse MD of 1.695, outperforming SparseFlex-1024 (2.165) which requires 210k tokens, demonstrating \(\sim\)10\(\times\) higher compactness.
• Reconstructed surfaces are topologically robust and watertight, completely avoiding the surface sticking, holes, and fragmented non-manifold components seen in discrete sparse voxel grids.

Highlights & Insights

• Decoupling Generation and Reconstruction Densities: By keeping initial latents compact (e.g., 4k tokens) for tractable diffusion and upsampling internally during decoding, MSVS-VAE reconciles generative scalability with sub-millimeter surface fidelity.
• Redefining Set-Based Decoding via Local Convolutions: It refutes the common assumption that continuous set-based fields require global Transformer cross-attention, demonstrating that geometry-aware local aggregation (AVS-Conv) delivers sharper edges and order-of-magnitude speedups.
• Test-Time Scalability: The representation can smoothly scale token budgets at inference time (4k to 80k), providing a flexible dial between computational budget and geometric detail.

Limitations & Future Work

• Industrial Sharp Corners: While significantly outperforming prior continuous representations, ultra-sharp CAD features may still suffer minor fillet rounding compared to dual contouring polygon meshes.
• Decoding Latency vs Dense Convolutions: Although up to 14.4\(\times\) faster than global cross-attention, dynamic BVH search and MLP queries remain slightly slower than direct regular 3D convolution grid sweeps.
• Absence of Appearance Modeling: Current formulation focuses entirely on geometry TSDF modeling; integrating PBR materials, albedo, and textures into the hierarchical VecSet remains unaddressed.

Related Work & Insights

• vs Lattice (ECCV 2024 / arXiv 2024): While Lattice introduced anchored VoxSets to improve spatial locality, it retained global cross-attention, taking over 3 seconds at 40k tokens and suffering from global oversmoothing. MSVS-VAE cuts MD from 2.838 to 1.395 and accelerates decoding by 14.4\(\times\).
• vs SparseFlex (ICCV 2025) & SparC3D: Discrete voxel grids frequently produce topological holes, open boundaries, and surface sticking; MSVS-VAE matches or exceeds their fidelity with only 10% to 40% of their token footprint while guaranteeing watertight manifolds.

Rating

• Novelty: ⭐⭐⭐⭐☆ Elegantly integrates point-shuffle upsampling and localized geometric convolution into set-based 3D VAEs.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluations across three complex benchmarks, sharp-edge subsets, detailed ablations, and query latency measurements.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-motivated problem framing, crisp architectural exposition, and thorough empirical comparisons.
• Value: ⭐⭐⭐⭐⭐ Provides a foundational, highly compact, and fast 3D geometry representation for next-generation 3D generative diffusion models.

Related Papers

• [CVPR 2026] LATTICE: Democratize High-Fidelity 3D Generation at Scale
• [ECCV 2026] Large-Scale High-Quality 3D Gaussian Head Reconstruction from Multi-View Captures
• [ECCV 2026] ROAR-3D: Routing Arbitrary Views for High-Fidelity 3D Generation
• [ECCV 2026] MAC-Splat: Multi-Attribute Consistency for High-Fidelity Sparse-View Reconstruction
• [ECCV 2026] Generalizable Neural Reconstruction of High-Fidelity Surfaces via Sparse Volumetric Representations