Ultra3D: Efficient and High-Fidelity 3D Generation with Part Attention

Conference: ECCV2026
Paper: ECCV Paper
Project: Ultra3D
Area: 3D Vision
Keywords: image-to-3D, sparse voxels, VecSet, part attention, flow matching

TL;DR

Ultra3D first establishes a coarse object layout with compact VecSet tokens, then generates sparse voxel latents using attention organized along part boundaries, supporting 1024-resolution meshes with 6.7× faster part self-attention and 3.3× faster DiT inference than a full-attention baseline.

Background & Motivation

Sparse voxel generation represents an object through active voxel coordinates and their latent features: coordinates locate the approximate surface, while features describe its local geometry. Methods such as Trellis first generate coordinates and then produce latents at those positions, making them better suited to fine details than a compact latent representation alone. However, sparsity does not imply a short sequence: the paper reports approximately 20K or 60K active voxels on average at sparse voxel resolutions of 64 or 128 when generating a 1024-resolution mesh. If every voxel attends to every other voxel, computation still grows quadratically with token count, turning higher resolution into substantial latency and memory costs.

The two generation stages do not require equally expensive representations. The first stage needs plausible outlines, holes, and major connections, without directly predicting numerous coordinates through a long sequence; the second stage already has a spatial layout and primarily refines local surfaces. For 128-resolution voxels, the original first-stage approach models a \(32^3\) dense grid with roughly 32K tokens; compressing it further to \(16^3\) is cheaper but harms final geometry. Ultra3D therefore does more than replace every layer with local attention: it assigns coarse structure to VecSet and detailed geometry to sparse voxels.

Local computation must also follow object structure. Fixed spatial windows may split a coherent part into separate groups that cannot communicate, producing inconsistent local styles; part grouping instead gathers geometrically related voxels. Different parts still need global consistency, so the model retains limited low-resolution global communication rather than treating parts as entirely independent. Core Idea: use coarse layout to determine which high-resolution interactions are worth computing, replacing pervasive global attention with compact structure generation, within-part refinement, and inexpensive cross-part communication.

Method

Overall Architecture

The input is a conditioning image, and the output is a refined 3D mesh. The first-stage VecSet generator predicts a compact latent set and decodes a 512-resolution coarse mesh plus camera parameters; the mesh is then converted into part-labeled sparse voxels. The second stage generates latent features at those voxel coordinates, followed by sparse VAE decoding into the final mesh. The default sparse voxel resolution is 128 and the output mesh resolution is 1024; these describe different grid scales.

The key changes, in order, are VecSet Coarse Layout, Part Annotation, Low-Resolution Cross-Part Communication, and Part Attention. The last two are not single passes: Part-DiT repeatedly alternates one global communication block with three part attention blocks. The diagram groups them into one repeated module to avoid suggesting that local attention and global communication are separate generators.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    Image["Conditioning Image"] --> Layout["VecSet Coarse Layout<br/>Coarse Mesh and Camera"]
    Layout --> Labels["Part Annotation<br/>Labeled Sparse Voxels"]
    subgraph Refinement["Part-DiT: Repeated Refinement Blocks"]
        Global["Low-Resolution<br/>Cross-Part Communication"] --> Local["Part Attention<br/>Three Consecutive Blocks"]
    end
    Labels --> Global
    Image -->|DINOv2 Image Features| Local
    Local --> Decode["Sparse VAE Decoding<br/>1024-Resolution Mesh"]

Key Designs

1. VecSet Coarse Layout: generate a shape before extracting coordinates

VecSet encodes an object as a fixed-length unordered latent set, typically requiring only a few thousand tokens. The paper follows the relevant Hunyuan3D 2.0 configuration: a shape VAE encodes points sampled from mesh surfaces and decodes the latent representation into a signed distance function (SDF). During generation, the VecSet model first samples a shape representation to obtain a coarse mesh instead of directly predicting high-resolution sparse coordinates. Voxelization then extracts the surface support needed by the second stage. This exploits an asymmetry in the task: the coarse mesh does not need final fine-scale geometry if it provides a credible layout at a voxel resolution of 64 or 128.

Camera parameters are also encoded into the latent vectors and estimated by the VecSet decoder during generation. They are not merely auxiliary viewing parameters: later stages need them to project 3D parts onto the conditioning image and select cross-attention tokens. This also limits what refinement can correct: the second stage generates features at existing voxel coordinates, so it should not be assumed to recover missing major parts or severely incorrect topology. That limitation is an architectural interpretation, not a separately measured failure rate in the paper.

2. Part Annotation: turn geometry groups into computational boundaries

Raw meshes usually lack part labels, so the method uses PartField to establish groups. It uniformly samples surface points, feeds them through PartField to obtain a triplane feature field, queries their features, and voxelizes the mesh. Features of points within each voxel are averaged into part-aware voxel features, which are then partitioned through agglomerative clustering. Training preprocessing fixes the number of groups at 8 to avoid searching for a suitable cluster count for every object. These groups are segmentation regions for attention routing, not necessarily manually named semantic classes.

Clustering is followed by filtering to reject shapes dominated by one group and shapes with spatially fragmented labels. Let \(r_k\) denote the fraction of all active voxels assigned to group \(k\); the paper measures imbalance through the sum of squared group fractions, which its prose defines as:

\[ B = \sum_{k=1}^{K} r_k^2. \]

The implementation removes samples with \(B>0.25\) and samples whose neighborhood inconsistency exceeds 25%. Neighborhood inconsistency measures local label disagreement, but the main text does not specify the exact adjacency stencil or counting procedure, so it cannot support a fully reproducible operator definition. Including filtering, annotation takes approximately 2 seconds per mesh on one A800 GPU. Training annotates ground-truth meshes, whereas inference applies the same procedure to the generated coarse mesh; test-time part labels are not supplied as ground-truth answers.

3. Low-Resolution Cross-Part Communication: retain global consistency without paying the global cost everywhere

Refining parts entirely independently could weaken stylistic and structural coordination between them. Part-DiT therefore retains some full attention: a residual block downsamples sparse voxels, applies full attention, upsamples the resulting features, and fuses them back at the original resolution. Cross-part information is exchanged through a short sequence, while the high-resolution sequence remains available to subsequent part attention. This differs from keeping the entire refinement stage at low resolution merely to save computation: local detail processing still accesses a denser set of active voxels.

The architecture repeatedly stacks one low-resolution full-attention block followed by three part attention blocks. This explains why 6.7× faster part self-attention does not imply the same speedup for the complete DiT: global communication, upsampling, downsampling, and other operations remain. It also explains why the paper does not dismiss all global dependencies; it removes repeated high-resolution interactions across many unrelated regions. The main text provides no numerical ablation that removes every global block, so its independent quality contribution cannot be quantified here.

4. Part Attention: constrain both 3D interactions and image conditioning

Part self-attention permits a voxel token to attend only to tokens in its own group. For group sizes \(n_k\), the number of attention pairs changes from \(L^2\) to \(\sum_k n_k^2\), where \(L\) is the total number of active voxels. When the \(K\) groups are approximately balanced, this is about \(1/K\) of global attention, matching the paper's conditional near-\(K\)-fold saving rather than guaranteeing it for arbitrary segmentations. The squared-fraction filter therefore does more than clean labels: it avoids a dominant part that would bring computation close to global attention. This last connection is an interpretation of group sizes and computational complexity.

Part cross-attention addresses a different correspondence: which conditioning-image tokens a 3D part should read. Using camera parameters, the model projects each part's voxels onto the image and assigns their part labels to the covered image tokens. An image token can be associated with multiple parts because occlusion and overlapping projections place different 3D regions at the same 2D location. A voxel attends only to image tokens whose associated label set contains its own part, so the mapping should not be interpreted as mutually exclusive single-label image segmentation. Training uses ground-truth cameras, while inference uses cameras estimated by the VecSet decoder, making projection quality dependent on camera estimation.

Equations (1) and (2) are corrupted in the cached extraction; the explanation above follows the intact surrounding prose without reconstructing missing mask-value expressions. The mechanism is nevertheless explicit: self-attention groups tokens by part, and cross-attention selects image tokens through part projections. Fixed windows also reduce pairs but do not necessarily follow part boundaries, which is the substantive distinction between these localization strategies.

A Worked Example

Consider a chair image containing a seat, backrest, and thin legs as an illustrative example, not an additional experiment. VecSet first produces a 512-resolution coarse mesh and estimates the input camera; voxelization and PartField annotation yield a 128-resolution layout with part groups. Fixing 8 groups does not require exactly 8 everyday named components, since clustering may subdivide a larger surface. Part-DiT exchanges global information through downsampled features, then refines surfaces within each group while reading conditioning features from the projected image tokens. Even if the backrest and legs overlap in the image, one image token can serve multiple parts rather than receiving a forced unique assignment. After generative sampling, sparse latents decode into a 1024-resolution mesh; if a leg was absent from the coarse layout, local refinement does not guarantee its recovery.

Loss & Training

The generators follow conditional flow matching, with their corresponding VAEs learning shape representations; the main contribution is representation allocation and attention routing, not a new loss. Conditioning uses DINOv2 image embeddings, and the part-aware DiT is scaled to 1.8B parameters. Training uses a private dataset of 1000k high-quality 3D objects processed by the part annotation and filtering pipeline. The sparse VAE trains for 2 days on 32 A800 GPUs; VecSet-DiT trains for 15 days on 128 GPUs; the part-aware DiT trains for 15 days on 256 GPUs with a total batch size of 256. The part-aware DiT uses AdamW with a learning rate of \(10^{-4}\), no weight decay, and an EMA rate of 0.9999. The unconditional training probability is 0.1, and timesteps follow a Logit-Normal distribution with mean 0 and standard deviation 1. Inference uses 25 sampling steps and classifier-free guidance of 3.5; training fixes 8 part groups, but inference supports other group counts.

Key Experimental Results

Main Results

The following efficiency results come from Table 1 on page 15, with full attention as the baseline and FlashAttention-2 used by both approaches. The default output is a 1024-resolution mesh with 128-resolution sparse voxels; the table does not provide complete timing hardware, token distributions, or absolute durations for each entry.

Measured Component	Speedup over Full Attention	Supported Conclusion
Part self-attention	6.7×	Within-part interactions substantially reduce attention computation
Part cross-attention	4.1×	Projection constraints also reduce cross-modal interaction costs
DiT training	3.1×	Block-level savings translate into faster training
DiT inference	3.3×	Model-level gains are smaller than those of individual attention operations

The text on page 13 separately states that the full-attention baseline often takes over 15 minutes per mesh, while this pipeline averages approximately 4 minutes. An often-exceeded duration and an average are not matched timing statistics, so their ratio should not replace the speedup in Table 1. Figure 5 compares geometric details with Trellis, Hi3DGen, Direct3D-S2, and an anonymous commercial model, but this cache provides no corresponding numerical quality table.

Ablation Study

The following table summarizes actual visual ablations rather than inventing numerical quality scores; sources are Figure 2 on page 3, the experimental description on page 12, and Figure 6 on page 14. The cache lacks the supplementary material mentioned by the paper, so no preference percentages, quality scores, or additional quantitative ablations are supplied here.

Ablation Dimension	Original Configuration	Authors' Reported Observation	Evidence Boundary
Attention replacement	Full attention	Quality comparable to part attention	Figure 2; further fine-tuned from a part-attention checkpoint
Attention replacement	3D window attention, space split into 8 fixed regions	Degraded geometry and inconsistent styles	Figure 2; also further fine-tuned
Attention replacement	Part attention	Preserves geometric continuity and local detail	Figure 2; no corresponding numerical quality metric
Inference group count	4, 8, 12, 16 groups, trained with 8	Good generation quality is illustrated under these settings	Figure 6; does not establish robustness to arbitrary segmentation

Key Findings

• Part self-attention provides the largest local speedup, while complete DiT inference improves by only 3.3×, showing that other computation remains relevant.
• The contrast between fixed windows and part groups concerns how local regions are defined, not simply whether local attention is better than global attention.
• Evidence for varying inference group counts consists of preserved visual quality, without speed curves, confidence intervals, or a sweep over segmentation errors.

Highlights & Insights

• Choose representations by stage responsibility. Limited VecSet detail is not necessarily fatal for coarse layouts, while expressive sparse voxels are reserved for the stage that needs fine geometry.
• Use part labels for computational routing. Segmentation can determine which attention interactions are worth executing rather than serving only as a final prediction task, making the strategy structurally informed rather than merely compressive.
• Separate the scales of global consistency and local precision. A few low-resolution global blocks coordinate parts, while high-resolution local blocks refine surfaces, avoiding an all-or-nothing choice about global modeling.

Limitations & Future Work

• The authors acknowledge remaining latency: sparse voxel sequences are long and locally constrained attention may still approach dense computation; approximately 4 minutes on average is not instantaneous interaction.
• The method depends on external PartField annotations and estimated inference cameras; coarse-mesh errors, fragmented groups, or projection errors may affect feature generation, an architectural risk rather than a quantified failure result.
• The private 1000k-object dataset and substantial training resources complicate independent reproduction; visual geometric advantages alone cannot separate method gains from data and model scale.
• Supplementary quantitative experiments are unavailable in this cache, so the abstract's preference claim cannot be expanded into a verified win rate, and visual robustness is not a statistical guarantee.
• Measuring quality and speed under group imbalance, camera perturbations, and missing coarse structures would be useful follow-up work; these are reader suggestions, not completed experiments.

Related Work & Insights

• Trellis: supplies structured sparse latents and the two-stage flow-matching framework; Ultra3D redesigns coordinate generation and reorganizes attention during latent feature generation.
• 3DShape2VecSet / Hunyuan3D 2.0: compact vector sets support efficient overall shape generation; Ultra3D places them before sparse voxel refinement instead of requiring one representation to capture all details.
• Sparc3D: its SparConv-VAE supplies the sparse shape encoder and decoder, so the detailed geometry representation is not introduced entirely from scratch.
• PartField / PartCrafter: the former currently supplies external segmentation, while the latter represents direct part-aware shape generation; the authors suggest such methods could replace external annotation rather than claiming that integration is already implemented.

Rating

These are reading assessments, not experimental metrics reported by the paper. - Novelty: 4/5. Mixed representations, part routing, and multiscale communication address a concrete high-resolution generation bottleneck. - Experimental Thoroughness: 3/5. Efficiency gains are explicit, but quality and grouping robustness in the available main text rely mainly on qualitative demonstrations. - Writing Quality: 4/5. Stage responsibilities and motivations are clear, while some reproducibility details and quantitative evidence require the supplement. - Value: 4/5. Useful guidance for allocating computation in high-resolution 3D generation, though low-resource and instantaneous generation remain unresolved.