3D-ReGen: A Unified 3D Geometry Regeneration Framework

Conference: ECCV2026
Paper: Official page / Paper PDF
Area: 3D Vision
Keywords: geometry regeneration, VecSet, conditional diffusion, sparse-view reconstruction, 3D editing

Naming and source note: the conference manifest calls this paper 3D-ReGen, whereas the supplied PDF cache uses MeshReGen for both its title and method. This note retains the manifest title and uses MeshReGen below. The cache contains the full main-text method, experiments, and references, but not the appendices cited by the paper. Appendix-only implementation details, an arXiv identifier, and a code URL are not inferred.

TL;DR

MeshReGen represents coarse meshes, incomplete point clouds, and masked geometry as VecSet conditions, concatenates their tokens with noisy target tokens, and learns image-guided 3D regeneration from automatically constructed degradation pairs, reaching CD 0.0081 and F-score 0.4913 on four-view GSO reconstruction while also supporting enhancement and local editing.

Background & Motivation

Image-to-3D systems usually produce an entire object from a reference image, but production workflows rarely start with an image alone. An artist may already have a blockout specifying volume and proportions, a scan may supply an incomplete point cloud, or an existing mesh may need only a local replacement. Images describe appearance and semantics without fully specifying hidden structure, precise proportions, or the identity of existing geometry. Generating again from the image can therefore produce a plausible object that is no longer the intended one.

The geometric input is not necessarily a reliable constraint either. Blockouts omit detail, scans miss occluded regions, and low-quality generated assets can contain distorted shapes and floating fragments. A control mechanism designed for clean voxels or bounding boxes may preserve those errors. Collecting separate high/low-quality pairs for enhancement, reconstruction, and editing would instead make every application depend on expensive supervision. The paper treats these settings as instances of recovering high-information geometry from low-information geometry and an optional image, rather than deterministically smoothing or subdividing an input mesh.

Its starting point is the VecSet space of a pretrained 3D generator: if target shapes can be encoded as sets of latent vectors, coarse conditions can use the same encoder, leaving the generative prior to determine which structure to preserve and which details to recover. Core idea: let degraded geometry act as a fixed condition during target denoising in a shared 3D latent space, and synthesize different degradations from complete objects so that one architecture can learn multiple regeneration tasks.

Method

Overall Architecture

The inputs are a low-information shape \(y_L\) and an optional guidance image \(I\); the output is a complete, detailed shape \(x_H\). The model learns the conditional distribution \(p(x_H\mid y_L,I)\), not a deformation function that moves each input vertex. It can consequently infer missing surfaces and alter local topology.

The pipeline begins with Self-Supervised Degradation Pairs, which provide training examples, followed by Shared VecSet Encoding and Fixed-Condition Concatenation Denoising. Coarse geometry becomes a small set of condition tokens, target generation starts from noisy tokens, and DINOv2 image features supply guidance through cross-attention. The denoised target tokens are decoded through the VAE's signed-distance queries into a shape. Mask Capacity Allocation is the editing branch of data preparation, converting a locally missing region into the same conditioning interface.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Source["Complete training objects"] --> Pairs["Self-Supervised<br/>Degradation Pairs"]
    Pairs -->|Enhancement or reconstruction| Encode["Shared VecSet<br/>Encoding"]
    Pairs -->|Editing| Mask["Mask Capacity<br/>Allocation"]
    Mask --> Encode
    Input["Inference input<br/>Coarse mesh or point cloud"] --> Encode
    Encode --> Denoise["Fixed-Condition<br/>Concatenation Denoising"]
    Image["Guidance image<br/>DINOv2 features"] --> Denoise
    Noise["Random target tokens"] --> Denoise
    Denoise --> Output["VAE decoding<br/>Complete 3D shape"]

Here, unified means a shared architecture and base data source, not one final checkpoint that performs every task without adaptation. The enhancement model, VGGT-based reconstruction model, and editing model undergo different training or fine-tuning procedures.

Key Designs

1. Self-Supervised Degradation Pairs: expose the model to the errors it must repair

Rather than collecting large manually annotated sets of bad/good shapes, the authors start with approximately 1 million proprietary artist-authored 3D objects, synthesize low-information versions, and retain the originals as targets. Enhancement degradation is not simply mesh smoothing. Random objects are arranged in a 2D grid to form a pseudo-scene, the entire scene is normalized to \([-1,1]^3\) and VecSet-encoded, and its tokens are noised at an intermediate timestep and denoised in an SDEdit-like procedure. Decoding and extracting individual objects yields damaged inputs. More objects share the same scene token budget, reducing the information available per object and approximating the loss of small-object detail in generated scenes. Denser grids generally impose stronger degradation; concrete grid sizes and corruption timesteps are not supplied in the cached main text.

For reconstruction, each ground-truth object is rendered from 1-4 views and passed through VGGT to produce a sparse, incomplete point cloud. VecSet requires point coordinates and normals, so covariance analysis supplies estimated normals, and the reconstruction's scale and pose are aligned with the target. One rendered view is randomly chosen as the guidance image. Thus, multi-view evidence primarily enters through the VGGT point cloud, rather than all views being supplied simultaneously to the image branch. Camera azimuth and focal length are sampled uniformly, with elevation in \([-10^\circ,30^\circ]\). The two constructions approximate latent compression damage and reconstruction incompleteness, respectively: the goal is to teach correction, not merely obedience to ideal control geometry.

2. Mask Capacity Allocation: size the editing freedom to the missing region

Editing training selects a random 3D region in a complete target and removes its points and normals. The remaining geometry carries object identity. With a total condition budget of \(C\) tokens, the visible region is encoded into \(C-k\) tokens, and \(k\) learnable mask positional tokens are appended to specify where reconstruction is needed. The training image depicts the complete target, making the task self-supervised completion from remaining geometry and a target image. Manually authored before/after editing mesh pairs are unnecessary.

The mask-token count adapts to the masked fraction \(r\) of source points. The rule given in the paper is:

\[ k=\max\{\lfloor Cr\rfloor,k_{\min}\}. \]

Here \(k_{\min}\) is a minimum editing capacity, not an upper bound. Small masks still receive sufficient representation capacity, while larger masks allow more substantial geometric changes. At inference, a source view is rendered, the projected 3D edit region is masked in 2D, and an image-inpainting model follows the requested text instruction to create the target image. The geometry branch preserves the unmasked context, while the image branch describes the replacement. The main text does not specify a numerical \(k_{\min}\). Removing points and regenerating in latent space also does not impose a strict guarantee that every vertex outside the mask remains unchanged.

3. Shared VecSet Encoding: represent coarse conditions and detailed targets in the same space

The VecSet encoder samples surface points with normals, uses farthest point sampling to initialize token locations, and pools information from the full point cloud through cross-attention. Its decoder reverses the interaction: arbitrary 3D query points attend to latent tokens to predict signed distances. This is a queryable shape representation, not a collection of latent tokens interpreted directly as output vertices. An incomplete conditioning point cloud can therefore guide a complete, continuous output surface.

Its useful property is variable token count. Low-information conditions use \(C=512\) tokens, whereas the final high-information target uses \(K=2048\). A coarse condition does not need a full high-resolution token allocation merely to support detailed output. Concatenated self-attention still grows quadratically with total sequence length, so this is not cost-free conditioning; it reduces the extra overhead relative to equally large condition and target sets. Reusing the encoder also places the two signals in compatible spaces instead of defining a separate task-specific point-cloud, voxel, or bounding-box interface.

4. Fixed-Condition Concatenation Denoising: introduce geometric control through existing self-attention

Condition and target tokens cannot simply be mixed without distinction. Prior work observed that concatenating two VecSets can approximate merging their underlying shapes, which is not the desired regeneration behavior. The authors first pass condition tokens through a zero-initialized MLP and add positional embeddings associated with their spatial locations. This eases the introduction of new control tokens into a pretrained generator and distinguishes conditioning content from content being generated. The processed conditions are concatenated with noisy target tokens along the token dimension and handled by the same DiT self-attention; image guidance continues to use the original cross-attention path.

The following equation reformats the denoising interface whose notation is damaged in the PDF extraction. It is not an additional proposed loss:

\[ \widehat z^H_{t-1}=\phi\!\left(z^H_t\mathbin{\|}\left[\operatorname{MLP}(E(P_L;C))+\operatorname{PE}(\widehat P_L)\right]\mid I,t\right). \]

\(P_L\) is the low-information point cloud, \(\widehat P_L\) denotes sampled positions associated with its condition tokens, and \(\|\) denotes token-axis concatenation. Only the target tokens are updated at each step. The condition tokens remain fixed throughout sampling rather than being noised or reconstructed. The model can consequently synthesize detail from random target latents while retaining access to the same coarse geometric reference. Unlike adding a new 3D cross-attention path or summing conditions elementwise into target tokens, concatenation lets sets with different sizes and information content interact through self-attention. The experiments support this choice for the authors' backbone and data, not universal optimality across architectures.

A Worked Example

Consider the four-view object-reconstruction workflow evaluated on GSO. VGGT first turns the input images into a point cloud covering observed surfaces. Point normals are estimated, and geometric alignment is handled as required by the training-pair construction; the cache does not specify the complete inference-time alignment implementation. The point cloud is compressed into 512 fixed condition tokens, while 2048 target tokens start from random noise.

The image branch receives a guidance view, with evidence from the other views already incorporated into the VGGT geometry. Across 100 DDPM sampling steps, the DiT continuously reads the conditions to retain observed global structure while using its generative prior to infer unobserved surfaces. Decoding produces a new complete shape, rather than directly applying Poisson reconstruction to the VGGT points. The corresponding four-view IoU in Table 2 is 0.7574; this is a dataset-level result, not the score of an individual object in this illustrative walkthrough.

Loss & Training

The backbone is an in-house pretrained VecSet-based 3D generator supplying initial VAE and DiT weights. Enhancement training combines enhancement and editing augmentations, progressively increasing target token counts to 512, 1024, and 2048 over 49K, 112K, and 390K steps, respectively. The reconstruction model fine-tunes the enhancement model on VGGT outputs for 25K steps. The editing model uses masked examples and 2048 target tokens, with convergence reported after 25K steps. Every task uses 512 condition tokens.

Training uses v-prediction, a scaled linear noise schedule, and image-condition dropout with probability 0.1. Additional geometric augmentation introduces Gaussian blobs that imitate floating fragments and perturbs normals; image augmentation applies cropping, color jitter, random backgrounds, and blur. Inference uses 100 DDPM steps and CFG scale 5. The cached main text does not provide an expanded training objective, optimizer configuration, or detailed compute budget, and the referenced appendix is absent. A conventional diffusion MSE or customary hyperparameters should therefore not be presented as equations or settings explicitly reported by this paper.

Key Experimental Results

Main Results

The enhancement evaluation uses 623 low-quality objects decomposed from 21 scene meshes. ULIP-3D measures shape-view alignment. MV-ImageReward uses concise condition-image captions generated with LLaMA 3.2 and averages ImageReward over four rendered views per object. The latter is a perceptual proxy, not a ground-truth surface distance.

Enhancement method (paper Table 1)	ULIP-3D (higher is better)	MV-ImageReward (higher is better)
Coarse input	0.2280	0.1716
Input + DetailGen3D	0.2294	0.1003
Input + MeshReGen	0.2626	0.3394

GSO geometry metrics are Chamfer Distance (CD, lower is better), F-score@1%, and IoU (higher is better). PSNR, SSIM, and LPIPS compare rendered normal maps with ground-truth normal maps, with unit normals rescaled to \([0,255]\). These are not texture or RGB reconstruction scores. The following selection from paper Table 2 retains view budgets and a representative unrefined VGGT baseline.

Method	Views	CD (lower)	F-score@1% (higher)	IoU (higher)	PSNR (higher)	SSIM (higher)	LPIPS (lower)
TRELLIS	1	0.0122	0.4216	0.6442	22.4605	0.9342	0.1059
AssetGen	4	0.0111	0.4663	0.6703	24.3809	0.9206	0.1024
MeshLRM	4	0.0115	0.4237	0.7035	22.8111	0.9290	0.1180
VGGT + Poisson	4	0.1276	0.0119	0.0193	8.7823	0.7704	0.5952
VGGT + MeshReGen	1	0.0135	0.3658	0.6438	21.8982	0.9315	0.1097
VGGT + MeshReGen	2	0.0112	0.4299	0.7053	22.9795	0.9353	0.0985
VGGT + MeshReGen	3	0.0090	0.4650	0.7372	23.6394	0.9385	0.0918
VGGT + MeshReGen	4	0.0081	0.4913	0.7574	24.2754	0.9408	0.0873

At four views, CD decreases from AssetGen's 0.0111 to 0.0081, an absolute reduction of 0.0030, but PSNR remains slightly below AssetGen. At one view, MeshReGen's CD and F-score are worse than TRELLIS. The results therefore do not establish superiority on every metric at every view budget. The VGGT row evaluates a Poisson mesh produced from its point cloud, not the error of VGGT's original point predictions alone.

Ablation Study

Paper Table 3 compares 3D-condition injection mechanisms. Additional CA introduces a parallel 3D cross-attention block; Extended CA feeds coarse point-cloud embeddings into the existing image cross-attention; Add \(z_L\) directly sums the condition into noisy latents. The full approach uses token concatenation. These are controlled comparisons in the authors' setup, not scores copied from official CLAY or Hunyuan3D-Omni checkpoints.

Conditioning mechanism (paper Table 3)	CD (lower)	F-score (higher)	IoU (higher)	PSNR (higher)	SSIM (higher)	LPIPS (lower)
Additional CA	0.0066	0.6107	0.6889	24.4104	0.9258	0.0900
Extended CA	0.0271	0.2374	0.2818	18.7145	0.9051	0.1410
Add \(z_L\)	0.0112	0.5259	0.5960	22.7384	0.9175	0.1060
MeshReGen	0.0044	0.7538	0.7735	26.3958	0.9338	0.0765

A separate experiment in paper Table 4 compares diffusion parameterizations. A Flow baseline that directly maps coarse latents toward the target distribution records CD 0.0061 and F-score 0.6026; the proposed approach records 0.0051 and 0.7003. The Ours rows in Tables 3 and 4 differ, and the main text does not provide enough configuration detail to reconcile them. They must not be combined into one ablation or used to calculate a Table 4 improvement from Table 3's CD of 0.0044.

Key Findings

• Condition injection matters substantially: within Table 3, concatenation raises F-score from Additional CA's 0.6107 to 0.7538, an absolute increase of 0.1431. The table does not separately isolate the contributions of zero initialization and positional embeddings.
• More views improve observed geometry before generative completion: MeshReGen's GSO IoU progresses through 0.6438, 0.7053, 0.7372, and 0.7574 for 1, 2, 3, and 4 views. Sparse observations remain useful rather than being entirely overridden by an image-generation prior.
• Editing uses Edit3D-Bench, and blockout generation is compared with CLAY, but the cached main text mainly provides qualitative evidence for these tasks. It does not support invented edit success rates, outside-mask errors, or inference speedup factors.

Highlights & Insights

• Treat geometric control as a repairable condition, not a hard constraint. The coarse shape guides generation without prescribing the entire output surface, which suits erroneous scans and low-quality assets. This requires training degradations that actually cover the relevant errors.
• Define the learning problem through the degradation process. Damage caused by sharing a pseudo-scene token budget matches a scene generator's bottleneck more closely than arbitrary smoothing. The transferable lesson is to model how the upstream system loses information before training its restorer.
• Use asymmetric token budgets for asymmetric information. Coarse geometry carries broad structure, while target tokens carry fine detail, avoiding mechanically equal capacities. This makes reuse of an existing generative backbone more natural than adding a separate module for every task.

Limitations & Future Work

• Reproduction depends on internal resources. Approximately 1 million training objects and the base generator are proprietary; scene visualizations also use a proprietary texture generator. Those textures serve visualization only and do not establish texture restoration as a capability of this method.
• A unified architecture is not unified training. The three tasks have different data construction and adaptation stages. A genuinely single-checkpoint model would still require evaluation of task mixing, interference, and condition generalization.
• Geometric fidelity is not a hard guarantee. Missing surfaces are inferred from a prior, and preservation outside an edit mask is mainly learned. The main text does not systematically establish limits under severe misalignment, abnormal normals, or extreme degradation on unseen categories.
• Quantitative coverage is uneven. Enhancement and reconstruction have metrics, whereas editing and blockout control are mainly demonstrated visually. Sampling takes 100 steps, but there is no comparable end-to-end latency or memory table. Missing appendices further limit verification of training cost, alignment implementation, and ablation settings.
• Author-proposed directions include dynamic geometry, richer controls such as skeletons, and alternative latent representations such as SLAT. Outside-mask error measurements and real-scan evaluations would help distinguish plausible completion from faithful recovery.

Related Work & Insights

• vs 3DShape2VecSet: VecSet supplies a variable-length latent shape representation. The contribution here is encoding degraded conditions and targets in that space and training the regeneration interface on paired degradations, not inventing a new shape VAE.
• vs CLAY / Hunyuan3D-Omni: These systems support 3D-aware control, whereas this work emphasizes repairing erroneous or incomplete geometry and compares corresponding cross-attention-style injection mechanisms. Controlled ablations should be distinguished from qualitative blockout comparisons using official services.
• vs DetailGen3D: Both exploit generative priors for geometry enhancement. MeshReGen keeps low-information geometry fixed as a condition, generates target latents from explicit noise, extends to scan reconstruction and editing, and separately compares a direct coarse-to-fine Flow parameterization.
• vs TRELLIS / VGGT: TRELLIS's sparse-structure conditioning does not automatically confer an ability to repair degraded inputs. VGGT predicts observed geometry but does not itself supply a complete surface. Their comparison motivates combining observed geometry with generative completion, without interpreting poor Poisson meshes as evidence that VGGT is useless.
• vs VoxHammer: VoxHammer follows a training-free native-3D editing route, whereas this method uses a specifically trained masked-regeneration model to avoid the corresponding inversion procedure. Cost shifts from per-instance processing to pretraining and adaptation, so editing-stage efficiency should not be assessed while ignoring training cost.

Rating

• Novelty: 4/5. The architectural change is restrained, but fixed-condition concatenation and task-specific degradation pairs form a useful unified regeneration formulation.
• Experimental Thoroughness: 3/5. Enhancement, reconstruction, and two mechanism comparisons have numerical support; editing, blockout control, efficiency, and public reproducibility lack equally complete quantification.
• Writing Quality: 4/5. The problem and three data constructions are clear. Naming differences, duplicate references, and damaged cached equations add reading friction, while missing appendices limit verification.
• Value: 4/5. The work offers practical ideas for iterative asset creation, point-cloud completion, and local editing, but private data, an internal backbone, and task adaptation raise deployment barriers.