Axolotl3D: a Unified Framework for Faithful 3D Shape Completion

Conference: ECCV2026
Paper: ECCV Paper
Area: 3D Vision
Keywords: shape completion, partial point cloud, multi-view reconstruction, occlusion awareness, geometry editing

TL;DR

Axolotl3D unifies images, visibility masks, cameras, and partial points as conditions for 3D generation, using geometric anchors to achieve a 0.9046 F-score on occluded single-view Toys4K while also supporting local shape edits that preserve unedited regions.

Background & Motivation

Image-to-3D models can generate plausible objects, but producing a plausible object is different from preserving the observed geometry of a particular object. Real captures often contain only a few views, occluded objects, and depth reconstructions covering only part of the surface. A generator conditioned on one complete object image cannot directly exploit additional views and lacks an explicit interface for specifying already known 3D locations. The shape prior in Hunyuan3D 2.1 can fill unknown regions, but that prior alone cannot ensure that known structures are not reimagined.

Existing methods address different parts of this problem: Amodal3R uses visibility information for occluded objects, ReconViaGen emphasizes multi-view consistency, and Instant3dit uses image inpainting for local editing. Editing and occlusion completion nevertheless share an information structure: some geometry should remain fixed while other geometry must be inferred from visual evidence. During editing, the region selected for replacement must no longer be supplied as fixed geometry; otherwise, the old points conflict with the new image guidance. For multi-view input, cameras connect 2D observations to 3D anchors so that structures across images can be interpreted in a common coordinate system.

The paper therefore defines which image regions are valid, which surface points are available, and how these observations correspond, instead of assigning each task a separate generator. This unified interface also requires an appropriate training distribution: training only on complete points risks failure under large missing regions, while training only on complete images leaves the model vulnerable to occlusion. Core Idea: anchor known geometry with partial points, align multi-view evidence with cameras, exclude invalid image conditions with visibility masks, and train one shape generator for completion and editing through mixed-condition synthesis.

Method

Overall Architecture

Inputs comprise up to 6 images, their valid-region masks and camera parameters, and a partial point cloud; the output is a complete object mesh. The backbone is Hunyuan3D 2.1: its DiT generates shape tokens in latent space, and Hunyuan3D-ShapeVAE decodes them into a mesh. The authors modify the conditioning interface and its training distribution rather than learning a new 3D representation from scratch. During training, Mixed-Condition Synthesis creates task-specific inputs; generation then uses Camera-Aligned Visual Encoding, Geometric Anchor Fusion, and Mask-Biased Generation. Points retain a separate set of geometric tokens instead of first becoming another guidance image, and shape latents query them together with image tokens. Inference does not run the synthetic augmentations; valid images, cameras, and partial points may come from real capture or editing workflows.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Augment["Mixed-Condition Synthesis<br/>Conditions from training meshes"] -.->|Training only| Input["Images, masks, cameras<br/>Partial point cloud"]
    Input --> Visual["Camera-Aligned Visual Encoding"]
    Visual --> Fusion["Geometric Anchor Fusion"]
    Input -->|Partial points| Fusion
    Fusion --> Generate["Mask-Biased Generation"]
    Input -->|Valid-region masks| Generate
    Generate --> Output["ShapeVAE decoding<br/>Complete mesh"]

Key Designs

1. Mixed-Condition Synthesis: expose one model to different forms of missing information

Training objects are normalized to \([-1,1]^3\), sampled into dense surface points, and rendered from 150 cameras sampled on a sphere. Each training example selects one of three conditioning tasks with equal probability: sparse views and occlusions, large-area point dropout, or local editing. The first task selects an unobserved view, then chooses non-nearby conditioning views using a 90-degree camera-direction threshold to retain genuinely unobserved regions. Brush-stroke or rectangular occlusions are sampled for conditioning views, and invisible points or points projecting into occluded regions are removed to produce patchy point clouds. Point removal and image occlusion are not always coupled: the same occlusion is applied to the images and their valid masks only with probability \(p_{\mathrm{mask}}=0.5\). This creates cases where an image still reveals a region whose points are missing, discouraging the network from merely reconstructing the available point cloud. Within this task, single-view and multi-view cases are sampled equally, with occlusion probabilities \(p_{\mathrm{occl}}=0.25\) and \(p_{\mathrm{occl}}=1.0\), respectively.

The second task selects a threshold along a random coordinate axis and direction, removes a large region with a distance-dependent probability, and further filters points using surface normals. It also excludes cameras directly observing the missing region and removes points unseen by the remaining views, preventing another complete image from trivially revealing the gap. The third task treats a randomly selected complete rendering as the modified view and places a 3D bounding box around a surface point visible in that view. Points inside the box are removed, and its projections are masked in other conditioning images; only the designated modified view shows the complete target. This does not require an image editor to alter the training object: the same complete mesh supplies a supervised task with local missing geometry and single-view target guidance. The three tasks consequently teach cross-modal complementarity, large-region inference, and local replacement, while the complete object remains the target for shape learning.

2. Camera-Aligned Visual Encoding: identify the viewing rays behind image features

DINOv2 first produces spatially aligned patch features for each image; when fewer than 6 views are available, missing slots are zero-padded and masked. Cameras are not merely view-identity labels: they provide pixelwise Plücker ray embeddings with geometric meaning. These embeddings contain camera-center and ray-direction information, are average-pooled to the spatial resolution of image features, and are linearly projected to the same channel dimension. Camera features are then added to image features, allowing similar local textures observed from different views to be associated with different 3D viewing directions. This matters especially for a single view, where the model cannot infer the relationship between point-cloud orientation and image appearance from correspondences across several images.

Valid-region masks are partitioned into DINOv2's \(14\times14\) patches, and a patch is valid only when it is entirely unoccluded. Background is also considered unoccluded because empty background supplies free-space evidence about where object geometry should not appear. For editing, regions to preserve are marked valid, while the modified guidance view receives a fully valid mask. The semantics differ but the interface is shared: occlusion masks exclude unseen content, whereas editing masks exclude old content that should no longer be preserved. Equation (1) is corrupted in the extracted text, so this note describes the camera-encoding procedure without reconstructing the authors' exact formula.

3. Geometric Anchor Fusion: constrain observed surfaces with separate point tokens

VecSetX encodes the partial point cloud into a fixed-length latent set; its ShapeNet-v2 pretraining provides a representation of local geometry. Visual and geometric tokens initially have different channel counts, so separate gated feedforward networks project them into a shared 1024-dimensional space. Each modality then receives its own learnable embedding before the tokens are concatenated along the token dimension. This lets the model distinguish 2D observation features from 3D surface evidence without requiring their original representations to have identical semantics. The resulting condition set contains every input view and the point cloud for joint access by the same DiT, rather than cycling image conditions during denoising.

Geometric anchors make observed structure a conditioning constraint so that the shape prior can primarily resolve unknown regions. However, an anchor here is a learned condition, not a mathematical guarantee that original vertices are copied into the output mesh. Incorrect points can therefore induce incorrect geometry when followed faithfully; the predicted-depth experiments probe this trade-off. The point ablation also removes the multimodal fusion layers, so it measures the contribution of the entire geometric conditioning path rather than one isolated layer.

4. Mask-Biased Generation: exclude invalid visual tokens when reading conditions

In DiT cross-attention, noisy shape latents supply queries and the fused conditions supply keys and values. Valid-region masks become attention biases: invalid visual tokens receive negative infinity, while valid visual tokens and point tokens receive zero. The textual definition on page 6 supports the following explicit bias rule without reconstructing the corrupted full attention equation:

\[ B_{\ell,t}=\begin{cases} -\infty, & t\text{ is an invalid visual token},\\ 0, & t\text{ is a valid visual or point token}. \end{cases} \]

Here \(\ell\) indexes shape queries and \(t\) indexes condition tokens; every query excludes the same invalid conditions. Point tokens are not individually masked by image masks because VecSetX outputs a fixed-length latent set rather than a preserved pixelwise or pointwise correspondence table. Invalid point regions have already been removed when constructing the input, whereas invalid image regions are excluded during conditional attention. This is neither clipping the final mesh nor inpainting an occluded image; it controls which evidence the generator can consult each time it reads its conditions. The masking ablation produces a smaller gain than the geometric path, consistent with a safeguard against invalid content rather than the primary source of geometric information.

A Worked Example

For local editing, a user first marks the region to change in a chosen view and uses an image inpainting model to produce the modified view. The original mesh is sampled into a point cloud, but points projecting into the modified region must be discarded so that the old structure no longer constrains generation. The modified image and its camera indicate where the new structure belongs, while the remaining points constrain the outline and detail of unedited regions. Additional views, when supplied, mask the old editing region and contribute only evidence about surrounding unchanged geometry. The generator reads these conditions, completes the shape latents, and uses ShapeVAE to decode a new mesh. This walkthrough explains the editing interface on page 14; it does not imply a quantitative preservation guarantee across all editing cases.

Loss & Training

Training uses 407k filtered shapes from TRELLIS-500K, excluding buildings and scenes from its ObjaverseXL subset. The authors fine-tune Hunyuan3D 2.1 pretrained weights, initialize added linear layers with Kaiming uniform initialization, and initialize new embeddings from a normal distribution. The main paper describes the flow-based shape-generation backbone and denoising training but does not provide a fully specified separate total-loss expression; this note does not invent an additional geometric constraint loss. Optimization uses AdamW with learning rate \(10^{-5}\) and batch size 16 for 500k steps on 8 A100 GPUs. Classifier-free guidance (CFG) uses a conditioning dropout rate of 0.1; inference uses guidance scale 5 and 50 sampling steps. The real-image application first segments with SAM 2, estimates points and cameras with MapAnything, and aligns and normalizes them before conditioning the model.

Key Experimental Results

Main Results

Source: Table 1, page 10; the following means are from occluded Toys4K, with 6 views in the multi-view setting. F-score uses threshold 0.05, vIoU is evaluated at \(64^3\) resolution, and the CD column retains the original \(\times10\) scale. Meshes are normalized and aligned by ICP before 1M points are sampled from each mesh for evaluation, so these scores are not unaligned absolute-pose accuracy measurements.

Method	View condition	F-score ↑	vIoU ↑	CD ×10 ↓
Amodal3R	Single-view	0.6877	0.1886	0.6144
SAM 3D	Single-view	0.6764	0.1836	0.6375
Hy3D-Omni	Single-view	0.6593	0.1802	0.8123
ShapeR	Single-view	0.7453	0.2285	0.4622
Axolotl3D	Single-view	0.9046	0.3265	0.2402
ShapeR	Multi-view	0.9190	0.4002	0.2040
Axolotl3D	Multi-view	0.9689	0.4187	0.1448

The single-view occluded F-score exceeds ShapeR by 0.1593, calculated by subtracting the two Table 1 means. The advantage is not universal across datasets and metrics: for occluded multi-view OmniObject3D, ShapeR has higher vIoU, 0.4123 versus 0.3967, and slightly better CD, 0.1443 versus 0.1467. In the main table, Axolotl3D and ShapeR receive 8,192 points, Hy3D-Omni accepts at most 2,048 points, and SAM 3D uses pointmaps, so conditioning budgets are not identical. Table 3 separately compares methods with 2,048 points; the occluded single-view Toys4K F-score remains 0.8738 for Axolotl3D versus 0.6956 for ShapeR. Hy3D-Omni's multi-view setting supplies only the image with highest object coverage, although points can come from multiple views; it should not be described as a native multi-image model.

Ablation Study

Source: Table 4, page 14, under occluded Toys4K conditions; entries are means and use the same metric definitions as the main results.

Config	Single-view F-score ↑	Single-view vIoU ↑	Single-view CD ×10 ↓	Multi-view F-score ↑	Multi-view vIoU ↑	Multi-view CD ×10 ↓
Remove points and fusion layers	0.8074	0.2253	0.3589	0.9097	0.2881	0.2231
Remove mask bias	0.8968	0.3225	0.2517	0.9655	0.4151	0.1463
Replace cameras with view embeddings	0.9002	0.3228	0.2509	0.9689	0.4153	0.1462
Full model	0.9036	0.3262	0.2423	0.9688	0.4185	0.1450

Removing the geometric conditioning path reduces single-view F-score from 0.9036 to 0.8074, a drop of 0.0962 and the largest single-view degradation in this table. The camera ablation replaces Plücker embeddings with learnable view embeddings rather than removing all information distinguishing views. Its multi-view F-score is 0.9689, slightly above the full model's 0.9688, so not every component improves every metric. The full-model values in Tables 4 and 1 differ slightly, including single-view F-score values of 0.9036 and 0.9046; the paper does not explain this difference here, and both original values are retained.

Key Findings

Source: Table 2, page 10, using single-view, unoccluded Toys4K with Depth Anything v3 predicted depth; the following entries exclude camera perturbation. Predicted depth is aligned to ground-truth depth by a per-scene scale and shift, then backprojected using ground-truth cameras, so this is not a fully calibration-free end-to-end test.

Method	F-score ↑	vIoU ↑	CD ×10 ↓
SAM 3D	0.7062	0.1993	0.5898
Hy3D-Omni	0.6837	0.1729	0.5926
ShapeR	0.5626	0.1407	0.7767
Axolotl3D	0.7616	0.1903	0.4235

Axolotl3D leads in F-score and CD with predicted depth but trails SAM 3D's vIoU of 0.1993; the prose claim of leading on every metric conflicts with the table, so the individual columns take precedence. Camera perturbations use 10-degree rotation noise, 10% translation-scale noise, and ±5% focal-length jitter; Axolotl3D obtains F-score 0.7620, close to the unperturbed 0.7616. This indicates limited sensitivity to moderate camera errors in this test, not immunity to severe depth errors. Qualitative results explicitly show degradation under strong depth errors and failure to connect some thin structures.

Highlights & Insights

• Treating an editing region as an old observation that is no longer valid lets editing and occlusion completion share a conditioning interface. The unification lies in defining evidence validity, not merely assigning several task names to one model.
• Partial points and visible background supply complementary constraints: points locate surfaces, while background helps exclude space the object should not occupy. The generative prior resolves the remaining unknown regions.
• Partially decoupling image and point augmentation is a targeted design choice. It encourages the model to use one modality when another lacks information rather than always relying on points.

Limitations & Future Work

• The authors acknowledge that faithful point following amplifies severe depth-prediction errors and call for a better balance between fidelity and robustness. Confidence-aware point conditioning is a reader-proposed direction, not a validated component of this paper.
• Training with spherical cameras, fixed distance, and fixed intrinsics limits capture diversity. The authors also identify more diverse camera positions and texture support as future work.
• Tables 1 and 2 do not support universal leadership across every condition and metric. Multi-view volumetric overlap and predicted-depth vIoU provide explicit counterexamples.
• Editing, real-image reconstruction, and physical simulation are mainly supported by application examples; the main paper does not provide corresponding large-scale edit-preservation or simulation-stability metrics.
• The available source is the main paper; referenced supplementary details were not independently checked, and two major equations have extraction corruption, so undisplayed implementation details cannot be reconstructed reliably.

Related Work & Insights

• vs Amodal3R: both exploit occlusion validity, but Axolotl3D adds explicit partial points and cameras so that 3D geometry need not be inferred only indirectly from images.
• vs Hy3D-Omni: the latter supports several geometric controls but still uses a single image here, with Pix2Gestalt preprocessing for occluded images. Axolotl3D incorporates occlusion handling into its conditioning interface and training distribution.
• vs ShapeR: both combine images, points, and cameras, while ShapeR targets SLAM outputs and Axolotl3D emphasizes unified occlusion completion and editing. Table 1 supports stronger single-view results, not dismissal of ShapeR's stronger multi-view metrics.
• vs Instant3dit: image inpainting can remain the upstream editing tool, while partial geometric conditions constrain unmodified regions. Future studies could evaluate edit magnitude and unedited-region preservation separately instead of relying only on whole-shape reconstruction scores.

Rating

• Novelty: 4/5. The main contribution is a coherent conditioning interface and training-task design rather than a new generation backbone.
• Experimental Thoroughness: 4/5. Two datasets, occlusion, point counts, predicted depth, and component ablations are covered, but application quantification and some comparison controls remain limited.
• Writing Quality: 4/5. The method is clearly organized, although several broad performance claims require correction against individual table columns.
• Value: 4/5. The framework offers reusable ideas for geometrically constrained completion and editing, while real deployment remains dependent on input geometry quality.