Rethinking Multimodal Point Cloud Completion: A Completion-by-Correction Perspective

Conference: AAAI 2026
arXiv: 2511.12170
Code: https://github.com/RobWonn/PGNet
Area: 3D Vision
Keywords: Point Cloud Completion, Multimodal Fusion, Generative Prior, Correction Paradigm, Feature Alignment

TL;DR

A new paradigm, Completion-by-Correction, is proposed. It utilizes a pretrained image-to-3D model to generate a topologically complete shape prior, which is then corrected in the feature space to align with partial observations, replacing the traditional Completion-by-Inpainting method. It achieves a 23.5% reduction in average CD and a 7.1% improvement in F-score on ShapeNetViPC.

Background & Motivation

Background

Point cloud completion aims to recover complete 3D shapes from partial observations, with wide applications in autonomous driving, augmented reality, and robotics. Recently, deep learning methods (e.g., PoinTr, SeedFormer) have made significant progress. However, single-modal methods still struggle to distinguish "occlusion-induced missingness" from "structural voids" under severe occlusion. Consequently, multimodal methods leverage RGB images to provide complementary texture and semantic information to assist in completion.

Limitations of Prior Work

Existing multimodal methods (CSDN, XMFNet, EGIInet, etc.) all follow the Completion-by-Inpainting paradigm—fusing image and point cloud features first, and then directly synthesizing the missing geometry from the fused latent features. The authors experimentally identify inherent flaws in this approach:

Structural Inconsistency: The network must "invent out of thin air" the missing structure without an explicit structural skeleton, which easily leads to topological artifacts.

Semantic Ambiguity: In severely degraded cases, the constraints provided by fused features are insufficient, resulting in generations that are semantically plausible but geometrically incoherent.

Unconstrained Synthesis: Synthesizing geometry from incomplete representations is inherently an ill-posed problem.

Key Challenge & Key Insight

The authors argue that the root of the problem lies in the extreme difficulty of performing "unconstrained synthesis" from incomplete representations. Instead, it is better to first provide a topologically complete initial shape (utilizing an image-to-3D model) and then "correct" this shape to align with observations. This shifts the task from "unconstrained synthesis" to "guided refinement," making the problem much more well-posed.

Method

Overall Architecture

PGNet (PriorGroundNet) consists of three stages:

1. Corrective Dual-Feature Encoding: Parallely encodes the generative prior and partial observations, correcting the prior in the feature space.
2. Grounded Seed Generation: Synthesizes coarse but topologically complete seed point clouds as the structural skeleton.
3. Hierarchical Grounded Refinement: Progressively refines geometric details through two layers of GRBs.

The inputs include a partial point cloud \(P_o \in \mathbb{R}^{M \times 3}\) and the corresponding single-view RGB image \(I\), aiming to reconstruct the complete point cloud. First, a pretrained Trellis image-to-3D model is used to generate a prior point cloud \(P_g\) from the image, and then \(P_g\) is aligned with the observation via a learned correction function \(\mathcal{T}\).

Key Designs

1. Corrective Dual-Feature Encoding

Core Idea: Since disparities exist between the prior \(P_g\) and the observation \(P_o\) in scale, pose, and point distribution, alignment is required in the feature space.

• Partial Point Cloud Encoder: Employs hierarchical local feature aggregation (FPS + DGCNN) to extract \(N_e = 128\) representative points and initial features. A learnable relative position encoding \(\Phi\) is incorporated to alleviate pose disparities. Then, global and local contexts are fused using the Salient Transformer (dual-branch structure):
  • Global Branch: MHSA generates long-range context \(A_o\).
  • Local Branch: kNN + shared MLP + max pooling generates local patterns \(X_o\).
  • Adaptive fusion is achieved via a learnable saliency gate \(G_o = \sigma(\text{MLP}([A_o, X_o]))\).

\[F_o = (1 - G_o) \odot A_o + G_o \odot X_o\]

• Generative Prior Encoder: Employs the same hierarchical encoding, but utilizes a Grounding Transformer to correct the prior in the feature space:
  • Self-attention branch captures the internal structure of the prior.
  • Grounding branch (cross-attention) uses \(F_g''\) as queries and \(F_o\) as keys/values to obtain observation-aligned features.
  • Similarly fused via a saliency gate.

Design Motivation: The Salient Transformer enhances the reliability of \(F_o\) (attending to the global context in sparse regions and local details in fine regions), while the Grounding Transformer injects reliable observation signals into the generative prior.

2. Grounded Seed Generation

Core Idea: Generate a coarse but topologically complete and geometrically grounded skeleton point cloud.

• Performs max pooling on \(F_g\) and \(F_o\) to extract global representations \(\hat{F}_g\) and \(\hat{F}_o\).
• Fuses global features via cross-attention to obtain \(\hat{F}_{\text{fused}}\).
• Inspired by PixelShuffle, expands the global features to \(N_c = 512\) seed features through MLP + reshape.
• Performs cross-attention again to align (ground) seed features with \(F_o\).
• Finally, an MLP generates the coarse point cloud \(P_c\):

\[P_c = \text{MLP}([\text{Replicate}(\hat{F}_{\text{fused}}, N_c), F_{\text{seed}}, F_{\text{gr}}])\]

3. Hierarchical Grounded Refinement

Core Idea: Progressively enhance geometric fidelity through \(K=2\) stacked Grounded Refinement Blocks (GRBs). Each GRB contains two components:

(a) Dual-Source Feature Association: - Querying from observations: Uses Inverse Distance Weighting (IDW) to interpolate features from the kNN of \(F_o\) for each point. - Querying from priors: Since \(P_o\) and \(P_g\) are spatially unaligned, kNN + IDW interpolation is conducted in the feature space instead. - Concatenating dual-source features: \(f_{as}(p_i) = [f_{\text{interp},o}(p_i), f_{\text{interp},g}(p_i)]\)

(b) Structure-Aware Upsampling: - Cross-Scale Shape Context (CSSC) module: Aggregates multi-scale shape contexts from the previous resolution for each point via geometric transformer attention. - Attention weights simultaneously consider feature similarity and relative spatial positions. - Predicts \(r\) displacement vectors (\(r=2\)), progressively upsampling from low to high resolution at each layer: \(512 \to 1024 \to 2048\).

Loss & Training

The L1 Chamfer Distance is adopted as the training objective, supervising both the coarse output and the upsampled output of each layer (multi-level supervision):

\[\mathcal{L} = \frac{1}{K+1}\left(\mathcal{L}_{\text{CD}}(P_c, P_{gt}) + \sum_{k=1}^{K}\mathcal{L}_{\text{CD}}(P^{(k)}, P_{gt})\right)\]

Training details: AdamW optimizer, initial learning rate of \(2 \times 10^{-4}\) with cosine annealing, trained for 100K steps per category individually, batch size 192, NVIDIA RTX 4090. Prior generation uses the Trellis model + Poisson disk sampling 2048 points.

Key Experimental Results

Main Results

Evaluated on the ShapeNet-ViPC dataset (38,328 objects, 13 categories):

Method	Type	Mean CD (×10⁻³) ↓	Mean F-score ↑
PoinTr	Single-modal	2.851	0.683
SeedFormer	Single-modal	2.902	0.688
ViPC	Multimodal	3.308	0.591
CSDN	Multimodal	2.570	0.695
XMFNet	Multimodal	1.454	0.797
EGIInet	Multimodal	1.211	0.836
PGNet (Ours)	Multimodal	0.926	0.895

Compared with the previous SOTA EGIInet: CD decreases by 23.5%, and F-score improves by 7.1%. The gains are particularly significant on heavily occluded categories such as cabinet (+42.2%) and sofa (+26.6%).

Ablation Study

Ablation study on the cabinet category (CD ×10⁻³ / F-score):

Configuration	CD ↓	F-score ↑	Description
w/o Prior Feature Grounding	1.185	0.827	Removes feature-space correction
w/o Seed Grounding	1.219	0.821	Removes seed grounding
w/o Dual-Source Association	1.324	0.803	Largest impact; dual-source association is the core component
w/o Structure-Aware	1.275	0.800	Removes structure-aware upsampling
PGNet (Full)	1.111	0.839	Full model

Paradigm comparison (Inpainting vs. Correction): The Inpainting variant achieves an average CD of 1.10 compared to 0.93 for PGNet. Inpainting's CD on the cabinet category is 41.4% higher.

Key Findings

1. The Completion-by-Correction paradigm is inherently more robust than Completion-by-Inpainting, proving that "correcting from a complete prior" outperforms "synthesizing from incomplete features."
2. Dual-source feature association is the most critical performance component (removing it increases CD by 19.2%), showing that simultaneously leveraging observation fidelity and prior structural information is essential.
3. The superiority is most pronounced on heavily occluded categories (cabinet, sofa), validating the core value of the prior skeleton when missing regions are large.

Highlights & Insights

1. Paradigm Innovation: Proposes a new paradigm for point cloud completion—shifting from "inpainting" to "correction," which converts the ill-posed synthesis problem into a well-posed refinement task.
2. Clever Utilization of Image-to-3D Models: Instead of direct geometric-level fusion (which is vulnerable to pose/scale disparities), it corrects features in the feature space, yielding a more elegant design.
3. Saliency Gating Mechanism: Unifies the gating design of both the Salient Transformer and the Grounding Transformer, which is simple yet effective.
4. Feature-Space Interpolation: Adopts feature-space kNN instead of spatial kNN for prior features in the Dual-Source Association, successfully bypassing geometric unalignment issues.

Limitations & Future Work

1. It relies on a pretrained image-to-3D model (Trellis). The quality of the prior directly determines the performance upper bound and increases inference overhead.
2. Training 100K steps per category individually involves high computational costs, and cross-category generalization capabilities have not been evaluated.
3. Evaluation is only conducted on the ShapeNetViPC synthetic dataset, lacking validation in real-world scenarios.
4. Hallucinations from the prior generation model may introduce systematic errors, which lacks in-depth analysis in the paper.

Related Work & Insights

• SymmCompletion (AAAI 2025): Utilizes symmetry priors for point cloud completion. This work can be combined with it—first generating symmetry-based priors and then correcting them.
• PCDreamer (CVPR 2025): Diffusion-based point cloud completion. Direct geometric fusion suffers from pose disparities, which this work bypasses via feature-space correction.
• Advances in image-to-3D models (e.g., Trellis, TripoSR) will directly elevate the upper bound of this framework.
• Similar "generate-then-correct" ideas can be extended to other 3D reconstruction tasks.

Rating

• Novelty: ⭐⭐⭐⭐⭐ — Paradigm-level innovation. The transition from Inpainting to Correction is highly convincing.
• Experimental Thoroughness: ⭐⭐⭐⭐ — Comprehensive ablation studies, but evaluated on only one dataset, and lacks real-world scenario validation.
• Writing Quality: ⭐⭐⭐⭐⭐ — Clear motivation, exquisite illustrations, and coherent narration.
• Value: ⭐⭐⭐⭐ — Opens up a new direction for multimodal point cloud completion, though practical deployment remains to be validated.

Related Papers

• [AAAI 2026] DAPointMamba: Domain Adaptive Point Mamba for Point Cloud Completion
• [AAAI 2026] Simba: Towards High-Fidelity and Geometrically-Consistent Point Cloud Completion via Transformation Diffusion
• [AAAI 2026] DANCE: Density-Agnostic and Class-Aware Network for Point Cloud Completion
• [AAAI 2026] TOSC: Task-Oriented Shape Completion for Open-World Dexterous Grasp Generation from Partial Point Clouds
• [AAAI 2026] Point Cloud Quantization through Multimodal Prompting for 3D Understanding