UniPart: Towards Zero-shot Language-Grounded 3D Part Segmentation for Embodied Interaction¶
Conference: ECCV 2026
Paper: ECCV Official
Full Cache: /Users/zy/workspace/paper_cache/ECCV2026/eccv-4472.txt
Code: [Pending Release]
Area: 3D Vision
Keywords: 3D Part Segmentation / Zero-Shot Learning / Embodied Interaction / Vision-Language Models / Point Cloud Representation
TL;DR¶
UniPart proposes a lightweight feed-forward cross-modal 3D Transformer with layer-wise additive text conditioning alongside LangPart-1M (160K+ objects and 8M text-part pairs generated via multi-view consistent merging) and a human-verified LangPart-4K subset, achieving real-time (0.4s) zero-shot 3D part segmentation and real-world robot grasping without multi-view backprojection at inference.
Background & Motivation¶
Building general-purpose embodied agents requires robots not only to detect macro-level object instances, but also to understand functional object parts and geometric composition at a granular level. In open-world robotic manipulation, performing downstream tasks such as grasping a screwdriver handle, pressing a specific microwave button, or holding the bail of a kettle fundamentally relies on precisely localizing instruction-relevant sub-regions. While 2D foundation models such as SAM and CLIP have unlocked impressive open-vocabulary zero-shot generalization, extending fine-grained part understanding into 3D point clouds remains an open dilemma.
On one hand, promptable 3D models like Point-SAM inherit SAM-style instance segmentation strengths, but they isolate whole objects from background clutter without resolving internal functional parts within an object. On the other hand, classic part segmentation benchmarks and models (e.g., ShapeNetPart, PartNet) provide part-level granularity, yet they strictly depend on closed-set taxonomies (e.g., 16 or 45 semantic classes) and completely fail under open-vocabulary, free-form linguistic queries. Existing open-vocabulary attempts (such as PointCLIPV2 or PartSLIP++) either necessitate rendering dozens of 2D images and computing voting through dense backprojection at inference time—leading to massive computational latencies up to hundreds of seconds—or suffer from severe 2D-to-3D projection artifacts and multi-view inconsistencies.
Addressing this fundamental tension, the authors conceptualize the ideal perception primitive for embodied manipulation as a direct semantic interface that maps a free-form natural language phrase into a 3D part segmentation mask. Realizing this vision requires conquering two critical barriers: the absence of large-scale, view-consistent 3D text-part paired supervision, and the lack of an efficient, direct feed-forward 3D multimodal segmentation network. The core idea is to build a scalable multi-view data engine featuring text-guided semantic merging to construct LangPart-1M, paired with a feed-forward 3D Transformer model UniPart that injects global CLIP text embeddings into each layer via simple addition, eliminating inference-time multi-view rendering entirely.
Method¶
Overall Architecture¶
The UniPart framework integrates an automated data engine pipeline with an efficient feed-forward 3D cross-modal neural network. On the data generation side, starting from 160K+ Objaverse 3D models filtered by a six-criteria VLM judge, the engine queries part candidate lists via GPT-5.2-Pro, renders multi-view imagery in Blender, generates 2D segmentations using SAM with Set-of-Mark (SoM) prompts, merges semantically coherent regions across views via GPT-5.2-Pro, and lifts the masks into 3D to yield LangPart-1M (8M text-part pairs), accompanied by a verified LangPart-4K subset. On the inference side, given an object point cloud and a text phrase, point coordinates are grouped via Farthest Point Sampling (FPS) and k-NN into geometric patches and mapped into 3D tokens using a PointNet encoder; these tokens enter an \(N\)-layer plain 3D Transformer encoder where frozen global CLIP text embeddings are injected additively across all layers; finally, an MLP-based decoder upsamples the tokens to produce per-point binary part masks at 0.4 seconds per query.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
A["Input: Point Cloud X + Text Query l"] --> B["Point Cloud Tokenization<br/>FPS + k-NN + PointNet Local Patch Encoding"]
A --> C["Language Encoding<br/>Frozen CLIP Text Encoder Extracts Global Embedding"]
B --> D["Cross-Modal 3D Transformer<br/>N-Layer Plain Transformer Backbone"]
C -->|Layer-wise Broadcast| E["Layer-wise Additive Text Injection<br/>Direct Token-Level Addition across All Layers"]
D <--> E
E --> F["PointNet-MLP Decoder<br/>Upsampling to Output Per-Point Binary Logits"]
F --> G["Output: Target 3D Part Segmentation Mask"]
Key Designs¶
1. LangPart-1M Data Engine with Multi-View Text-Guided Merging: Bridging 2D Foundation Models and 3D Part Supervision at Scale
The central obstacle in distilling fine-grained part semantics from vast web-crawled 3D repositories (Objaverse) into 3D point clouds is the severe multi-view inconsistency and over-segmentation when 2D mask predictions are lifted independently. To overcome this, the authors develop a four-stage automated data engine. First, 3D assets undergo a six-criteria VLM automated filtering (removing ground planes, non-distinguishable shapes, full scenes, and evaluation leaks). Next, GPT-5.2-Pro generates a comprehensive part checklist for each valid object. Twelve viewpoints are rendered under controlled lighting in Blender, where SAM generates granular masks tagged with Set-of-Mark (SoM) IDs. Crucially, in the third stage, GPT-5.2-Pro inspects the SoM-labeled images, selects visible functional parts matching the checklist, and merges fragmented region IDs belonging to the identical functional part. Finally, depth-calibrated backprojection produces consistent per-point part annotations, resulting in 160K+ objects, 250+ categories, 1,000+ part categories, and 8M text-part pairs.
2. Layer-wise Additive Text Injection: A Minimalist and Highly Responsive Cross-Modal Fusion Mechanism
For fusing 3D point tokens with language features, common cross-modal architectures often deploy bidirectional cross-attention modules or channel concatenation. However, empirical findings reveal that because embodied interaction commands are typically concise phrases (e.g., "handle", "bottom switch"), the text semantic content is already compactly encapsulated by the CLIP global text embedding. Complex cross-attention layers introduce heavy parameter and memory footprints and tend to overfit sparse point distributions. UniPart introduces a minimalist layer-wise additive injection strategy, directly performing element-wise addition between the CLIP text embedding vector and each point token feature across all \(N\) Transformer layers. This continuous conditioning reinforces linguistic guidance throughout deep geometric reasoning while preserving a clean feed-forward speed of 0.4s.
3. Three-Stage Progressive Alignment and Fine-Tuning Strategy: Harmonizing Cross-Modal Semantics and Boundary Fidelity
To mitigate the residual noise and boundary ambiguity inherent in fully automated synthetic labels, UniPart establishes a three-stage learning curriculum. Stage 1 executes cross-modal geometric-visual pre-alignment: by projecting single-view partial point clouds to corresponding rendered RGB patches, the network minimizes the cosine distance between point features and CLIP visual patch embeddings, initializing the 3D backbone with rich visual-linguistic priors. Stage 2 performs large-scale representation pretraining on the full LangPart-1M dataset using binary cross-entropy (BCE) loss, grounding open-vocabulary semantic parts across diverse categories. Stage 3 conducts high-fidelity fine-tuning on LangPart-3K (manually verified by ten human annotators), sharpening part boundaries and substantially boosting accuracy on fine, complex structures.
Loss & Training¶
UniPart is trained using specialized objectives tailored to each stage: 1. Stage 1: Cross-Modal Feature Alignment: For a single-view point cloud \(X^{sini}\) and its corresponding rendered image \(I\), with a known point-to-patch mapping \(\pi(i) \in \{1,\dots,M\}\), the objective aligns normalized point features \(\hat{z}_i\) with normalized CLIP image patch features \(\hat{c}_{\pi(i)}\): $\(\mathcal{L}_{align} = 1 - \frac{1}{N}\sum_{i=1}^N \langle \hat{z}_i, \hat{c}_{\pi(i)} \rangle\)$ 2. Stages 2 & 3: Binary Segmentation Fine-Tuning: Given a text prompt \(\ell_{ij}\), the network predicts per-point probabilities \(\hat{S}_{ij} \in [0, 1]^N\) supervised against ground-truth binary mask \(S_{ij}\) via Binary Cross-Entropy (BCE) loss: $\(\mathcal{L}_{BCE}(\hat{S}_{ij}, S_{ij}) = -\frac{1}{N}\sum_{k=1}^N \left[ S_{ij}^k \log(\hat{S}_{ij}^k) + (1 - S_{ij}^k) \log(1 - \hat{S}_{ij}^k) \right]\)$ The global optimization objective aggregates BCE losses across all objects and valid part masks: $\(\min_{\theta_{UP}} \sum_{X_i \in \mathcal{D}} \sum_{S_j^i \in S_i} \mathcal{L}_{BCE}(\hat{S}_j^i, S_j^i)\)$
Key Experimental Results¶
Main Results¶
The authors evaluate UniPart on open-world benchmarks (Objaverse-General), classic part transfer benchmarks (ShapeNetPart, PartNet-E), and the newly curated benchmark LangPart-1K. The primary evaluation metric is mean Intersection over Union (mIoU, %). UniPart consistently outperforms prior 2D-to-3D projection and 3D feed-forward baselines across both compositional ({part} of a {object}) and short ({part}) prompt templates, running in 0.4s (over \(400\times\) faster than PartSLIP++).
| Benchmark | Subset / Setting | Prompt Format | UniPart (Ours) | FIND3D (SOTA) | OpenMask3D | PointCLIPV2 | PartSLIP++ |
|---|---|---|---|---|---|---|---|
| Objaverse-General | Seen Categories | {part} of a {object} |
46.43 | 33.78 | 11.81 | 9.81 | 2.69 |
| Seen Categories | {part} |
49.27 | 34.10 | 11.93 | 11.27 | 15.03 | |
| Unseen Categories | {part} of a {object} |
41.46 | 26.21 | 7.01 | 10.27 | 0.57 | |
| Unseen Categories | {part} |
45.23 | 27.41 | 10.31 | 11.09 | 10.43 | |
| ShapeNetPart | Canonical | {part} |
41.75 | 24.09 | 10.37 | 20.22 | 6.46 |
| Rotated | {part} of a {object} |
42.43 | 29.64 | 6.75 | 16.88 | 0.94 | |
| ShapeNetPart-V2 | {part} of a {object} |
55.63 | 42.15 | 15.87 | 15.14 | 1.54 | |
| LangPart-1K | Single-View (Partial) | {part} |
33.56 | 8.97 | - | - | - |
| Whole Object (Complete) | {part} of a {object} |
28.69 | 14.36 | - | - | - | |
| PartNet-E | Canonical | {part} of a {object} |
23.15 | 16.86 | 12.54 | 11.28 | 5.12 |
| Rotated | {part} of a {object} |
24.28 | 17.62 | 11.93 | 10.32 | 3.87 |
Ablation Study¶
To isolate the respective contributions of the proposed architecture and dataset, the paper provides a controlled cross-evaluation on Objaverse-General (Table 6), alongside ablations on multimodal fusion strategies (Table 8) and stage-1 cross-modal pretraining (Table 7).
1. Controlled Study Disentangling Architecture vs. Data (Objaverse-General, Zero-shot mIoU %)
| Model Architecture | Training Data | mIoU (%) | Relative Gain | Note |
|---|---|---|---|---|
| FIND3D Model | FIND3D Data (36k) | 30.75 | Baseline (0.00) | Standard prior baseline |
| UniPart (Ours) | FIND3D Data (36k) | 35.27 | +4.52 (+14.69%) | Validates additive Transformer architecture superiority |
| FIND3D Model | LangPart-1M (Ours) | 38.42 | +7.67 (+24.94%) | Demonstrates superior quality and scale of LangPart-1M |
| UniPart (Ours) | LangPart-1M (Ours) | 47.25 | +16.50 (+53.66%) | Synergy of architecture and scaled supervision |
2. Multimodal Fusion Mechanism and Pretraining Ablation
| Study Group | Ablation Setting | mIoU (%) | Analysis |
|---|---|---|---|
| Fusion Method | Cross-attention | 29.22 | Overfits sparse point clusters and introduces heavy parameters |
| Multiplication | 28.13 | Suppresses inactive geometric features too aggressively | |
| Concatenation (Concat) | 30.31 | Increases decoder feature dimensions with limited gain | |
| Addition (Ours) | 33.56 | Outperforms Cross-attn by +4.34% with zero extra attention overhead | |
| Pretraining Effect | With pretraining + partial PC | 33.56 | Directly aligns 3D geometric tokens with CLIP visual-semantic space |
| Without pretraining + partial PC | 26.34 | Significant drop of -7.22%, proving pre-alignment necessity | |
| With pretraining + complete PC | 27.17 | Single-view patch pretraining generalizes stably to complete shapes | |
| Without pretraining + complete PC | 23.39 | Noticeable drop of -3.78% |
Key Findings¶
- Dual Engine Synergy of Data and Architecture: The controlled cross-study reveals that upgrading the training data from FIND3D (36k) to LangPart-1M yields a +7.67% boost on the FIND3D model, while upgrading the model architecture alone gives +4.52%. When both are combined, UniPart reaches 47.25% mIoU (+16.50% total gain), confirming that the additive Transformer is exceptionally effective at absorbing scaled 3D part supervision.
- Additive Injection Beats Complex Cross-Attention: In multimodal fusion ablations, simple layer-wise vector addition achieves 33.56% mIoU, significantly surpassing Cross-Attention (29.22%) and Concat (30.31%). Because language instructions for parts are compact, additive injection uniformly broadcasts semantic intent into all layers without diluting local spatial representations.
- Sim-to-Real Embodied Manipulation Transfer: Integrated with Franka Panda (gripper) and UR5 (Shadow Hand) arms using a single wrist-mounted RealSense D415 camera, UniPart delivers 90.5% part segmentation accuracy across 20 physical objects and powers an 85.0% grasp success rate in language-conditioned grasping via GraspNet/DexGraspNet2.0, with an ultra-low latency of 0.4s.
Highlights & Insights¶
- Feed-Forward Efficiency Overcoming Multi-View Rendering Bottlenecks: Prior open-vocabulary 3D methods suffered from unbearable inference latency (e.g., PartSLIP++ at 174.3s per object due to multi-view 2D rendering and voting). UniPart demonstrates that a pure feed-forward 3D network with layer-wise text addition cuts inference time to 0.4s while achieving superior accuracy.
- VLM-Guided Cross-View Region Merging: Leveraging GPT-5.2-Pro and Set-of-Mark prompts to select and merge fragmented SAM masks across multi-view renders overcomes traditional geometric projection tearing, offering a robust paradigm for 3D data generation.
- Decoupled Perception Primitive for Dexterous Manipulation: Formulating part segmentation as an open-vocabulary semantic interface allows seamless plug-and-play integration with downstream robotic grasp planners (e.g., DexGraspNet), bypassing the need to retrain task-specific manipulation policies for every new object.
Limitations & Future Work¶
- Object-Centric Assumption and Clutter Sensitivity: LangPart-1M is curated from segmented, isolated Objaverse objects. Consequently, UniPart expects pre-segmented or cropped point clouds and relies on an external 3D object detector when deployed in heavily cluttered, unsegmented indoor scenes.
- Geometric Fragility on Thin and Deformable Structures: FPS sampling and k-NN neighborhood aggregation remain vulnerable to delicate topological features (such as thin cables, wires, or slender handles), where downsampling can cause disconnected components.
- Future Directions: Future avenues include generalizing UniPart to scene-level unbounded point clouds, exploring 3D Gaussian Splatting representations, and integrating closed-loop tactile feedback during dynamic interactive manipulation.
Related Work & Insights¶
- vs FIND3D: Both target open-vocabulary 3D part segmentation. FIND3D utilizes a smaller dataset (~36k objects) and cross-attention fusion; UniPart scales data over \(4\times\) via multi-view consistent merging (LangPart-1M) and proves that layer-wise additive text injection is faster (0.4s vs 0.9s) and dramatically more accurate (47.25% vs 30.75% zero-shot mIoU).
- vs PartSLIP / PartSLIP++: PartSLIP relies on 2D GLIP detection and superpoint lifting, exhibiting severe vulnerability to view occlusions and random object rotations (dropping to 0.94% mIoU under rotation). UniPart operates directly on 3D geometry in a feed-forward manner, maintaining robust performance across arbitrary poses (42.43% mIoU under rotation).
- vs Point-SAM / OpenMask3D: Point-SAM focuses primarily on coarse instance-level object boundaries or spatial point prompt propagation; UniPart specifically addresses fine-grained, functional, action-relevant parts essential for embodied grasping.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Introduces a scalable multi-view text-guided merging data engine alongside an effective layer-wise additive cross-modal 3D Transformer]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated across Objaverse-General, ShapeNetPart, PartNet-E, LangPart-1K, with rigorous architecture-data decoupling and real physical robot trials]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary clarity and coherence, seamlessly bridging the gap between computer vision representations and embodied manipulation needs]
- Value: ⭐⭐⭐⭐⭐ [Provides a practical sub-second open-vocabulary 3D part perception primitive and a landmark dataset for the embodied AI community]