Ground3D-LMM: Fine-Grained 3D Point Grounding and Spatial Reasoning with LMM¶

Conference: ECCV 2026
Paper: ECCV 2026 Official Poster
Code: https://github.com/AmolHarsh/ground3d-lmm
Area: Multimodal VLM / 3D Vision
Keywords: 3D Point Grounding, Part Segmentation, Spatial Reasoning, 3D Multimodal LLM, Embodied AI

TL;DR¶

Ground3D-LMM introduces a unified point-cloud multimodal architecture and a 2.5M QA dataset that jointly outputs point-level 3D masks and physical metric measurements at both object and part granularity within interactive dialogue.

Background & Motivation¶

Natural-language interaction with 3D physical environments becomes actionable in robotics, AR/VR, and assistive technologies only when responses fulfill two foundational criteria: verifiability and metric awareness. Verifiability requires the model to ground its conversational utterances to explicit 3D regions in the physical scene, avoiding hallucinated statements. Metric awareness requires system answers to report measurements in standardized real-world physical units (e.g., meters or centimeters) rather than vague qualitative descriptors such as "larger" or "nearer", providing actionable information for robotic grasping, navigation, and obstacle avoidance.

However, existing 3D vision-language models struggle to unify both capabilities within a single interface. On one hand, conversational 3D QA systems (e.g., ScanQA, SQA3D) generate purely textual outputs without explicit point-level grounding, rendering their spatial claims difficult to verify. On the other hand, 3D referring expression models and open-vocabulary segmentation networks (e.g., Open3DIS, SegPoint, Reason3D) output isolated bounding boxes or masks, but lack interactive dialogue capabilities and cannot formulate metric numeric predictions. Furthermore, open-vocabulary part-level grounding (e.g., segmenting the graspable handle of a drawer or the cushion of a chair) remains severely underexplored despite its practical significance in physical affordance reasoning.

To bridge this gap, this paper unifies explicit point-cloud grounding, part-level spatial discrimination, and metric numeric reasoning into an interactive multimodal dialogue framework. Core idea: embed superpoint geometric features extracted by a sparse 3D U-Net as soft tokens into a multimodal LLM, trigger a lightweight attention-based segmentation head dynamically via auto-regressive <SEG> tokens, and jointly optimize point-level mask prediction and physical metric reasoning.

Method¶

Overall Architecture¶

Ground3D-LMM consists of three primary components: a 3D point cloud encoder that extracts hierarchical scene geometric representations, a Large Multimodal Model (Point LMM) serving as the core reasoning engine, and a lightweight query-driven segmentation head that produces point-level binary masks. The input comprises a colored 3D point cloud \(\mathbf{X} \in \mathbb{R}^{L \times 6}\) (coordinates and RGB colors), a user query \(q\), and an optional camera-aligned reference RGB image \(I\).

The inference pipeline operates end-to-end: the raw point cloud is downsampled into \(M\) superpoints via voxelization and processed by a sparse 3D U-Net to extract geometric feature representations; a linear projector maps these features into the LMM's token space alongside textual tokens and visual tokens from the reference image; during text generation, when the LMM refers to a specific target entity, it emits a wrapped tag <p>entity_name</p><SEG>; the hidden state of <SEG> is projected into a segmentation query and refined through cross-attention with point features to yield a 3D binary mask, while the language stream simultaneously articulates exact metric dimensions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Point Cloud X (L×6) + Reference Image I + Query q"] --> B["Superpoint Voxel Downsampling & Sparse 3D U-Net"]
    B --> C["Multimodal Alignment Projector<br/>Point Features Fp mapped to Tokens Tp"]
    C --> D["Point LMM Autoregressive Reasoning<br/>Qwen3-VL processes text/image/point tokens"]
    D --> E{"Check if output contains &lt;SEG&gt; token"}
    E -->|Contains grounding entity| F["Query Mapping & Cascaded Attention Head<br/>Cross-Attn interacts with dense Fp"]
    E -->|Pure text/numeric values| G["Language Stream: Verbalize metric dimensions & spatial relations"]
    F --> H["Mask Decoder: Output 3D Point-level Binary Mask M"]
    G --> I["Final Multimodal Response: Verifiable 3D Mask + Precise Metric Numbers"]
    H --> I

Key Designs¶

1. Hierarchical Superpoint Encoding & Multimodal Projection: Balancing efficiency and fine geometric resolution
Feeding hundreds of thousands of raw 3D scene points directly into an LMM causes GPU memory overflow while neglecting local geometric topology. To address this issue, the point cloud is first voxelized and pooled into \(M\) superpoints (\(M \ll L\)), which aggregate local structural primitives while filtering sensory noise. A sparse convolutional 3D U-Net (SpConv) extracts hierarchical representations \(\mathbf{F}_p \in \mathbb{R}^{M \times d}\) that balance local geometry with global scene context. A learnable linear projector \(\mathcal{P}\) then projects \(\mathbf{F}_p\) into the LMM embedding space as point tokens \(\mathcal{T}_p = \mathcal{P}(\mathbf{F}_p)\). In the instruction prompt, \(M\) reserved <|point|> placeholder tokens are dynamically populated with \(\mathcal{T}_p\). Incorporating an optional camera-aligned 2D reference image \(\mathcal{T}_{img}\) supplies high-resolution texture and perspective orientation cues, resolving geometric ambiguities in sparse point clouds.

2. Triggered Query Segmentation Head: Seamless decoupling of linguistic reasoning and point-level masking
To equip the conversational language model with spatial localization without compromising language generation capabilities, the framework employs an in-context signaling mechanism using tags like <p>object_name</p><SEG>. When the LMM determines that a referred object or part requires spatial grounding, it generates the special token <SEG>. The last-layer hidden state \(\mathbf{t}\) at that position is mapped by an MLP query embedding function \(Q\) into an initial segmentation query \(\mathbf{q}^{(0)} = Q(\mathbf{t})\). The query is refined iteratively through cross-attention with dense point features \(\mathbf{F}_p\):

\[\mathbf{q}^{(1)} = \text{Cross-Attn}(\mathbf{q}^{(0)}, \mathbf{F}_p), \quad \mathbf{q}^{(2)} = \text{Self-Attn}(\mathbf{q}^{(1)}), \quad \mathbf{q}^{(f)} = \text{FFN}(\mathbf{q}^{(2)})\]

A lightweight mask decoder then computes the dot-product similarity between the refined query \(\mathbf{q}^{(f)}\) and \(\mathbf{F}_p\), followed by a sigmoid activation to yield the point-level binary mask \(\mathcal{M} = \text{Decoder}(\mathbf{q}^{(f)}, \mathbf{F}_p) \in \{0, 1\}^N\). This design cleanly decouples language generation from dense 3D point classification while preserving fine-grained geometric alignment.

3. Ground3D Large-Scale Metric Benchmark: Multi-stage scalable synthesis and rigorous manual verification
Because existing 3D datasets lack concurrent annotations for 3D part-level segmentation and metric physical quantities, the authors constructed the Ground3D benchmark using ~2.5K scenes from ScanNet and ScanNet++. The construction pipeline spans five automated stages: (1) extracting hierarchical object and part bounding box proposals using Qwen3-VL-30B on keyframes; (2) generating fine-grained 2D masks via SAM2 with boundary erosion; (3) back-projecting depth pixels to 3D camera coordinates and applying percentile filtering to eliminate depth outliers; (4) deriving ground-truth metrics (oriented bounding box dimensions, camera distances, clearance between objects) strictly on the visible viewport geometry; and (5) synthesizing ~2.5M multi-turn question-answer pairs covering 8 distinct tasks. The evaluation benchmark underwent 100% manual inspection across all 128 evaluation scenes (removing 26% noisy samples), establishing a verified standard evaluation protocol.

Loss & Training¶

Ground3D-LMM is trained end-to-end using a multi-task objective function balancing 3D point localization and linguistic generation:

\[\mathcal{L}_{total} = \lambda_{seg}\mathcal{L}_{seg} + \lambda_{text}\mathcal{L}_{text}\]

Both loss weights are unified to \(\lambda_{seg} = \lambda_{text} = 1.0\). The text generation loss \(\mathcal{L}_{text}\) is the standard autoregressive next-token cross-entropy loss over output sequences. For mask prediction, to handle the extreme class imbalance in sparse 3D point clouds (especially for slender or small functional parts), the segmentation loss combines point-wise Binary Cross-Entropy with Dice loss:

\[\mathcal{L}_{seg} = \mathcal{L}_{BCE}(\mathcal{M}, \mathcal{M}_{gt}) + \mathcal{L}_{Dice}(\mathcal{M}, \mathcal{M}_{gt})\]

Optimization utilizes AdamW with weight decay 0.05 and a stratified learning rate schedule: the point encoder is fine-tuned at \(1\times 10^{-6}\) initialized from SSTNet; the LMM backbone (Qwen3-VL-4B-Instruct) is trained with LoRA at \(1\times 10^{-5}\); and newly introduced projection and segmentation modules are trained at \(1\times 10^{-4}\).

Key Experimental Results¶

Main Results¶

On the proposed Ground3D benchmark, models are evaluated across object-level and part-level mean Intersection-over-Union (mIoU), along with metric estimation performance via Mean Absolute Percentage Error (Mean APE) and \(\delta\) success rate (\(\max(s/\hat{s}, \hat{s}/s) \le 1.25\)). Table 1 and Table 2 summarize the segmentation and metric results.

Dataset / Setting	Method	Modality	Overall mIoU (%)	Func. Obj / Part (%)	Gnd. Mea. (%)	Relative Gain vs SOTA
ScanNet (Object)	Image baseline (Qwen-VL + Grounded-SAM)	2D	32.52	31.46	36.67	baseline
ScanNet (Object)	Reason3D	3D	9.31	–	6.09	-23.21
ScanNet (Object)	UniSeg3D	3D	22.97	23.64	24.00	-9.55
ScanNet (Object)	Ground3D-LMM (Ours)	3D	37.40	37.79	37.64	+14.43 vs UniSeg3D
ScanNet (Object)	Ground3D-LMM (Ours)	3D+2D	42.22	41.37	43.73	+9.70 vs 2D baseline
ScanNet (Part)	Image baseline	2D	21.52	20.70	22.94	baseline
ScanNet (Part)	Reason3D	3D	8.19	–	7.04	-13.33
ScanNet (Part)	UniSeg3D	3D	11.28	11.71	10.14	-10.24
ScanNet (Part)	Ground3D-LMM (Ours)	3D	30.06	30.47	27.27	+18.78 vs UniSeg3D
ScanNet (Part)	Ground3D-LMM (Ours)	3D+2D	36.57	37.33	31.70	+15.05 vs 2D baseline

In metric estimation on the Ground3D-ScanNet subset (Table 4): - Image baseline (2D): Mean APE is 166.91%, \(\delta\) success rate is 25.05%; - SD-VLM (2D): Mean APE is 231.03%, \(\delta\) success rate is 20.12%; - Ground3D-LMM (3D+2D): Mean APE drops to 74.03%, and \(\delta\) success rate rises to 43.55% (Hallucination score 9.16/10).

On the external ScanRefer benchmark (Table 6), Ground3D-LMM achieves 38.72% mIoU under pure 3D input ([email protected] of 55.73%), outperforming prior 3D SOTA UniSeg3D (29.10%) by 9.62% and Reason3D (13.05%) by 25.67%. Under 3D+2D multimodal input, Ground3D-LMM reaches 41.30% mIoU, outperforming MLLM-For3D+VideoLISA (30.45%) by 10.85%.

Ablation Study¶

Table 7 investigates the impact of dataset size, multi-task supervision, and label noise on Ground3D-ScanNet object-level performance (using the 3D+2D model):

Configuration	mIoU (%)	APE (%)	\(\delta\) Success Rate (%)	Hallucination Score (0-10)	Note
Full model (3D+2D)	42.22	76.07	46.34	9.16	full model setting
(a) Trained with 25% data	36.93	99.23	40.26	8.86	data reduction hurts metric estimation sharply
(a) Trained with 50% data	39.64	81.89	40.84	8.93	monotonic improvement across data scales
(b) Text-only supervision	–	77.41	45.62	9.02	cannot predict masks; metric accuracy degrades
(c) 25% noise in GT masks	41.43	77.97	44.09	8.91	modest drop of only 0.79% mIoU
(c) 50% noise in GT masks	41.06	79.60	43.84	9.01	retains over 41% mIoU; robust geometric prior

Key Findings¶

3D Geometry is Indispensable for Metric Understanding: Pure 2D vision models and depth-augmented models suffer from 166%~231% Mean APE on metric estimation due to monocular depth and scale ambiguity. Direct point cloud features cut this error in half (down to 74%), demonstrating that native 3D representations are required for metric grounding.
Mutual Synergy Between Grounding and Metric Measurement: When mask supervision is removed and the model is trained with text-only supervision, metric error increases and \(\delta\) success rate degrades. Explicitly predicting point masks forces the network to attend to the exact spatial boundaries of the referent, which directly benefits physical size estimation.
2D-3D Complementarity: While 3D point clouds provide metric coordinates, adding reference 2D images provides vital semantic context and boundary sharpness, lifting object mIoU from 37.40% to 42.22% and part mIoU from 30.06% to 36.57%.

Highlights & Insights¶

Triggered In-Context Mask Generation: Employing <SEG> tokens inside the autoregressive text stream decouples conversational flow from dense mask generation, offering an elegant interface for verifiable multimodal grounding.
Viewport-Visible Metric Ground Truth: Grounding metrics to the visible point cloud oriented bounding box (OBB) rather than complete CAD meshes accurately reflects real-world sensor constraints in robotics and embodied navigation.
Scalable Multi-Stage Part Annotation Pipeline: Combining open-vocabulary proposals from large VLMs, SAM2 mask segmentation, and depth back-projection provides an effective recipe for generating millions of fine-grained 3D point annotations.

Limitations & Future Work¶

Reliance on Bounding-Box Approximations: Metric dimensions for non-rigid or complex curved objects (e.g., curved cables, irregular chair arms) are currently approximated via 3D OBBs, which can overestimate true physical thicknesses.
Sensitivity to Occlusion: Because metrics are measured over visible points from the current camera viewpoint, heavy occlusions yield measurements of the visible fragment rather than the complete amodal object.
Future Directions: Incorporating neural implicit representations or 3D Gaussian Splatting (3DGS) to reconstruct amodal 3D geometry across multiple viewpoints could enable robust metric estimation in unconstrained outdoor scenes.

vs Reason3D (Huang et al., 2024): Reason3D targets open-vocabulary category-level semantic segmentation and degrades significantly on fine-grained part localization and multi-instance scenes (mIoU < 10% on Ground3D); Ground3D-LMM incorporates superpoint hierarchical representations to handle both object and part granularity effectively.
vs SD-VLM (Chen et al., 2025): SD-VLM encodes depth into 2D vision-language models for metric queries, but still experiences high metric error (>180% APE) and lacks 3D point mask outputs; Ground3D-LMM operates natively on 3D coordinates via sparse 3D U-Nets to maintain metric scale fidelity.

Rating¶

Novelty: ⭐⭐⭐⭐ [First unified LMM coupling 3D point-level mask grounding with physical metric numerical reasoning]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Introduces 2.5M QA benchmark across 8 sub-tasks and achieves substantial gains on ScanNet, ScanNet++, ScanRefer, and Reason3D]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear technical exposition, crisp pipeline figures, and transparent task formulations]
Value: ⭐⭐⭐⭐⭐ [Establishes a solid foundational baseline for embodied robotic manipulation and verifiable 3D spatial intelligence]