title: >- [Paper Note] MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshes description: >- [ECCV 2026][3D Vision][Geometric Entity Localization] MV-GEL introduces a prompt-conditioned view ranking framework (GELviews) and CAD-adapted reasoning segmentation (LISA-CAD) to accurately ground fine-grained CAD faces and edges directly on unstructured triangle meshes without requiring B-Rep data. tags: - ECCV 2026 - 3D Vision - Geometric Entity Localization - Multi-View Selection - Referring Segmentation date: 2026-09-19 content_hash: cd3a1cce02c7238a

MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshes¶

Conference: ECCV 2026
Paper: ECCV Original
Code: https://github.com/kbali1297/MV-GEL
Area: 3D Vision
Keywords: Geometric Entity Localization, Multi-View Viewpoint Selection, Vision-Language Models, CAD Mesh, 3D Referring Expression Segmentation

TL;DR¶

Addressing severe viewpoint occlusions and topological boundary ambiguity in 3D CAD grounding, MV-GEL decouples prompt-conditioned geometric view ranking (GELviews) from domain-adapted 2D reasoning segmentation (LISA-CAD), accurately localizing natural-language-queried faces and edges directly on unstructured polygon meshes via geometry-aware ray casting.

Background & Motivation¶

In computer-aided design (CAD), robotic manipulation, and scientific engineering simulation, grounding and identifying precise geometric entities—such as fillets, chamfers, planar mounting pads, and curved inner cylindrical surfaces—is a foundational capability. In standard industrial workflows, these targets are strictly defined as discrete topological elements within boundary representations (B-Reps). However, when downstream pipelines receive unstructured polygon meshes or raw point clouds lacking analytical topology metadata, identifying which discrete elements belong to a language-described feature becomes remarkably difficult.

While contemporary vision-language models (VLMs) have achieved remarkable reasoning segmentation performance on 2D images, transferring this capability to structured 3D geometry is bottlenecked by acute viewpoint sensitivity. The visual appearance of a 3D object changes drastically across camera angles: a viewpoint aligned head-on renders a queried entity clearly observable, whereas oblique or rear angles lead to severe self-occlusion, depth ambiguity, and projective foreshortening. Moreover, existing open-vocabulary 3D grounding frameworks predominantly operate on unstructured point clouds or Gaussian splats; lacking explicit topological connectivity, they frequently suffer from massive over-segmentation bleed across textureless industrial surfaces, yielding dismal edge-level IoU figures below 0.02.

The key insight of this work is that language-driven geometric localization is not merely an image-text segmentation problem, but rather an observability-aware viewpoint prioritization challenge—the core tension lies in rendering structured 3D geometry visually and geometrically digestible to 2D reasoning VLMs. The core idea is to decouple prompt-conditioned geometric view ranking from reasoning segmentation, introducing a cross-view self-attention ranking module (GELviews) and domain-adapted segmentation (LISA-CAD) that lifts 2D masks back onto discrete mesh faces and edges via geometry-aware ray casting.

Method¶

Overall Architecture¶

The input to MV-GEL is an unstructured triangle mesh $M = (V, E, F)$ (comprising vertex set $V$, edge set $E$, and face set $F$) and a natural language query $q$ specifying a target geometric entity; the output is a binary labeling vector $\hat{y} \in \{0, 1\}^{|X|}$ defined directly on the discrete mesh primitives ($X \in \{E, F\}$ for faces or edges). The overall processing pipeline proceeds across four coordinated stages: multi-view rendering, prompt-conditioned geometric view ranking, image-space reasoning segmentation, and geometry-aware topological lifting.

Initially, a set of 60 calibrated candidate views is rendered by uniformly sampling azimuths in $[0^\circ, 360^\circ)$ and elevations in $[-60^\circ, 60^\circ]$ at $30^\circ$ increments. Next, the prompt-conditioned ranking module GELviews evaluates the geometric observability of the queried entity across all candidates using multimodal conditioning and cross-view self-attention, greedily selecting the top-$K$ viewpoints. Domain-adapted LISA-CAD then processes each selected 2D rendering independently to infer binary target masks. Finally, a geometry-aware ray casting operator projects foreground mask pixels back onto the 3D surface, intersecting mesh primitives and aggregating predictions into topological face or edge assignments.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Triangle Mesh M & Language Query q"] --> B["Uniform Candidate View Rendering<br/>60 calibrated azimuth/elevation camera poses"]
    B --> C["GELviews Geometric View Ranking<br/>LoRA-CLIP encoding + Cross-View Transformer"]
    C --> D["Greedy Selection of Top-K Views<br/>Retain angular neighbors as fallback redundancy"]
    D --> E["LISA-CAD 2D Reasoning Segmentation<br/>LLaVA SEG token generation + SAM mask decoding"]
    E --> F["Geometry-Aware Ray Topological Lifting<br/>Camera ray casting & intersection onto F and E"]
    F --> G["Output: Mesh Topological Entity Binary Labeling ŷ"]

Key Designs¶

1. GELviews Geometric View Ranking: Fusing Spatial Position Embeddings with Cross-View Self-Attention Conventional view selection techniques rely on computing the global cosine similarity between raw CLIP image embeddings and textual descriptions. On textureless, homogeneous CAD parts, however, global matching is blind to localized geometric exposure and structural occlusions. GELviews remedies this by adapting CLIP vision and text encoders using low-rank adapters (LoRA) to capture fine-grained CAD semantics. To inject viewpoint geometry into the visual token stream, learnable positional embeddings $p_v$ are added to image features $f_v$, producing pose-aware representations $\tilde{f}_v = f_v + p_v$. These are conditioned on the global query embedding $t$ through Feature-wise Linear Modulation (FiLM): $$\text{FiLM}(\tilde{f}_v, t) = \gamma(t) \odot \tilde{f}_v + \beta(t)$$ Because entity visibility is fundamentally relative across different angles, the modulated tokens are processed by a multi-layer Transformer encoder operating across the viewpoint dimension. This enables cross-view self-attention, allowing each viewpoint to contextualize its exposure against all candidate angles. A linear projection head outputs a geometric adjustment score $s_v^{\text{geo}}$, yielding the composite ranking criterion: $$s_v = \alpha (s_v^{\text{clip}} + s_v^{\text{geo}})$$ This mechanism actively downweights grazing angles and severely occluded perspectives while elevating direct, unobstructed head-on views.

2. Soft-Margin Pairwise Ranking Objective: Isolating Informative Perspectives from Background Noise Across a large camera sphere, the vast majority of arbitrary viewpoints either completely occlude the target entity or observe it from uninformative grazing angles; supervising the precise relative order among these weak views introduces substantial gradient noise. MV-GEL addresses this by using Farthest Point Sampling (FPS) on ground-truth CAD entities to sample reference points $P_E$, casting rays to establish exact visibility scores $r_v$. The top 5 highest-visibility views ($\rho = 0.09$) form the positive set $V^+$, while all remaining views form $V^-$. Optimization is driven by a soft-margin pairwise ranking loss over positive-negative pairs $\mathcal{P}$: $$\mathcal{L}_{\text{rank}} = \frac{1}{|\mathcal{P}|} \sum_{(i,j) \in \mathcal{P}} \ln\left(1 + \exp(-(s_i - s_j - m))\right)$$ with a margin parameter $m = 0.1$. This objective cleanly separates top informative vantage points from uninformative ones without over-penalizing trivial differences among background perspectives.

3. Geometry-Aware Ray Lifting and View Retention Strategy: Rejecting View-NMS To map 2D pixel masks back into discrete mesh topology, MV-GEL casts camera rays through foreground pixels using calibrated camera projection matrices, directly intersecting the triangular faces and continuous boundary edges of mesh $M$. Crucially, while standard 3D vision pipelines habitually enforce View Non-Maximum Suppression (View-NMS) to eliminate angularly redundant viewpoints, the authors find that View-NMS severely degrades CAD entity grounding performance. Because fine geometric entities (particularly 1D thin edges) suffer from discrete pixel discretization and stochastic VLM false-negatives in single views, closely aligned neighboring viewpoints serve as indispensable "fallback redundancy." Preserving them through greedy top-$K$ selection significantly enhances multi-view recall accumulation.

Loss & Training¶

MV-GEL maintains a completely decoupled two-stage training scheme: 1. GELviews Optimization: Trained on 54k natural query-geometric entity pairs generated from over 5,000 ABC CAD models using the pairwise ranking loss $\mathcal{L}_{\text{rank}}$. LoRA adapters on CLIP encoders use rank $r=8$, $\alpha=16$, and dropout 0.1. 2. LISA-CAD Fine-Tuning: Built upon LLaVA-7B, the model is fine-tuned on the same 54k dataset using autoregressive language modeling cross-entropy, binary cross-entropy (BCE), and DICE mask supervision losses on SAM's mask decoder while keeping the vision backbone frozen. Training is conducted on 4 NVIDIA H100 GPUs with a uniform learning rate of $1 \times 10^{-4}$.

Key Experimental Results¶

Main Results¶

Evaluation is performed on a held-out test benchmark consisting of 218 CAD meshes and 1535 natural language queries (655 face targets and 880 edge targets). Metrics are calculated directly on 3D mesh entities using area-weighted (faces) and length-weighted (edges) IoU, Precision, Recall, and F1.

Segmentation VLM	View Selector	Face IoU	Face Prec.	Face Rec.	Face F1	Edge IoU	Edge Prec.	Edge Rec.	Edge F1
LISA (Vanilla)	Random	0.122	0.159	0.287	0.174	0.028	0.034	0.261	0.046
LISA (Vanilla)	Only-CLIP	0.201	0.239	0.460	0.272	0.035	0.039	0.388	0.061
LISA (Vanilla)	FiLM	0.293	0.342	0.647	0.391	0.046	0.053	0.503	0.079
LISA (Vanilla)	GTviews (Oracle)	0.293	0.340	0.645	0.389	0.046	0.051	0.510	0.079
LISA-CAD	Random	0.196	0.262	0.293	0.250	0.071	0.097	0.156	0.101
LISA-CAD	Only-CLIP	0.305	0.374	0.464	0.377	0.160	0.193	0.327	0.210
LISA-CAD	No-Fusion	0.490	0.570	0.695	0.584	0.283	0.324	0.541	0.355
LISA-CAD	Cross-Attention	0.486	0.570	0.694	0.582	0.287	0.327	0.547	0.359
LISA-CAD (Ours)	FiLM (Ours)	0.501	0.588	0.706	0.598	0.284	0.326	0.551	0.358
LISA-CAD	GTviews (Oracle)	0.529	0.619	0.744	0.632	0.311	0.355	0.598	0.390

Ablation Study & Point Cloud Baselines¶

A comparative analysis against open-vocabulary 3D point cloud segmentation frameworks projected onto the mesh, alongside view selection filtering ablations:

Model / Configuration	Face IoU	Face Prec.	Face Rec.	Face F1	Edge IoU	Edge Prec.	Edge Rec.	Edge F1	Note
PartSLIP (Default)	0.073	0.083	0.368	0.114	0.012	0.013	0.422	0.022	Zero-shot point cloud baseline, lacks fine edge topology
Find3D (Default)	0.171	0.172	0.955	0.260	0.017	0.017	0.937	0.032	Massive over-segmentation bleed across surfaces
PatchAlign3D (Default)	0.159	0.161	0.893	0.242	0.017	0.017	0.895	0.031	Fails on thin boundary discretization
MV-GEL (FiLM@top3, ViewNMS)	0.372	0.400	0.846	0.495	0.180	0.194	0.695	0.266	45° angle View-NMS drops critical fallback coverage
MV-GEL (FiLM@top3)	0.419	0.469	0.810	0.540	0.228	0.248	0.708	0.317	Greedy selection accumulates multi-view recall
MV-GEL (FiLM@top1)	0.501	0.588	0.706	0.598	0.284	0.326	0.551	0.358	Optimal single-view configuration balancing precision & IoU

Key Findings¶

Reaching Oracle Headroom: Operating on a single view (top-1), MV-GEL (FiLM) attains a face F1 score of 0.598, capturing 95% of the theoretical ground-truth oracle limit (GTviews F1 of 0.632). On the challenging edge localization task, it achieves an F1 of 0.358, exceeding 91.8% of the oracle upper bound (0.390), indicating near-optimal viewpoint identification.
Pathological Point Cloud Bleed: Foundation 3D point cloud models like Find3D achieve high recall (0.955 on faces, 0.937 on edges) but suffer from abysmal precision (0.172 on faces, 0.017 on edges), resulting in an edge IoU of only 0.017 due to unconstrained bleeding across discrete boundaries.
Counter-Intuitive View-NMS Degradation: Imposing a $45^\circ$ View-NMS constraint on top-3 views reduces Face IoU from 0.419 to 0.372 and Edge F1 from 0.317 to 0.266, quantitatively confirming that adjacent viewpoints provide indispensable segmentation redundancy.
Exceptional Latency-Coverage Efficiency: Profiling on an NVIDIA H100 GPU shows GELviews introduces only ~0.1 s ranking overhead (total ~0.25 s per query) while matching the face recall of 5 randomly sampled views, which demand >0.6 s latency.

Highlights & Insights¶

Decoupling Viewpoint Digestibility from 2D Reasoning: Rather than forcing VLMs to reason over convoluted 3D point tokens, MV-GEL demonstrates that engineering prompt-conditioned viewpoint observability provides a vastly superior inductive bias for fine-grained 3D grounding.
Unveiling the Value of Viewpoint Fallback Redundancy: The study disproves the universal efficacy of View-NMS in multi-view reasoning segmentation, proving that correlated adjacent views act as crucial safety nets for thin, view-sensitive geometry.
High Transferability to Embodied Systems: The self-attention view-ranking mechanism can be directly generalized to active robotic viewpoint selection, industrial inspection drones, and Next-Best-View exploration in physical environments.

Limitations & Future Work¶

Single-Entity Prompt Assumption: Current data generation and benchmarking assume queries target a single isolated face or edge; handling compound multi-instance prompts (e.g., "all counterbore holes on the top flange") remains unaddressed.
Occlusion within Deep Concavities: Spherical exterior camera sampling cannot effectively probe deeply recessed interior channels or tortuous cavities without adaptive focal zooming or cross-sectional mesh slicing.
Future Directions: Extending the framework toward active closed-loop viewpoint trajectory planning and interactive conversational mesh manipulation.

vs PartSLIP / PartSLIP++: PartSLIP relies on GLIP 2D bounding box back-projections on point clouds, struggling with non-semantic or untextured geometric transitions; MV-GEL leverages dense pixel-level reasoning masks and ray-mesh intersections, boosting edge F1 by over 4.5×.
vs Cap3D / ViewRefer: These methods optimize global CLIP image-text alignment for camera selection, which fails on textureless mechanical parts; MV-GEL integrates spatial position encoding and cross-view self-attention, dramatically outperforming raw CLIP selection.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formulates language-driven CAD entity localization on raw meshes and introduces a dedicated prompt-conditioned geometric view-ranking paradigm.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 1,535 queries on 218 CAD meshes, featuring point cloud SOTA comparisons, oracle ceilings, and extensive component ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Clear pedagogical narrative, meticulous mathematical formulations, and compelling ablation visualizations.
Value: ⭐⭐⭐⭐⭐ Bridges vision-language multimodal reasoning with practical downstream CAD modeling, robotic grasping, and reverse engineering.