LangLoc: “Tell Me What You See”

Conference: ECCV 2026
Paper: ECCV Official
Code: https://rzninvo.github.io/Lang-Loc/
Area: AI Safety / 3D Vision / Multimodal VLM
Keywords: indoor localization / natural language localization / 3D scene graphs / ray-cast visibility scoring / Bayesian dialog disambiguation

TL;DR

LangLoc is the first indoor localization pipeline that estimates an observer's 2D floor position and heading direction directly from natural language descriptions without any camera images, combining dual-branch GATv2 scene retrieval, ray-cast visibility floor scoring, and an interactive Bayesian dialog module to achieve sub-meter median accuracy.

Background & Motivation

Knowing an observer's precise spatial coordinates and heading is fundamental to location-aware services such as indoor robot assistance, emergency dispatch, and augmented reality. Conventional localization paradigms rely almost exclusively on visual data, requiring client devices to capture high-resolution images or video streams and query a server via 2D-3D matching, scene coordinate regression (such as ACE or DSAC), or hierarchical re-localization. However, visual capture presents severe operational and ethical bottlenecks: transmitting images over bandwidth-constrained networks is costly; more critically, capturing photos in private residences, hospital rooms, or confidential research labs risks leaking sensitive personal identities; and in low-texture or dimly lit environments (such as blank walls or uniform corridors), users frequently fail to frame a sufficiently discriminative view.

Natural language descriptions provide a compelling, privacy-preserving, and sensor-free alternative. Telling a system "I am standing in front of a bookshelf, with a blue sofa on my left and a TV across the room" conveys high-density spatial and semantic cues without transmitting visual identifiers or requiring specialized optical cameras. Despite this potential, prior language-based localization methods have remained confined to coarse scene retrieval: outdoor methods such as Text2Pos only index coarse point-cloud cells, while indoor works such as Text2SGM retrieve room-level scene graphs without estimating any continuous metric pose. Resolving an intra-scene pose from text alone is difficult because multiple vantage points in the same room share nearly identical object semantics, differing only in subtle relative geometry and physical line-of-sight visibility.

The core insight of this work is that an observer's verbal description is strictly governed by physical visibility and line-of-sight occlusion from their standing vantage point, and remaining multi-modal spatial ambiguities can be systematically dissolved by actively querying the observer with targeted binary questions. Core idea: build a three-stage pipeline integrating dual-branch GATv2 cross-modal scene retrieval, ray-cast visibility floor-grid scoring, and information-gain-driven Bayesian dialog disambiguation to achieve accurate 2D coordinates and heading estimation purely from natural language.

Method

Overall Architecture

LangLoc operates across three sequential stages: given a free-form textual description \(T\) and a known database of 3D indoor scenes \(\mathcal{D}\), the system parses \(T\) into a text scene graph \(G^t\) and retrieves the most compatible 3D scene using a dual-branch graph attention encoder. Next, it establishes a dense walkable floor grid within the retrieved 3D mesh, evaluating candidates via ray-cast physical visibility and distance proximity to predict an initial 2D coordinate \(\hat{\mathbf{c}}\) and viewing direction \(\hat{\theta}\). Finally, if the posterior distribution reveals multi-modal ambiguities, an interactive dialog module selects targeted binary questions maximizing expected information gain, iteratively updating a Bayesian belief over reference frames until the pose converges.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Natural Language Description T"] --> B["Text Scene Graph Parsing<br/>Extract object labels and spatial relations"]
    B --> C["Dual-Branch GATv2 Scene Retrieval<br/>Gated fusion of geometry and relation branches"]
    C --> D["Retrieved 3D Mesh & Floor Grid Sampling"]
    D --> E["Ray-Cast Visibility & Proximity Scoring<br/>Dense evaluation of candidate 2D pose and heading"]
    E --> F{"Posterior Multi-Modal Ambiguity?"}
    F -->|Unimodal / High Confidence| G["Output Predicted Pose (c, θ)"]
    F -->|Multi-Modal / Ambiguous| H["Bayesian Dialog Disambiguation<br/>Greedy max-information-gain yes/no questions"]
    H -->|Posterior Concentration| G

Key Designs

1. Dual-Branch GATv2 Scene Retrieval: Resolving Asymmetry Between Sparse Text and Dense 3D Graphs

While both 3D scene graphs and text-derived scene graphs share a node-edge graph abstraction, they exhibit profound geometric and semantic asymmetries. 3D scene graph nodes carry physical 3D centroids \(\mathbf{c}_i \in \mathbb{R}^3\), mean RGB colors \(\mathbf{k}_i \in \mathbb{R}^3\), and CLIP text embeddings of their labels \(\phi(l_i) \in \mathbb{R}^{512}\) (totalling 518 dimensions), whereas text scene graphs lack metric coordinates and raw photometric properties entirely. Rather than naively projecting them into a symmetric space, the pipeline zero-pads missing fields in text queries and processes graphs through two specialized branches. The geometric branch connects each node to its \(k=5\) nearest spatial neighbors using relative displacement vectors \(\Delta\mathbf{c}_{ij}\), Euclidean distances, and bounding radii via GATv2 message passing, while the relation branch embeds relational predicates with CLIP text embeddings. A learned per-node gating vector fuses the two representations dynamically:

\[\mathbf{h}_i = \boldsymbol{\alpha}_i \odot \mathbf{g}_i + (1 - \boldsymbol{\alpha}_i) \odot \mathbf{t}_i, \quad \boldsymbol{\alpha}_i = \sigma(\text{MLP}([\mathbf{g}_i \mathbin{\|} \mathbf{t}_i]))\]

For text graphs lacking explicit coordinates, the gate learns to downweight the geometric branch and rely on relational topology. The graph representation is pooled and concatenated with a global scene-level CLIP descriptor \(\mathbf{u}(G)\) summarizing unique object categories, trained under an InfoNCE contrastive objective. At test time, candidate scenes are ranked via a combined metric incorporating learned embedding cosine similarity, global CLIP similarity, and label Jaccard/F1 overlap, ensuring robustness against lexical variations.

2. Ray-Cast Visibility and Proximity Grid Scoring: Inferring 2D Floor Pose and Heading without Images

Once the target scene is retrieved, the central challenge is grounding abstract verbal descriptions into continuous metric coordinates. The system builds on the physical principle that an observer can only mention objects that are simultaneously visible and unobstructed from their vantage point. Query entities are first greedily mapped to 3D scene instances using Word2Vec semantic cosine similarities. The system then samples candidate floor coordinates \(\mathbf{c}_k \in \mathbb{R}^2\) on a uniform grid with spacing \(\Delta=0.25\text{ m}\) over the walkable floor area, lifting each point to a fixed eye height \(h=1.6\text{ m}\). For each candidate and matched object centroid \(\boldsymbol{\mu}_o\), a single geometric ray is cast against the scene mesh to verify physical line-of-sight visibility. The candidate score balances total visible objects with a proximity exponential reward:

\[s_k = \sum_{o} \mathbb{I}[\text{vis}(o, \mathbf{c}_k)] + w_d \sum_{o: \text{vis}(o, \mathbf{c}_k)} \exp\left(-\frac{d_{ko}}{\tau_d}\right)\]

where \(d_{ko}\) denotes the horizontal distance and \(w_d, \tau_d\) govern proximity weighting. Scores are converted to spatial probabilities via Softmax. For heading estimation, the system evaluates viewing frustums centered on top-scoring locations, selecting the horizontal field-of-view orientation that encompasses the maximum number of visible matched objects and taking the normalized average direction as the predicted heading vector \(\hat{\theta} \in \mathbb{S}^2\).

3. Maximum Information Gain Bayesian Dialog: Actively Collapsing Spatial Multi-Modal Ambiguity

Indoor environments frequently contain symmetric layouts or duplicate furniture clusters (e.g., two identical sofas facing identical monitors on opposite sides of a room), causing single-turn descriptions to produce multiple distant posterior peaks. To actively collapse such ambiguities, LangLoc deploys an interactive dialog agent. The continuous pose space is mapped to representative reference keyframes \(\{(\mathbf{t}_j, \theta_j)\}\) selected offline via a two-stage determinantal point process (DPP) maximizing semantic diversity and spatial separation. At each dialog round, the module evaluates a pool of candidate yes/no questions concerning object visibility or pairwise spatial relations. It greedily selects the question \(q\) maximizing expected mutual information (information gain):

\[s(q) = H(p) - \sum_{a \in \{\text{yes}, \text{no}, \text{unknown}\}} p(a \mid q) H(p(\cdot \mid a, q))\]

Upon receiving user response \(a\), a soft observation likelihood \(\mathcal{L}(a \mid j)\) accounting for user unreliability and unanswerability updates the frame belief via Bayes' rule \(p(j) \leftarrow p(j)\mathcal{L}(a \mid j) / \sum_{j'} p(j')\mathcal{L}(a \mid j')\). Once the posterior probability concentrates above a confidence threshold \(\tau\) or reaches a turn budget, the maximum a posteriori (MAP) reference frame defines the final localized 2D position and orientation.

A Worked Example

Consider an open-plan apartment where a user states: "I am next to a wooden dining table, with a chair on my left and a floor lamp ahead." 1. Scene Retrieval: Extracted entities and spatial relations match the apartment graph, and the dual-branch GATv2 ranks the target scene Top-1. 2. Floor Grid Scoring: On the walkable mesh, both the dining table and an identical wooden work desk in the study satisfy the "table + chair" description. Consequently, the posterior forms two distinct probability peaks separated by 4.2 meters. In single-shot mode, the estimated mode may land on the incorrect desk cluster, yielding a large raw position error (~1.7 m), though the true dining location remains within the Top-10 scoring grid cells. 3. Active Bayesian Dialog: Detecting the bimodal posterior, the system calculates expected information gain across candidate questions and poses: "Can you see a refrigerator?" (visible from the dining area, occluded from the study, providing an even 50/50 prior split). The user replies "Yes". The posterior immediately collapses onto the dining area mode, followed by a second verification question confirming "Is the lamp to the left of the window?". The final posterior confidence exceeds 0.94, pinpointing the position with error dropping from 1.74 m to 0.28 m.

Loss & Training

The dual-branch GATv2 scene retrieval encoder is trained end-to-end using an InfoNCE contrastive loss:

\[\mathcal{L}_{\text{ret}} = -\log \frac{\exp(\cos(\mathbf{z}(G^t), \mathbf{z}(G^s)) / \tau)}{\sum_{s' \in \mathcal{B}} \exp(\cos(\mathbf{z}(G^t), \mathbf{z}(G^{s'})) / \tau)}\]

with temperature \(\tau=0.07\). Optimization utilizes AdamW with an initial learning rate of \(5 \times 10^{-4}\), weight decay of \(10^{-4}\), 50% negative sampling, and gradient clipping at maximum norm 1.0 for 70 epochs. The subsequent visibility-based fine localization and Bayesian dialog disambiguation operate entirely via geometric ray-tracing and probabilistic inference, requiring zero fine-tuning on downstream environments.

Key Experimental Results

Main Results

Retrieval performance was evaluated on ScanScribe (derived from 3RScan with 3DSSG annotations) across both 10-candidate pools and challenging LLM-generated zero-shot descriptions.

Table 1: Scene Retrieval Recall@k (%) on ScanScribe (from paper Table 1 and Table 3)

Protocol / Method	Top-1	Top-2	Top-3	Top-5	Mechanism & Notes
10-Scene Pool (Table 1)
Text2Pos (fine-tuning)	10.30 ± 0.93	20.74 ± 1.32	31.65 ± 1.45	53.42 ± 1.68	Cross-modal point cloud coarse cell retrieval
CLIP2CLIP	33.09 ± 0.21	52.53 ± 0.24	65.98 ± 0.21	82.69 ± 0.13	Rendered multi-view visual-text alignment
Text2SGM (cos-sim)	68.27 ± 2.05	84.95 ± 1.62	91.65 ± 1.23	97.14 ± 0.76	Prior SOTA graph matching
Text2SGM (ret-based)	68.61 ± 0.04	85.00 ± 0.02	91.57 ± 0.01	96.99 ± 0.00	Specialized retrieval baseline
LangLoc (Ours)	76.70 ± 4.58	90.40 ± 2.73	96.10 ± 1.58	98.90 ± 1.22	Dual-branch GATv2 + CLIP (+8.1 pp over SOTA)
LLM Queries (Table 3)					Reduced lexical overlap, testing natural phrasing
CLIP2CLIP	33.07 ± 0.21	52.52 ± 0.23	66.00 ± 0.21	82.78 ± 0.13	Image-text matching under vocabulary shift
Text2SGM (cos-sim)	34.22 ± 1.77	56.67 ± 1.70	68.78 ± 2.35	82.11 ± 1.23	Severe degradation due to missing canonical tags
LangLoc (Ours)	59.50 ± 5.26	76.40 ± 4.94	87.80 ± 3.12	96.20 ± 2.27	Outperforms SOTA by +25.3 pp Top-1

Ablation Study

Fine localization accuracy was evaluated on 100-scene subsets of 3RScan and ScanNet, benchmarking against non-language geometric baselines and zero-shot VLMs.

Table 2: Fine Localization Accuracy and Ablations on 100-Scene Subsets (from paper Table 4)

Dataset	Configuration	Top-10 Pos. (m) ↓	Pos. Mean (m) ↓	Pos. Median (m) ↓	Angle Mean (deg) ↓	Angle Median (deg) ↓	3D IoU ↑
3RScan	Midpoint (Floor Centroid)	–	1.416	1.347	–	–	–
	VLM (Qwen2.5-VL-2B Zero-Shot)	–	1.582	1.420	85.48	85.54	0.062
	LangLoc w/o dialog	1.037	1.712	1.551	46.07	37.24	0.172
	LangLoc w/ dialog	0.983	0.926	0.799	39.52	33.37	0.342
ScanNet	Midpoint (Floor Centroid)	–	1.279	1.098	–	–	–
	VLM (Qwen2.5-VL-2B Zero-Shot)	–	1.382	1.099	93.96	91.27	0.030
	LangLoc w/o dialog	1.254	1.676	1.314	42.67	34.66	0.236
	LangLoc w/ dialog	0.532	0.593	0.069	23.77	5.12	0.593

Key Findings

• Dialog Disambiguation Dramatically Reduces Error: On 3RScan, interactive Bayesian dialog cuts median position error from 1.551 m to 0.799 m (a 48.5% reduction) and doubles 3D frustum IoU from 0.172 to 0.342. On ScanNet, where scenes contain more visible objects per view (average 6 vs. 4), each question is highly discriminative; the median position error plunges to 0.069 m (under 7 cm) with median angular error dropping to 5.12°.
• Zero-Shot Top-Down VLMs Fail at Spatial Metric Orientation: Zero-shot Qwen2.5-VL-2B achieves angular errors exceeding 85°–93° (virtually random guessing) and near-zero 3D IoU (0.030–0.062), proving that vision-language self-attention alone cannot deduce metric occlusions and lines of sight from top-down floor maps.
• The Midpoint Fallacy in Indoor Rooms: The unguided floor midpoint yields a deceptively competitive raw distance error (~1.3 m) merely because indoor rooms are compact. However, it provides zero orientation and zero viewpoint evidence. In contrast, LangLoc's Top-10 position error (0.983 m on 3RScan, 0.532 m on ScanNet) confirms that true poses reside firmly within top-ranked candidate cells, which dialog subsequently pinpoints.

Highlights & Insights

• Privacy-Preserving and Low-Bandwidth Localization: By eliminating image capture, LangLoc transmits lightweight text strings rather than megabyte-scale video feeds, making indoor positioning viable in camera-prohibited settings such as medical wards, government facilities, or secure research cleanrooms.
• Training-Free Physical Visibility Scoring: Leveraging exact mesh ray-casting to evaluate physical line-of-sight constraints grounds natural language into metric space deterministically, avoiding black-box regression failures.
• Active Uncertainty Resolution: Instead of treating localization as a passive one-shot prediction, formulating disambiguation as an information-theoretic game allows the system to actively ask the user discriminative questions, providing a template for human-agent embodied cooperation.

Limitations & Future Work

• Dependency on Pre-Annotated 3D Semantic Meshes: The system requires pre-scanned 3D meshes with annotated object instances and scene graphs, limiting immediate deployment in unmapped environments.
• Interactive Latency Under Dense Layouts: In highly repetitive or cluttered scenes lacking distinctive landmarks, multiple dialog turns are required to prune ambiguity, which may tax human user patience.
• Future Directions: Exploring localization against coarse architectural 2D floor plans or integrating open-vocabulary 3D foundational models could eliminate manual annotation barriers.

Related Work & Insights

• vs. Text2Pos [CVPR 2022]: Text2Pos learns joint embeddings between text and outdoor point clouds for coarse cell retrieval; LangLoc advances indoor localization from coarse retrieval to fine-grained 2D coordinates and heading estimation.
• vs. Text2SGM [ECCV 2024]: Text2SGM pioneered 3D scene-graph retrieval from text queries but stopped at room identification; LangLoc improves Top-1 retrieval recall by 8.1 percentage points via dual-branch GATv2 and extends the task to continuous intra-scene pose estimation.
• vs. DiaLoc [CVPR 2024] & SQA3D [ICLR 2023]: While DiaLoc and SQA3D study dialog-guided navigation or embodied question answering with camera/point-cloud inputs, LangLoc uses dialog specifically as an active Bayesian mechanism to resolve geometric ambiguities in camera-free localization.

Rating

• Novelty: ⭐⭐⭐⭐⭐ (First end-to-end framework resolving continuous indoor 2D position and heading purely from natural language)
• Experimental Thoroughness: ⭐⭐⭐⭐½ (Rigorous validation across 3RScan, ScanNet, and human interactive pilots with well-defined controlled protocols)
• Writing Quality: ⭐⭐⭐⭐⭐ (Clear mathematical formulation, well-structured pipeline diagrams, and insightful experimental analysis)
• Value: ⭐⭐⭐⭐½ (Highly valuable for privacy-sensitive environments, camera-less devices, and assistive human-robot collaboration)

Related Papers

• [NeurIPS 2025] Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research
• [ICML 2026] SemGrad: Gradients w.r.t. Semantics-Preserving Embeddings Tell LLM Uncertainty
• [CVPR 2026] What Your Features Reveal: Data-Efficient Black-Box Feature Inversion Attack for Split DNNs
• [ICLR 2026] Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models
• [ICLR 2026] LitmusValues: Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmas