Abstract the Layout, Focus the Detail: A Dual-Granularity Representation Framework for Zero-Shot 3D Visual Grounding¶
Conference: ECCV2026
Paper: ECCV Official Page · Paper PDF
Code: https://github.com/VIPL-VSU/ALFD
Area: 3D Vision
Keywords: Zero-shot visual grounding, semantic spatial layout, visual detail patches, query-guided filtering, joint spatial-visual reasoning
TL;DR¶
The framework recasts a 3D scene as an abstract layout for spatial reasoning and object-centric patches for appearance verification, enabling an off-the-shelf VLM to select targets in a spatial-first, visual-second workflow and achieve 45.0% [email protected] on ScanRefer and 63.2% accuracy on Nr3D without GT object classes.
Background & Motivation¶
3D visual grounding identifies an instance in a 3D scene from a natural-language description and returns its bounding box. The difficult part is often not recognizing the category "cabinet," but resolving "the brown cabinet under the bed opposite the door" among several cabinets. Supervised methods can learn language-conditioned 3D relations, but require costly instance-level language annotations. Zero-shot approaches instead reuse existing LLMs and VLMs by converting 3D scenes into familiar text or images. Methods such as ZSVG3D and LaSP express relations through coordinate operations or programs, whereas SeeGround and related approaches expose scene views to a VLM.
These alternatives preserve different kinds of evidence. Coordinates provide global positions, but fixed rules can struggle to capture visually contextual relations such as "opposite" or "tucked under." Camera images preserve color, material, and shape, but limited fields of view and occlusion can prevent a bed and a distant door from appearing together. Adding local views still leaves the model responsible for reconstructing the room layout. Even rendering the whole scene from above can retain substantial background clutter and object occlusion. The bottleneck is therefore whether the input representation clearly exposes the required evidence, not simply whether the reasoning prompt is elaborate enough.
The paper draws on the visualization principle of overview first, filtering next, and details on demand, giving different evidence types their own representations. Core idea: retain the target and reference objects required by the query, expose global relations through a semantic spatial layout, and verify appearance through separate object detail patches, so the VLM need not reconstruct global space from local images.
Method¶
Overall Architecture¶
The inputs are a scene point cloud, its corresponding RGB-D frames, and a grounding query. The output is a target instance ID, which identifies the associated 3D bounding box. Query-Guided Filtering performs instance labeling, linguistic entity analysis, and soft semantic matching; Semantic Spatial Layout (SSL) and Visual Detail Patches (VDP) are then constructed in parallel, followed by Joint Spatial-Visual Reasoning.
The two images are not ordinary alternative camera views of the same scene. SSL deliberately removes texture and draws relevant instances as top-down polygons with IDs. VDP deliberately relinquishes the task of representing global inter-object positions and retains local images suitable for checking color, shape, and state. Instance IDs link these evidence sources, while optional height descriptions compensate for information lost in the top-down projection.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
INPUT["Point cloud, RGB-D frames<br/>and natural-language query"] --> FILTER["Query-Guided Filtering"]
FILTER --> SSL["Semantic Spatial Layout"]
FILTER --> VDP["Visual Detail Patches"]
SSL -->|"Layout and optional height text"| REASON["Joint Spatial-Visual Reasoning"]
VDP -->|"Appearance evidence with instance IDs"| REASON
REASON --> OUTPUT["Target instance ID<br/>and 3D bounding box"]
Key Designs¶
1. Query-Guided Filtering: preserve the instances needed by the relation chain, not just the target category
The system first obtains object proposals from an off-the-shelf 3D instance segmentation network, but does not treat its closed-set categories as final semantics. For each proposal, it selects a suitable 2D frame and projects the 3D mask into that image to produce a visual highlight. The original RGB frame, annotated frame, and predicted 3D category are supplied to a VLM. The category acts as a soft prior: it helps distinguish books on a shelf from the entire bookshelf without restricting open-vocabulary recognition to a fixed label list. The resulting instance labels support subsequent language alignment. This initialization reuses the view-selection mechanism described under Visual Detail Patches.
An LLM then extracts target and reference entities from the query and determines their evidence requirements: whether an entity is a target candidate, needs fine-grained appearance verification, involves height, or imposes viewpoint or position conditions. The paper denotes these flags as \(\tau_{tgt},\tau_{vis},\tau_{hgt},\tau_{cam},\tau_{pos}\). They are switches controlling downstream input construction, not additional learning objectives. For example, "under the bed" requires height evidence, "brown cabinet" requires appearance verification, and "facing the door" requires observer-direction alignment. Entity parsing and label alignment are implemented together in a single LLM reasoning pass. Synonyms, hypernyms, hyponyms, and functional equivalence support soft matching, after which all instances bearing matched labels are retrieved. This preserves the full door-bed-cabinet relation chain and avoids missing synonymous descriptions through exact string matching. The corresponding risk is that an upstream semantic omission removes evidence that later reasoning needs.
2. Semantic Spatial Layout: turn global positions into an abstract map the model can read directly
For each retained instance, the system projects its 3D points onto the ground XY plane and computes a 2D convex hull. This transformation preserves approximate footprints and relative distributions while discarding texture and point-cloud noise. The paper's central geometric operation is:
Here, \(P_k^{xy}\) is the XY projection of the point cloud for instance \(k\), and \(H_k\) is the convex hull rendered on the global canvas. Category colors, numeric IDs, a coordinate grid, and peripheral axes make object identity, position, and distance references explicit. For queries such as "on the left when facing the door," viewpoint conditions identify a reference anchor and direction. The layout is rotated so that the implied observer direction aligns with the upward image axis. This replaces mental rotation with input preprocessing, reducing the burden of translating between the original coordinate system and the linguistic observer perspective.
A top-down image cannot fully express above-below relations. The authors therefore compute a topological rendering order from object Z-axis bounds, using semi-transparent polygons with solid black borders to expose overlapping instances. When the height flag is active, they also provide the relevant bounding-box \((z_{min},z_{max})\) values as text. Transparency makes planar overlap readable, whereas height text distinguishes vertical relations; neither substitutes for the other. SSL is not a complete 3D reconstruction or a relation graph containing the answer in advance. It is an intermediate representation that exposes spatial evidence on a common canvas.
3. Visual Detail Patches: choose views for appearance recognition without making local crops solve spatial reasoning
The detail branch first filters the RGB sequence for sharpness. Within small temporal windows, it measures sharpness through the variance of the grayscale Laplacian and retains the sharpest frame. Candidate views are then evaluated for geometric exposure and image composition, and the view maximizing the product of the two scores is selected. The geometry score determines point visibility by checking whether projected depth and sensor depth differ by no more than a margin \(\delta\). It combines the visible-point fraction with normal-weighted surface exposure, favoring views facing broad object surfaces. The composition score uses the projected box-to-image area ratio, with a trapezoidal decay penalizing objects that are too small or too large, together with a boundary-truncation penalty. The best frame is therefore not necessarily the one with the largest object projection: the object must also be clear and sufficiently complete.
After selecting a frame, the system crops the projected 2D box, expanding it by a factor of 1.5 to preserve local context, and assembles crops with instance IDs into a compact image. At the reasoning stage, this detail extraction is activated only for target candidates or instances explicitly requiring appearance verification, reducing irrelevant visual input and token usage. This does not imply that scene initialization avoids per-instance visual labeling altogether. Equations (1) and (2) are damaged in the cached text extraction, so this note retains only the view-selection mechanism supported by the surrounding prose rather than guessing the complete normal-direction convention, weights, or thresholds. The main text also does not specify the temporal-window length, depth margin, score weights, or trapezoidal breakpoints.
4. Joint Spatial-Visual Reasoning: identify relationally plausible candidates before checking appearance
The VLM jointly receives the query, SSL, VDP, and any requested height descriptions. The prompt asks it to infer spatial relationships from the layout first, verify visual attributes from the detail image second, and provide analysis before returning a target ID. This ordering concerns evidence use within a joint inference step; it is not a multi-round agent that repeatedly calls tools or searches for new viewpoints. The abstract representation shifts much of the difficulty into input construction, so the final prompt does not require separate elaborate execution templates for relations such as "opposite" or "middle."
The spatial step excludes instances that violate layout constraints. The visual step then checks color, shape, or state for the remaining IDs and can verify that a candidate really belongs to the requested category. This separation discourages the VLM from treating the crop montage as a map of global positions or guessing color from a texture-free layout. Because the authors use free-form reasoning rather than a deterministic relation solver, structured evidence reduces ambiguity but does not guarantee correct spatial judgments.
A Worked Example¶
Figure 1 uses a query describing the brown cabinet under the bed opposite the door. Filtering retains doors, beds, and cabinets, rather than cabinets alone. "Under" calls for height evidence, while "brown" requires visual details. The layout contains two beds: bed 1 is opposite the door, and cabinets 3 and 4 are under bed 1, narrowing the spatial candidates to 3 and 4.
The detail image identifies cabinets 4 and 5 as brown. Matching the spatial and color evidence through instance IDs leaves target 4. Cabinet 3 satisfies the spatial relation but not the color constraint; cabinet 5 satisfies the color constraint but occupies the wrong location. This example comes directly from the paper's illustration, not a new measured result, and its instance counts are not dataset statistics.
Loss & Training¶
This is a zero-shot inference framework without task-specific training: it introduces neither a new grounding loss nor a fine-tuning procedure, but depends on pretrained perception and foundation models. The default core engine is gpt-4o-2024-08-06, with temperature 0.1 and top_p 0.3 across prompts. ScanRefer proposals come from pretrained Mask3D, and the detail-crop expansion factor is 1.5. The authors place full prompts and additional implementation details in supplementary material. The local cache available for this note contains only the main paper, so unobserved configurations are not filled in.
Key Experimental Results¶
Main Results¶
ScanRefer contains 51,583 descriptions of 11,046 objects and reports accuracy at specified IoU thresholds between predicted and ground-truth boxes. Nr3D contains 41,503 queries and reports selection accuracy among supplied object proposals. The table below selects the most informative results from Tables 1 and 2 of the paper; all scores are percentages. GT classes indicate access to ground-truth category labels, which is different from access to instance proposals.
| Dataset and setting | Metric | Ours | Comparator | Comparator score | Difference (percentage points) |
|---|---|---|---|---|---|
| ScanRefer, Overall | [email protected] | 51.1 | CSVG | 49.6 | +1.5 |
| ScanRefer, Overall | [email protected] | 45.0 | CSVG | 39.8 | +5.2 |
| ScanRefer, Multiple | [email protected] | 39.7 | SeeGround | 30.0 | +9.7 |
| ScanRefer, Unique | [email protected] | 66.8 | SeeGround | 68.9 | -2.1 |
| Nr3D, without GT classes | Overall Acc | 63.2 | LaSP | 52.9 | +10.3 |
| Nr3D, with GT classes | Overall Acc | 74.7 | Transcrib3D | 70.2 | +4.5 |
These are not controlled replacements under a common foundation model. SeeGround uses Qwen2-VL-72B, CSVG uses Mistral-family models, Transcrib3D uses GPT-4, and both the proposed system and LaSP use GPT-4o. The evidence therefore supports a system-level comparison among the reported zero-shot methods, not attribution of every gain to a single representation component or a claim of winning every subset. Its lower Unique score than SeeGround is consistent with the main benefit being disambiguation among same-category distractors rather than uniformly stronger category recognition.
Ablation Study¶
The following results come from Table 4 and are measured on the sampled Nr3D subset without GT classes. They should be interpreted separately from the full-validation results above.
| Query filtering | Spatial input | Detail patches VDP | Overall Acc (%) |
|---|---|---|---|
| No | Full-scene Raw BEV + ID | No | 48.8 |
| Yes | SSL | No | 60.8 |
| Yes | None, local crops only | Yes | 56.0 |
| Yes | SSL | Yes | 66.0 |
The full system exceeds filtering plus SSL by 5.2 percentage points and filtering plus VDP by 10.0 points, supporting the complementarity of local appearance and global layout. Moving from Raw BEV to filtering plus SSL improves accuracy by 12.0 points, but adds both filtering and layout abstraction. It cannot be interpreted as a 12.0-point contribution from filtering alone. The table also does not independently disable viewpoint alignment, transparent rendering, or the view-selection score.
Efficiency & Key Findings¶
Table 3 compares the same sampled subsets used by agent-based baselines. Nr3D does not provide GT classes, and all three systems use GPT-4o. Time is measured in seconds per query; these subset accuracies must not be directly subtracted from full-validation scores.
| Method | ScanRefer subset [email protected] (%) | Nr3D subset Acc (%) | Mean inference time (seconds/query) |
|---|---|---|---|
| VLM-Grounder | 33.5 | 48.0 | 50.3 |
| SPAZER | 48.8 | 63.8 | 23.5 |
| Ours | 49.2 | 66.0 | 10.9 |
Compared with SPAZER, the proposed system gains 0.4 and 2.2 percentage points on the two subsets while reducing average time from 23.5 to 10.9 seconds. The authors attribute this advantage to compact layouts and a small set of object crops replacing long multi-view inputs. The main paper does not break down per-stage latency, token counts, or initialization amortization. The results support an inference-efficiency advantage under this evaluation, not a claim of real-time robotic deployment.
Highlights & Insights¶
- The spatial image deliberately removes appearance, and the appearance image deliberately drops the global spatial task. Rather than merely adding modalities, the framework assigns low-distraction evidence to distinct questions, an input-design principle that could transfer to object retrieval or embodied question answering.
- Query-conditioned viewpoint alignment makes coordinate transformation explicit. Compared with asking a VLM to perform mental rotation from text alone, this computable preprocessing leaves more reasoning capacity for interpreting relations.
- The ablation supports the importance of a readable global layout over simply supplying more realistic imagery, while showing that layout cannot replace appearance verification. Foundation-model applications may benefit from reconsidering how evidence is presented before increasing reasoning steps.
Limitations & Future Work¶
- The following are reading-based assessments of the method and experiments; the main paper has no dedicated limitations section. Instance segmentation, open-vocabulary labeling, and query filtering form a chain of dependencies. Once a target or reference object is omitted, neither downstream image can restore it; low-confidence cases could retain a broader candidate set instead of discarding uncertain instances.
- Convex-hull projection removes concavities and some 3D detail, while height intervals restore only limited vertical information. Complex stacking or non-ground-plane arrangements may need supplementary side-view sections, but the paper does not evaluate this extension.
- Nr3D improves from 63.2% without GT classes to 74.7% with them, indicating that category information remains important. This 11.5-point gap is not a strict measure of perception error because the settings also change the evidence available to downstream reasoning.
- Foundation models are not uniformly matched, and the experiments lack an isolated filtering ablation, individual view-selection ablations, and uncertainty intervals. The evidence supports the complete design but cannot precisely rank the independent value of each engineering choice.
- Self-scanned real-world demonstrations are assigned to supplementary material. The main text provides no verifiable robotic task-success rate or scale of open-environment evaluation, so deployment should be described as preliminary exploration rather than demonstrated closed-loop navigation capability.
Related Work & Insights¶
- Compared with ZSVG3D / LaSP: These methods use visual programs or spatial code to execute relation judgments, whereas this framework asks a VLM to read an abstract spatial layout. Program execution is easier to inspect, while the layout approach avoids rigid rules for each natural-language relation but remains susceptible to reasoning mistakes.
- Compared with CSVG: CSVG organizes global symbolic reasoning as constraint satisfaction, so it would be inaccurate to characterize every symbolic approach as lacking global context. The distinction here is a visually expressed global layout complemented by object crops for fine-grained appearance.
- Compared with SeeGround / VLM-Grounder / SPAZER: These methods use local renderings, video frames, or broader scene views. The proposed method instead presents a geometrically abstracted global layout and reserves local imagery for attribute verification. The transferable lesson is to match representations to evidence requirements rather than making one realistic image solve every reasoning problem.
Rating¶
- Novelty: 4/5. Dual-granularity input and query adaptation form a clear methodological contribution centered on representation and workflow rather than a new foundation model.
- Experimental Thoroughness: 4/5. Two benchmarks, GT-class settings, component ablations, and efficiency comparisons are covered, but independent component attribution and statistical stability remain incomplete.
- Writing Quality: 4/5. Motivation, illustrations, and reasoning flow align well, although several implementation parameters and real-world details depend on supplementary material.
- Value: 4/5. The framework offers a practical way to turn 3D tasks into VLM-readable evidence, with deployment value still constrained by perception quality and model-call costs.