Holo-Captioning: A Comprehensive Textual View of 3D Scenes

Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://visual-ai.github.io/holocap/
Area: 3D Vision
Keywords: 3D Scene Captioning, Holo-Captioning, 3D Vision-Language Model, Instance Decoupling, Relation Modeling

TL;DR

This paper introduces Holo-Captioning, a novel task striving for the textual equivalent of 3D scenes, and develops HoloScribe, a decoupled 3D-LLM that jointly predicts semantic categories, 3D bounding boxes, fine-grained attributes, and inter-entity relations in pure text without external detectors.

Background & Motivation

Describing the 3D physical world using natural language is a foundational frontier in computer vision and embodied artificial intelligence, serving as an essential building block for 3D scene reconstruction, embodied robot navigation, and spatial manipulation. However, traditional 3D dense captioning approaches remain severely restricted to narrow closed vocabularies (e.g., only 18 object categories in ScanRefer) and primarily produce brief, coarse phrases that fail to convey detailed object appearances and spatial-physical configurations. Crucially, these earlier paradigms heavily depend on pre-trained discrete 3D detectors or segmenters to generate candidate proposals, fragmenting the intrinsic spatial continuity between 3D geometric structures and natural language representations. While recent 3D large language models (3D-LLMs) have attempted to autoregressively output structured scene layouts, they predominantly concentrate on primary architectural boundaries and salient foreground objects, overlooking fine-grained entity surface attributes and inter-entity spatial interactions.

This oversight prevents 3D scenes from establishing a truly comprehensive, high-fidelity "text equivalent." Existing pipelines attempting to supply missing attributes and spatial relations typically cascade heterogeneous standalone detectors, attribute classifiers, and scene graph networks, which introduces significant cumulative errors, leads to pipeline fragmentation, and struggles to generalize to open-world real environments. Conversely, attempting to make a generalist 3D-LLM predict such lengthy, information-dense text in a single forward pass causes perceptual overload, numeric divergence across long 3D coordinate sequences, and a combinatorial explosion of pairwise entity relations.

The angle of attack in this work is to formulate the text equivalent of 3D scenes as a structured generation task covering four fundamental dimensions, resolving the perceptual bottleneck through an instance-centric decoupled generation pipeline. Core idea: decompose holo-captioning into a three-stage decoupled pipeline comprising grounded instance discovery, anchor-aware instance linking, and grounded description generation, enabling a single 3D-LLM to end-to-end predict comprehensive structured captions—including 3D bounding boxes—without auxiliary detectors.

Method

Overall Architecture

The Holo-Captioning task requires a model to take a raw 3D point cloud as input and directly generate a structured, comprehensive holo-caption in pure text format without any pre-trained 3D object detector. The resulting output spans semantic tags, spatial locations (represented as 7-DoF/9-DoF 3D oriented bounding boxes), free-form fine-grained attributes, and inter-entity physical relations. To prevent attention dispersion and combinatorial explosion during end-to-end multi-entity generation, HoloScribe builds upon an "Encoder-MLP-LLM" architecture (utilizing Sonata as the point cloud encoder and Qwen as the autoregressive language decoder) and organizes generation across three decoupled instance-centric phases.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input 3D Point Cloud Qv"] --> B["Grounded Instance Discovery<br/>Prompt-driven structured 3D box and tag prediction"]
    B --> C["Anchor-aware Instance Linking<br/>Sampled subgraphs around anchor entities to prune candidate pairs"]
    C --> D["Grounded Description Generation<br/>Conditioned decoding of attributes and bidirectional relations"]
    D --> E["Output Structured Holo-Caption"]

Key Designs

1. Grounded instance discovery: predicting continuous 3D spatial coordinates directly in pure text Conventional 3D vision-language systems rely on auxiliary 3D proposal networks like PointGroup or VoteNet, bypassing the language model's native capacity for spatial reasoning. HoloScribe discards external detectors entirely, formalizing each 3D entity directly as a text token sequence bbox_i=Bbox(t_i, c_{i,x}, c_{i,y}, c_{i,z}, e_{i,x}, e_{i,y}, e_{i,z}, r_{i,z}), where \(t_i\) denotes open-vocabulary category tags, and the 3D box center, extent, and yaw angle are represented as textual coordinates. To handle dense indoor scenes where generating all entities at once risks decoder truncation and omissions, HoloScribe introduces a category-tiered query mechanism. Entities are categorized into architectural structures (walls, floors, ceilings), common furniture, and smaller objects. The model generates each tier independently via tailored prompts before merging and deduplicating the results, substantially improving instance recall and numerical stability in complex indoor scans.

2. Anchor-aware instance linking: mitigating the Cartesian combinatorial explosion of entity pairs Exhaustively enumerating all pairwise relations among \(N\) detected instances incurs an \(O(N^2)\) candidate space, imposing prohibitive autoregressive decoding costs while introducing numerous ungrounded negative pairs. HoloScribe resolves this challenge via an anchor-aware subgraph sampling strategy. For each detected instance designated as an anchor, the model samples \(n_g - 1\) spatial neighbor candidates to form a compact local subgraph. This serialized subgraph text is fed to the LLM, which outputs a concise linking token sequence such as bbox_1, bbox_2, bbox_3 (constrained to begin with the anchor bbox_1 to ensure stable decoding). This approach restricts the relational search space to localized star subgraphs, effectively filtering out non-interacting entity pairs and boosting relational precision.

3. Grounded description generation: instance-conditioned dual semantic completion To eliminate context crosstalk and attribute confusion when synthesizing long narrative paragraphs, this phase conditions the language decoder directly on the identified spatial boxes and relational links. For the attribute dimension, the prompt specifies the 3D location and category tag of an individual target entity, guiding the LLM to focus exclusively on that geometric region to detail its color, geometric shape, material texture, and constituent parts. For the relation dimension, the prompt supplies the paired 3D bounding boxes and tags, prompting the model to produce bidirectional descriptions (e.g., "the bed supports the helmet" alongside "the helmet is placed on the bed"). This conditioned generation converts global scene complexity into manageable, localized perceptual subtasks, ensuring rich descriptive detail while preventing hallucinated attribute bleed.

4. Viewpoint-aware aggregation engine and hierarchical multi-dimensional evaluation: establishing the data and benchmarking closed loop To train and rigorously assess holo-captioning, the authors develop HoloEngine and the HoloScore evaluation protocol. HoloEngine projects annotated 3D bounding boxes onto multi-view images, prompts vision-language models with highlighted bounding boxes to produce rotation-invariant entity attributes and paired relation texts, and uses an LLM to consolidate multi-view descriptions into holistic captions across 15K scenes in the HoloScan benchmark. For evaluation, traditional n-gram metrics (BLEU, CIDEr) fail on long structured documents. HoloScore addresses this by computing an optimal bipartite matching over grounded instances:

\[ \Theta = \arg\max_\Theta \sum_{i=1}^n \sum_{j=1}^m \left(\alpha \cdot \text{sim}_{i,j} \cdot \text{IoU}(l_i, \hat{l}_j)\right) \cdot \Theta_{i,j} \]

where \(\alpha=10\) prioritizes semantic category alignment. Following grounded matching, an LLM decomposes lengthy attribute descriptions into atomic descriptors to perform dual descriptor embedding matching. Finally, HoloScore aggregates the F-scores across semantic tagging (\(s_t\)), spatial localization (\(s_l\)), entity attributes (\(s_a\)), and inter-entity relations (\(s_r\)) into an overall metric \(s = s_t + s_l + s_a + s_r\).

Loss & Training

HoloScribe is initialized with pre-trained SpatialLM1.1-Qwen-0.5B weights, using Sonata as the 3D point cloud encoder. The model is trained using the standard autoregressive language modeling cross-entropy objective:

\[ \mathcal{L} = -\sum_{t=1}^{T} \log P(w_t \mid w_{<t}, Q_v, \text{Prompt}) \]

Fine-tuning is performed on the HoloScan training set for one epoch using LoRA. During training, 3D bounding boxes are constrained from 9-DoF to 7-DoF (fixing pitch and roll angles \(r_x = r_y = 0\)), reducing the learning complexity of continuous 3D coordinate regression. During inference, predicted instances and links are parsed sequentially to synthesize complete holo-captions.

Key Experimental Results

Main Results

On the expert-curated HoloScan test set (comprising 83 rigorously verified real indoor scenes), HoloScribe is evaluated against prominent 3D dense captioning models and generalist 3D-LLMs across all four dimensions of HoloScore:

Model	LLM Backbone	Semantic Tag (\(s_t\))	Localization (\(s_l\))	Attribute (\(s_a\))	Relation (\(s_r\))	Overall HoloScore
Vote2Cap-DETR	-	32.50	17.71	2.70	-	-
Vote2Cap-DETR++	-	31.39	17.04	2.67	-	-
LEO	Vicuna-7B	25.91	14.93	1.60	-	-
LL3DA	OPT-1.3B	31.28	16.89	3.27	-	-
SpatialLM-Tuned	Qwen2.5-0.5B	49.82	9.23	7.01	3.57	69.63
LL3DA-Tuned	Qwen2.5-0.5B	31.28	16.88	9.54	3.97	61.67
HoloScribe (Ours)	Qwen2.5-0.5B	61.83	22.75	23.72	8.74	117.04

Note: Dash ("-") indicates that the baseline architecture cannot output open-vocabulary relation descriptions.

Ablation Study

On the validation set of HoloScan, the authors conduct an ablation study isolating the decoupled pipeline components and the anchor-aware linking mechanism:

Config	Semantic Tag (\(s_t\))	Localization (\(s_l\))	Attribute (\(s_a\))	Relation (\(s_r\))	Overall HoloScore	Note
Direct fine-tuning Baseline	46.98	8.23	2.39	1.04	58.64	Single-stage joint generation
+ Relation decoupling	51.73	9.94	6.74	2.60	71.01	Isolates relation generation
+ Instance-aware prediction	59.20	20.45	22.41	7.92	109.98	Conditions on detected bounding boxes
Full w/o anchor mechanism	58.67	17.94	22.46	9.20	108.27	Dense all-pair subgraph modeling
Full Model (HoloScribe)	60.43	22.33	22.52	11.08	116.36	Incorporates anchor-aware subgraph linking

Key Findings

• Decoupled formulation is decisive: Direct end-to-end tuning of SpatialLM achieves only 58.64 overall score due to attention dilution over long structured outputs. Introducing instance-aware conditional prediction yields an immediate leap to 109.98, with the attribute score surging from 6.74 to 22.41, demonstrating the necessity of dividing complex scene comprehension into instance-centric subtasks.
• Anchor-aware linking suppresses false relational edges: Removing the anchor mechanism leads to spurious pair identification in dense candidate subgraphs, dropping the relation score to 9.20. The anchor-centric approach raises relation modeling to 11.08, while reinforcing spatial localization consistency (from 17.94 to 22.33).
• Strong zero-shot generalization to unseen real scenes: When evaluated on MultiScan without any task fine-tuning, HoloScribe achieves a tagging F-score of 53.5 and a localization F-score of 20.2, substantially outperforming LL3DA (24.9 / 17.3) and SpatialLM (20.2 / 9.6). In downstream mobile robotic object navigation, HoloScribe attains a 75% goal success rate compared to 25% for SpatialLM.

Highlights & Insights

• Pioneering 3D scene text equivalence: Transcends traditional 3D dense captioning by unifying open-vocabulary semantic tags, metric 3D oriented bounding boxes, granular surface physical attributes, and bidirectional topological relations.
• Detector-free end-to-end 3D localization: Autoregressively generates exact metric 3D bounding box coordinates entirely in pure text, avoiding cascading errors from pre-trained 3D detectors or multi-stage modular pipelines.
• Anchor-centric subgraph pruning: Transforms an intractable \(O(N^2)\) global matching space into localized star graph classifications, offering an effective paradigm for scene-graph parsing in dense 3D visual environments.

Limitations & Future Work

• Variance in continuous coordinate regression: Pure text tokenization can struggle to match the high precision of specialized geometric regression heads in heavily cluttered or occluded environments, leaving headroom in 3D bounding box IoU (\(s_l \approx 22.75\)).
• Absence of fine-grained geometric texture in point clouds: Sparse point cloud inputs lack photorealistic texture and optical reflectance cues, requiring projection from 2D images for certain material details. Future iterations could explore RGB point cloud fusion or 3D Gaussian Splatting backbones.
• Static scene assumption: Current formulations target static indoor environments and do not yet model dynamic scene motion, articulated object kinematics, or temporal human-object interactions.

Related Work & Insights

• vs Scan2Cap / Vote2Cap-DETR++: Prior 3D dense captioners depend on specialized 3D proposal backbones, support only ~18 categories, and generate brief sentences (~15 words); Holo-Captioning covers 700+ categories and ~90-word comprehensive descriptions generated purely end-to-end via an autoregressive 3D-LLM.
• vs SceneScript / SpatialLM: Earlier structured 3D language models focus strictly on architectural room layouts and coarse bounding boxes using synthetic data; Holo-Captioning enriches descriptions with fine-grained attributes and relational graphs trained on over 15K real and synthetic scenes.
• vs 3D Scene Graphs (3DSG): 3DSG formulations restrict relations to discrete predicate triplets and omit natural language descriptive attributes; Holo-Captioning provides flexible, free-form linguistic characterizations of physical attributes and topological interactions directly consumable by downstream reasoning agents.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Formulates the holo-captioning paradigm for 3D visual-text equivalence with a detector-free 3D-LLM.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Curates the 15K-scene HoloScan benchmark, introduces the HoloScore metric, and validates on 3D reconstruction and embodied robotics.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear logical motivation, comprehensive mathematical formalization, and clean architectural presentation.
• Value: ⭐⭐⭐⭐⭐ Establishes a concrete path toward holistic 3D scene textualization for embodied agents and spatial computing.

Related Papers

• [ECCV 2024] View Selection for 3D Captioning via Diffusion Ranking
• [ECCV 2026] Argus: Metric Panoramic 3D Reconstruction for Indoor Scenes
• [CVPR 2026] 3D-LATTE: Latent Space 3D Editing from Textual Instructions
• [ECCV 2026] AnyView: Synthesizing Any Novel View in Dynamic Scenes
• [ECCV 2026] AVSplat:Dense-View Feed-Forward 3D Gaussian Splatting with Assist-View Preconditioning