3D-Aware VLMs with Implicit and Explicit Geometries¶
Conference: ECCV2026
Official Paper: 3224
Paper: PDF
Code: https://github.com/Vegetebird/VLM-IE3D
Area: Multimodal VLM
Keywords: 3D scene understanding, implicit geometry, explicit geometry, cross-attention, spatial reasoning
TL;DR¶
VLM-IE3D injects both implicit geometric features and explicit tokens derived from reconstructed depth into Qwen2.5-VL, raising 3D video detection [email protected] from the backbone's 30.9 to 42.8, although its average spatial reasoning gain over an existing geometry-enhanced model is modest.
Background & Motivation¶
Recognizing an object in an image does not guarantee that a VLM can accurately report its 3D location, size, or relative distance. Appearance alone does not directly establish the relationship between camera motion and spatial structure. Methods such as Video-3D LLM supply depth or point clouds to address this problem, but require additional 3D data. RGB-only approaches such as VG LLM instead obtain geometric priors from a pretrained geometry encoder operating on video.
The issue is not that implicit features contain no geometry. Rather, their compressed, abstract form may make quantitative details difficult for the language model to access. Knowing that a sofa stands in front of a television is different from predicting the sofa's precise boundaries and dimensions. Meanwhile, geometry encoders already have prediction heads for depth, poses, and other attributes. Using their hidden features while discarding reconstruction outputs may overlook a useful source of local spatial measurements.
The paper therefore reuses both types of output from the same geometry model: hidden representations for global structural priors and explicit predictions for local detail. Core idea: complement rather than replace implicit geometry by lightly embedding reconstructed measurements, letting implicit tokens query those explicit features, and then fusing the result with 2D semantics.
Method¶
Overall Architecture¶
The input is an RGB video and a natural-language question. Depending on the task, the output is a set of 3D boxes, the first frame containing a target and its location, an object caption, or an answer to a spatial question. The Qwen2.5-VL visual encoder extracts appearance features; AnySplat supplies Implicit Geometry Tokens (IGTs) and reconstructed attributes that are encoded as Explicit Geometry Tokens (EGTs). A 3D-aware adapter combines the geometric streams, adds the 2D features, and passes the unified visual representation to the VLM.
Here, "explicit" describes the physically meaningful source of the tokens, such as depth. It does not mean that depth matrices are written into a text prompt or that inference requires a depth sensor. The default pipeline reconstructs its geometry from RGB. This input claim must still be distinguished from 3D task annotations and proposal assistance in the evaluation protocols.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["RGB video"] --> B["2D visual encoder"]
A --> C["AnySplat"]
C --> D["Implicit Geometry Tokens"]
C --> E["Reconstructed depth<br/>or other attributes"]
E --> F["Explicit Geometry Tokens"]
B --> G["3D-Aware Adapter<br/>Compression, IEA, 2D fusion"]
D --> G
F --> G
G --> H["VLM + language query<br/>3D prediction or text answer"]
Key Designs¶
1. Implicit Geometry Tokens: use cross-frame reconstruction features for global structure
AnySplat processes images through a per-frame encoder, a cross-frame fusion decoder, and task-specific prediction heads. Its fusion decoder alternates frame-wise and global self-attention to connect observations across views. Instead of treating the final reconstruction as the only geometric information source, VLM-IE3D takes the fusion decoder output as IGTs, retaining structural priors learned through multi-view geometry.
These tokens are useful for overall layouts and object relationships, reducing reliance on appearance-based guesses from individual frames. They are not directly readable coordinate tables: even when a distance is encoded in a latent vector, the language model may not extract it reliably. IGTs therefore form the main geometric foundation, with the explicit branch supplying additional detail. The argument is not that implicit features are inherently unable to represent local geometry.
2. Explicit Geometry Tokens: preserve quantitative attributes with lightweight embedding
AnySplat's prediction heads produce depth, camera poses, and 3D Gaussian attributes. Reconstructed depth can also be back-projected using reconstructed poses to form point maps. The experiments separately consider depth maps, point maps, and Gaussian attributes, with 1, 3, and 86 channels per pixel, respectively. Depth is the default because it has a compact representation and performs at least comparably to the more elaborate alternatives.
The explicit embedding module consists of a single patch-embedding layer followed by a two-layer MLP, mapping dense attributes into features aligned with the visual tokens. The model does not default to passing depth through another large semantic encoder. IGTs already provide high-level abstraction; aggressively abstracting EGTs as well could obscure the local measurements that this branch is supposed to contribute. Their complementarity comes from the attributes and encoding design, not from EGTs remaining untransformed numerical depth values.
The explicit embedding itself introduces only 0.008B parameters. Replacing it with a DepthAnything V2 deep encoder applied to depth maps substantially reduces performance. The authors attribute this to over-processing and redundancy with IGTs, but the experiments do not separately rule out optimization difficulties. This explanation should be treated as a hypothesis rather than a demonstrated causal mechanism.
3. 3D-Aware Adapter: query explicit detail with implicit tokens, then retain 2D semantics
Each token stream is first spatially compressed. Following Qwen2.5-VL, neighboring \(2\times2\) features are concatenated and processed by a two-layer MLP. Images have resolution \(392\times518\) with patch size 14, giving 1036 tokens per frame before compression and 252 afterward, with channel dimension 2048. Matching spatial layouts make interaction and element-wise fusion possible without concatenating all three streams into a longer language-model input sequence.
The Implicit-Explicit Attention module, or IEA, uses compressed IGTs as queries and compressed EGTs as both keys and values for each frame. Multi-head cross-attention retrieves explicit geometric detail, with a residual connection. Intuitively, a feature describing how the scene is organized reads additional evidence about its local depth structure. Cross-frame relationships have already been modeled by AnySplat; the adapter text describes frame-wise geometric interaction, not another full global video-attention stage.
The fused 3D tokens are then added element-wise to the compressed 2D visual tokens. This retains appearance cues needed for object identification while enriching the representation with spatial structure. Two distinct fusion sites matter: IEA combines IGTs and EGTs, whereas non-parametric addition combines 3D and 2D tokens. The method does not apply pairwise cross-attention among every branch.
Equations (1)-(3) are visibly corrupted in the cached text, so their algebraic forms are not reconstructed here. The query/key/value assignments, residual connection, and final addition are explicitly described in the surrounding intact prose; the cache does not support a reliable transcription of the precise residual equation.
A Worked Example¶
Consider one sample under the 3D video detection protocol: 4 consecutive frames sampled at 1 FPS, with the task of predicting boxes for visible objects. This is an illustration of the evaluation workflow, not an additional quantitative result.
The 2D branch captures the appearance of objects such as chairs. AnySplat uses the views to produce IGTs and reconstruct per-frame depth. The depth embedding supplies EGTs, and each stream is compressed from 1036 to 252 tokens per frame. IEA lets implicit features read depth detail before 2D semantics are added back and the VLM generates 3D boxes. The model thus receives both identification cues and geometry that can constrain box size and position.
Most tasks use the first frame as the reference coordinate system. Visual grounding is an exception: boxes are expressed in each frame's local camera coordinates. If a question asks for the first frame containing a target, the model must also predict its temporal location; detection and grounding therefore have different output protocols.
Loss & Training¶
The backbone is Qwen2.5-VL-3B and the geometry encoder is AnySplat. Training freezes the 2D visual encoder and geometry encoder while updating the VLM backbone and explicit embedding. The adapter is a newly introduced fusion module, but the implementation paragraph does not separately specify its freezing status. The cache does not define a training-loss equation or loss weights, so no assumed geometric reconstruction objective is added here.
Training uses Adam for 1 epoch, a warmup ratio of 0.03, and a learning rate that rises to \(10^{-5}\) before decaying to 0. Batch size is 1 per GPU on 8 H100 80G GPUs, and maximum video length is 32 frames. Freezing the geometry encoder means the training primarily teaches the VLM to use existing geometric outputs rather than jointly learning reconstruction from scratch.
The 3D scene-understanding model is trained on a mixture of Scan2Cap, ScanRefer, and 3D video detection tasks. Spatial reasoning uses a separately trained model with 234K SPAR-7M samples (3%) and 63K samples from the LLaVA-Hound split of LLaVA-Video-178K (25%). The results across all four tasks should not be described as evaluations of one identical checkpoint.
Key Experimental Results¶
Main Results¶
The table selects the principal RGB-input comparisons from paper Tables 1-4. For Scan2Cap, C/M denote CIDEr/METEOR and @0.5 specifies the 3D-box IoU evaluation threshold. ScanRefer Acc is grounding accuracy at the indicated IoU threshold. For 3D video detection, P25, R25, and F125 denote precision, recall, and F1 at IoU=0.25. VSI-Bench Avg. is the reported aggregate across numerical and multiple-choice tasks, not a single classification accuracy.
| Task and metric | Qwen2.5-VL-3B | VG LLM | VLM-IE3D |
|---|---|---|---|
| Scan2Cap [email protected] | 58.0 | 78.6 | 80.4 |
| Scan2Cap [email protected] | 26.9 | 28.6 | 28.8 |
| ScanRefer [email protected], raw output | 34.0 | 36.4 | 43.2 |
| ScanRefer [email protected], raw output | 10.6 | 11.8 | 16.9 |
| ScanRefer [email protected], proposal refinement | 50.7 | 53.5 | 55.4 |
| ScanRefer [email protected], proposal refinement | 44.7 | 47.5 | 48.9 |
| 3D video detection P25 / R25 / F125 | 32.1 / 30.1 / 30.9 | 41.7 / 35.7 / 38.2 | 44.2 / 41.9 / 42.8 |
| VSI-Bench Avg. | 30.6 | 47.3 | 47.6 |
The VSI-Bench table labels the geometry-enhanced models VG LLM-4B and VLM-IE3D-4B. Separately, Table 3 reports trainable parameter counts of 3.09B, 3.13B, and 3.23B for 3D detection; these should not be conflated with exact total model sizes. Speeds on a single H100 are 14, 7, and 6 FPS, respectively.
Scan2Cap uses Mask3D proposals provided by LEO and generates captions conditioned on object centers. ScanRefer refinement matches predicted boxes to pre-detected proposals. Thus, RGB-only describes the core model input, not proof that the entire evaluation pipeline from proposal generation onward avoids external 3D information. The detection data is derived from EmbodiedScan, with 958 training scenes and 243 evaluation scenes, 150 and 10 samples per scene respectively, and 20 evaluated categories.
Ablation Study¶
The following results come from paper Tables 5-8 and all use 3D video detection F125. The default full model scores 42.8. "IGTs only" in Table 5 is the authors' controlled ablation, not the VG LLM comparison with 38.2 in the main results.
| Ablation group | Configuration | F125 | Interpretation |
|---|---|---|---|
| Geometry source | Backbone without geometry | 30.9 | Starting point |
| Geometry source | EGTs only | 34.7 | Explicit geometry alone |
| Geometry source | IGTs only | 40.5 | Implicit geometry alone |
| Geometry source | IGTs + EGTs | 42.8 | Better than either stream alone |
| Geometry fusion | Concatenation and projection | 41.5 | Non-attention fusion |
| Geometry fusion | Direct addition | 42.4 | Only 0.4 below IEA |
| Geometry fusion | Addition with two learned weights | 41.2 | Not necessarily better than fixed addition |
| Geometry fusion | IEA | 42.8 | Default |
| Explicit attribute | Point / depth / Gaussian | 42.6 / 42.8 / 42.5 | Small differences among attributes |
| Explicit embedding | Average pooling / with sinusoidal positions | 42.0 / 42.5 | Lightweight alternatives remain effective |
| Explicit embedding | DepthAnything V2 deep encoder | 35.9 | Worse than IGTs alone |
Key Findings¶
- Implicit geometry supplies the foundation: IGTs alone improve over the backbone by 9.6 points, and adding EGTs contributes a further 2.3. This supports complementarity, not the claim that explicit geometry is superior to implicit geometry.
- IEA improves on direct addition by only 0.4 points, so attention cannot explain the entire gain. Exposing both types of geometry is the more consequential design decision.
- In Table 9, the full framework with ฯ3, VGGT, and AnySplat scores 42.1, 41.7, and 42.8, respectively. Combining DepthAnything V2 reconstructed depth with AnySplat IGTs scores 42.5. Changing the source of depth and using a deep encoder to re-encode depth are different experiments.
- VSI-Bench average improves over VG LLM by just 0.3. Relative distance rises from 44.6 to 47.7, while appearance order falls from 36.4 to 31.9. Better local geometry does not produce uniform gains across reasoning subtasks.
Highlights & Insights¶
- A reconstruction model's hidden features and prediction outputs can both serve as interfaces to a VLM. One retains learned structure, while the other exposes physically meaningful attributes without requiring another large 3D network for the explicit branch.
- Preserving complementary information can matter more than encoder depth. The lightweight embedding's advantage suggests checking whether an encoder abstracts away the measurements needed when fusing geometry, pose, or motion attributes.
- Geometry-to-geometry interaction and semantic-geometric fusion need not have equal complexity. IEA for the former and simple addition for the latter provide a useful starting point for multi-branch designs.
Limitations & Future Work¶
- The authors do not provide a dedicated limitations section. The following points are primarily assessment of the experimental scope, not failure mechanisms already established by the authors.
- There is no systematic robustness analysis for reconstruction errors, textureless regions, occlusion, or changing camera motion. Since explicit geometry is estimated rather than ground truth, controlled noise and confidence-weighted fusion could test whether the adapter amplifies incorrect geometry.
- Precise grounding still depends substantially on proposal quality: [email protected] rises from a raw 16.9 to a refined 48.9. Fine localization remains unresolved, and complete pipelines with and without proposals should be evaluated separately.
- Scene-understanding experiments concentrate on indoor data, reasoning gains are modest and uneven, and the cache reports no repeated-run variance. The results do not establish superiority over every 3D-input method; Video-3D LLM still reports 58.1 / 51.7 on the ScanRefer comparison with proposal-refined methods.
- The additional branch is relatively lightweight compared with VG LLM, but speed falls from the backbone's 14 to 6 FPS and training uses 8 H100 80G GPUs. A small relative increment is not evidence of low-resource real-time deployment.
Related Work & Insights¶
- Compared with VG LLM: both use video geometry priors, while VLM-IE3D adds explicit encoding of reconstructed attributes and IEA. The main comparison measures system-level differences; the controlled IGT/EGT ablation more directly tests complementarity.
- Compared with Video-3D LLM: that method injects 3D positions obtained by depth back-projection and requires additional 3D inputs. VLM-IE3D reconstructs its own attributes from RGB, although external proposal assistance must still be disclosed for the relevant evaluations.
- Connection to AnySplat, VGGT, and ฯ3: these models can supply both geometric features and explicit attributes. Cross-encoder experiments support the interface's portability, but do not imply that arbitrary reconstruction backbones can be swapped without adaptation.
Rating¶
- Novelty: 4/5. The complementary roles of explicit and implicit geometry are clear; the main contribution is the representation interface and fusion rather than a new reconstruction algorithm.
- Experimental Thoroughness: 4/5. Multiple tasks and ablations are covered, but error robustness, variance, and fully proposal-independent evaluation remain limited.
- Writing Quality: 4/5. The argument and ablations are well organized, although readers must distinguish the evaluation boundaries of the RGB-only claim.
- Value: 4/5. A reusable extension for geometry-enhanced VLMs with clear detection gains, while deployment and broader reasoning capabilities require further validation.