Open-Vocabulary BEV Segmentation with 3D-Aware Geometric Constraints¶

Conference: ECCV 2026
arXiv: 2606.24353
Code: To be confirmed
Area: Autonomous Driving
Keywords: Bird's-Eye-View Segmentation, Open-Vocabulary, 3D Gaussian Splatting, Geometric Consistency, Knowledge Distillation

TL;DR¶

This work proposes the first open-vocabulary BEV segmentation framework, OVBEVSeg. By reversing the geometric reasoning path (establishing 2D-BEV correspondence with unsupervised 3D detection \(\rightarrow\) optimizing geometry via BEV-aware 3DGS \(\rightarrow\) distilling to a feed-forward network), it lifts VLM semantics from 2D to the BEV space, outperforming closed-set methods by 15.3 mIoU on novel classes in nuScenes with faster inference speed.

Background & Motivation¶

Bird's-eye-view (BEV) perception has become the core paradigm of autonomous driving, fusing multi-camera images into a unified ego-centric top-down view representation to support downstream tasks such as 3D object detection, semantic segmentation, and trajectory planning. However, a long-ignored fundamental assumption is that existing BEV perception pipelines are closed-set systems—both training and inference assume the same finite set of semantic classes. In real driving scenarios, vehicles continuously encounter objects never seen during training (e.g., trucks, strollers, wheelchairs). Although these novel classes exist in sensor data, they are silently ignored by the models, creating blind spots on BEV semantic maps. This is not only a technical deficiency but also directly threatens the safety of downstream tasks like path planning and collision avoidance.

An intuitive approach is to graft 2D open-vocabulary detectors onto the BEV pipeline: running an OV detector for each camera, lifting 2D regions into 3D space via depth estimation, and then projecting them to the BEV. However, this "2D-OV-then-BEV" scheme suffers from a structural issue—the 2D-to-3D lifting process itself is ill-posed. Small 2D localization errors or semantic misclassifications are severely amplified during the lifting stage, especially in autonomous driving settings with large open scenes and sparse views. Even state-of-the-art Gaussian Splatting methods are not immune, as they directly lift 2D features into 3D Gaussian primitives and are similarly plagued by lifting errors.

The key insight of this paper is that the direction of geometric reasoning must be reversed: rather than "lifting" noisy 2D predictions into 3D, it is better to "project" reliable 3D structures back into 2D and BEV spaces. Based on this projection-first principle, the authors propose the OVBEVSeg framework, which progressively injects 3D geometric constraints into the BEV perception pipeline through three sequential stages (semantic pseudo-label generation \(\rightarrow\) scene-level geometric optimization \(\rightarrow\) efficient knowledge distillation). Core Idea: Utilize unsupervised 3D detection to establish stable 2D-BEV correspondences, transfer 2D VLM semantics to the BEV space through reliable 3D projections instead of ill-posed 2D unprojection, and finally achieve high-efficiency open-vocabulary segmentation via BEV-aware 3DGS optimization and knowledge distillation.

Method¶

Overall Architecture¶

OVBEVSeg uses the "projection-first" principle as a unifying guideline, building a three-stage hybrid 3DGS framework. The first stage, PBL, serves as a semantic bridge: it uses an unsupervised 3D detector, UNION, to extract class-agnostic 3D candidate boxes from LiDAR point clouds, projects each box onto 2D images, utilizes SAM to obtain precise instance masks, and extracts CLIP embeddings to calculate similarity with text templates to generate pseudo-labels. Finally, the labeled 3D boxes are projected onto the BEV plane to generate BEV-level pseudo-labels—thereby bypassing the depth ambiguity of 2D unprojection at its source. The second stage, BAGS, is responsible for geometric refinement: taking the BEV structural layout (occupancy map) obtained in the first stage as a strong geometric prior, it simultaneously constrains 2D image rendering and BEV occupancy consistency during per-scene 3DGS optimization to recover high-fidelity 3D geometry from sparse views. The third stage, BAGD, distills the optimized geometric knowledge back into a feed-forward student 3DGS network, allowing the student to predict Gaussian primitives and rasterize BEV features in a single forward pass, meeting real-time inference demands.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-camera Images <br/>+ LiDAR Point Cloud"] --> B["PBL: Pseudo-BEV Labels<br/>3D Detection → 2D Segmentation → CLIP Classification → BEV Projection"]
    B --> C["BAGS: BEV-Aware 3DGS<br/>per-scene Optimization<br/>Joint 2D + Depth + BEV Occupancy Supervision"]
    C --> D["BAGD: BEV-Aware Gaussian Distillation<br/>top-1 Gaussian Correspondence →<br/>Symmetric KL Distillation to Student"]
    D --> E["Online Inference:<br/>Feed-forward 3DGS + OV Segmentation Head<br/>Real-time BEV Semantic Map"]

Key Designs¶

1. PBL: Replacing 2D Lifting with 3D Projection to Establish 2D-BEV Semantic Correspondence

For open-vocabulary BEV semantic segmentation, the most critical bottleneck is how to stably map the semantic clustering results of 2D VLMs to the BEV space without introducing the geometric errors of ill-posed lifting. The fatal weakness of existing schemes (performing 2D OV detection first, then lifting based on depth) is unreliable depth estimation—the sparse, large-baseline camera views in autonomous driving scenarios lead to multi-modal ambiguity in pixel-wise depth distributions. Subtle errors amplified by unprojection cause object boundaries to collapse on the BEV grid. The design of PBL does the opposite: it first runs an unsupervised 3D detector, UNION (which uses spatiotemporal clustering to obtain class-agnostic 3D candidate boxes from point clouds without requiring any semantic annotations), projects each 3D box onto multi-view camera images to obtain 2D projection regions, and then leverages SAM to extract precise pixel-level instance masks within these regions. Since 3D boxes are already stable geometric entities in metric space, the projected 2D regions naturally possess cross-view consistency. Subsequently, CLIP is utilized to extract background-masked, cropped image embeddings for each instance mask. The cosine similarity between these embeddings and a text template set containing both base and novel classes is calculated, taking the maximum as the pseudo-label. Finally, the labeled 3D boxes are projected onto the BEV grid to generate BEV pseudo-labels. To suppress false positives from the 3D detector, the authors design a filtering strategy: if the 2D mask area deviates significantly from the projected 2D box area, it indicates inaccurate 3D box scale, and it is discarded. The core advantage of this workflow is that the semantic propagation path always uses 3D geometric entities as intermediates and no longer relies on error-prone pixel-wise depth distributions, thereby resolving the 2D-BEV semantic correspondence inconsistency at the source.

2. BAGS: Constraining 3D Gaussian Geometric Structure with BEV Occupancy Supervision

Standard 3DGS in autonomous driving scenarios has a fatal but rarely discussed issue: optimization driven by photometric loss faithfully reconstructs views from near-field cameras, but on the BEV plane, Gaussian primitives scatter into non-object regions, leading to geometric collapse. This is because sparse cameras have limited overlapping regions, and 2D image signals alone cannot constrain the accurate xy-plane positions of Gaussian primitives. The core idea of BAGS is to introduce the BEV structural layout (occupancy map) as a new supervision signal into 3DGS optimization. Specifically, for each 3D Gaussian primitive, a differentiable rasterization pipeline is used to simultaneously render a 2D occupancy map (pixel-level density) and a BEV occupancy map (top-view density), which are then supervised by the instance masks and BEV pseudo-labels from the PBL stage using a smooth L1 loss:

\[\mathcal{L}_{\text{occ}} = \left\|O^{\text{img}}(v) - i(v)\right\|_{\text{SL1}} + \left\|O^{\text{bev}}(v) - b(v)\right\|_{\text{SL1}}\]

In addition, dense depth supervision is introduced: LiDAR data is used to calibrate the output of the depth estimator ZoeDepth, which serves as a dense depth prior to compute L1 loss with the depth rendered by 3DGS. The final total loss is a weighted sum of color loss (original 3DGS), SSIM loss, depth loss, and occupancy loss. After joint optimization, the Gaussian primitives are constrained near the physical boundaries of objects, effectively suppressing floating artifacts on the BEV plane. Notably, occupancy supervision is color-agnostic—it only cares about whether "there is an object" geometrically, regardless of surface appearance, rendering it particularly suitable for BEV representations where texture is irrelevant.

3. BAGD: Symmetric Knowledge Distillation Based on Dominant Gaussians

The offline-optimized BAGS teacher contains extremely high-fidelity geometric information, but the computational cost of per-scene optimization is too high to be directly used for online perception. The key challenge BAGD needs to address is that the teacher's Gaussian primitives form an unstructured, unordered dense set, while the student's feed-forward network outputs a neat, gridded feature tensor—there is no one-to-one correspondence between them. The authors propose an ingenious solution: for each pixel, find the Gaussian that contributes the most to the rendering weight along that ray (the top-1 dominant Gaussian) and use the properties of this "most influential" Gaussian as the supervision target for that pixel. This assumption is reasonable in autonomous driving BEV scenarios since objects rarely stack in the vertical direction. Consequently, an H×W image corresponds to H×W teacher Gaussian attributes, perfectly aligning with the student grid. Distillation employs a symmetric KL divergence loss:

\[\mathcal{L}_{\text{BAGD}} = \frac{1}{2|\mathcal{V}|}\sum_{v \in \mathcal{V}}\left[ \text{KL}(\hat{G}(v)\|G^*(v)) + \text{KL}(G^*(v)\|\hat{G}(v)) \right]\]

The symmetric formulation ensures that distribution alignments in both directions are constrained, which is more stable than one-way KL. Experiments show that the top-1 configuration is sufficient, and increasing K brings diminishing returns because the dominant Gaussian already captures the core geometry and semantics of the target scene.

Loss & Training¶

The overall training is divided into two phases: offline (per-scene optimization) and online (feed-forward network training). The loss for offline BAGS is a weighted sum of four components: \(\mathcal{L}_{\text{BAGS}} = (1-\lambda_{\text{ssim}})\mathcal{L}_{\text{color}} + \lambda_{\text{ssim}}\mathcal{L}_{\text{D-SSIM}} + \lambda_{\text{depth}}\mathcal{L}_{\text{depth}} + \lambda_{\text{occ}}\mathcal{L}_{\text{occ}}\), with all weights set to 1.0. In the online stage, the optimization objective of the student network is a weighted sum of the OV segmentation loss (cross-entropy on CLIP cosine similarity logits) and the BAGD distillation loss, with weights of 1.0 and 0.01, respectively. Using the AdamW optimizer, learning rate 3e-4, weight decay 1e-7, batch size 8, 50 epochs, training takes about 6 hours on 2x A100 GPUs.

Key Experimental Results¶

Main Results¶

Comparison of OV BEV segmentation performance on the nuScenes validation set. Ground-truth annotations of novel classes (truck, bus, motorcycle) are strictly excluded during training.

Method	Novel mIoU	Base mIoU	Inference FPS	Memory (MiB)
GaussianLSS (Closed-set Baseline)	0.0	40.0	80.2	33.0
TaDe (Closed-set Baseline)	0.0	42.8	51.4	41.5
OVBEVSeg (Ours)	19.3	41.8	79.6	32.4
Δ vs GaussianLSS	+19.0	+1.8	-0.6	-0.6

On novel classes, OVBEVSeg achieves a breakthrough performance of 19.3 mIoU (while closed-set methods are all 0), while maintaining inference speed (79.6 FPS) and GPU memory consumption highly comparable to purely closed-set methods. Base class performance does not degrade significantly.

Ablation Study¶

Configuration	Novel mIoU	Description
Full Model (PBL+BAGS+BAGD)	19.3	All modules
w/o BAGS (PBL only + direct student training)	9.2	Remove geometric optimization, novel class performance halved
w/o BAGD (PBL + BAGS inference, no distillation)	16.5	No distillation, but per-scene optimization cannot be used online
w/o PBL (BAGS + BAGD only)	5.1	Without pseudo-label framework, degrades to closed-set
LangSplat replacing Lang-BAGS	8.7	Disentanglement rate of baseline language embedding method is only 43.7%

Key Findings¶

BAGS is the core source of performance gain: without the geometric optimization of BEV occupancy constraints, the novel class mIoU drops from 19.3 to 9.2, indicating that PBL pseudo-labels alone are insufficient to train high-quality OV BEV segmentors, and accurate 3D geometric signals are indispensable.
The quality of PBL's pseudo-labels directly affects final performance: in the absence of GT annotations for novel classes, PBL achieves a pseudo-label quality of 13.0 mIoU (compared to GT) on the training set, covering 88.5% of the scenes.
The decoupled distillation of Lang-BAGS achieves a 99.4% hit rate among vehicle subclasses (car vs. truck vs. bus), which is much higher than LangSplat's 43.7%, indicating that directly transferring 2D language embedding methods to autonomous driving scenes leads to semantic entanglement.
OVBEVSeg also extends to open-vocabulary 3D object detection (30.8 NDS / 24.1 mAP), outperforming BEVFormer and GaussianLSS.

Highlights & Insights¶

Generality of the "Projection-First" Principle: Reversing the direction of geometric reasoning from unprojection to projection is an idea that not only works for BEV semantic propagation but can also theoretically generalize to any 2D-to-3D scene understanding task where 3D priors are available.
Ultra-Simple Cross-Modal Propagation Link: Using unsupervised 3D detection (no annotation required) as an intermediate relay establishes 2D-BEV correspondence in a single projection, bypassing the complexity and cumulative error of depth estimation.
Occupancy as Color-Agnostic Geometric Supervision: Imposing occupancy constraints on BEV is both simple and effective—it only cares about "whether there is an object" without caring about color, neatly avoiding the failure of 3DGS color optimization on the BEV plane.
Top-1 Dominant Gaussian Distillation: Using the single Gaussian with the highest rendering contribution to establish the correspondence between continuous 3D Gaussians and discrete grids is concise and effective, avoiding complex Gaussian matching or clustering.

Limitations & Future Work¶

Per-scene optimization remains time-consuming: BAGS requires hundreds of thousands of optimization iterations for each subset of scenes (approx. 4 days on 8 GPUs for the full training set). Although online inference after distillation is fast, its generalization and rapid adaptation to new scenes are limited. Combining it with lightweight 3DGS models like Scaffold-GS or Mini-GS is a natural direction for future work.
Lack of temporal consistency: The current framework only processes single frames (or independent frames) for BEV segmentation, without utilizing 4D spatiotemporal consistency modeling. For dynamic obstacles and long-range tracking, temporal modeling is a critical capability.
Dependence on UNION as an unsupervised detector: PBL relies on the quality of 3D proposal boxes from UNION. In scenarios where UNION is unavailable or performs poorly (e.g., extreme weather, LiDAR degradation), performance may decrease.
Limited class coverage ratio: PBL's pseudo-labels cover 39.4% of GT instances (only 32.2% for trucks and 41.2% for motorcycles among novel classes), leaving a significant number of small objects un-recalled.

vs GaussianLSS: GaussianLSS is the base feed-forward 3DGS framework used in this work but is limited to closed-set scenarios. This work adds the PBL pseudo-label pipeline, BAGS geometric optimization, and BAGD distillation on top of it, imparting open-vocabulary capabilities while maintaining comparable inference speed.
vs TaDe: TaDe is one of the state-of-the-art closed-set BEV segmentors. Taking TaDe as a baseline, this method improves novel class mIoU by 5.7, with negligible loss (or even gains) on base classes (-1.0 to +1.7 depending on the class).
vs LangSplat: LangSplat compresses language embeddings into a low-dimensional latent space and then decodes them, but semantic entanglement occurs among vehicle subclasses in autonomous driving scenarios. The decoupled distillation loss proposed in this work (KL alignment based on class similarity distribution) effectively solves this issue.
vs OpenScene: OpenScene achieves open-vocabulary 3D semantic segmentation through co-embedding of 3D points with text/pixels. This paper, conversely, focuses on the BEV representation, which is an intermediate representation unique to autonomous driving, offering superior efficiency.

Rating¶

Novelty: ⭐⭐⭐⭐ For the first time introducing open-vocabulary segmentation to BEV areas, the proposed "projection-first" principle and the design of "BEV occupancy as 3DGS geometric constraint" are ingenious and effective.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Conducted detailed main experiments, ablations, pseudo-label quality analysis, 3D detection extension, and Lang-BAGS decoupling verification on nuScenes, with rich tables and visualizations, and clear ablation of each module's contribution.
Writing Quality: ⭐⭐⭐⭐ Clear logical chain (three problems \(\rightarrow\) three modules \(\rightarrow\) step-by-step progression), strong motivation, and good correspondence with diagrams. Tables and formula details in the supplementary material are complete.
Value: ⭐⭐⭐⭐⭐ Open-vocabulary BEV segmentation is a necessary capability for autonomous driving safety. This paper proposes the first feasible solution with significant efficacy, which could potential become a new baseline for future BEV perception upon open-sourcing.