BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations

Conference: ECCV 2026
Paper: Official paper page · PDF
Project: BEVLM
Area: Autonomous Driving
Keywords: Bird's-eye view representations, semantic distillation, visual question answering, spatial reasoning, closed-loop safety evaluation

TL;DR

BEVLM first establishes that fused BEV tokens support stronger spatial reasoning in language models, then trains the BEV encoder with frozen-LLM question-answering gradients and detection supervision, raising UniAD's NeuroNCAP score from 2.38 to 3.05 and reducing collision rate from 0.56 to 0.47 without introducing an LLM into driving-time inference.

Background & Motivation

Multimodal language models for driving commonly encode each camera view and time step independently before passing visual tokens to a language model. This inherits the advantages of large-scale image-text pretraining, but splits one driving scene into image fragments with different coordinate systems. The language model must reconstruct where an object lies across views and how far two objects really are from one another. Adding frames also repeats processing of correlated content, so rich semantics do not automatically produce strong cross-view geometric reasoning.

BEVFormer, UniAD, and VAD use a different intermediate representation: a bird's-eye grid that fuses multi-view and temporal information in the ego-vehicle coordinate system. This grid supports spatial relationships and can be shared by detection, prediction, and planning modules, but it is usually trained with geometric labels such as 3D boxes. It receives direct supervision for where a vehicle is, but not necessarily for why a blocked lane requires early evasive action. Conversely, image-text foundation models have broader semantic knowledge without necessarily understanding the geometry between cameras accurately.

Rather than asking a language model to control the vehicle directly, the paper first asks whether BEV is already an effective language-model input, then whether the safety semantics required by that model can remain inside the BEV encoder. Core idea: use a frozen LLM as a differentiable semantic teacher, shape BEV features through driving questions while preserving spatial structure with the original detection task, and return the enriched encoder to a conventional end-to-end driving system.

Method

Overall Architecture

The paper has two complementary parts that should not be conflated. Its representation study holds the driving scenes and questions fixed while comparing the original VLM's image features, UniAD image features before fusion, and BEV features after fusion. Semantic distillation goes further: it updates the BEV encoder through safety-related question answering, then tests whether the resulting representation improves driving.

During distillation, multi-frame, multi-view images enter a BEV encoder to produce a shared grid. One branch compresses and projects this grid into a frozen LLM; another continues object detection. Their supervision jointly updates the encoder. After distillation, the enriched encoder is frozen and the task heads of UniAD or VAD are retrained; the deployed driving pipeline no longer calls the language model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Multi-frame multi-view images"] --> BEV["BEV encoder<br/>Shared spatial grid"]
    BEV --> ALIGN["BEV-language interface<br/>Pooling and MLP projection"]
    DATA["Driving QA and 3D boxes"] --> COORD["Ego-centric question answering<br/>Spatial reference conversion"]
    ALIGN --> JOINT["Frozen teacher and geometric constraints<br/>LLM QA + object detection"]
    COORD --> JOINT
    BEV -->|Detection branch| JOINT
    JOINT -->|Joint gradients update encoder| DISTILL["Semantically enriched BEV encoder"]
    DISTILL -->|Freeze encoder and train task heads| DRIVE["UniAD or VAD<br/>No LLM at deployment"]

Key Designs

1. BEV-language interface: make a unified spatial representation consumable by a language model

The authors use the BEV encoder from UniAD, max-pool the grid to \(50\times50\), obtain 2,500 visual tokens, and map them into the language model's embedding space through a learnable MLP projector. When BEV tokens are used, the original VLM vision encoder is disabled; the method does not concatenate two image-token streams with BEV tokens. The language component comes from InternVL3 or DeepSeek-VL, whose prior cross-modal alignment motivates the choice and whose native image inputs provide useful comparison settings.

The interface must first pass a concrete test: can the language model identify whether an object category exists in a specified direction instead of merely guessing the dataset's most common answer? On DriveLM object-existence questions, the authors compare majority-class priors, a BEV linear probe, rule-based answers from UniAD's detection head, and BEVLM. Average accuracy over 10 categories is 78.2%, 88.7%, and 92.8% for the first three, while InternVL3-8B with BEV reaches 95.3%. This suggests that a simple projection retains useful cues for this task, not that every piece of geometric information survives without loss.

The representation study then connects UniAD features before and after BEV fusion to the same type of language model, reducing the confound that one system simply starts with a much larger backbone. Cross-view Ego3D questions require relating objects in different cameras rather than inspecting a single direction, precisely where a common BEV coordinate system should help. Max pooling is not a new reasoning module; it is a low-cost interface. In the downsampling ablation, average and max pooling both yield 95.3% accuracy with InternVL3-8B, and the authors select the latter based on the language-metric differences.

2. Ego-centric question answering: express textual references in the BEV grid's coordinate system

DriveLM questions can refer to objects using image-plane pixels, whereas the BEV grid is ego-centric. Passing pixel-based references directly to the BEV-language interface would require recovering camera projection relationships as an additional step. The authors therefore convert spatial references: they project ground-truth 3D boxes into the image plane, match objects through IoU, retain matches above a predefined threshold, and rewrite locations as metric offsets relative to the ego vehicle.

The ground-truth boxes support training-data conversion; they are not additional deployment inputs. This step fixes a mismatch between how supervision describes the scene and how the input represents it, rather than handing the language model a separate detection output. The cached main text does not specify the IoU threshold, so a seemingly standard value should not be supplied. Likewise, an unreliable match should not be assigned an invented 3D location.

3. Frozen teacher and geometric constraints: change BEV to meet semantic requirements instead of teaching a new decoder to memorize answers

During distillation, LLM parameters remain frozen, but the loss still backpropagates through the LLM into the projector and BEV encoder. For a question about a safe ego action, cross-entropy on the correct answer demands input features that convey lane blockage, potential hazards, and behavioral consequences. The teacher is not primarily generating text for a student to imitate. Its pretrained semantic computation converts question-answering errors into constraints on the student's input representation.

Freezing distinguishes this approach from a conventional auxiliary task. A language decoder trained from scratch could adapt to the existing BEV through answer templates without forcing the encoder to acquire useful semantics. The authors test a randomly initialized auxiliary Transformer with approximately 13M effectively trainable parameters, matched to BEVLM's projector; its closed-loop performance deteriorates. This supports the importance of pretrained semantics, but does not establish that every internal feature has an interpretable correspondence to a particular safety concept.

Question answering also constrains only information relevant to the answer, not every object location and spatial relationship. To avoid sacrificing geometry for semantics, the method retains the original object-detection supervision and jointly trains it with VQA distillation using equal weights. The enriched representation can therefore feed the original driving task heads instead of becoming a global scene vector useful only for conversation.

A Worked Example

Consider the paper's coordinate-conversion illustration: a reference to (450 px, 360 px) can be rewritten as an object located 3 meters ahead of the ego vehicle and 1.5 meters to its left. These are illustrative values from the paper, not a measured matching result. Multi-view images form the BEV grid, and the textual query is placed in the same ego-centric frame. The 2,500 pooled tokens enter the frozen LLM through the MLP, allowing the answer loss to backpropagate against that spatial relationship while the detection branch continues to constrain object positions.

Now consider the qualitative scenario in Figure 4: an excavator blocks the rightmost lane after a right turn, while a vehicle approaches in the adjacent lane. The enriched driving system adjusts its trajectory early; the baseline hesitates and collides. There is no online sequence in which an LLM explains the scene and a controller executes that explanation. Nor should the coordinate illustration above be treated as a measurement from this scenario: the two examples explain supervision alignment and a possible behavioral manifestation of offline semantic enrichment, respectively.

Loss & Training

Equation (1) is incomplete in the cached text and cannot be reliably reconstructed. The expression below is only an explanatory formulation of the explicitly described frozen-LLM cross-entropy proxy and equally weighted detection objective, not a reproduction of Equation (1):

\[ \mathcal{L}_{\mathrm{train}}=\mathcal{L}_{\mathrm{VQA}}+\mathcal{L}_{\mathrm{det}},\qquad \mathcal{L}_{\mathrm{VQA}}=\operatorname{CE}\!\left(p_{\phi}(\cdot\mid P_{\psi}(E_{\theta}(X)),q),a\right),\quad \phi\ \text{frozen}. \]

Here, \(E_{\theta}\) is the BEV encoder, \(P_{\psi}\) summarizes pooling and learnable projection, \(q,a\) are the question and answer, and \(p_{\phi}\) is the frozen language model. Parameter updates are frozen, not input gradients: the computation graph must remain connected at the LLM input. There is also no directly accessible ideal semantic-token target for pointwise regression. The paper uses that ideal representation as an interpretation and implements answer cross-entropy as the differentiable proxy.

Training starts from BEVFormer detection-pretrained weights and performs 1 epoch of distillation on the full DriveLM-nuScenes perception, prediction, behavior, and planning questions. The distilled encoder is then frozen, and task heads are trained for 20 epochs in UniAD or 12 epochs in VAD. The baseline likewise freezes its undistilled pretrained encoder and trains its task heads, allowing the evaluation to focus on changes to the representation.

Experiments use 8 NVIDIA A100 80GB GPUs. Full-data distillation takes approximately 35 hours with a 1B teacher and 100 hours with an 8B teacher; subsequent UniAD training remains 115 hours. The method therefore avoids bringing LLM cost into driving-time inference but adds offline training cost. To match total iterations in the data-subset ablation, the smaller behavior-and-planning subset is trained for 3 epochs, while the perception-and-prediction subset uses 1 epoch. The subsets should not be described as all receiving an identical number of epochs.

Key Experimental Results

Main Results

Ego3D evaluates cross-view, object-centric questions with InternVL3-1B as the language model. MCQ denotes multiple-choice accuracy; numerical distance error is L1 in meters. The following values are selected from the paper's Table 3. I denotes image-space input, B denotes BEV input, and ft. denotes the fine-tuned configuration.

Representation	Visual encoder parameters	Visual tokens	Average MCQ accuracy (%), higher is better	Distance L1 (m), lower is better
Original VLM image features \(I_{\mathrm{ViT}}\)	400M	4,608	28.57	14.15
\(I_{\mathrm{ViT}}\) w/ ft.	400M	4,608	62.19	7.42
UniAD before fusion \(I_{\mathrm{UniAD}}\)	40M	2,250	42.02	9.01
UniAD after fusion \(B_{\mathrm{UniAD}}\)	44M	2,500	61.34	7.05

Moving from 42.02% to 61.34% adds 19.32 percentage points, approximately a 46.0% relative improvement, not a 46.0-percentage-point increase. Distance error decreases from 9.01 to 7.05, approximately 21.8%. BEV accuracy remains slightly below the fine-tuned ViT's 62.19%, so the result is comparable performance with a smaller visual encoder, not uniform superiority over that baseline.

Driving results below are selected from Table 4. Open-loop nuScenes evaluation measures trajectory L2 error; closed-loop evaluation uses NeuroNCAP. All closed-loop entries report mean and standard deviation over 50 random-seed runs. NeuroNCAP scores range from 0 to 5 and account for impact velocity, with 5 indicating no collision. CR is collision rate and is recorded as a proportion rather than a percentage.

Driving pipeline and configuration	Average L2 (m), lower is better	NeuroNCAP score, higher is better	CR, lower is better
UniAD, baseline BEV	1.05	2.38 ± 1.52	0.56 ± 0.31
UniAD, random auxiliary QA head	1.16	1.75 ± 0.96	0.73 ± 0.19
UniAD, VLM-AD trans.	1.07	2.02 ± 1.18	0.70 ± 0.23
UniAD, VLM-AD CLIP	1.07	2.07 ± 1.13	0.67 ± 0.23
UniAD, 1B-distilled BEV	0.97	2.93 ± 0.69	0.54 ± 0.19
UniAD, 8B-distilled BEV	1.00	3.05 ± 1.26	0.47 ± 0.30
VAD, baseline BEV	0.82	3.24 ± 1.21	0.37 ± 0.24
VAD, 8B-distilled BEV	0.74	3.42 ± 0.97	0.34 ± 0.20

For UniAD, 8B distillation increases the score by approximately 28.2% relatively and reduces collision rate by approximately 16.1% relatively, corresponding to an absolute reduction of 9 percentage points. VAD also improves on both closed-loop metrics, but by smaller margins. The 8B model's open-loop average L2 of 1.00 is worse than the 1B model's 0.97 despite better closed-loop performance, showing why trajectory imitation error should not substitute for safety evaluation.

Ablation Study

The following values come from Table 6 and all use UniAD; distilled configurations use a 1B teacher. Impact velocity is measured only on cases that collide, in m/s, not as average vehicle speed over every scenario.

QA supervision	Average L2 (m), lower is better	NeuroNCAP score, higher is better	CR, lower is better	Impact velocity (m/s), lower is better
No distillation	1.05	2.38 ± 1.52	0.56 ± 0.31	6.11
Perception + prediction	1.02	2.77 ± 1.11	0.54 ± 0.25	4.64
Behavior + planning	1.01	2.75 ± 1.30	0.54 ± 0.26	3.97
All four question types	0.97	2.93 ± 0.69	0.54 ± 0.19	4.08

Key Findings

• The semantic source matters. A random auxiliary head on the same questions does not reproduce the gains. The authors' VLM-AD-style implementations also trail BEVLM, supporting the role of the frozen pretrained LLM without constituting a direct reproduction of the official VLM-AD system.
• Safer does not necessarily mean fewer collisions. With the 1B teacher, CR changes only from 0.56 to 0.54, while impact velocity falls from 6.11 to 4.08. The score gain primarily reflects reduced collision severity; the 8B teacher also provides a more substantial collision-rate reduction.
• Question subsets are complementary, but not every metric is simultaneously optimal. Full-data supervision yields the best score of 2.93, whereas behavior-and-planning supervision produces the lowest impact velocity of 3.97. Best aggregate score should not be paraphrased as best on every safety metric.

Highlights & Insights

• The teacher can exist only during training. Input gradients place semantics inside the existing BEV representation instead of requiring the planner to generate text on every step, making the approach compatible with established driving architectures.
• Validate the interface before claiming knowledge transfer. Object-existence and cross-view experiments establish that a language model can use BEV, then closed-loop evaluation tests whether semantic enrichment changes behavior. This is a more complete argument than reporting question-answering scores alone.
• Geometry constrains semantic learning rather than becoming obsolete. The detection branch lets language supervision add safety-relevant information without taking over the entire representation. A related approach could preserve spatial structure in robotic scene representations, although that transfer is not evaluated here.

Limitations & Future Work

• The authors acknowledge limited data coverage. Distillation primarily relies on DriveLM-nuScenes, selected for compatibility with nuScenes and curated annotations; scaling behavior with more diverse semantic data remains untested.
• Closed-loop evidence is still simulation-based. NeuroNCAP results in hazardous scenarios do not guarantee safety in road deployment. Standard deviations are substantial, and the main text does not report paired significance tests, so mean differences alone should not be treated as stable superiority over every baseline.
• Causal controls are incomplete. The visual encoders before and after fusion have 40M and 44M parameters and different token counts. The modest additional capacity and large-ViT comparison support the BEV argument, but this is not a strictly parameter- and token-matched experiment.
• Alternative-supervision comparisons have reproduction boundaries. The original VLM-AD system and supervision data were unavailable, so the comparisons use author-constructed variants. There is also no direct ablation removing the detection constraint in the main text, preventing an isolated estimate of its contribution.
• Cost and reproducibility details remain relevant. The 8B distillation adds approximately 100 hours under the stated training setup. The available cache contains no appendix and has a damaged Equation (1); details such as the IoU threshold could not be verified. This note does not fill those gaps, and a project link is not a verified code repository.

Related Work & Insights

• BEVFormer, UniAD, and VAD: BEVFormer supplies the geometric fusion encoder, while UniAD and VAD supply end-to-end driving pipelines. BEVLM inserts semantic distillation between detection pretraining and task-head training rather than proposing an entirely new controller.
• DriveLM and Ego3D: DriveLM supplies driving questions and distillation data, whereas Ego3D probes cross-view object relationships. Correct answers and safe driving are different evaluation levels and should not substitute for one another.
• VLM-AD: The paper's alternatives transfer semantics through text embeddings or a random Transformer, while BEVLM passes BEV tokens directly through a frozen LLM's question-answering computation. The transferable principle is to constrain the representation that the downstream model actually consumes, not simply to add text supervision.
• Language-model driving approaches such as LMDrive and EMMA: These explore language models in driving outputs, while BEVLM separates semantic learning from online execution. Broader safety-question coverage and teacher-scale benefits can therefore be studied without first accepting the cost of online language reasoning.

Rating

• Novelty: 4/5. Connects a study of BEV's spatial advantages with frozen-LLM semantic distillation; the innovation primarily lies in the combination and its empirical argument.
• Experimental Thoroughness: 4/5. Covers representation controls, two driving pipelines, teacher scale, and question subsets, but lacks real-road generalization and an isolated ablation of the geometric constraint.
• Writing Quality: 4/5. Research questions build progressively, and training versus deployment is clearly separated; some stronger mechanistic interpretations still require more direct evidence.
• Value: 4/5. Offers a useful route to safety-oriented representation learning without online LLM overhead, while benefits should remain bounded by the evaluated settings.