Plug-and-Play Traffic Element Awareness for End-to-End Autonomous Driving¶
Conference: ECCV 2026
Paper: ECCV Official
Project: https://zzongzheng0918.github.io/TE-Aware-E2E-AD/
Area: Autonomous Driving
Keywords: End-to-End Autonomous Driving, Traffic Elements, Topology Reasoning, Plug-and-Play, Trajectory Planning
TL;DR¶
Addressing the widespread neglect of regulatory traffic lights and road signs in mainstream end-to-end autonomous driving systems, this paper introduces a lightweight, plug-and-play auxiliary 3D traffic element supervision along with a language-guided ego-centric topology conditioning mechanism, consistently boosting safety and compliance across nuScenes, NAVSIM-v1/v2, and Bench2Drive.
Background & Motivation¶
End-to-end (E2E) autonomous driving has rapidly transitioned from exploratory laboratory prototypes to real-world industrial deployments, spanning a broad spectrum of paradigms including perception–prediction–planning pipelines, direct trajectory regression, candidate trajectory scoring, generative diffusion policies, and recent vision–language–action (VLA) foundation models. Despite this architectural variety, a fundamental regulatory signal that strictly governs human driving decisions remains surprisingly under-discussed and marginalized in the E2E literature: traffic elements (TE), specifically traffic lights and road signs. In real-world driving environments, traffic elements are not optional ambient background features; they are statutory regulatory signals that directly dictate the feasible action space (e.g., stop/go directives, lane-level turning prohibitions, and right-of-way permissions). Benchmark statistics underscore that over 55% to 65% of driving scenarios in widely used datasets such as nuScenes and NAVSIM-v1 contain traffic elements, demonstrating that interacting with traffic elements is a pervasive daily driving requirement rather than a rare long-tailed edge case.
Nevertheless, existing end-to-end driving frameworks largely lack systematic modeling and quantitative assessments of traffic elements. This persistent gap stems from two practical bottlenecks: first, mainstream public driving benchmarks rarely provide complete, structured 3D spatial annotations for traffic elements along with their semantic topological associations; second, modern driving models exhibit high architectural diversity, making empirical conclusions derived from a single bespoke architecture difficult to generalize. Consequently, current models disproportionately allocate their representation capacity to dynamic foreground obstacles (vehicles and pedestrians) and dense geometric occupancy. In complex urban intersections, this bias frequently leads to the generation of trajectories that are geometrically smooth but violate traffic laws, such as running red lights or turning from prohibited lanes.
Rather than proposing yet another hyper-customized planner, this work establishes a minimal and universal research infrastructure to systematically investigate traffic element awareness. The angle of attack centers on building cross-benchmark 3D traffic element annotations and devising an unobtrusive integration pipeline. Core idea: by establishing standardized cross-dataset 3D traffic element annotations, introducing a lightweight plug-and-play auxiliary 3D detection objective, and conditioning the planner on compact, language-encoded ego-centric lane-element topology, end-to-end planners achieve substantial improvements in regulatory awareness and driving safety with negligible runtime overhead.
Method¶
Overall Architecture¶
The overall method operates across two primary stages: automated cross-benchmark 3D traffic element ground-truth and pseudo-label construction, and plug-and-play rule injection during end-to-end planner training. Given multi-view and front-view camera streams, a visual backbone extracts latent queries (such as bird's-eye-view queries \(Q_{\text{bev}}\)). To prevent tiny and spatially sparse traffic elements from being overwhelmed by the gradients of dense background geometry, a decoupled 3D traffic element prediction head is attached for multi-task auxiliary supervision. In parallel, global centerline-to-centerline connectivity and centerline-to-element governance graphs are filtered around the ego vehicle to isolate active local rules. This filtered subgraph is serialized into structured natural language and transformed by a frozen language encoder into compact topology queries \(Q_{\text{topo}}\). Finally, the visual and topology queries are concatenated and fed into the trajectory planning decoder to output safe and compliant future trajectories.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
A["Multi-View Images + Point Clouds"] --> B["Automated 3D Traffic Element Extraction<br/>2D Detection + Monocular Depth + LiDAR Projection"]
B --> C["Independent 3D TE Auxiliary Supervision<br/>3D Center L1 Loss and Category Focal Loss"]
A --> D["Multi-View Vision Backbone<br/>Generate BEV Feature Queries Q_bev"]
C -.->|Back-propagation gradient enhancement| D
A --> E["Language-Guided Ego-Centric Topology Conditioning<br/>Filter ego-active lanes and elements into text"]
E --> F["Frozen BERT Language Encoder<br/>Generate Topology Queries Q_topo"]
D --> G["Dual Spatial Stream Concatenation<br/>Q_plan = [Q_bev; Q_topo]"]
F --> G
G --> H["Trajectory Planning Decoder<br/>Generate Rule-Compliant Ego Trajectory"]
Key Designs¶
1. Automated 3D Traffic Element Extraction: Multi-Modal Geometric Fusion Pipeline To resolve the absence of 3D spatial annotations for traffic elements in public benchmarks, the authors build an automated labeling pipeline that reliably recovers 3D center coordinates \((x, y, z)\) from 2D images and LiDAR point clouds. Front-view camera images are simultaneously processed by a 2D traffic element detector and a monocular metric depth foundation model (UniDepthv2). Utilizing camera intrinsic and extrinsic parameters, detected 2D bounding boxes are projected into the vehicle's coordinate frame, where dense depth estimations are fused with spatially corresponding LiDAR point clusters to calculate precise 3D centroids. For benchmarks entirely devoid of traffic annotations such as NAVSIM, the authors iteratively optimize a YOLOv8 detector on OpenLane-V2 using class-balanced resampling, loss reweighting, pseudo-labeling, and test-time augmentation (TTA), thereby generating high-fidelity 3D traffic element pseudo-labels across the entire NAVSIM-v1 and NAVSIM-v2 suites.
2. Independent 3D Traffic Element Auxiliary Supervision: Decoupled Dual Spatial Stream Because traffic elements occupy only a minute fraction of the spatial BEV grid, naively treating them as an additional semantic segmentation class causes their gradients to be entirely suppressed by dominant background terrain and dynamic objects. To overcome this gradient imbalance, the framework establishes an independent detection head trained with Focal Loss for multi-class identification and L1 loss for 3D coordinate regression. During spatial feature downsampling, conventional average pooling severely dilutes localized peak activations; hence, adaptive max pooling is deployed to preserve sharp feature responses. The downsampled traffic element features are then concatenated directly with the primary BEV feature map to construct an aligned "Dual Spatial Stream." Compared to unconstrained cross-attention mechanisms, direct feature concatenation enforces strict spatial correspondence, allowing the trajectory decoder to bind regulatory constraints to relevant lane coordinates reliably.
3. Language-Guided Ego-Centric Topology Conditioning: Structured Textualization and Compact Injection Global topological representations (centerline connectivity \(R_{\text{LCLC}}\) and element-to-lane associations \(R_{\text{LCTE}}\)) typically contain hundreds of distant, irrelevant interactions that distract the planner's attention mechanism. To eliminate this noise, the model identifies the ego vehicle's current lane ID and retrieves only the directly connected centerlines and governing traffic elements. This localized causal subgraph is mapped into a structured natural language template (e.g., detailing active traffic light states, turning prohibitions, and downstream lane connectivity). A frozen pretrained BERT-base encoder transforms this text into compact topology queries \(Q_{\text{topo}}\). In contrast to Graph Convolutional Networks (GCNs), which suffer from feature over-smoothing when aggregating heterogeneous nodes (continuous road centerlines versus discrete semantic sign categories), language encoding maps multi-modal relations into a unified semantic embedding space while preserving strict causal dependencies.
Loss & Training¶
The overall training objective combines the primary planning loss with auxiliary multi-task supervision: $$ \mathcal{L}{\text{total}} = \mathcal{L}}} + \sum_k \lambda_{\text{aux}}^{(k)} \mathcal{L{\text{aux}}^{(k)} + \lambda}} \left( \mathcal{L{\text{L1}}^{\text{TE}} + \mathcal{L} \right) $$ where }}^{\text{TE}\(\mathcal{L}_{\text{plan}}\) denotes the baseline trajectory imitation or generative loss, \(\mathcal{L}_{\text{aux}}^{(k)}\) encompasses existing auxiliary objectives (such as 3D vehicle bounding box detection, occupancy prediction, and motion forecasting), and the traffic element supervision weight \(\lambda_{\text{TE}}\) is set equal to the baseline auxiliary tasks. Models on nuScenes and Bench2Drive are trained across 8 NVIDIA A100 GPUs with batch size 1, while NAVSIM experiments run on 4 RTX 3090 GPUs with batch size 32, strictly adhering to the original optimization schedules of each baseline planner.
Key Experimental Results¶
Main Results¶
The methodology was comprehensively evaluated on three open-loop benchmarks (nuScenes, NAVSIM-v1, NAVSIM-v2) and one closed-loop simulator (Bench2Drive). Across all paradigms, integrating traffic element awareness produces consistent and significant gains.
| Benchmark / Task | Metric | Baseline Model | Base Score | + Ours | Gain |
|---|---|---|---|---|---|
| nuScenes (Open-Loop) | Avg. L2 (m) ↓ | VAD-Base | 0.72 | 0.60 | -0.12 m |
| nuScenes (Open-Loop) | Avg. Collision (%) ↓ | VAD-Base | 0.22 | 0.17 | -0.05% |
| nuScenes (VLA Architecture) | Avg. L2 (m) ↓ | Orion | 0.34 | 0.26 | -0.08 m |
| nuScenes (VLA Architecture) | Avg. Collision (%) ↓ | Orion | 0.37 | 0.23 | -0.14% |
| NAVSIM-v1 (navtest) | PDMS ↑ | LTF (Regression) | 84.1 | 85.2 | +1.1 |
| NAVSIM-v1 (navtest) | PDMS ↑ | DiffusionDrive (Diffusion) | 86.0 | 87.7 | +1.7 |
| NAVSIM-v1 (navtest) | PDMS ↑ | DrivoR (Scoring) | 93.1 | 94.4 | +1.3 |
| NAVSIM-v1 (+SimScale) | PDMS ↑ | DrivoR + SimScale | 94.6 | 95.1 | +0.5 (Surpasses human 94.8) |
| Bench2Drive (Closed-Loop) | Driving Score ↑ | VAD | 42.3 | 56.4 | +14.1 |
| Bench2Drive (Closed-Loop) | Driving Score ↑ | DriveTransformer-L | 63.46 | 68.29 | +4.83 |
On the challenging NAVSIM-v2 (navhard-two-stage) pseudo closed-loop benchmark, traffic element awareness yields pronounced improvements in EPDMS across all architectures: - Regression-based LTF: improves from 25.1 to 28.9 (+15% relative gain); further increases to 36.9 when co-trained with SimScale data. - Diffusion-based DiffusionDrive: improves from 29.4 to 32.7 (+11% relative gain); reaches 37.9 when combined with SimScale. - Scoring-based DrivoR: improves from 48.3 to 51.8 (+7.2% relative gain); reaches 57.9 with SimScale, setting a new state-of-the-art record.
Ablation Study¶
Systematic ablations conducted on NAVSIM-v2 using the LTF baseline analyze the impact of different spatial representations and design choices.
| Config ID | Model Variant | Core Modification | EPDMS ↑ | Comparison vs Base |
|---|---|---|---|---|
| 0 | LTF Baseline | Standard regression planner | 25.1 | Base |
| 1 | + TE(2D) | 2D bounding boxes in front-view image only | 28.1 | +3.0 |
| 2 | + depth(FV) | Global dense front-view depth auxiliary task | 27.7 | +2.6 |
| 3 | + TE(LiDAR) | Direct 2D box projection to LiDAR point clustering | 27.9 | +2.8 |
| 4 | + TL Only | Traffic lights only (excluding road signs) | 26.7 | +1.6 |
| 5 | + Class-Agnostic TE | Supervised with 3D L1 loss only without semantic classes | 26.2 | +1.1 |
| 6 | Ours (Full TE) | Independent head + Focal Loss + MaxPool + Concat | 28.9 | +3.8 |
In the topology integration ablation on nuScenes (using VAD as baseline, where baseline L2 Avg is 0.72m and Collision Rate Avg is 0.22%): - LCLC connectivity only: L2 drops to 0.63m, CR drops to 0.16%. - LCTE element governance only: L2 drops to 0.62m, CR drops to 0.18%. - GCN graph encoding: CR increases to 0.23% due to node over-smoothing. - BERT language encoding: CR decreases to 0.16% while maintaining 0.62m L2. - Global vs. Ego-centric scope: global topology introduces irrelevant distant noise, degrading CR to 0.50%; ego-centric filtering achieves 0.16% CR.
Key Findings¶
- Road signs are as critical as traffic lights: Supervising traffic lights alone yields only +1.6 EPDMS, whereas incorporating road signs expands the gain to +3.8, proving that turn restrictions and right-of-way signs are essential for eliminating illegal maneuvers.
- Object-centric depth beats global geometry: Explicitly estimating depth for traffic elements outperforms supervising a global dense depth map (28.9 vs. 27.7 EPDMS). Dense depth introduces redundant surface geometry noise, while object-level 3D modeling isolates high-value regulatory cues.
- Language encoding surpasses GCNs for topology: Driving scene topology involves heterogeneous nodes (continuous polyline lanes versus discrete categorical lights). GCN message-passing causes semantic blurring, whereas structured language models serialize relationships into coherent causal prompts without loss.
- High resilience to perception errors: Artificially injecting perception noise at inference time—such as up to 10m depth perturbation, 55% missed detections, or 20% false positives—results in graceful, linear performance degradation, consistently outperforming the baseline planner across all corruption levels.
Highlights & Insights¶
- Minimalist plug-and-play architecture: Avoids invasive structural modifications to planner backbones. By pairing a lightweight auxiliary 3D detection objective with language-projected ego topology, the method effortlessly integrates into perception-planning, regression, diffusion, trajectory scoring, and VLA models.
- Negligible computational latency: Unlike compute-heavy end-to-end VLM planners that introduce latency of several hundred milliseconds, this approach reduces inference throughput by only ~0.3 FPS (an added latency of merely 4-5ms), making it highly suitable for real-time automotive deployment.
- Generalizable neuro-symbolic conditioning: The strategy of serializing graph-structured road topologies into structured natural language for ingestion by small language models provides an effective blueprint for autonomous delivery robots, HD-map routing, and embodied AI systems navigating rule-governed spaces.
Limitations & Future Work¶
- Performance in adverse environmental conditions: While resilient to common detection inaccuracies, extreme weather (heavy rain, severe fog, intense glare, or darkness) may enlarge monocular depth estimation errors, potentially undermining 3D traffic element localization.
- Absence of fine-grained temporal modeling: The current framework primarily processes single-frame or short-horizon front-view imagery. Complex dynamic signal transitions, such as flashing yellow phases or countdown timers, lack dedicated temporal sequence modeling.
- Future directions: Jointly modeling temporal signal dynamics within end-to-end world models, and directly interfacing V2X digital infrastructure signals with language-conditioned planning streams.
Related Work & Insights¶
- vs. OpenLane-V2 & TopoMLP: OpenLane-V2 introduced benchmarks for lane and traffic element topology, while TopoMLP achieved high topology recognition scores. However, both remained confined to perception without quantifying downstream planning impact. This work bridges that divide, establishing the first empirical validation of topology's causal utility in planning.
- vs. UniAD / VAD: Mainstream planners focus predominantly on dynamic obstacle occupancy and continuous map boundaries, leaving discrete regulatory signals under-represented. This plug-and-play framework fills that blind spot, eliminating intersection traffic violations while maintaining original inference speeds.
- vs. DriveVLM / OmniDrive: Large vision-language driving models possess rich semantic reasoning but operate at prohibitive latencies (< 1 FPS). By combining auxiliary 3D supervision with a compact language encoder, this approach delivers comparable regulatory adherence at over 10× higher throughput.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ The first systematic study to quantify and integrate traffic element awareness across diverse modern end-to-end driving paradigms.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations spanning 4 mainstream benchmarks, 6 diverse planner architectures, open-loop, closed-loop CARLA simulation, and large-scale data engine scaling.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, structured motivations, and thorough ablation studies dissecting every architectural choice.
- Value: ⭐⭐⭐⭐⭐ High industrial relevance and practical engineering value for deploying safe and rule-compliant end-to-end autonomous driving systems.