PriorEye: Geospatial Visual Priors for End-to-End Autonomous Driving¶

Conference: ECCV 2026
arXiv: 2606.31830
Code: No public code yet (Project page: https://ori-mrg.github.io/PriorEye)
Area: Autonomous Driving
Keywords: End-to-End Autonomous Driving, Geospatial Visual Priors, Memory Augmentation, Robustness, NAVSIM

TL;DR¶

PriorEye anchors offline-collected street-view images to planned routes to construct "geospatial visual priors." These priors are injected into end-to-end driving policies via a dual-memory architecture, equipping the agent with human-driver-like prospective perception while maintaining robustness under both sensor failures and prior corruption.

Background & Motivation¶

End-to-end (E2E) autonomous driving, with its elegant paradigm of mapping "sensor inputs directly to trajectory outputs," has rapidly become mainstream, with methods like LTF, GTRS, and DrivoR continuously advancing state-of-the-art results on benchmarks such as NAVSIM and nuPlan. However, such systems suffer from an overlooked structural limitation: they rely solely on a few seconds of online streaming perception data at the current step for decision-making, making them fundamentally reactive. Complex real-world driving scenarios—such as speed bumps hidden around corners, vanishing lanes, or occluded intersections—require drivers to decelerate or change lanes before obstacles are even visible. Humans achieve this because the brain stores visual memories of driven routes. Substantial cognitive neuroscience research (e.g., on the hippocampus and spatial navigation) indicates that such "route foresight" is core to safe and efficient driving.

For E2E systems, the solution is not simply feeding longer historical frames to the model, as occlusions and sensor noise will inevitably contaminate any real-time inputs. In traditional methods, High-Definition (HD) maps supply road structure and topologies but are costly to construct and maintain. Furthermore, they do not encode visual appearance. Conversely, pure retrieval-augmented methods (e.g., RAD-Driver) are typically confined to high-level reasoning and do not directly modify the planner's internal state. Visual priors (e.g., street views) and spatial priors (route anchors) have not previously been systematically fused and injected into E2E planning backbones.

This work's Key Insight aims to fill this gap: street-view images along driving routes are collected offline from Google Street View to bind visual priors with route coordinates, forming "Geospatial Visual Priors." A lightweight, model-agnostic memory augmentation module is then designed to inject these priors into the intermediate states of any E2E planner. Core Idea: Treat offline segment-level street views as a retrievable long-term memory. Use a dual-memory architecture (contextual memory + persistent memory) and an adaptive gating mechanism to fuse them into the E2E planning state, allowing the model to rely on these priors for prospective driving even when real-time perception fails.

Method¶

Overall Architecture¶

PriorEye consists of two decoupled components: an offline-constructed Memory Bank and an online-queried Memory Augmentation Module. In the deployment area, the system pre-traverses every lane centerline, gathering street-view images at 5-meter intervals. It embeds them using a frozen SigLIP2 visual encoder to formulate a "position-to-visual embedding" mapping table. This yields a ~939 MB memory bank covering 6.5 km². During online inference, based on the current vehicle location and high-level navigation intent (left/right turn/straight), the system retrieves the forward \(N=20\) nodes (a ~100m look-ahead) along the predicted route. This yields visual priors \(\mathbf{V} \in \mathbb{R}^{N \times D_m}\) and corresponding relative coordinate prior \(\mathbf{X} \in \mathbb{R}^{N \times 2}\), which are jointly fed into the memory augmentation module. Taking the E2E model's intermediate driving state \(\mathbf{S}\), this module outputs the augmented state \(\mathbf{S}'\) to the downstream trajectory decoder, leaving the parameters of the E2E backbone intact.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Street-view Image Library<br/>(Collected offline every 5m)"] --> B["Memory Bank Construction<br/>SigLIP2 encoding + Position anchoring"]
    B --> C["Intent-guided Retrieval<br/>DFS route + Navigation intent filtering"]
    C --> D["Geospatial Visual Priors<br/>V (Visual embedding) + X (Coordinates)"]
    E["E2E Intermediate State S<br/>(BEV/Image/Scene features)"] --> F["Dual-memory Architecture<br/>Contextual Memory C + Persistent Memory P"]
    D --> F
    F --> G["Cross-attention Memory Fusion<br/>S as query, P+C as key/value"]
    G --> H["Adaptive Memory Gating<br/>Cosine similarity + Euclidean distance"]
    E --> H
    H --> I["Augmented State S'<br/>Passed to downstream decoder"]

Key Designs¶

1. Intent-Guided Route Retrieval: Aligning Priors with Planning Intent

The most intuitive approach is retrieving the spatially nearest \(N\) street-view nodes. However, this proximity-based retrieval frequently aggregates images from multiple directions near the vehicle, most of which are irrelevant to the forthcoming driving actions. Ablation experiments indicate that proximity-based retrieval improves the LTF baseline by only 0.4 EPDMS, whereas intention-guided retrieval improves it by 2.7 EPDMS. The proposed method: Starting from the current lane, a depth-first search (DFS) is executed on the lane connectivity graph to construct a set of forward candidate routes. Based on the high-level navigation intent (left turn, right turn, straight), the corresponding route branch is filtered from the candidate set, and street-view embeddings are retrieved solely along this branch. Consequently, the retrieved \(N\) priors closely align with the semantic intent of the actual maneuvers, drastically reducing noise.

2. Dual-Memory Architecture: Integrating Contextual and Persistent Memory

Contextual Memory \(\mathbf{C}\) is jointly constructed from the retrieved visual and coordinate priors:

\[\mathbf{C} = \phi(\mathbf{V}) + \psi(\mathbf{X})\]

where \(\phi(\cdot)\) is a linear layer mapping visual embeddings to the internal dimension of the E2E model, and \(\psi(\cdot)\) is a function encoding relative coordinates into 2D sinusoidal position embeddings. Their addition enables contextual memory to preserve both semantic appearance and coarse geometric layout. Ablation studies show that visual encoding contributes +2.3 EPDMS and spatial encoding +1.0 EPDMS, whereas their combination yields +2.7 EPDMS, displaying clear complementarity.

However, street-view images can fail due to seasonal changes, geometric discrepancies, or collection gaps, introducing noisy or misleading context. This is fundamentally identical to the "attention sink" problem in Transformers, where attention is forced to allocate weight to useless tokens when valid keys are absent. Inspired by Titans and persistent memory mechanisms, this work introduces Persistent Memory \(\mathbf{P} \in \mathbb{R}^{K \times D}\)—a set of input-independent learnable tokens—acting as a stable backup when contextual memory fails. Both memory types are concatenated to serve as keys and values for cross-attention. Under normal conditions, the model relies primarily on contextual priors, but naturally redirects its attention to persistent memory when priors are corrupted. Experiments reveal that when visual priors are severely corrupted, the model's attention weight on \(\mathbf{P}\) surges from 0.29 to 0.97, reducing performance degradation from 19.6% to just 6.8%.

3. Adaptive Memory Gating: Measuring Compatibility for Fusion Calibration

After obtaining the memory-weighted state \(\tilde{\mathbf{S}}\) via cross-attention, directly adding it as a residual to the original state risks instability—especially when retrieved priors do not match the current scene, making forced injection harmful. The authors design a lightweight adaptive gating mechanism \(\mathbf{G}\) that leverages both cosine similarity \(c\) and normalized Euclidean distance \(d\) to assess semantic compatibility between \(\mathbf{S}\) and \(\tilde{\mathbf{S}}\):

\[\mathbf{G} = \sigma\!\left(f_g\!\left([\mathbf{S};\, \tilde{\mathbf{S}};\, d;\, c]\right)\right), \quad \mathbf{S}' = \mathbf{S} + \mathbf{G} \odot \tilde{\mathbf{S}}\]

\(f_g\) is a lightweight MLP with its bias initialized near zero, keeping the gate largely closed during early training phases to prevent random priors from disrupting backbone gradient signals. Cosine similarity and Euclidean distance capture directional alignment and magnitude differences, respectively. Separately, they improve performance by 0.4 EPDMS each, while their combination yields the optimal +2.7 EPDMS, indicating they capture different aspects of compatibility. The entire module introduces only 713K extra parameters, representing a negligible 0.6%–1.7% overhead relative to the LTF/GTRS/DrivoR backbones.

Loss & Training¶

The memory augmentation module is inserted directly into the intermediate layers of each baseline model. It is trained end-to-end jointly using the baseline's original training objectives without additional loss terms. The definition of "intermediate states" varies lightly across baselines: LTF and GTRS-DP utilize BEV features + ego-status, GTRS-Dense uses image features + ego-status, and DrivoR utilizes scene tokens + ego-status. Training was conducted on 8 NVIDIA RTX 5090 GPUs, with training epochs configured to 100, 80, 40, and 20 for the respective baselines.

Key Experimental Results¶

Main Results¶

NAVSIM-v2 navhard-two-stage (EPDMS, Two-Stage Product)

Method	Stage1 EPDMS	Stage2 EPDMS	Gain
LTF	24.7	73.2	—
LTF + PriorEye	32.4	72.1	+31.2%
GTRS-DP	26.3	62.2	—
GTRS-DP + PriorEye	30.1	66.9	+14.4%
GTRS-Dense	44.9	49.6	—
GTRS-Dense + PriorEye	48.6	51.3	+8.2%
DrivoR	48.9	76.5	—
DrivoR + PriorEye	49.6	74.5	+1.4%

NAVSIM-v2 navtest (EPDMS, Stage 1 Evaluation)

Method	Baseline	+PriorEye	Gain
LTF	84.3	86.8	+2.5
GTRS-DP	82.2	82.5	+0.3
GTRS-Dense	85.4	88.8	+3.4
DrivoR	87.2	89.9	+2.7

Ablation Study¶

Configuration	EPDMS	Description
LTF Baseline	81.7	Reference baseline
DINOv2 Encoder	83.4	Limited impact from visual backbone
SegFormer Encoder	82.9
SigLIP2 Encoder (Selected)	84.4	Most semantically rich
Proximity-based retrieval	82.1	Almost ineffective
Intent-guided retrieval (Selected)	84.4	Key design
Persistent memory P only	82.2	Added parameters but lacks context
Contextual memory C only	83.9	Effective but lacks backup
P + C (Selected)	84.4	Complementary
Gate using only S and S̃	83.0	Lacks similarity signals
With cosine similarity c	83.4
With Euclidean distance d	83.4
With c and d (Selected)	84.4	Best

Sensor Robustness (GTRS-Dense, navhard-two-stage EPDMS)

Corruption Type	Baseline	PriorEye	Baseline Drop	PriorEye Drop
Clean	44.9	48.6	—	—
Fingerprint	40.0	45.1	-10.9%	-7.2%
Handprint	40.3	45.4	-10.2%	-6.6%
Frost	34.6	38.1	-22.9%	-21.6%
Dirt (Light)	42.3	47.9	-5.8%	-1.4%
Dirt (Heavy)	21.0	32.8	-53.2%	-32.5%

Key Findings¶

Intent-guided retrieval is the most critical individual design. The failure of proximity-based retrieval indicates that "semantic alignment with current actions" is far more crucial than "spatial proximity."
Visual encoding (appearance) contributes more than spatial encoding (coordinates) (+2.3 vs +1.0), but their combination yields complementary gains, showing that geometric structure and semantic appearance carry distinct information.
When priors are severely corrupted (e.g., random visual offset of 500 m), attention weights automatically shift to persistent memory (0.71 -> 0.97), shrinking the performance drop from 19.6% to 6.8%. Persistent memory provides a vital safety backup.
PriorEye achieves the most notable gains on Ego Progress (+3.3) and Extended Comfort (+11.6), while collision-related metrics (NC, TTC) slightly decrease. This occurs because the priors encode static road features rather than dynamic agents, and more aggressive progress occasionally compresses the safety margins with dynamic obstacles.
During inference, the module adds 9.3 ms of latency to GTRS-Dense (3.7 ms CPU + 5.6 ms GPU), leading to a total latency of 67.2 ms, which is well below the 100 ms real-time limit for 10 Hz operation.
PriorEye (+3.7 EPDMS) outperforms vectorized HD maps (+1.5) and rasterized HD maps (+2.2) while only requiring lane centerlines instead of complete HD maps.

Highlights & Insights¶

Decoupled offline prior + online fusion design holds significant engineering value: Street-view image encoding is fully executed offline, leaving only lightweight retrieval and attention operations for inference. This ensures minimal real-time overhead and enables plug-and-play integration without altering E2E backbones.
The analogy of persistent memory as an "Attention Sink" is precise and insightful: when the quality of contextual keys is extremely poor, attention has nowhere to go. Persistent tokens serve as a "safe absorption pit," preventing a complete breakdown of attention quality. This mechanism is highly generalizable to other retrieval-augmented Transformer scenarios.
Intent-guided retrieval unveils a vital design principle: retrieval units for driving planning must align with planning intent rather than merely spatial positioning. This offers clear transferability for other retrieval-augmented driving approaches.
The finding that street-view priors outperform HD map priors is counter-intuitive: although HD maps are more structured and precise, semantic appearance (visual textures, traffic signs, pedestrian crossings, road conditions) delivers the actual cues required by end-to-end planners, which are entirely absent in HD maps.

Limitations & Future Work¶

Current retrieval relies entirely on vehicle locations (GPS anchors), which can drift in regions with poor GPS signals or inaccurate maps. The authors suggest that incorporating visual appearance matching could boost retrieval robustness.
The memory bank is built on static street-view images, failing to account for seasonal changes (e.g., snow, construction), temporary signs, or accident scenes. Future work could integrate fleet logs to construct dynamically updated memory banks.
\(N=20\) is slightly beyond necessary in low-speed urban scenarios of NAVSIM (\(N=10\), i.e., 50 m, is already optimal). Its applicability in long-horizon highway scenarios remains unverified.
The slight drop in NC/TTC under prior corruption suggests that interaction with dynamic agents remains a blind spot, requiring orchestration with trajectory prediction modules.

vs. Spatial Retrieval-Augmented Driving (SRAD): While both leverage street-view embeddings, SRAD employs proximity-based retrieval primarily for upstream tasks like mapping. Under contrast, Ours explicitly uses intent-guided retrieval, directly injecting priors into E2E planning states while introducing a robust dual-memory mechanism.
vs. RAD-Driver / MTRDrive: These methods leverage past driving experiences for high-level semantic reasoning (VLMs). Ours utilizes the visual appearance of road segments rather than historical trajectories, and modulates the internal state representation of the planner rather than prompts.
vs. Titans / Persistent Memory Transformers: Ours transfers the dual-memory philosophy of Titans from language sequence modeling to autonomous driving, demonstrating that persistent tokens are highly effective in visual-spatial retrieval-augmented scenarios.
vs. HD Map Prior Methods: Methods like P-MapNet and SatMap use aerial or satellite images as priors, while Ours employs street-level perspective imagery. The two approaches are complementary: street views are visually richer, whereas HD maps are geometrically more precise.

Rating¶

Novelty: ⭐⭐⭐⭐ Integrating geospatial visual priors directly into E2E planning states through intent-guided retrieval and dual-memory fusion is a highly novel design.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Validation across 4 distinct baselines, 2 evaluation splits, and extensive robustness evaluations against sensory corruption and prior noise makes the experiments remarkably robust.
Writing Quality: ⭐⭐⭐⭐ Well-structured with abundant auxiliary experiments, though notations are slightly dense at times.
Value: ⭐⭐⭐⭐⭐ Extremely high engineering relevance given its lightweight, plug-and-play architecture, outperforming HD-map solutions with low deployment costs.