Learning Ego-Centric BEV Representations from a Perspective-Privileged View: Cross-View Supervision for Online HD Map Construction¶

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/DriverlessMobility/CrossViewSupervision
Area: Autonomous Driving
Keywords: BEV perception, online HD map construction, cross-view supervision, privileged information, feature alignment

TL;DR¶

Treating ego-aligned aerial imagery as a "perspective-privileged view" available only at training time, a frozen aerial encoder supplies dense BEV feature-level alignment supervision to a camera BEV encoder (stabilized by channel normalization and an affine adapter), lifting StreamMapNet on nuScenes/AID4AD from 34.1 to 38.0 mAP (60×30 m) and from 22.4 to 32.3 mAP (100×50 m, +44% relative) without changing the inference architecture.

Background & Motivation¶

HD maps have long been a key enabler of autonomous driving, supplying geometric and semantic priors for localization, motion forecasting and behavior planning, but they are expensive to produce, labor-intensive to maintain, and hard to scale in dynamic environments with construction, temporary lane changes or seasonal variation. Research has therefore shifted toward map-less or map-light perception, where the vehicle infers structural priors directly from onboard sensors. Online HD map construction is the most prominent instance: the vehicle predicts structured, vectorized map elements (road boundaries, lane dividers, pedestrian crossings) from current observations in real time. BEV-based methods such as StreamMapNet, MapTR/MapTRv2 and VectorMapNet already convert multi-camera input into structured vector output and perform strongly on standard benchmarks.

The difficulty is that camera-based BEV perception is inherently ego-centric: the vehicle observes the scene through a limited field of view, so global spatial structure must be inferred from partial, perspective-distorted evidence. As a result, BEV encoders tend to produce locally inconsistent or fragmented representations, and the problem worsens at long range, where visual cues become sparse and information density drops sharply. Supervision compounds the issue: in most methods the loss acts only after decoding into semantic or vectorized map outputs, imposing no direct constraint on the geometry of the intermediate BEV feature. How the encoder's representation is shaped is thus left almost unmanaged during training. A natural question follows: can a BEV encoder be steered toward globally consistent spatial representations during training?

Aerial imagery offers exactly the complementary overhead view that ego-centric sensing lacks — road layout, connectivity and long-range spatial relationships are directly observable from above, unaffected by occlusion or perspective distortion. The AID4AD dataset registers aerial imagery to the nuScenes ego coordinate frame, establishing pixel-level correspondence between aerial imagery and ego-centric BEV representations, which lets aerial imagery serve not only as a fusion input but also as a cross-view training signal. Existing fusion-based approaches do improve structural completeness and large-scale geometric consistency, but they require dual encoders and continuous aerial availability at inference time, which is costly to deploy. The core idea of this paper is to treat the ego-aligned aerial view as a perspective-privileged signal — it exists only during training — and to distill its global structural prior into the camera encoder through dense BEV feature alignment, so that at inference the privileged branch is removed entirely and the model matches the original camera-only model in architecture, inputs and runtime.

Method¶

Overall Architecture¶

The method is a dual-branch framework used exclusively during training, instantiated on top of StreamMapNet. The student branch is StreamMapNet's BEVFormer-style camera encoder: six synchronized surround-view images are aggregated via deformable attention into a single unified BEV feature map. The teacher branch is an aerial encoder pretrained on AID4AD (ResUNet, a ResNet backbone with a U-Net decoder) that takes aerial crops registered to the ego coordinate frame and outputs another BEV feature map. Both branches produce feature maps of exactly the same shape, \(C\times H\times W = 256\times 50\times 100\). Because AID4AD already performs pixel-level registration, the same spatial location in the two feature maps corresponds to the same patch of ground, so a direct position-wise comparison is possible without any projection, resampling or spatial transform. During training a lightweight MSE alignment loss pulls the student features toward the teacher features, while the teacher stays frozen throughout; at inference only the camera branch remains, identical in structure, input and runtime to StreamMapNet.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["6 surround-view images"] --> B["Camera BEV encoder<br/>StreamMapNet backbone"]
    C["Ego-aligned aerial crop"] --> D["Perspective-privileged aerial teacher<br/>ResUNet, frozen"]
    B --> E["Channel normalization + affine adapter<br/>active only in the loss path"]
    D --> F["Dense BEV alignment loss"]
    E --> F
    F -->|"weighted by λ_bev"| G["Total loss<br/>classification + regression + BEV"]
    B --> G
    G --> H["Inference: camera branch only<br/>same architecture / inputs / runtime"]

Key Designs¶

1. Perspective-privileged aerial teacher: freezing the overhead view's dense structural prior into a BEV teacher

"Perspective-privileged view" has a concrete meaning here: information that is available during training but unavailable at deployment, which in this paper refers specifically to aerial imagery precisely registered to the ego local coordinate frame. Making it the teacher is worthwhile because structural information in the camera view is spatially non-uniform — dense nearby, sparse and occluded at distance — whereas from above, road layout and connectivity are equally visible across the whole image and remain stable over long ranges. The authors reuse a mapping network trained on AID4AD as the teacher: it follows a ResUNet architecture (ResNet backbone plus U-Net decoder) and produces metrically consistent BEV features from high-resolution aerial imagery, features that already encode lane boundaries, road edges and intersection topology. The teacher is frozen throughout training: it only supplies a feature target and receives no gradient, so its behavior cannot drift and the student regresses toward a stable objective. This also explains why the work differs from LiDAR-camera distillation or diffusion-based refinement — those supervise from within the ego sensing domain, whereas the overhead perspective carries structural information that is stable over large spatial extents, a prior that comes from the viewpoint itself rather than from reprocessing information available to the ego view.

2. Channel normalization + affine adapter: making heterogeneous view features comparable

Aligning two feature maps from entirely different modalities with a pointwise MSE carries an implicit assumption — that their numerical scales are comparable — which does not hold in practice: the aerial branch processes orthorectified RGB imagery and is produced by a convolutional encoder-decoder, while the camera branch processes multi-view images and is aggregated by deformable attention, so their feature statistics and inductive biases differ substantially. One-for-All systematically shows that mismatched feature statistics and inductive biases hinder direct feature transfer in heterogeneous distillation, which explains why naive alignment captures only part of the gain. The remedy is lightweight: before computing the alignment loss, both BEV feature maps are channel-wise normalized to remove the magnitude gap; then an affine adapter is added on the student branch, applying a per-channel scale and shift \(\tilde F_c = \gamma_c F_c + \beta_c\) to the camera features used for supervision, with learnable \(\gamma_c,\beta_c\) per channel. The adapter is active only within the loss path, and at inference it is removed together with the teacher branch, leaving the encoder itself untouched.

The ablation makes the effect of this design directly visible. In the 100×50 m setting, cross-view supervision with neither normalization nor adapter already pushes mAP from 22.4 to 28.9, showing that the aerial signal carries usable structural information despite modality-induced feature shifts; adding normalization jumps to 31.5, showing that the scale discrepancy is the dominant source of mismatch in cross-view alignment; the affine adapter adds a further refinement to 32.3, compensating residual per-channel offsets. More interesting is the explanation provided by feature similarity analysis: as normalization and the adapter are introduced, the median CKA (linear Centered Kernel Alignment, invariant to isotropic scaling and orthogonal transformations, measuring structural rather than pointwise similarity between two feature spaces) between student and teacher stays stable, while R² (how well teacher features can be linearly reconstructed from student features) progressively decreases. In other words, these two components deliberately relax exact pointwise value copying, steering alignment from "imitating specific values" toward "preserving spatial structural correspondence" — consistent with prior findings in heterogeneous distillation that transferable spatial structure matters more than maximizing value reconstruction, and explaining why stronger numerical fitting does not translate into better mAP.

3. Dense BEV alignment loss: moving supervision from the semantic output back into feature space

The third design concerns where the supervision lands. Existing approaches either take the semantic route — attaching a lightweight segmentation head and supervising BEV features with rasterized ground-truth maps — or the in-domain distillation and diffusion-refinement routes, but they either compress dense spatial structure into discrete semantic categories (geometric detail near category boundaries is lost) or remain confined to the ego sensing domain. This paper aligns directly in the shared BEV feature space, with a loss that is simply the element-wise mean squared error between the two feature maps:

\[\mathcal{L}_{\text{bev}}=\frac{1}{CHW}\sum_{c=1}^{C}\sum_{h=1}^{H}\sum_{w=1}^{W}\left(\tilde{F}^{\text{cam}}_{c,h,w}-F^{\text{aerial}}_{c,h,w}\right)^2\]

The supervision signal is therefore per BEV position and per channel dense geometric structure rather than semantic labels that have passed through decoding and quantization. This was tested explicitly: under the same 100×50 m setting, a MapTRv2-style auxiliary loss (lightweight segmentation head with ground-truth-derived targets) raises mAP only from 22.4 to 23.3, and increasing the weight to 3 and 5 gives 22.4 and 22.8 respectively, all far below the 32.3 of CVS. This rules out the explanation that the gain merely comes from adding a BEV-level auxiliary objective — the improvement comes from the cross-view dense structural supervision itself. The authors read it as follows: aerial supervision supplies dense structural guidance that supervision based on discretized semantic map targets cannot reach.

A Worked Example¶

Take a single nuScenes training sample through the pipeline. Six surround-view images enter the BEVFormer-style camera encoder and are aggregated by deformable attention into one \(256\times 50\times 100\) BEV feature map; the ego-aligned aerial crop of the same scene at the same timestamp (provided by AID4AD, already registered to the same local coordinate frame) enters the frozen ResUNet aerial encoder and yields another \(256\times 50\times 100\) feature map. Because both maps share the same ego-aligned BEV grid, location \((h,w)\) naturally corresponds to the same patch of ground in both, and the two can be paired directly with no projection or resampling. The camera features are channel-wise normalized and then passed through the affine adapter to give \(\tilde F^{\text{cam}}\); the squared difference between the two maps is averaged over the full grid to give \(\mathcal{L}_{\text{bev}}\), which is combined with the decoder's classification loss and geometric regression loss, weighted by \(\lambda_{\text{bev}}\). During back-propagation gradients flow only into the camera encoder and the adapter, while the entire aerial teacher branch remains frozen and is not updated. At inference the teacher branch and the affine adapter are both removed, and the forward path collapses back to the original StreamMapNet — the input is still six images, the output is still a vectorized map, and neither the parameter count nor the runtime changes.

Loss & Training¶

The decoder follows StreamMapNet: a classification head trained with Focal Loss (\(\mathcal{L}_{\text{cls}}\)) and a geometry head trained with a line-based L1 loss (\(\mathcal{L}_{\text{reg}}\)). The overall objective extends this baseline with a weighted BEV alignment loss:

\[\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{cls}}+\mathcal{L}_{\text{reg}}+\lambda_{\text{bev}}\,\mathcal{L}_{\text{bev}}\]

\(\lambda_{\text{bev}}\) controls the relative strength of aerial supervision, balancing feature-level guidance against the primary map prediction objectives. Training uses AdamW with an initial learning rate of \(1.25\times10^{-4}\), cosine annealing and a batch size of 4 on a single NVIDIA L40S GPU; the learning rate is adjusted for single-GPU training while all other architectural, training and evaluation settings remain identical to StreamMapNet, so that observed differences can be attributed solely to aerial-guided supervision. The supervision weight is \(\lambda_{\text{bev}}=60\) for the 60×30 m RoI and 70 for the 100×50 m setting.

Key Experimental Results¶

Main Results¶

The dataset is nuScenes with the AID4AD cross-view extension, using the geographically separated split introduced by Roddick and Cipolla and adopted by StreamMapNet (StreamMapNet quantified that the original nuScenes split has roughly 84% geographic overlap between train and test frames, which is why generalization under geographic separation matters). Metrics are per-class and overall mAP over pedestrian crossings (APped), road dividers (APdiv) and lane boundaries (APbound); mAP is averaged over distance thresholds \(\{0.5,1.0,1.5\}\) m for the 60×30 m region and \(\{1.0,1.5,2.0\}\) m for the 100×50 m region. All results are obtained under identical camera-only inference conditions.

RoI	Method	APped↑	APdiv↑	APbound↑	mAP↑	Relative gain
60×30 m	StreamMapNet	32.2	29.3	40.8	34.1	–
60×30 m	StreamMapNet + CVS	40.1	30.3	43.5	38.0	+11% (+3.9)
100×50 m	StreamMapNet	25.6	17.4	24.3	22.4	–
100×50 m	StreamMapNet + CVS	40.3	25.8	30.7	32.3	+44% (+9.9)

Cross-view supervision consistently beats the baseline in both regions, and the gain grows markedly with spatial extent: in the 60×30 m region mAP rises from 34.1 to 38.0 with all three semantic categories improving; in the 100×50 m region it rises from 22.4 to 32.3 (+9.9 absolute), with pedestrian crossings alone gaining 14.7 points. This matches the motivation's claim that ego-centric observations become sparse and structurally most ambiguous at long range.

Ablation Study¶

With everything else fixed in the 100×50 m setting, the role of normalization and the affine adapter (all models use \(\lambda_{\text{bev}}=70\)):

Config	mAP↑	Δ (vs. 22.4 baseline)	Note
w/o normalization, w/o adapter	28.9	+6.5	Direct alignment already helps, but is held back by modality shift
+ normalization only	31.5	+9.1	Correcting the cross-modal scale discrepancy is the single largest step
+ normalization + affine adapter	32.3	+9.9	Full CVS, compensating residual per-channel offsets

Two further controlled comparisons (also 100×50 m):

Axis	Config	mAP	Note
Supervision form	MapTRv2-style auxiliary loss (weight 1 / 3 / 5)	23.3 / 22.4 / 22.8	Lightweight segmentation head with ground-truth targets, far below CVS's 32.3
Supervision form	CVS (ours)	32.3	Dense feature-level cross-view alignment
Teacher architecture	ResUNet (original)	teacher-side 47.3 → student 32.3	Lowest teacher accuracy, best student
Teacher architecture	ResUNet++	teacher-side 52.2 → student 30.5	More accurate teacher, worse student
Teacher architecture	UNet++	teacher-side 55.0 → student 32.2	Roughly on par with the ResUNet teacher

Key Findings¶

The gain grows with the RoI, and this is the paper's most convincing piece of evidence: +3.9 mAP in the standard 60×30 m region versus +9.9 mAP when extended to 100×50 m. Aerial supervision addresses precisely the structural shortfall of ego-centric observations at large extents rather than simply improving nearby predictions.
Normalization is what stabilizes cross-view alignment; the adapter is a fine-grained supplement: 28.9 without normalization → 31.5 with normalization only (+2.6) → 32.3 with the adapter (+0.8). The scale mismatch dominates; per-channel offsets are secondary.
CKA stays stable while R² decreases: the three supervised variants have comparable median CKA, yet R² falls progressively as normalization and the adapter are added. Combined with the mAP ablation, this suggests that preserving spatially aligned activation structure (CKA) matters more than maximizing pointwise value reconstruction (R²), which explains why stronger numerical fitting does not mean better mapping.
Teacher-side accuracy does not predict student gains: the ResUNet++ teacher is 4.9 points more accurate than the original ResUNet teacher, yet its student drops from 32.3 to 30.5; the UNet++ teacher, the most accurate (55.0), yields a student of 32.2, merely on par with the original. Teacher-feature visualizations in the supplementary material offer an explanation — the ResUNet++ teacher produces less localized, higher-frequency activations that transfer less well. Increasing the BEVFormer student encoder from 1 to 2 layers changes little, indicating that student capacity is not the bottleneck; transferability of the teacher representation is.
Relative degradation in rainy scenes is substantially reduced: a preliminary analysis in the supplementary material shows CVS cutting the relative performance drop in rain from 21.2% to 10.0%. This is a notable side effect — perspective-privileged supervision appears to improve robustness when visual evidence degrades.
The supervision weight has to be set per RoI (60 and 70), and the paper reports no weight sweep, leaving that to the reader.

Highlights & Insights¶

The training-inference decoupling of the "privileged view" paradigm: an extra viewpoint available only during training is treated as a source of privileged information and compressed into the student encoder via feature-level alignment, with the whole branch removed at inference. Architecture, inputs and runtime are all unchanged, which means it can be plugged into any existing multi-camera BEV pipeline without new sensors or infrastructure — the most essential advantage over fusion-based approaches.
"No projection needed" is the precondition that makes this supervision work: aerial imagery is already registered to the ego local coordinate frame and the teacher and student emit feature maps on the same BEV grid, so cross-view supervision collapses into a single element-wise MSE with none of the engineering complexity of cross-view geometric transforms. Conversely, feasibility depends heavily on registration quality — the design's Achilles' heel.
The CKA/R² diagnostic pair is transferable: using "stable structural similarity (CKA) plus falling value reconstruction (R²)" to explain why adding an adapter helps turns the vague question of "what should distillation actually imitate" into two measurable dimensions. Any cross-modal or cross-architecture distillation work can reuse this diagnostic to tell whether alignment is learning structure or memorizing values.
A supervision-form control experiment that rules out the trivial explanation: a MapTRv2-style BEV-level auxiliary loss is trained and compared at several weights, proactively closing off the "the gain is just an extra auxiliary task" objection — good experimental design awareness.
The teacher-selection conclusion is counter-intuitive and useful: "the more accurate the teacher, the better the student" does not hold in this cross-view setting; transferability depends on how much spatial structure in the intermediate representation the student can actually receive. This observation is directly useful to anyone pursuing perspective-privileged supervision or cross-modal distillation.

Limitations & Future Work¶

Dependence on precisely ego-aligned aerial imagery (the authors' stated primary limitation). Sources of overhead imagery are not scarce — public orthophotos, commercial satellite imagery, targeted drone captures — but precise metric registration to the ego local coordinate frame is a hard requirement, and AID4AD provides a semi-automatic high-precision alignment workflow for it. Scaling to larger datasets and more deployment regions requires automating that alignment process.
Narrow validation scope: AID4AD currently provides cross-view aligned aerial imagery only for nuScenes, so experiments are limited to that benchmark, and only three map categories (pedestrian crossings, road dividers, lane boundaries) are covered. The authors propose extending to datasets such as Argoverse 2 to cover more sensor setups and geographic environments, but have not done so yet.
Teacher training cost is not quantified: the method needs a pretrained aerial mapping teacher, yet the paper reports neither the cost of training it nor a sensitivity analysis of how registration error affects downstream student performance — both relevant to deployment decisions.
No quantitative report of training overhead: the paper only states that inference is free of overhead, but the teacher branch's extra forward pass and the aerial input during training have memory and per-step time costs that are never given, so readers cannot estimate reproduction cost directly.
Hyper-parameters need manual tuning per RoI: \(\lambda_{\text{bev}}\) is 60 for 60×30 m and 70 for 100×50 m, with no sensitivity sweep, so it is unclear how wide that range is.
The supervision-form comparison is not exhaustive: the contrast with a MapTRv2-style auxiliary loss only tries weights 1/3/5 and does not explore other design axes such as segmentation-head capacity or target encoding; whether "auxiliary losses don't work" still holds over a wider design space deserves re-checking.
Extensions: since perspective-privileged supervision improves the spatial structure of the encoder, could it also serve tasks that likewise rely on BEV spatial reasoning, such as occupancy prediction or motion forecasting? The authors list this as future work, and it is the natural extension of the paradigm.

vs StreamMapNet: this is the baseline, and the method changes nothing in its architecture or training configuration (aside from a single-GPU learning-rate adjustment), only attaching an aerial teacher branch and a feature alignment loss during training, so the gain is cleanly attributable to cross-view supervision. The advantage is plug-and-play, zero inference cost; the drawback is a new dependence on cross-view registered aerial data.
vs MapDistill / DistillBEV / BEV-LGKD: these also "train with a stronger modality, infer with cameras only", but teacher and student live in the same ego sensing domain (LiDAR, or LiDAR-camera fusion). This paper swaps the supervision source for a different viewpoint — structural information from above is unaffected by occlusion and perspective distortion over large spatial extents, giving it long-range global structure that in-domain distillation cannot provide.
vs AID4AD's fusion setup: fusion requires dual encoders and continuous aerial availability at inference; structural completeness and large-scale consistency are better, but deployment is expensive. CVS uses aerial imagery purely as a training signal, trading "one-time data availability during training" for "zero extra inference-time dependency".
vs diffusion-based methods such as BEVDiffuser / MapDiffusion / DifFUSER: those apply diffusion at inference for feature denoising or decoding refinement, remaining inside the ego sensing domain and adding inference cost; all of CVS's gains come from shaping the representation during training and leave the inference path untouched.
vs One-for-All: this paper explicitly inherits its finding that mismatched feature statistics and inductive biases hinder direct transfer in heterogeneous distillation, and uses it as a design rationale — normalization and the affine adapter are exactly the lightweight alignment mechanisms added to mitigate that mismatch.

Rating¶

Novelty: ⭐⭐⭐⭐ The combination of training-time privileged view and dense BEV feature alignment for online mapping is clearly framed and explicitly distinguished from in-domain distillation and auxiliary losses; the individual mechanism is not brand new, but the problem framing and cross-view angle have independent value.
Experimental Thoroughness: ⭐⭐⭐ Main results span two RoIs with ablations on normalization/adapter, supervision form and teacher architecture — focused and on point; but only one dataset, and no quantitative report on training overhead or weight sensitivity.
Writing Quality: ⭐⭐⭐⭐ The motivation chain is clear, and the feature-space analysis (CKA/R²) explains "why this design" at the representation level rather than leaning on mAP alone; some equations are corrupted in the cached text, though the mechanism remains fully recoverable.
Value: ⭐⭐⭐⭐ Training-inference decoupling lets it drop into existing multi-camera BEV pipelines with zero inference overhead, and the two conclusions — teacher-side accuracy does not predict student gains, and structural alignment beats value imitation — are directly informative for broader work on perspective-privileged supervision and cross-modal distillation.