Beyond Imitation: Learning Safe End-to-End Autonomous Driving from Hard Negatives¶

Conference: ECCV 2026
Paper: Official paper page · PDF
Code: https://github.com/wjl2244/BeyondDrive
Area: Autonomous Driving
Keywords: Imitation learning, hard negatives, flow matching, safety boundaries, trajectory planning

TL;DR¶

BeyondDrive trains a planner to approach expert trajectories while avoiding generated trajectories that are nearby but unsafe, improving optimized LTF from 88.7 to 89.7 PDMS on NAVSIM v1 and from 89.1 to 90.1 EPDMS on NAVSIM v2, with additional evidence from other planners and zero-shot HUGSIM evaluation.

Background & Motivation¶

End-to-end driving commonly treats planning as trajectory regression: given sensor observations and a navigation command, a prediction receives a smaller training loss when it stays closer to the expert demonstration. Geometric deviation, however, is not risk. Similar deviations in opposite directions can have different consequences: one stays inside the lane, while the other crosses a road boundary or approaches a vehicle. The paper calls this safety asymmetry. Successful demonstrations alone provide limited information about which side of the expert trajectory tolerates deviations and which side does not.

DiffusionDrive, MeanFuser and WoTE address some limitations of single-trajectory regression through multiple candidates and scoring. Yet discarding a low-scoring trajectory at inference does not itself make the planner learn from that failure. This paper focuses on local boundary learning: an obviously bad trajectory far from the expert may contribute little, whereas a nearly correct imitation that crosses a safety boundary provides a more informative contrast. Simply drawing many samples does not solve this problem either, because a generator can concentrate its outputs around a few high-probability modes.

Core idea: first filter diverse candidates by low safety score, then select the one nearest to the expert, and train with joint expert attraction and bounded failure repulsion to learn safety boundaries in trajectory space.

Method¶

Overall Architecture¶

BeyondDrive is a training framework, not a new online obstacle-search routine that must run in the vehicle. Its first stage uses conditional flow matching, diversity-aware sampling and two-stage negative selection to construct a hard negative for a scene. Its second stage adds these negatives to the existing planner's training. Deployment uses the safety-enhanced planner; negative generation and selection belong to training-data construction.

The output is a future trajectory in ego coordinates over 4 seconds: 8 waypoints at 2 Hz, each containing position and heading. The conditional generator uses the front-view image, ego velocity and acceleration, and navigation command, with the same perception module as LTF. The downstream planner can be camera-only LTF or TransFuser with additional LiDAR input. These downstream sensor configurations should not be conflated with the generator's stated inputs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Scene context and<br/>expert trajectory"] --> GEN["Conditional flow matching"]
    GEN --> DIV["Diversity-aware sampling<br/>64 candidates"]
    DIV --> SELECT["Two-stage negative selection<br/>Nearest among low scores"]
    SELECT --> TRAIN["Clipped repulsive training"]
    INPUT -->|Expert imitation supervision| TRAIN
    TRAIN --> OUTPUT["Safety-enhanced planner<br/>8 future waypoints"]

Key Designs¶

1. Conditional flow matching: learn the driving distribution before mining failures

The generator is trained toward expert trajectories, not recorded accident trajectories. It linearly interpolates between Gaussian noise and a vectorized expert trajectory, then learns the velocity field that moves noise toward the trajectory. A scene encoder supplies context, while a Transformer Decoder, LayerNorm and feed-forward network form the velocity network. Importantly, the method uses L1 velocity regression rather than the conventional squared error introduced in the preliminary section. This produces scene-aware candidates near expert behavior, but does not by itself ensure that they are unsafe.

During training, conditions are dropped with probability 0.1, allowing the same network to learn conditional and unconditional velocity estimates. This enables classifier-free guidance (CFG) at sampling time without separately training an unconditional model. The following equations retain the mechanism-defining relationships. Equation extraction in the cached text is damaged: these are readable reconstructions grounded in the adjacent prose of Section 4, not verbatim equation transcriptions; consult the PDF for exact notation.

\[ x_t=(1-t)x_0+tx_1,\qquad \mathcal{L}_{\mathrm{FM}}= \mathbb{E}\left[\left\|v_\theta(x_t,t\mid\mathcal{C}_{\mathrm{in}})-(x_1-x_0)\right\|_1\right]. \]

Here \(x_0\) comes from the Gaussian prior, \(x_1\) is the vectorized expert trajectory, and \(t\) is sampled uniformly from \([0,1]\); \(\mathcal{C}_{\mathrm{in}}\) denotes retained or dropped scene context. Intuitively, the generator first learns how vehicles typically move in a scene, and subsequent sampling and selection mine failures from that neighborhood. This differs from adding arbitrary noise to an expert trajectory and immediately labeling the result unsafe.

2. Diversity-aware sampling: weaken conditioning while expanding initial noise

A well-fitted conditional generator can repeatedly produce similar safe trajectories. Without varied candidates, it cannot expose different failure directions around the expert. The authors therefore combine a lower CFG weight with larger sampling noise: the training prior has standard deviation \(\sigma\), while negative generation uses \(2.0\sigma\). With \(w=0.5\), the generator samples 64 candidates per scene in parallel using 5 flow-matching sampling steps to solve the ODE.

\[ \widetilde v_\theta(x_t,t\mid\mathcal C) =v_\theta(x_t,t)+w\left[v_\theta(x_t,t\mid\mathcal C)-v_\theta(x_t,t)\right]. \]

\(w=1.0\) recovers ordinary conditional generation, whereas \(w=0.5\) reduces concentration on the conditional modes. This is deliberately different from the common use of guidance above 1 to strengthen conditioning. Larger noise is not the end goal either: it expands exploration, after which irrelevant candidates must still be filtered. Valid-negative scene coverage rises from 8.0% to 78.8% and then to 96.4%, illustrating that sample count and failure-mode coverage are distinct; 64 nearly repeated trajectories can still be poor supervision.

3. Two-stage negative selection: require low scores before seeking proximity

Every candidate is evaluated by a safety and driving-quality function \(S\), instantiated as PDMS. It considers collisions, drivable-area compliance, time-to-collision, comfort and progress. The procedure first retains candidates satisfying \(S(\tau)<\xi\), then selects the one closest to the expert trajectory, using Euclidean distance accumulated over waypoints. The order matters: nearest-neighbor selection alone may return another safe demonstration, while selecting the lowest score alone may return an easy negative far from normal driving.

Hardness here means proximity to the expert under a low-score constraint, not simply a worse score. PDMS is also a composite driving metric: a low score is not logically equivalent to a collision, so not all selected negatives should be described as accident trajectories. The scoring function provides an operational definition for supervision, without proving that it captures every real-world safety risk.

The readable body does not specify the threshold \(\xi\) or what happens when the low-score set is empty. Valid-negative coverage of 96.4% does not imply that every scene has a negative; skipping scenes, resampling and loss masking are therefore not implementation details that can be supplied by assumption. The physical executability of generated candidates and the scorer's reliability for longer interactions also remain prerequisites for useful supervision.

4. Clipped repulsive training: discourage proximity to failure without rewarding unlimited escape

The downstream planner represents each waypoint through first-order position differences and heading sine/cosine: \((\Delta x_k,\Delta y_k,\sin h_k,\cos h_k)\). Displacements describe movement over 0.5 seconds, rather than unscaled velocity. Sine and cosine avoid angle-wrap discontinuities. Both imitation and repulsion operate in this representation, so geometric neighbor selection during generation and differential L1 distance during training are different quantities and should not be conflated.

\[ \mathcal{L}_{\mathrm{imi}}=\left\|\Delta\hat\tau-\Delta\tau^{\mathrm{exp}}\right\|_1, \qquad \mathcal{L}_{\mathrm{rd}}=-\min\left(\left\|\Delta\hat\tau-\Delta\tau^{\mathrm{neg}}\right\|_1,C\right). \]

Minimizing negative distance pushes the prediction away from the negative, but only until the clipping constant \(C\) is reached. Beyond that bound, repulsion stops encouraging further deviation, while imitation continues to attract the output toward expert behavior. This is not ordinary label separation in an embedding space and does not introduce a new inference-time scorer: it directly changes the gradients applied to predicted trajectories. The resulting separation is empirical, not a formal safety guarantee.

A Worked Example¶

One example in Figure 3 is an intersection where the vehicle should stop. LTF* instead predicts forward motion, receiving NC and TTC scores of 0. The training mechanism can be understood as follows: generate 64 candidates for the same scene, filter them by low PDMS, and select the failure nearest to the expert stopping trajectory. Expert supervision attracts the planner toward stopping, while negative supervision identifies the danger of the local deviation toward continued forward motion.

The planner still outputs a 4-second, 8-waypoint trajectory; it has not gained an online emergency-braking rule. The paper does not report how many candidates survived filtering in this example, their precise spacing or their speeds, so none are supplied here. The example illustrates the mechanism but does not replace large-scale collision-risk evaluation.

Loss & Training¶

The overall objective is a weighted sum of imitation, RD and semantic auxiliary losses. The paper initially calls the map-reconstruction auxiliary term \(\mathcal L_{\mathrm{map}}\), but uses \(\mathcal L_{\mathrm{sem}}\) in the total objective. This is a notation inconsistency: rasterized HD-map auxiliary supervision is described, but there is insufficient evidence to invent two separate auxiliary heads. The readable body does not clearly list the exact loss weights or clipping constant \(C\).

Training uses batch size 32 for 100 epochs, cosine annealing, 3 warmup epochs and peak learning rate \(2\times10^{-4}\). Tuning also reduces auxiliary-task loss weights. The text accompanying Figure 4 reports improvements as the RD weight increases, provided that it remains below the imitation weight; otherwise training fails to converge. Exact curve values are not recoverable from the cached text, so no optimal weight ratio is asserted.

Key Experimental Results¶

Main Results¶

NAVSIM v1 contains approximately 85k navtrain scenes and 12k navtest scenes, and evaluates short-horizon, non-reactive pseudo-simulation. NAVSIM v2 uses traffic agents that respond to the ego vehicle. PDMS multiplies NC and DAC by a weighted mean of EP, TTC and comfort with weights 5:5:2. EPDMS additionally covers driving direction, traffic lights, lane keeping and historical/extended comfort. These are different metrics; their values should not be compared directly across benchmark versions.

The table below selects results from Tables 1, 2 and 4. Higher is better throughout; C means camera and C+L means camera with LiDAR. Differences are score points, not relative percentages.

Model	Input	NAVSIM v1 PDMS	NAVSIM v2 EPDMS
TransFuser	C+L	84.0	84.4
TransFuser v7	C+L	89.6	90.0
LTF	C	83.8	83.6
LTF*, tuning and trajectory normalization	C	88.7	89.1
LTFv7, LTF* + BeyondDrive	C	89.7	90.1
DiffusionDrive	C+L	88.1	88.3
DiffusionDrive + BeyondDrive	C+L	89.2	88.9
MeanFuser	C	89.0	89.6
MeanFuser + BeyondDrive	C	90.3	90.5
WoTE	C+L	88.3	87.6
WoTE + BeyondDrive	C+L	89.2	88.7

On zero-shot interactive closed-loop HUGSIM evaluation, LTF to LTFv7 improves overall RC from 41.4 to 46.2 and HD-Score from 24.8 to 34.8. RC denotes route completion; HD-Score multiplies RC by a time-averaged safety and comfort score. At Hard difficulty, RC actually falls from 36.9 to 35.5, although HD-Score rises from 19.8 to 26.3. Safety-related gains therefore do not imply improved progress at every difficulty.

Ablation Study¶

Table 4 separates engineering improvements from BeyondDrive's incremental contribution. This is the most important comparison for attribution.

Progressive LTF configuration	PDMS	EPDMS	Incremental PDMS gain
Original baseline	83.8	83.6	Not applicable
Add learning-rate and loss-weight tuning	87.4	87.8	+3.6
Add first-order trajectory normalization, LTF*	88.7	89.1	+1.3
Add BeyondDrive, LTFv7	89.7	90.1	+1.0

The sampling ablation from Table 5 follows; all rows use 64 candidates. Valid-negative coverage is the proportion of scenes yielding a valid negative. Negative distance is the average distance from the selected negative to the expert; its unit is not clearly specified in the cache, so it is not labeled in meters. Negative PDMS is the selected negatives' average score, while the final column evaluates the downstream LTF planner.

CFG weight	Sampling standard deviation	Valid-negative coverage	Negative distance	Negative PDMS	Downstream PDMS
1.0	\(1.0\sigma\)	8.0%	0.40	23.9	88.8
0.5	\(1.0\sigma\)	78.8%	1.83	22.8	89.1
0.5	\(2.0\sigma\)	96.4%	1.68	19.8	89.7

Key Findings¶

Of the total 5.9 PDMS gain from original LTF to LTFv7, tuning contributes 3.6, normalization 1.3 and BeyondDrive 1.0. LTF* is the appropriate primary baseline for the new method, rather than attributing the entire gain to negative learning.
Diversity enhancement does more than generate distant mistakes: increasing sampling noise raises coverage from 78.8% to 96.4%, while the selected negative distance falls from 1.83 to 1.68. This supports expanding the candidate pool before mining nearby failures.
Cross-architecture PDMS gains are +1.1 for DiffusionDrive, +1.3 for MeanFuser and +0.9 for WoTE. However, NC falls from 98.6 to 98.5 for MeanFuser and from 98.5 to 98.4 for WoTE, so not every safety submetric improves monotonically.

Highlights & Insights¶

Failure supervision near expert behavior targets the actual error regime of imitation learning more directly than adding extreme negatives. The reusable principle is to require failure first and then find the closest counterexample, rather than copying a particular scoring function.
CFG below 1 serves failure-mode exploration here, rather than the conditional fidelity usually emphasized when displaying generated samples. The most plausible sample and the most useful training sample need not optimize the same objective.
Clipped negative distance gives repulsion a stopping condition, preventing escape from negatives from overwhelming attraction to the expert. Integration into continuous planners is relatively simple, but generation, scoring and storage remain real costs; adding one loss term does not make the entire pipeline free.

Limitations & Future Work¶

The authors explicitly acknowledge that long-horizon interaction and high-level semantic constraints are not modeled. A 4-second trajectory target cannot fully cover negotiated yielding or long-term traffic efficiency; interactive, longer-horizon negatives deserve study beyond increasing short-trajectory sample counts.
Vocabulary-based selectors such as VADv2 require an additional trajectory-refinement module after scoring to enable this safety training. The method therefore does not integrate into every discrete planner without architectural changes.
Reproducibility details are incomplete: equation extraction is damaged, algorithm/supplement references appear as "??", and the readable text does not clearly specify the selection threshold, empty-set handling, clipping constant or exact loss weights. Equations above are grounded reconstructions from the prose, and missing parameters have not been invented.
Some reported claims require caution: the text says both NAVSIM v2 variants exceed 90, but Table 2 lists TransFuser v7 at 90.0. In HUGSIM Table 3, MeanFuser's Overall values repeat the VAD row and appear inconsistent with its difficulty-level results. This note neither repairs that row speculatively nor uses it to claim an overall HUGSIM ranking against all models.
The paper provides no readable multi-seed error bars, real-road deployment results or formal safety guarantee. HUGSIM also lacks a separate LTF* comparison, preventing isolation of BeyondDrive's contribution to the total transfer gain. Strong-baseline transfer tests, variance, scorer transfer and generation-cost analysis remain useful next steps.

Versus TransFuser / LTF: The underlying end-to-end planner is retained, but supervision expands from successful demonstrations to local failures. The strong baseline in Table 4 underscores the need to optimize ordinary training before attributing improvements to a new signal.
Versus DiffusionDrive / MeanFuser / WoTE: These methods emphasize diffusion generation, flow generation and world-model evaluation, respectively; BeyondDrive can supplement their training. The distinction is whether failure changes planner parameters, rather than merely being filtered at inference.
Versus DriveDPO / TakeAD: As characterized in this paper's related work, these approaches primarily use pre-collected or predefined preference/takeover trajectory sets. BeyondDrive actively generates low-scoring trajectories near expert behavior and applies repulsion to continuous outputs. Mixing real takeover failures with generated negatives is a concrete way to test whether the two sources complement each other.

Rating¶

Novelty: 4/5. Expert-proximate failure mining and clipped repulsion form a targeted combination, although the underlying generative and contrastive tools are established.
Experimental Thoroughness: 4/5. Two NAVSIM versions, cross-architecture ablations and zero-shot HUGSIM provide breadth, but variance, full costs and strong-baseline transfer comparisons are missing.
Writing Quality: 3/5. The central problem and examples are clear; unresolved references, missing parameters and table/text inconsistencies reduce reproducibility.
Value: 4/5. The method supplies a useful safety-training signal beyond an optimized baseline, warranting further validation rather than being treated as a deployable safety guarantee.