Towards Metric-Agnostic Trajectory Forecasting¶

Conference: ECCV 2026
arXiv: 2607.01133
Code: https://vision.rwth-aachen.de/TraDiE-policies
Area: Autonomous Driving / Trajectory Prediction
Keywords: Trajectory Prediction, Predictive Distribution, Metric Decoupling, Sampling Policy, Probabilistic Training

TL;DR¶

This paper advocates that trajectory forecasting should only learn a "well-calibrated predictive distribution," treating conflicting metrics like minFDE and soft mAP as downstream tasks. By utilizing a set of retraining-free sampling policies (TraDiE policies) during evaluation, the optimal \(K\) trajectories for each metric are extracted from the same distribution, allowing a single DONUT-NLL model to simultaneously achieve SOTA on both distance-based and window-based metrics on Waymo.

Background & Motivation¶

Trajectory forecasting is a core component of the perception-planning pipeline in autonomous driving. Given the road graph and historical states of surrounding traffic participants, the model must predict \(K\) future trajectories (usually \(K=6\)) for each agent and assign a confidence score to each. Ideally, these \(K\) trajectories should faithfully reflect the future multi-modal uncertainty. However, the authors observe an awkward reality: current SOTA models are heavily trained specifically for metrics. Methods evaluated on Argoverse 2, where the primary metric is Brier-minFDE (focusing only on endpoint distance), tend to use the winner-takes-all (WTA) strategy to pull the closest mode toward the ground truth, making the training loss a virtual copy of minFDE. Conversely, methods evaluated on Waymo, where the primary metric is soft mAP (assessing coverage of the future space), output 64 candidates and employ Non-Maximum Suppression (NMS) to scatter clustered endpoints. Some even parameterize the NMS radius to vary with velocity to fit Waymo's velocity-dependent rectangular window. Some works simply train multiple models to target different metrics separately.

The core issue is that these two types of metrics reward conflicting behaviors. Distance-based metrics award high scores as long as a single trajectory hits the ground truth, which encourages models to cluster predictions around the main modes, even if redundant. Window-based metrics care about coverage; only the highest-confidence prediction within a window counts as a hit, while the rest are either ignored (soft mAP) or penalized as false positives (mAP), which encourages scattering the predictions. Clustering benefits minFDE but hurts coverage, whereas scattering benefits mAP/miss rate but increases distance error—no single model can satisfy both. A deeper complication is that these benchmark metrics do not directly evaluate the predictive distribution itself. As a result, it is unclear whether performance gains stem from "better predictions" or "aggressive metric tuning," and models become rigidly tied to specific evaluation protocols, hindering transferability to new scenarios.

The core insight of this paper is to completely decouple the training target from the evaluation metrics. During training, the model is only required to output a well-calibrated predictive distribution that faithfully represents uncertainty—this is the most general representation. During evaluation, selecting \(K\) trajectories for a specific metric is treated as a post-processing step applied to this distribution. The core idea is to train the model to learn the predictive distribution using a metric-agnostic probabilistic loss (Negative Log-Likelihood, NLL), and then design a Monte Carlo sampling-based evaluation policy (TraDiE policy) for each of minFDE, (soft) mAP, and miss rate. This allows extracting the optimal \(K\) trajectories and confidence scores for each metric from the exact same distribution without retraining. If the distribution is well-calibrated, these policies will yield excellent performance across any metric; conversely, if the distribution is poorly learned, the policies will expose this miscalibration.

Method¶

Overall Architecture¶

This paper addresses the issue of models being hijacked by evaluation metrics and unable to handle multiple metrics with a single distribution. The general framework consists of two parts: On the training side, the original WTA distance loss of DONUT is replaced with direct optimization of the negative log-likelihood (NLL) of the mixture distribution, forcing the model to focus on learning a well-calibrated predictive distribution (resulting in DONUT-NLL). On the evaluation side, a TraDiE policy is designed for each benchmark metric. This policy draws a batch of Monte Carlo samples from the predicted (endpoint) distribution and "solves" for the \(K\) trajectories and their confidence scores based on the metric definition. Crucially, with a single training run generating a single distribution, switching between the distance policy for minFDE and the window policy for soft mAP/mAP/miss rate during evaluation is sufficient—the model weights remain completely frozen. The authors also investigate the choice of distribution families (Laplace, Generalized Gaussian, and Gaussian Scale Mixture) as a design dimension, finding that the Generalized Gaussian distribution performs the best.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Road Graph + Agent History"] --> B["DONUT-NLL Probabilistic Training<br/>Directly learn predictive distribution via NLL"]
    B --> C["Predicted Endpoint Distribution p(x_T)"]
    C --> D["Monte Carlo Sampling<br/>|S|=3000 endpoint samples"]
    D -->|Distance-based Metrics| E["minFDE Policy<br/>Gradient descent optimization of K endpoints"]
    D -->|Window-based Metrics| F["(soft) mAP / Miss Rate Policy<br/>Greedy selection of endpoints covering the most windows"]
    E --> G["K=6 Trajectories + Confidences<br/>Submitted to benchmark"]
    F --> G

Key Designs¶

1. Metric-Agnostic Probabilistic Training: Replacing WTA Distance Loss with Direct NLL Optimization

To score high on Argoverse 2's distance metrics, the original DONUT uses WTA: it selects the mode \(k_n^*\) closest to the ground truth based on average Euclidean distance and optimizes the negative log-likelihood only for that mode. This effectively mirrors minFDE as the training loss. While this yields accurate endpoints, it causes modes to cluster together, leading to terrible window-based metrics (e.g., when native DONUT is evaluated on Waymo, its soft mAP is extremely poor). This paper instead bypasses the "winner selection" and optimizes the full mixture distribution against the ground truth via NLL. The authors present two variants: Traj-NLL shares the mixture weights \(\pi_{nk}\) across all time steps, meaning each component \(k\) represents a complete global trajectory candidate:

\[-\log p(\bm{Y})=-\sum_{n}\log\sum_{k=1}^{K}\pi_{nk}\prod_{t=1}^{T}p_{nkt}(\bm{y}_{nt})\]

Step-NLL is more aggressive, defining a separate set of weights \(\pi_{nkt}\) for each time step. This allows the importance of different modes to evolve dynamically over time, enhancing expressiveness (at the cost of weakening implicit temporal step coupling). This modification is effective because NLL forces the model to faithfully model "how uncertain the future actually is" rather than heuristically squeezing points near the ground truth. Experiments also confirm that when paired with the proposed policies, directly optimizing NLL outperforms optimizing surrogate WTA losses.

2. minFDE Distance Policy: Sampling from the Distribution and Placing \(K\) Endpoints via Gradient Descent

The distance metric minFDE only measures "how close the nearest model prediction among the \(K\) endpoints is to the ground truth." This paper formalizes it as an expectation under the predictive distribution: given the endpoint distribution \(p_{nT}\), the chosen \(K\) endpoints should minimize \(\mathbb{E}_{\bm{x}_{nT}\sim p_{nT}}[\min_k\|\hat{\bm{x}}_{nkT}-\bm{x}_{nT}\|_2]\), which represents the "expected minFDE of these endpoints assuming the ground truth is sampled from the model's distribution." Since a closed-form solution is intractable, the authors approximate this expectation via Monte Carlo sampling with \(|S|=3000\) endpoints from the distribution. They then treat the \(K=6\) endpoints as free variables and directly apply Adam for 300 gradient descent steps to minimize the empirical target, repeating this process with 10 random initializations to find the best result. The beauty of this approach is that it "aligns perfectly with the metric definition" without relying on model-specific heuristics. Consequently, it acts as a diagnostic test: if the distribution is miscalibrated (such as QCNet, as discussed later), the endpoints chosen by this policy perform poorly on real data, immediately exposing the miscalibration.

3. (soft) mAP / Miss Rate Window Policy: Greedily Selecting Endpoints that "Cover the Most Windows"

Window-based metrics measure coverage: a window (a rectangle that varies with orientation and velocity in Waymo) is placed around the ground-truth endpoint, and only the highest-confidence prediction that falls inside the window counts as a true positive. Computing the globally optimal AP is difficult (as it depends on global sorting across all agents and scenarios). Thus, the authors design a per-agent heuristic: they first sample a candidate pool of endpoints from the predictive distribution, associate an evaluation window with each candidate, and then greedily select endpoints that "are covered by the most windows." Once an endpoint is chosen, the covered windows are deactivated to prevent duplicate counting. Because ground-truth endpoints are unavailable at inference time, samples \(\tilde{\bm{x}}_{nT}\) from the distribution are used as surrogates for the ground truth to estimate the probability of "a selected endpoint being a true positive," denoted as \(\overline{\text{TP}}_{nk}\). The endpoints are chosen as:

\[\hat{\bm{x}}_{nkT}^{\text{pos}}=\operatorname*{arg\,max}_{\bm{x}_{nT}\in S}\overline{\text{TP}}_{nk}(\bm{x}_{nT}^{\text{pos}})\]

and this probability is directly used as the endpoint's confidence. This greedy approach aligns naturally with the three requirements for improving AP: prioritizing points most likely to act as "highest-confidence true positives" pushes true positives higher in the global ranking, boosting precision at various thresholds; it serves as a standard greedy approximation for the maximum coverage problem, thereby increasing recall by covering more unvisited windows; and the "deactivation of hit windows" naturally avoids wasting multiple predictions inside the same window. Since the optimal policy for miss rate is likewise to "have \(K\) endpoints cover as many windows as possible" (maximizing \(\sum_k\overline{\text{TP}}_{nk}\)），this single window policy is universally applicable to soft mAP, mAP, and miss rate.

4. Positional Distribution Families: Generalized Gaussian Outperforms Laplace and Gaussian Scale Mixture

Since evaluation has been completely decoupled from training, the choice of distribution to represent agent positions becomes a design choice that can be systematically compared. While the original DONUT used 1D Laplace, this paper treats distribution family as a plug-and-play component, comparing Generalized Gaussian and Gaussian Scale Mixture (\(J=5\)). The Generalized Gaussian features an extra shape parameter \(\beta\) (\(\beta=1\) corresponds to Laplace, \(\beta=2\) to Gaussian), offering a more flexible density shape. Interestingly, "raw evaluation" (without policies) and "policy-based evaluation" exhibit different preferences: in raw evaluation, Laplace achieves the best minFDE because its sharper peak pulls modes closer to the ground-truth endpoint, while the Gaussian Scale Mixture gets the best raw soft mAP due to its smoother, wider coverage. However, once policies are applied, Generalized Gaussian consistently wins across all metrics. This indicates that its additional shape flexibility helps the model learn a "better-calibrated predictive distribution that is easier to exploit by downstream policies," rather than merely catering to a specific raw metric.

Loss & Training¶

DONUT-NLL retains DONUT's architecture and most training hyperparameters, only replacing the loss with Traj-NLL or Step-NLL. A valuable improvement is rotating the 1D positional distributions from "aligned with the global x/y axes" to "aligned with the agent's local longitudinal (lg) and lateral (lt) directions" (using the predicted average heading), which aligns positional uncertainty better with kinematics. Heading is still modeled with a von Mises distribution. Key strategy hyperparameters include: all Monte Carlo sets use \(|S|=3000\) samples; the minFDE policy uses Adam for 300 steps with a learning rate of 0.2 and 10 random restarts. Since Waymo evaluates at 3s/5s/8s horizons and averages them, the authors run the policies on each horizon separately, pair the endpoints with matching confidence ranks across horizons to form a trajectory, and use the confidence of the 8s horizon (the longest, hardest, and most informative horizon) as the entire trajectory's confidence score.

Key Experimental Results¶

Main Results¶

Evaluated on the Waymo motion prediction benchmark (~487k training scenarios, \(K=6\), 8s horizon). The core table below compares training objectives on Waymo validation (arrows \(\rightarrow\) represent "before policy \(\rightarrow\) after policy"):

Training Objective	Soft mAP ↑	mAP ↑	Miss rate ↓	minFDE ↓
WTA	0.3427 → 0.4764	0.2995 → 0.4736	0.1310 → 0.1044	1.0520 → 1.0824
Traj-NLL	0.3401 → 0.4921	0.2851 → 0.4892	0.1286 → 0.0946	1.0874 → 1.0374
Step-NLL	0.3595 → 0.5082	0.3163 → 0.5053	0.1444 → 0.0902	1.2236 → 1.0200

Both NLL variants show substantial improvements across all metrics after applying the policies, with Step-NLL leading across all four dimensions. Notably, applying the distance policy to WTA actually degrades minFDE from 1.0520 to 1.0824. Because WTA directly optimizes for minFDE instead of calibrating the distribution, the policy fails to find high-quality endpoints in this miscalibrated distribution, directly validating the claim that "policies expose miscalibration."

Comparison with SOTA on Waymo test, where a single DONUT-NLL (Step-NLL + Generalized Gaussian) uses two different policies:

Model	Soft mAP ↑	mAP ↑	Miss rate ↓	minFDE ↓
IMPACT (e2e)	0.4434	0.4253	0.1274	1.0497
ModeSeq	0.4487	0.4450	0.1244	1.0836
IMPACT (ensemble)	0.4801	0.4598	0.1087	1.1295
DONUT-NLL (Window Policy)	0.5018	0.4987	0.0900	1.3280
DONUT-NLL (Distance Policy)	0.1649	0.1135	0.1199	1.0304

Using the exact same weights, the window policy yields soft mAP, mAP, and miss rate that outperform even ensemble-based methods. Meanwhile, the distance policy achieves a minFDE superior to IMPACT (e2e) with roughly 5x fewer parameters. While existing methodologies are typically strong in only one metrics category, ours achieves SOTA in both from a single model.

Ablation Study¶

Positional distribution comparison (Step-NLL, Waymo val, showing before \(\rightarrow\) after policy):

Positional Distribution	Soft mAP ↑	mAP ↑	Miss rate ↓	minFDE ↓	Remarks
Laplace	0.3595 → 0.5082	0.3163 → 0.5053	0.1444 → 0.0902	1.2236 → 1.0200	Native DONUT distribution
Scale Mixture	0.3837 → 0.5053	0.3533 → 0.5024	0.1492 → 0.0909	1.2722 → 1.0218	Best raw soft mAP
Gen. Gaussian	0.3759 → 0.5101	0.3439 → 0.5070	0.1561 → 0.0876	1.3127 → 1.0103	Best overall after policy

Cross-model transferability (MTR / QCNet, Waymo val): All metrics for MTR steadily improve after applying our policies, even outperforming its native NMS post-processing, and changing the loss has minimal impact. In contrast, running the distance policy on QCNet causes its minFDE to collapse from 1.2254 to 2.2805.

Key Findings¶

Step-NLL is the best overall training objective, but this only becomes apparent after applying the policies—in raw evaluation, WTA achieves the best minFDE. This highlights that the design choices favored by "optimizing distribution" versus "optimizing metrics" are fundamentally different: from a distribution standpoint, WTA is a sub-optimal choice.
Policies act as a "distribution calibration diagnostic." QCNet's Laplace distribution is aligned with the final historical heading, which tends to misalign with the actual trajectory on curves. To compensate, the model expands the uncertainty into a near-circular shape, scattering the minFDE-optimal endpoints widely. The distance policy directly amplifies this miscalibration into a collapsed minFDE, whereas window-based metrics remain relatively unaffected because the primary mass is still centered near each mode.
The extra shape freedom in the Generalized Gaussian translates into a better-calibrated distribution: although it does not rank first on any raw metrics, it dominates across all metrics after applying the policies. This suggests that the performance gain comes from improved distribution quality rather than overfitting to a specific raw metric.

Highlights & Insights¶

The decoupled paradigm of "learning distributions during training, extracting trajectories during evaluation" is highly elegant. By relegating benchmark metrics (previously treated as training objectives) to evaluation-time post-processing, it elegantly solves the long-standing conflict between "distance" and "coverage" metrics that historically forced researchers to train separate models. This mindset is transferable to any task where evaluation metrics do not perfectly align with the true objective.
The "policy-as-diagnostic" concept is a key "aha!" moment: the same set of policies can either extract SOTA performance or expose hidden defects when the distribution is miscalibrated (e.g., QCNet's minFDE collapse). This separates "high benchmark scores" from "genuinely high-quality distributions," two concepts long conflated in the field.
The minFDE policy formulates the selection of \(K\) endpoints as a differentiable objective solved via gradient descent, while the window policy leverages a greedy approximation of the maximum coverage problem. Both conform strictly to the metric definitions without model-specific tricks, making them truly model-agnostic—yielding out-of-the-box improvements even over MTR's native NMS.

Limitations & Future Work¶

The authors acknowledge that the primary drawback is the additional computational overhead during evaluation: the minFDE policy takes ~0.26s per scenario, and (soft) mAP takes ~0.03s. Since these policies are designed solely for evaluation and not intended as deployable planning stack components on vehicles, efficiency was not optimized. Real-world deployment will require an explicit trade-off between quality and speed.
The evaluation results are mainly demonstrated on Waymo. Preliminary experiments on Argoverse 2 suggest that Step-NLL is prone to overfitting (possibly due to the extra degrees of freedom introduced by step-wise mixture weights), leaving its generalizability an open question.
The endpoints produced by the minFDE/mAP policies are temporally discontinuous (the authors stitch temporal horizons together and interpolate intermediate steps). Downstream planning/control modules requiring \(K\) temporally continuous optimal trajectories cannot directly utilize these policies. However, the authors' stance is that once a well-calibrated distribution is provided, downstream tasks can define their own application-specific policies.
The current policies only process endpoint marginal distributions for a single agent. Extending this framework to joint multi-agent forecasting and continuous perception-aware objectives remains a key direction for future research.

vs HOME: HOME similarly decouples prediction by first learning an endpoint distribution (dense heatmap) and then using metric-specific post-processing. However, its post-processing is tightly bound to its discrete endpoint representation and relies on model-specific heuristics (e.g., using a modified k-means for minFDE, and using a 1.8m instead of 2m standard window to artificially boost its miss rate scores). In contrast, the sampling-based policies in this paper are derived directly from metric definitions, are applicable to any probabilistic predictor, and strictly stick to standard window parameters—attributing performance gaps purely to distribution quality rather than hidden hyperparameters in selection policies.
vs WTA-based methods (DONUT / QCNet / HiVT, etc.): These methods employ winner-takes-all loss to optimize only the closest mode, making the training loss a surrogate for minFDE. This causes predictions to cluster, leading to poor window-based metrics. This paper optimizes the entire distribution using NLL; once paired with the proposed policies, minFDE remains competitive while other metrics dramatically improve.
vs NMS-based methods (MTR / MTR++ / RMP-YOLO, etc.): These methods output up to 64 candidates and utilize NMS to scatter endpoints to cater to soft mAP, sometimes making the NMS radius velocity-dependent to match Waymo's windows, which sacrifices minFDE. Instead of modifying training for better coverage, this paper allows the window policy to retrieve the optimal covering endpoints at inference, while the same distribution can be switched to the distance policy to target minFDE.
vs ModeSeq: ModeSeq closely aligns its training objectives with the hit criteria of window-based metrics (early-match-takes-all), further biasing the model toward benchmark metrics. This paper takes the opposite route, advocating for metric-independent training that focuses entirely on modeling the complete predictive distribution.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Proposes a paradigm shift of "treating metrics as downstream tasks and training only to learn distributions" alongside hands-on sampling policies. It offers a fresh yet practical perspective.
Experimental Thoroughness: ⭐⭐⭐⭐ Solid ablation studies on training targets and distribution families on Waymo, along with cross-model validation on MTR/QCNet. However, the evaluation is limited to Waymo, and over-fitting issues have already surfaced in preliminary Argoverse 2 trials.
Writing Quality: ⭐⭐⭐⭐⭐ Well-structured motivation with thorough analysis of metric conflicts. The mathematical derivations of policies and the concept of "policies as diagnostics" are clearly and persuasively presented.
Value: ⭐⭐⭐⭐⭐ Achieves simultaneous SOTA on both metric categories using a single model, and provides the community with a new evaluation protocol based on distribution quality rather than metric tuning. It holds significant practical and methodological value.