Unveiling Transferability in Trajectory Prediction via Latent Scene Embeddings¶

Conference: ECCV2026
arXiv: 2606.30777
Code: https://github.com/westny/transferatlas
Area: Autonomous Driving
Keywords: Trajectory Prediction, Transfer Learning, Dataset Embedding, KL Divergence, Cross-domain Generalization

TL;DR¶

This paper proposes the TransferAtlas framework, which employs a Graph Neural Network encoder to map 24 trajectory prediction datasets jointly into a 32-dimensional unified latent space. In this space, each dataset is represented as a Gaussian distribution. The KL divergence between two distributions quantifies the difficulty of transferring from the source domain to the target domain. This achieves a Spearman rank correlation coefficient of 0.811 for zero-shot transfer, offering principled guidance for dataset selection, pre-training source recommendation, and foundation model construction.

Background & Motivation¶

Trajectory prediction is a core task in autonomous driving and robotics. Recently, deep learning advancements and the emergence of various open-source datasets (such as ETH/UCY, Argoverse, nuScenes, WOMD, etc.) have driven significant progress in data-driven prediction models. However, an deep-seated issue has emerged: these datasets differ drastically in terms of collection methods (onboard cameras, drones, static cameras), scene types (urban, highway, mixed), agent types (vehicles, pedestrians, mixed), and map information. Consequently, a model performing exceptionally well on one dataset often suffers from severe performance degradation when transferred to another. Domain adaptation and generalization have thus become critical bottlenecks for deploying trajectory prediction systems in practice.

Existing transfer learning studies typically conduct experiments on limited source-target dataset pairs (e.g., Argoverse→nuScenes, WOMD→Argoverse), leading to fragmented and unsystematic conclusions. Some works compare datasets based on superficial statistical features (agent density, velocity distribution, number of scenes). However, these coarse-grained statistics fail to capture deep "behavioral features" such as interaction patterns, social norms, and intent. The core challenge is that there is no a priori method to determine the extent to which a model trained on dataset A can transfer to dataset B. Currently, this question can only be answered through exhaustive cross-dataset experiments, which is extremely costly. As the number of trajectory datasets continues to grow, this issue of "blindly selecting pre-training sources" will become increasingly prominent.

The key insight of this paper is: instead of directly resolving transferability, it is better to first learn to "predict transferability." Inspired by feature-space transfer learning (such as Task2Vec, Clark et al.), the authors design a unified latent embedding model that projects all datasets into a shared 32-dimensional space. Probability divergence is then used to quantify the differences between datasets and systematically test the correlation between these differences and actual transfer performance. Core Idea: Represent trajectory datasets as Gaussian distributions in a latent space, and leverage the asymmetric KL divergence between two distributions to predict the transfer difficulty from a source domain to a target domain. This metric's strong correlation is validated on 552 cross-dataset transfer pairs across three transfer scenarios: zero-shot transfer, fine-tuning, and forgetting (achieving a maximum \(\rho=0.874\)).

Method¶

Overall Architecture¶

At the core of TransferAtlas is a unified scene encoder trained to map interaction scenes from different datasets into a 32-dimensional normalized latent space. Then, scene-level embeddings are aggregated into a Gaussian distribution for each dataset using their mean and covariance. The KL divergence is then used as an a priori measure of cross-dataset transferability. The overall framework operates in two stages: the first stage is embedding model training (jointly trained once across all datasets), and the second stage is transferability querying (once trained, given any source and target dataset, the transfer performance is predicted directly using KL divergence without additional cross-domain experiments).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["24 Datasets<br/>Trajectories + Map (optional)"] --> B["Graph-GRU Encoder<br/>Constructs Interaction Graph per Scene"]
    B --> C["Node-level Embeddings<br/>→ Scene-level Embeddings"]
    C --> D["Dataset-level Distributions<br/>Gaussian Modeling (\mu, \Sigma)"]
    D --> E["KL Divergence<br/>D_KL(Source || Target)"]
    E --> F{"Predict Transferability"}
    F --> G["Zero-shot Transfer"]
    F --> H["Fine-tuning Performance"]
    F --> I["Catastrophic Forgetting"]

Key Designs¶

1. Dual-Task Guided Latent Space Learning: Retaining Structural and Predictive Information

To make the latent space reflect not just superficial features (position, velocity) but also behavioral patterns (interaction styles, driving manners), the encoder is trained with two complementary supervisory signals: input reconstruction and future prediction. The reconstruction branch forces the embedding to retain the full structural information of the input scene, while the prediction branch ensures that the embedding carries behavioral information useful for predicting future trajectories—the latter being the core of trajectory prediction. Both decoding heads are discarded after training, and only the encoder is utilized during inference.

Additionally, the embedding vectors are constrained to be normalized onto a unit hypersphere (\(\|\mathbf{z}\|_2=1\)). This not only prevents degenerate solutions (all scenes collapsing to the origin) but also ensures that embeddings across different datasets and scenes share a comparable scale, which provides a stable foundation for subsequent probabilistic modeling.

2. Hierarchical Aggregation: Probabilistic Representation from Nodes to Scenes to Datasets

After agent-level embeddings \(\mathbf{z}_{s,i}\) are generated, they are first aggregated into scene-level distributions \((\boldsymbol{\mu}_s, \boldsymbol{\Sigma}_s)\) (Eq. 4), and then further aggregated into dataset-level distributions \((\bar{\boldsymbol{\mu}}_\mathcal{D}, \boldsymbol{\Sigma}_\mathcal{D})\) (Eq. 5). The decomposition of the covariance is particularly elegant: the dataset-level covariance is split into the sum of inter-scene variance (the dispersion of different scene means) and intra-scene variance (the fluctuation of agent embeddings within the same scene). The former reflects dataset diversity, while the latter reflects the level of noise and uncertainty within the interaction scenes. To prevent the high-dimensional covariance matrix from being ill-conditioned, a low-rank approximation with \(r=16\) is applied along with jitter regularization (Eq. 6). The resulting \(\widetilde{\boldsymbol{\Sigma}}_\mathcal{D}\) is numerically stable and preserves meaningful differences across distributions.

3. KL Divergence as an Asymmetric Transferability Metric: Asymmetry Reflecting Real-World Transfer

Given the Gaussian approximations of two datasets \(\mathcal{D}_i\) and \(\mathcal{D}_j\), this paper uses the KL divergence \(\mathrm{D_{KL}}(\mathcal{D}_i \| \mathcal{D}_j)\) to measure the transfer performance from the source \(\mathcal{D}_j\) to the target \(\mathcal{D}_i\). The asymmetry of KL divergence naturally captures the asymmetry of actual transfer performance. For instance, the divergence from nuScenes to WOMD is significantly larger than that from WOMD to nuScenes (Eq. 9), which aligns with actual observations: pre-training on WOMD and fine-tuning on nuScenes yields much better results than the reverse. Compared with symmetric metrics such as L1 distance, Wasserstein distance, and Maximum Mean Discrepancy (MMD), KL divergence achieves the highest rank correlation coefficients in both zero-shot and fine-tuning scenarios (Tab. 2), demonstrating that directional differences in probability distributions are the core signals of transferability.

Lasso regression analysis further reveals (Fig. 8) that when KL divergence competes with five explicit statistics (number of scenes, average velocity, number of agents, agent type distribution, and observation duration), the coefficient of KL divergence remains non-zero even under the strongest regularization, while almost all explicit statistics (except average velocity) are regularized to zero. This proves that the distribution differences learned in the latent space capture deep behavioral features far beyond simple statistics.

Loss & Training¶

The encoder is trained end-to-end using a joint reconstruction and prediction loss (Eq. 3), both of which are Mean Squared Errors. Training employs a weighted sampling strategy (Eq. 7): the weights of large-scale datasets are exponentially scaled down with \(\alpha=0.5\) to prevent massive datasets like WOMD (576k samples) and openDD (370k samples) from dominating the embedding space. The training runs for 100 epochs, with a cosine annealing learning rate decaying from \(10^{-3}\) to \(10^{-5}\), and teacher forcing used for the first 25 epochs (with probability linearly decaying from 1 to 0). The QCNet predictor is independently fine-tuned and trained on each dataset (Appendix Tab. 5).

Key Experimental Results¶

Main Results¶

Transfer Scenario	Metric	Spearman \(\rho\)	95% CI	Number of Dataset Pairs
Zero-Shot Transfer	\(\mathrm{D_{KL}}(\mathcal{D}_{eval}\\|\mathcal{D}_{train})\)	0.811	(0.782, 0.840)	552
Forgetting after Fine-tuning	\(\mathrm{D_{KL}}(\mathcal{D}_{source}\\|\mathcal{D}_{target})\)	0.729	(0.403, 0.913)	20
Fine-tuning Performance	Same as KL (opposite direction)	Positive correlation	—	5 Sources → Argoverse

Ablation Study¶

Metric	Zero-Shot \(\rho\) (L=16)	Zero-Shot \(\rho\) (L=32)	Zero-Shot \(\rho\) (L=64)	Zero-Shot \(\rho\) (L=128)
L1 Distance	0.475	0.468	0.470	0.425
Wasserstein Distance	0.489	0.482	0.482	0.448
MMD	0.410	0.412	0.410	0.362
KL Divergence	0.781	0.811	0.746	0.736

Metric	Forgetting \(\rho\) (L=16)	Forgetting \(\rho\) (L=32)	Forgetting \(\rho\) (L=64)	Forgetting \(\rho\) (L=128)
KL Divergence	0.504	0.729	0.794	0.874

Key Findings¶

For zero-shot transfer, the optimal dimension for KL divergence is \(L=32\) (\(\rho=0.811\)), where higher dimensions lead to overfitting and decreased correlation. However, in fine-tuning forgetting scenarios, higher dimensions perform better (\(L=128\), \(\rho=0.874\)), indicating that a finer-grained latent space is more suitable for distinguishing inter-class variations.
Regarding latent space clustering of datasets, Lasso regression verifies that KL divergence provides a significant incremental contribution over five explicit statistics (size, velocity, number of agents, types, and duration) (Fig. 8).
The latent space unveils intriguing relationships: drone-collected datasets (e.g., inD, openDD) are very close to onboard vehicle datasets (e.g., nuScenes, Argoverse 2) in the embedding space, signaling strong potential for cross-modal transfer. On the other hand, pedestrian-only datasets (ETH/UCY) are significantly detached from others, suggesting limited benefits from direct transfer.
In single-source selection experiments (Tab. 6), the top-3 hit rate of KL divergence-guided selection reaches 83.3% (with an average rank of 3.00), far outperforming selection based on average velocity (37.5%), largest dataset size (20.8%), and random selection (13.2%).

Highlights & Insights¶

Predicting Transferability Instead of Trial-and-Error: The core idea is simple yet practical. Instead of directly solving the transfer problem, the paper learns to "predict transfer performance" first, enabling researchers to select the optimal pre-training source at zero experiment cost. This is the first large-scale, systematic validation of the correlation between latent space divergence and transferability in trajectory prediction.
Asymmetric Divergence Naturally Matches Asymmetric Transfer: The asymmetry of KL divergence directly corresponds to the real-world phenomenon where transfer from "Source A → Target B" is not equivalent to "Source B → Target A" (e.g., WOMD transfers well to nuScenes, but not vice versa). This is critical information that symmetric metrics (L1, Wasserstein, MMD) fail to capture.
Offline Replay, Train Once and Query Multiple Times: Once the embedding model is trained, analyzing a new dataset only requires a single forward inference and covariance estimation to obtain transferability predictions against all existing datasets, with no need to retrain models or run cross-domain experiments.
Latent Space Analysis Unveils Unexpected Relationships: The t-SNE projection shows that traditional statistical classifications (onboard vehicle vs. drone vs. pedestrian) are not the sole determinants of transferability. The paper presents several cases where different collection modalities lie close to each other in the latent space, pointing out under-utilized "hidden" data sources for the research community.

Limitations & Future Work¶

Although the Gaussian assumption for the latent space is experimentally validated (outperforming non-parametric methods such as MMD), it may not be highly precise for highly multimodal datasets. Certain datasets might present a multi-cluster structure rather than a unimodal Gaussian.
This study evaluates the framework using QCNet as the sole predictor. The correlation between KL divergence and transferability under other architectures (such as diffusion models or GNN interaction models) remains unverified.
The single-source selection experiments (Tab. 6) only cover the zero-shot single-source scenario. Predicting transferability under joint multi-source training remains an open challenge—how the distributions of multiple source datasets can be "fused" and matched with the target is currently unsolved.
The training of the embedding model relies on joint access to all datasets. In privacy-sensitive scenarios where raw trajectory data cannot be shared across institutions, researching how to implement decentralized/distributed embedding learning represents a valuable future direction.

vs [Task2Vec / Clark et al. 2022]: Task2Vec and Clark project time-series datasets into a latent space via a shared autoencoder and calculate the L1 distance to predict transferability. This work tailors the concept specifically for the trajectory prediction domain: it utilizes a Graph Neural Network to encode agent interactions, introduces a prediction head to guide the learning of behavioral features, and replaces symmetric metrics with an asymmetric KL divergence, improving the Spearman \(\rho\) from 0.468 (L1) to 0.811 (KL).
vs [ScenarioNet / TrafficGen]: ScenarioNet similarly visualizes dataset discrepancies using a latent space, but functions essentially as a qualitative analysis tool. In contrast, this work establishes a quantitative statistical correlation between latent space differences and actual transfer performance, and provides practical tools for source selection and zero-shot assessment.
vs [Dronalize / Trajdata / UniTraj]: These works focus on unifying dataset interfaces and formats to lower the threshold of cross-domain experimentation. This paper is complementary: instead of formatting data, it quantifies the higher-level decision-making problem of "to what extent the transfer is worthwhile."

Rating¶

Novelty: ⭐⭐⭐⭐ The idea of predicting trajectory prediction transferability via latent space divergence is elegant and effective. However, it represents a domain-specific adaptation of existing directions like Task2Vec/Clark, meaning the core concept is not entirely new.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extremely thorough, evaluating 24 datasets, 552 zero-shot transfer pairs, Lasso regression validation, ablations on various covariance estimations, and single-source selection experiments. The experimental design is systematic and rigorous, boasting complete and transparent confidence intervals.
Writing Quality: ⭐⭐⭐⭐⭐ Highly structured. It moves logically from "why predicting transferability is necessary" to "how to design the latent space" and "how to validate." The reasoning is tight, and the appendix is highly comprehensive.
Value: ⭐⭐⭐⭐⭐ It provides a prior tool for pre-training source selection that has the potential to become a standard pre-processing step for trajectory prediction research, carrying practical significance for reducing trial-and-error costs in the community.