MetaDAT: Generalizable Trajectory Prediction via Meta Pre-training and Data-Adaptive Test-Time Updating

Conference: CVPR 2026 arXiv: 2603.09419 Code: N/A Area: Autonomous Driving Keywords: Trajectory Prediction, Test-Time Training, Meta-Learning, Distribution Shift, Online Adaptation

TL;DR

This paper proposes the MetaDAT framework, which obtains a model initialization amenable to online adaptation via meta pre-training, and achieves data-adaptive model adjustment at test time through dynamic learning rate optimization (DLO) and hard-sample-driven updates (HSD). MetaDAT surpasses all existing TTT methods under cross-dataset distribution shift settings across nuScenes, Lyft, and Waymo.

Background & Motivation

Background: Data-driven trajectory prediction methods (e.g., ForecastMAE) achieve strong performance on pre-collected datasets, but suffer significant degradation at test time when facing distribution shifts (e.g., changes in road structure, interaction patterns, or driving styles), posing safety risks.

Unique Advantage of Test-Time Training (TTT): Trajectory prediction has a natural "self-labeling" property — observations at the current timestep serve as ground-truth labels for past predictions. This enables online model updating using real observations at test time, without requiring additional annotation.

Two Key Bottlenecks of Existing TTT Methods: - Offline–Online Objective Misalignment: Existing offline pre-training objectives optimize only in-distribution prediction accuracy, ignoring the model's capacity for online adaptation. The resulting initialization is suboptimal for online updating, leading to slow adaptation and rapid representation degradation. - Fixed Online Update Strategy: Conventional methods apply fixed learning rates and update frequencies, without adapting to the characteristics of test data (e.g., degree of distribution shift, sample difficulty).

Distinction from AML: AML applies meta-learning adaptation only to the final Bayesian linear regression layer of the decoder, limiting deep representation adaptability. MetaDAT performs meta pre-training over all model parameters, unlocking full adaptation potential.

Method

Overall Architecture

MetaDAT consists of two stages: (1) Meta Pre-training (MP): TTT tasks are simulated on source data via bi-level optimization to obtain a model initialization \(\theta^*\) suited for online adaptation; (2) Data-Adaptive Test-Time Updating: the model is adapted on target data via DLO and HSD. The predictor follows the ForecastMAE architecture (embedding + encoder + decoder + MAE reconstruction branch).

Key Designs

1. Meta Pre-training (MP)

• Function: Solves the offline–online objective misalignment by using bi-level optimization (MAML-style) to directly optimize model parameters toward being a "good starting point" for online updating.
• Mechanism:
  • TTT Task Simulation: The source dataset is partitioned into sub-domains by driving scene (each scene exhibits distinct behavioral patterns and road structures). Within each scene, samples are organized in temporal order to form an online sequence \(\mathbf{S} = \{\mathbf{X}_0, \mathbf{X}_1, \ldots, \mathbf{X}_{t_s}\}\).
  • Bi-level Optimization: The inner loop simulates \(K\) TTT update steps on a single scene: \(\theta'_{i,\tau} = \theta'_{i,t-\tau} - \alpha_{in} \nabla \mathcal{L}^{i,t-\tau}_{mae}\); the outer loop evaluates post-adaptation performance across multiple scenes and updates the initial parameters: \(\theta \leftarrow \theta - \beta \nabla_\theta \sum_i \mathcal{L}^{i,K\tau}_{mae}\).
  • A first-order approximation is employed to avoid the computational cost of Hessian computation. Training is initialized from an offline pre-trained model \(\theta_{off}\) to accelerate convergence.
• Design Motivation: Standard pre-training minimizes training loss, whereas TTT requires parameters that can reach low loss rapidly after a few gradient update steps — these are fundamentally different objectives.

2. Dynamic Learning Rate Optimization (DLO)

• Function: Dynamically adjusts per-layer learning rates using online partial derivatives to match the characteristics of test data.
• Mechanism: Assuming the optimal learning rate changes slowly between adjacent steps, the partial derivative of the loss with respect to the learning rate is computed via the chain rule: $\(\frac{\partial \mathcal{L}_{mae}(\theta_{p-1})}{\partial \alpha} = -\nabla_\theta \mathcal{L}_{mae}(\theta_{p-1}) \cdot \nabla_\theta \mathcal{L}_{mae}(\theta_{p-2})\)$ The learning rate is then updated via gradient descent: \(\alpha_p = \alpha_{p-1} + \gamma \nabla_\theta \mathcal{L}_{mae}(\theta_{p-1}) \cdot \nabla_\theta \mathcal{L}_{mae}(\theta_{p-2})\). In practice, gradients are averaged over an interval \(\tau_\alpha\) to stabilize training. Independent learning rates are maintained for each network layer.
• Design Motivation: The degree of distribution shift is unknown a priori; a fixed learning rate may be too large (causing training instability) or too small (causing insufficient adaptation). The response of the learning rate to the consistency of gradient directions across two steps naturally encodes whether the data is changing in a consistent direction.

3. Hard-Sample-Driven Model Updates (HSD)

• Function: Identifies samples with significantly elevated prediction errors and performs additional update steps on them.
• Mechanism: The current prediction error \(e\) is compared against the running mean \(m\) and standard deviation \(\sigma\); an additional update is triggered when \(e > m + k\sigma\) (with \(k=3\)).
• Design Motivation: Autonomous driving data follows a long-tail distribution, where scenes involving dense interactions or heavy reliance on road maps are rare but safety-critical. These hard samples best represent the information the model needs to learn in the current domain; focusing updates on them improves performance without sacrificing efficiency.

Loss & Training

• Pre-training Loss: \(\mathcal{L}_{mae} = \mathcal{L}_{reg}(\mathbf{X}, \mathbf{Y}) + \mathcal{L}_{recon}(\mathbf{X}, \mathbf{Y})\), i.e., joint training with a prediction regression loss and a MAE reconstruction loss.
• TTT Loss: Same as above; observations from \(\tau\) timesteps prior are used as pseudo-labels for online updating.
• Training Pipeline: Offline pre-training → Meta pre-training (8 epochs, batch=4, inner loop \(K=4\) steps, AdamW + cosine decay) → Online DLO + HSD adaptive updating at test time.
• Actor-specific tokens \(a_n\) are used to learn individual driving habits during the TTT phase.

Key Experimental Results

Main Results

Setting	Metric	MetaDAT	T4P (Prev. SOTA)	Gain
Short-term (1/3/0.1), avg. over 3 scenarios	mADE₆/mFDE₆	0.301/0.648	0.339/0.744	11.2%/12.9%
Lyft→nuScenes Short-term	mADE₆/mFDE₆	0.332/0.683	0.357/0.770	7.0%/11.3%
nuScenes→Waymo Short-term	mADE₆/mFDE₆	0.305/0.712	0.336/0.807	9.2%/11.8%
Waymo→nuScenes Short-term	mADE₆/mFDE₆	0.266/0.548	0.323/0.656	17.6%/16.5%
Long-term (2/6/0.5), avg.	mADE₆/mFDE₆	0.912/2.011	1.014/2.311	10.1%/13.0%

Ablation Study

Configuration	Short-term mADE₆/mFDE₆ (Lyft→nuS)	Long-term mADE₆/mFDE₆ (nuS→Lyft)	Note
Baseline (T4P*)	0.408/0.847	0.711/1.578	No improvements
+MP	0.355/0.734	0.672/1.491	MP yields the largest gain
+DLO	0.376/0.776	0.684/1.538	Dynamic LR is effective
+HSD	0.400/0.836	0.707/1.552	Hard-sample updates provide marginal improvement
+MP+DLO	0.347/0.702	0.650/1.468	Two modules are complementary
+MP+DLO+HSD (Full)	0.332/0.683	0.648/1.472	Full model achieves best results

Key Findings

1. Meta pre-training is the dominant contributor: MP alone yields the largest improvement in both short- and long-term settings (13% reduction in short-term mADE₆), validating that offline–online objective misalignment is the core bottleneck.
2. Learning rate robustness: DLO substantially improves T4P performance under suboptimal learning rates (mADE₆ drops from 0.518 to 0.452 at α=0.01); MetaDAT further reduces this to 0.407.
3. Efficiency advantage: Under the same FPS constraint, MetaDAT consistently dominates T4P on the accuracy–efficiency Pareto frontier.
4. Few-shot adaptation: MetaDAT with only 2,000 samples matches or exceeds T4P with 10,000 samples (0.327 vs. 0.343 mADE₆).
5. Multimodal prediction: MetaDAT outperforms across mADE₁ and MR₆ metrics as well, with improved prediction diversity in both lateral and longitudinal directions.

Highlights & Insights

1. Precise problem formulation: Attributing TTT failure to "offline–online objective misalignment" and addressing it elegantly with the MAML framework represents a key insight for the trajectory prediction TTT field.
2. Full exploitation of the self-labeling property: The "observations as labels" property of trajectory prediction is naturally suited to TTT; this paper maximizes its potential through meta-learning.
3. Theoretically grounded and practically efficient DLO: Automatically adjusting the learning rate based on gradient direction consistency across adjacent steps avoids hyperparameter sensitivity with minimal computational overhead.
4. Strong complementarity among three modules: MP governs initialization, DLO governs learning rate, and HSD governs sample selection — each addresses a distinct aspect of the problem, and their combination yields consistent cumulative gains.

Limitations & Future Work

1. Dependence on accurate online detection/tracking: TTT relies on observed trajectories as training labels, but real-world perception systems are noisy — noisy labels may degrade online adaptation.
2. First-order MAML approximation: While the first-order approximation reduces computational cost, it may sacrifice meta-learning accuracy.
3. Only cross-dataset distribution shift is considered: Intra-dataset domain shifts (e.g., weather, lighting conditions) are not evaluated, leaving practical generalizability to be further validated.
4. Single-predictor online updating: In multi-agent systems, updating a single predictor online may be insufficient.

Related Work & Insights

• T4P [AAAI'24]: The current state-of-the-art TTT trajectory prediction method, introducing MAE loss and actor-specific tokens — MetaDAT's direct competitor and baseline.
• AML: Another meta-learning method that adapts only the final decoder layer — MetaDAT demonstrates that full-model meta pre-training is more effective.
• MAML [Finn et al.]: The classic meta-learning framework — MetaDAT applies it to the TTT setting, with novelty in TTT task simulation and data-adaptive updating.
• Insight: The DLO principle of "optimizing hyperparameters via partial derivatives" is generalizable to other online learning scenarios (e.g., online detection, online mapping).

Rating

• Novelty: ⭐⭐⭐⭐ Meta pre-training as a solution to offline–online misalignment is a clear contribution; the DLO derivation is concise and principled.
• Experimental Thoroughness: ⭐⭐⭐⭐ Three large-scale datasets, multiple shift configurations, comprehensive ablation and robustness analyses.
• Writing Quality: ⭐⭐⭐⭐ Problem motivation is clearly articulated and the framework diagram is intuitive, though some mathematical derivations could be further streamlined.
• Value: ⭐⭐⭐⭐ Strong practical relevance (robustness, efficiency, few-shot adaptation) with direct implications for online deployment in autonomous driving.
• Value: TBD

Related Papers

• [CVPR 2026] Den-TP: A Density-Balanced Data Curation and Evaluation Framework for Trajectory Prediction
• [CVPR 2026] FoSS: Modeling Long-Range Dependencies and Multimodal Uncertainty in Trajectory Prediction via Fourier–State Space Integration
• [CVPR 2026] DLWM: Dual Latent World Models enable Holistic Gaussian-centric Pre-training in Autonomous Driving
• [CVPR 2026] Recover to Predict: Progressive Retrospective Learning for Variable-Length Trajectory Prediction
• [CVPR 2026] TT-Occ: Test-Time 3D Occupancy Prediction