PersonaDrive: Controllable Trajectory Prediction with Multi-Dimensional Driving Personas¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/VisualAIKHU/PersonaDrive
Area: Autonomous Driving
Keywords: Trajectory Prediction, Driving Style, Controllable Planning, Multimodal Fusion, Closed-Loop Planning

TL;DR¶

PersonaDrive establishes a multi-dimensional driving persona framework by decomposing driving behavior into a 3×3 grid of Temporal Urgency and Ride Comfort, enabling natural-language controllable trajectory prediction via hierarchical anchor transformation (PCAT), multimodal gated fusion (PCMF), and axis-aligned ordering and diversity objectives.

Background & Motivation¶

End-to-end autonomous driving and trajectory prediction models have made remarkable progress in urban perception and closed-loop navigation. However, existing benchmarks and models are almost exclusively trained on datasets collected under ordinary, uniform driving conditions. As a result, planners collapse to a single conservative "average driver" policy, incapable of adapting to heterogeneous user demands. In real-world transportation, driving styles are shaped by multifaceted contextual factors such as trip urgency, passenger health, cargo fragility, route familiarity, and emotional state. Without controllable conditioning, autonomous planners cannot accommodate personalized user needs across varied scenarios.

Prior attempts to introduce persona-conditioned planning generally reduce driving style to a single coarse spectrum such as emergency, normal, and relaxed. This one-dimensional formulation suffers from severe representational collapse: within the same high-urgency regime, an ambulance transporting a critically injured patient demands ultra-smooth, jerk-free maneuvering to avoid secondary trauma, whereas a fire engine rushing to an active blaze permits aggressive turns and hard accelerations. Lumping both into a single "emergency" category erases their fundamental dynamic divergence and negates the expressive power of natural-language commands over discrete categorical labels.

To resolve this limitation, this paper decouples driving style into two orthogonal behavioral dimensions: Temporal Urgency (governing longitudinal speed and progress) and Ride Comfort (governing lateral acceleration and jerk), each discretized into three levels to yield a 3×3 grid of nine distinct personas accompanied by the newly curated PCT dataset. Core idea: hierarchically modulate trajectory anchor prototypes along decoupled urgency and comfort axes from natural-language descriptions, inject persona cues via scene-adaptive BEV gated fusion, and enforce physical ordering and anti-diagonal mode collapse losses to achieve controllable, physically grounded trajectory planning.

Method¶

Overall Architecture¶

The PersonaDrive framework takes multi-view camera streams $I_{\text{cam}}$, LiDAR point clouds $I_{\text{lidar}}$, and a natural-language description $T$ expressing passenger intent. A visual backbone extracts bird's-eye-view (BEV) query tokens $Q_{\text{bev}}$, while a frozen sentence encoder (MiniLM) generates text embeddings $Q_{\text{txt}}$. The core model operates through a two-stage modulation pipeline: the Persona-Conditioned Anchor Transform (PCAT) modulates trajectory anchor priors along both behavioral axes, while Persona-Conditioned Multi-Modal Fusion (PCMF) integrates the transformed anchors, ego status, and text cues with spatial BEV features. Finally, a diffusion-based trajectory decoder outputs an 8-waypoint trajectory spanning a 4-second planning horizon.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Sensor Inputs<br/>Surround Cameras + LiDAR"] --> B["BEV Visual Backbone<br/>Extract Scene Query Qbev"]
    C["Passenger Text Request T<br/>e.g., Rush to hospital / Fragile cargo"] --> D["Text Encoder (MiniLM)<br/>Extract Persona Query Qtxt"]
    D --> E["PCAT Anchor Transform<br/>Decouple Urgency αu + Comfort αc"]
    E --> F["Modulate Prior Anchors P̃<br/>Global Scaling + Per-Step Curvature Tuning"]
    B --> G["PCMF Multimodal Fusion<br/>Prototype Compression + Gated Cross-Attention"]
    D --> G
    F --> G
    G --> H["Diffusion Trajectory Head<br/>Predict 8-Waypoint Persona Trajectory"]
    H --> I["Axis-Aligned Supervisions<br/>Lguide Physical Ordering + LAD Diversity Loss"]

Key Designs¶

1. PCAT Anchor Transform: Two-Axis Decoupling and Hierarchical Prototype Modulation Conventional trajectory predictors rely on static anchor prototypes derived from dataset-wide clustering, making it difficult for downstream attention layers to forcefully manipulate spatial trajectories via text tokens. PCAT splits the text query $Q_{\text{txt}}$ into two independent MLP projection heads: an urgency branch computing an unconstrained global scaling scalar $\alpha_u \in \mathbb{R}$ to govern longitudinal travel distance, and a comfort branch generating a per-timestep vector $\alpha_c \in \mathbb{R}^T$ bounded to $[0.8, 1.2]$ via a sigmoid-affine mapping ($\gamma_c = 0.8, \Delta_c = 0.4$) to adjust local curvature and smoothness. Given base trajectory anchor $p_i$, the transformation hierarchically scales the global length before shaping each waypoint: $$\tilde{p}_i^{(u)} = p_i \times \alpha_u, \quad \tilde{p}_i(t) = \tilde{p}_i^{(u)}(t) \times \alpha_c(t)$$ For neutral driving requests (medium urgency, medium comfort), the transformation is bypassed ($\alpha_u = 1, \alpha_c = 1$), guaranteeing exact preservation of baseline performance.

2. PCMF Multimodal Fusion: Query Prototype Pool and Scene-Adaptive Gating Directly injecting global text tokens into dense spatial BEV grids risks visual distraction and feature corruption. PCMF performs a structured three-step cross-modal alignment: first, a Query Prototype Pool with 8 learnable seed queries compresses $Q_{\text{bev}}$ into a compact scene context vector $C = \text{Attn}(Q_{\text{seed}}, Q_{\text{bev}})$; next, a Conditional Resampler adds the context vector to slot queries $Q'_{\text{slot}} = Q_{\text{seed}} + \tilde{C}$ and resamples agent priors, ego status, and text tokens into persona-aligned representations $R_j$; finally, a Source Gate dynamically computes scene-dependent gating weights $g = \text{softmax}(\text{MLP}(Q_{\text{bev}})) = [g_{\text{agent}}, g_{\text{ego}}, g_{\text{text}}]$ to scale the contribution of each modality before updating $Q_{\text{bev}}$ through cross-attention, harmonizing environmental safety constraints with user preferences.

3. Hierarchical Guide Loss and Axis-Decomposed Diversity Supervision End-to-end training easily suffers from behavioral collapse, where models ignore subtle text differences or collapse into coarse binary clusters. PersonaDrive introduces two structured supervisory signals tailored for the 3×3 grid: - Hierarchical Guide Loss ($\mathcal{L}_{\text{guide}}$): Enforces physical monotonicity across adjacent grid neighbors. In the urgency dimension, trajectory length $l_m$ must increase monotonically with urgency within each comfort column; in the comfort dimension, third-order finite difference jerk magnitude $j_m$ must decrease monotonically with comfort within each urgency row. Twelve one-sided margin constraints across adjacent pairs prevent contradictory behaviors: $$\mathcal{L}_{\text{urg}} = \frac{1}{|\mathcal{C}_u|} \sum_{(m_l, m_h) \in \mathcal{C}_u} \text{ReLU}\Big(\big[l^{\text{gt}}_{m_h} - l^{\text{gt}}_{m_l}\big] - \big[l^{\text{pred}}_{m_h} - l^{\text{pred}}_{m_l}\big]\Big)$$ - Axis-Decomposed Diversity Loss ($\mathcal{L}_{\text{AD}}$): Applying an unconstrained pairwise diversity objective across all 9 personas triggers diagonal mode collapse, where models distinguish fast/aggressive from slow/gentle but fail on cross-axis combinations. $\mathcal{L}_{\text{AD}}$ decomposes supervision into an intra-axis loss $\mathcal{L}_{\text{Intra}}$ (aligning softmax distance distributions via symmetric KL divergence and distance margins across 6 single-axis triplets) and an inter-axis loss $\mathcal{L}_{\text{Inter}}$ (regularizing diagonal separation with weight $w_{\text{Inter}} = 0.2$).

Loss & Training¶

The overall training objective combines trajectory reconstruction and classification loss from DiffusionDrive with the proposed structural objectives: $$\mathcal{L}_{\text{Total}} = \lambda_1 \mathcal{L}_{\text{traj}} + \lambda_2 \mathcal{L}_{\text{guide}} + \lambda_3 \mathcal{L}_{\text{AD}}$$ Hyperparameters are set to $\lambda_1 = 10, \lambda_2 = 1, \lambda_3 = 1$, with softmax temperature $\tau = 0.25$. The model employs a ResNet-34 backbone and is trained from scratch on NAVSIM navtrain for 100 epochs using 6 NVIDIA RTX A6000 GPUs, AdamW optimizer with a learning rate of $6 \times 10^{-4}$, and a global batch size of 128.

Key Experimental Results¶

Main Results¶

Evaluated on the NAVSIM navtest closed-loop planning benchmark (derived from OpenScene), models are assessed using Average Displacement Error (Avg. ADE), Final Displacement Error (Avg. FDE), and closed-loop planning score (Avg. PDMS), averaged across all nine persona categories.

Table 2: Closed-loop planning performance comparison on NAVSIM navtest (averaged over all 9 personas)

Methods	Conference / Journal	Avg. ADE (m) ↓	Avg. FDE (m) ↓	Avg. PDMS ↑
LTF	TPAMI 2022	2.49	3.87	52.0
Transfuser	TPAMI 2022	2.53	3.96	51.9
UniAD	CVPR 2023	2.47	3.83	52.5
Hydra-MDP	arXiv 2024	6.70	11.85	41.4
PARA-Drive	CVPR 2024	4.84	9.00	51.9
Traj-LLM	T-IV 2024	2.66	4.20	54.1
VisionTRAP	ECCV 2024	5.17	9.95	50.7
WoTE	ICCV 2025	3.35	5.71	52.4
DiffusionDrive (Base)	CVPR 2025	3.93	5.79	55.3
VADv2	ICLR 2026	6.18	10.76	47.9
PersonaDrive (Ours)	ECCV 2026	2.39	3.71	57.7

PersonaDrive achieves the lowest displacement errors across all baselines, reducing Avg. ADE by 3.2% (2.47m → 2.39m) and Avg. FDE by 3.1% (3.83m → 3.71m) over the strongest baseline UniAD, while boosting Avg. PDMS to 57.7.

Ablation Study¶

Ablation studies analyze the cumulative impact of each architectural module and loss formulation.

Table 4: Ablation study of proposed components on NAVSIM navtest (all include text encoder)

Configuration	PCAT	PCMF	$\mathcal{L}_{\text{AD}}$	Avg. ADE (m) ↓	Avg. FDE (m) ↓	Avg. PDMS ↑	Note
Base (Text-only)	-	-	-	3.93	5.79	55.3	Naive concatenation of text and BEV
+ PCAT	✓	-	-	3.52	5.43	56.0	Hierarchical anchor modulation only
+ PCAT + PCMF	✓	✓	-	3.23	4.85	56.9	Adding gated multimodal feature fusion
+ PCMF + $\mathcal{L}_{\text{AD}}$	-	✓	✓	2.55	3.87	57.2	Lacks explicit anchor physical scaling
+ PCAT + $\mathcal{L}_{\text{AD}}$	✓	-	✓	2.61	3.97	57.5	Lacks PCMF feature refinement
Full Model	✓	✓	✓	2.39	3.71	57.7	Optimal synergy across all components

Table 3: Comparison between one-dimensional proxies and multi-dimensional conditioning (ADE / FDE in meters)

Setting	UH-CL	UH-CH	UM-CM	UL-CL	UL-CH	9-Cell Avg. ADE	9-Cell Avg. FDE
1D-Urgency (Urgency only)	3.30 / 5.81	3.07 / 5.42	2.35 / 3.52	2.73 / 4.06	2.56 / 4.21	2.67	4.27
1D-Comfort (Comfort only)	4.79 / 11.76	4.66 / 10.32	2.35 / 3.52	5.70 / 9.84	7.11 / 12.87	4.50	8.64
Multi-D (Ours)	2.98 / 5.25	2.69 / 4.47	2.35 / 3.52	2.02 / 2.82	1.82 / 2.33	2.39	3.71

Key Findings¶

Superior Handling of Against-the-Grain Scenarios: In opposing persona cells such as High Urgency + High Comfort (UH-CH, e.g., transporting delicate medical specimens rapidly yet smoothly) and Low Urgency + Low Comfort (UL-CL, e.g., crawling cautiously while scanning curbsides), 1D proxies collapse catastrophically (1D-Comfort yields FDE of 10.32m on UH-CH). PersonaDrive's orthogonal decoupling drops FDE to 4.47m, proving that neither axis alone is sufficient.
Sentence-Level Embeddings Drastically Outperform Token-Level Models: Comparing frozen text encoders reveals that the compact sentence-level model MiniLM (33M params, ADE 2.39m / FDE 3.71m) substantially beats larger masked language models like RoBERTa (125M params, ADE 4.72m / FDE 8.76m) and DeBERTa (140M params, ADE 5.99m / FDE 11.56m). Sentence-level contrastive pretraining maps semantically related row/column driving requests nearby in latent space, whereas token-level objectives inject high-frequency lexical noise.
Negligible Latency Overhead: The architecture adds merely 1.8M parameters (61.4M → 63.2M, +2.75%) and increases inference latency by only 1 ms per frame (45 FPS → 43 FPS), making it readily deployable for real-time vehicular control.

Highlights & Insights¶

Orthogonal 3×3 Behavioral Decomposition: Decouples subjective driving personas into two well-defined, physically grounded axes—Temporal Urgency (longitudinal scaling) and Ride Comfort (lateral smoothness and jerk)—providing an actionable blueprint for customizable autonomous driving.
Explicit Geometric Modulation Meets Implicit Feature Gating: Combining PCAT's bounded geometric anchor deformation with PCMF's scene-adaptive gating provides physical safety bounds while preserving the expressive flexibility of diffusion planners.
Distribution-Preserving Metric Alignment: The intra- and inter-axis KL divergence loss effectively prevents multi-conditional trajectory diffusion from collapsing into dominant modes, a technique directly extensible to human-robot interaction and multi-agent gaming.

Limitations & Future Work¶

Reliance on LLM-Synthesized Supervision: The PCT dataset trajectories are synthesized using GPT-4o-mini and verified via GPT-4o-based validation. While validated by human perceptual studies (85.7% classification accuracy), real-world trajectory validation with human driving instrumentation remains desirable.
Static Request Assumption: The model assumes a fixed passenger request per driving clip, lacking the capability for dynamic, mid-trajectory persona switching (e.g., sudden emergency transitions).
Future Directions: Extending to closed-loop interactive simulations with dynamic, real-time persona updates and multi-agent interactive game-theoretic scenarios.

vs DiffusionDrive / UniAD: Mainstream end-to-end planners optimize for a singular conservative average path; PersonaDrive preserves closed-loop driving safety while introducing predictable, continuous dynamic control.
vs LangTraj / Traj-LLM: Prior language-guided planners primarily execute coarse topological navigation directives (e.g., "turn left at the intersection"); PersonaDrive enables fine-grained control over continuous vehicle dynamics (acceleration, speed factor, jerk).

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Elegant 3×3 orthogonal decomposition bridging natural-language passenger requests with physical vehicle dynamics.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous closed-loop NAVSIM benchmarks, extensive ablations across encoders and dimensions, and human validation.
Writing Quality: ⭐⭐⭐⭐⭐ Clear narrative progression, clean mathematical formulation, and well-structured empirical analysis.
Value: ⭐⭐⭐⭐⭐ Highly valuable for customized ride-hailing services, smart cockpits, and human-centric autonomous driving systems.