Long-term Traffic Simulation via Structured Autoregressive Modeling

Conference: ECCV 2026
arXiv: 2606.31209
Code: https://sephirex-x.github.io/rosettasim/
Area: Autonomous Driving
Keywords: Traffic Simulation, Autoregressive Modeling, Large Language Models, Long-term Simulation, Retrieval-based Evaluation

TL;DR

RosettaSim models long-term multi-agent traffic simulation as a structured autoregressive sequence generation problem. By utilizing frozen/partially frozen LLMs (Qwen2.5-0.5B) as structural priors, it projects scene topology, agent states, and generation intents into variable-length token sequences to uniformly process motion prediction and agent population dynamics. Meanwhile, it proposes a Retrieval-based Traffic Evaluation (RTE) framework, using semantically similar real-world scenes as reference anchors instead of global distribution matching, achieving SOTA on both short-term and long-term simulation on WOSAC.

Background & Motivation

Traffic simulation is a critical world model for the scalable and safe development of autonomous driving systems. Existing methods perform excellently in short-term simulation (<8s), but long-term, trip-level simulation faces two fundamental challenges: agents continuously entering and exiting scenes, which leads to dynamic token cardinality, and the lack of a reliable long-term evaluation protocol.

Existing long-term works model scene generation (agent injection) and motion generation jointly in a single rollout process. However, scene generation focuses on spatial distribution (predicting when and where new agents appear), whereas motion generation emphasizes temporal consistency (predicting how agents move). These two types of objectives are fundamentally different. The recent work SceneStreamer lags significantly behind specialized short-term models (such as UniMM) on short-term metrics, demonstrating that the current modeling paradigms cannot adequately capture both agent population dynamics and long-term motion dependencies simultaneously. In parallel, large sequence models (LLMs) have shown powerful capabilities in long-range dependency modeling, but their application in traffic simulation remains at the level of "using language for conditional control" or "drawing inspiration from architecture design." No prior work has demonstrated whether the structured sequence modeling capability of LLMs can be directly transferred to traffic simulation.

The core finding of this paper is that the frequency distribution of discretized traffic motion tokens strictly follows Zipf's law (highly consistent with the statistical characteristics of natural language tokens), and LLMs exhibits highly similar attention locality when processing language and motion data. This implies that the internal representation of pretrained LLMs already contains highly transferable structure-aware capabilities—traffic dynamics can be regarded as a "foreign language," and the LLM is the optimal initialization manifold. Based on this insight, this paper proposes RosettaSim: projecting geometric topology, existing agent states, and generation intents into a unified structured autoregressive token sequence to simultaneously handle parallel motion updates and autoregressive agent generation, which significantly mitigates the performance trade-off between scene generation and motion generation. Additionally, addressing the issue where one-to-one agent matching fails over time in long-term evaluation, this paper proposes RTE (Retrieval-based Traffic Evaluation), which retrieves the most semantically similar real-world scenes from the latent space of a pretrained scene VAE as context-aware reference anchors.

Core Idea: Leveraging the structural priors of LLMs (rather than language semantics) to model multi-agent traffic scene evolution as a structured autoregressive token sequence, uniformly processing motion generation and agent population dynamics.

Method

Overall Architecture

RosettaSim formalizes long-term traffic simulation as a conditional sequence generation problem: given the current scene \(\mathcal{S}_t\) (\(N_t\) active agent state tokens) and map context \(\mathcal{C}\), it predicts the next scene \(\mathcal{S}_{t+1}\). The core innovation lies in decomposing the standard transition probability into two tractable processes—parallel motion update (all existing agents predict their next states simultaneously) and autoregressive agent generation (conditional on the intermediate scene after motion updates, generating new agents token-by-token). Both processes share the same frozen/partially frozen LLM backbone \(f_\theta\), and all tokens are organized into a unified structured sequence fed into the LLM for end-to-end single-stage training.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Current Scene S_t<br/>+ Map C"] --> B["Scene Encoder E<br/>Extract Agent Token a_t"]
    B --> C["Parallel Motion Generation<br/>LLM f_θ: Motion Token × Agent Token → Motion Update"]
    C --> D["Intermediate Scene S̃_{t+1}"]
    D --> E["Construct Map Token + Layout Token<br/>Spatial Topology + Grid/Yaw Encoding"]
    E --> F["Autoregressive Agent Generation<br/>&lt;SOS&gt; → Type → Grid → Yaw → &lt;EOS&gt;"]
    F --> G["Final Scene S_{t+1}<br/>with Newly Generated Agents"]

The scene encoder \(\mathcal{E}\) (a pretrained Transformer layer based on SMART) encodes the agent-centric scene context into Agent Tokens \(\mathbf{a}_t^{1:N_t}\), whose dimensions are aligned with the LLM via an MLP. Subsequently, a set of learnable Motion Tokens \(\mathbf{mo}^{1:N_t}\) is sent to the LLM along with the Agent Tokens. Through self-attention interaction to extract motion context, it outputs updated Motion Tokens, which are projected into motion logits via an MLP to achieve parallel state updates for all agents. On this basis, Map Tokens (road network geometric topology) and Layout Tokens (discretized encoding of grid positions + yaw angles) are constructed from the intermediate scene and concatenated with existing tokens as a contextual prefix to conditionally guide autoregressive agent generation: the LLM sequentially samples agent type token \(\mathbf{c}^j\), grid token \(\mathbf{g}^j\), and yaw token \(\mathbf{y}^j\) until <EOS> is sampled. To distinguish different agents, all tokens are superimposed with agent-wise sinusoidal position embeddings \(\text{SinPE}(i)\).

Key Designs

1. Adaptability of LLM Structural Priors: Isomorphism between Motion Token and Natural Language Distributions

This paper systematically reveals the fundamental reason why LLMs can adapt to traffic simulation through two sets of probing experiments. First, at the data distribution level: statistical analysis of the frequency distribution of discretized motion tokens on the WOMD dataset shows that it strictly follows Zipf's law, almost identical to the distribution characteristics of natural language tokens—a few common actions (cruising, stationary) appear with high frequency, while complex maneuvers (sharp turns) appear with low frequency. When the Zipf distribution of motion tokens is artificially disrupted (e.g., forced uniform distribution, where the Zipf coefficient drops to 0), the model's performance drops significantly, directly validating the hypothesis that "distributional isomorphism is a prerequisite for adaptation." Second, at the attention mechanism level: taking attention locality (defined as the proportion of attention weights on the main diagonal and its adjacent diagonals, measuring the token's focus on itself and its immediate neighbors) as a probe, the locality curves of Qwen2.5-0.5B across different layers are compared under different data sources (text vs. motion). The two are highly consistent; in contrast, Llama and TinyLlama exhibit large differences. This explains why the Qwen series performs best in the frozen setup—its multilingual pretraining (29 languages, 18T tokens) fosters a more generalized sequence modeling capability, allowing motion tokens to be effectively treated as a "foreign language." These two findings together indicate that: pretrained LLMs provide structured sequence priors (statistical distributions + attention patterns), rather than language semantic knowledge.

2. Structured Autoregressive Sequence: Parallel Motion Update + Conditional Agent Generation

This is the core architectural contribution of RosettaSim. Unlike approaches like InfGen that "generate the entire scene topology at once," RosettaSim decomposes the scene transition into two sequential steps. Parallel motion update stage: All \(N_t\) existing agents conditionally independently predict their next states simultaneously, given the rich historical scene context \(\mathcal{S}_t\) and map \(\mathcal{C}\), approximated as \(\prod_{i=1}^{N_t} p(\tilde{s}_{t+1}^i \mid \mathcal{S}_t, \mathcal{C})\), which guarantees simulation efficiency. The intermediate scene \(\tilde{\mathcal{S}}_{t+1}\) output by this step represents pure motion-updated states, without topology changes yet.

Autoregressive agent generation stage: The intermediate scene is encoded into Map Tokens \(\mathbf{m}^{1:M_t}\) (road network geometric relationships extracted from a pretrained map encoder) and Layout Tokens \(\mathbf{l}^{1:N_t}\) (consisting of grid tokens \(\mathbf{g}^i\) and yaw tokens \(\mathbf{y}^i\), obtained via embedding \(3\text{m}\) grid quantization and \(3^\circ\) yaw quantization, respectively). With the tokens of the existing scene serving as the context prefix, the LLM sequentially samples three structured tokens for new agents—type (vehicle/pedestrian/cyclist), grid position, and yaw direction—using standard causal attention to form the sequence \([\texttt{<SOS>}, \mathbf{c}^1, \mathbf{g}^1, \mathbf{y}^1, \ldots, \mathbf{c}^J, \mathbf{g}^J, \mathbf{y}^J, \texttt{<EOS>}]\), where \(J\) is the number of newly generated agents, resulting in \(N_{t+1} = N_t + J\). This agent-level autoregressive insertion, compared to InfGen's scene-level one-time generation, captures complex asynchronous interactions among agents, and causal attention naturally ensures that new agents do not interfere with the motion trajectories of existing agents.

3. Retrieval-based Traffic Evaluation Framework (RTE)

In long-term simulation, the dynamic entry and exit of agents invalidate strict one-to-one matching. Previous works sliced the rollout into short windows and calculated the divergence against the global distribution of the entire validation set. However, global matching suffers from severe statistical bias: forcing high-speed scenes to match a global distribution primarily composed of low-speed scenes penalizes realistic behaviors (empirical findings in the paper show that log-based Kinematic metrics are negatively correlated with standard WOSAC metrics, \(r \approx -0.3\)). RTE's solution is inspired by the LPIPS metric in image generation: given a simulation rollout \(\mathcal{S}_{0:T}\), it is first sliced into short segments \(\{\mathcal{S}_{t:t+\tau}\}\) of duration \(\tau\). Each segment is encoded into a latent representation \(\mathbf{z} = [\mathbf{z}_{\text{object}}, \mathbf{z}_{\text{map}}]\) (object-level and map-level Gaussian parameters) by a pretrained scene VAE \(\mathcal{E}_\phi\) (reusing the official checkpoint without fine-tuning). Then, within a retrieval library constructed from all log segments in the validation set, the top-K most semantically similar real-world scenes are retrieved as context-aware reference anchors according to the Wasserstein distance \(\text{similarity}(\mathbf{z}, \hat{\mathbf{z}}) = W_2(\mathbf{z}_{\text{object}}, \hat{\mathbf{z}}_{\text{object}}) + \lambda \cdot W_2(\mathbf{z}_{\text{map}}, \hat{\mathbf{z}}_{\text{map}})\). Finally, evaluation metrics are computed between the simulated segments and the retrieved references. RTE proposes RMM-F1 (Realism Meta Metric F1) as a comprehensive metric: it takes the harmonic mean (\(\beta=1\)) of Behavior Realism (aggregating WOSAC-like metrics) and Traffic Flow Realism (macro distribution of agent entry/exit counts), strictly penalizing models that perform well on only a single dimension.

Loss & Training

Training adopts a minimalist, single-stage cross-entropy loss. All tokens are concatenated into a unified sequence \(\texttt{seq} = [\mathbf{a}^{1:N_t}, \mathbf{mo}^{1:N_t}, \mathbf{m}^{1:M_t}, \mathbf{l}^{1:N_t}, \mathbf{s}]\) (where \(\mathbf{s}\) is the spawn token sequence). The causal cross-entropy loss is computed only at the positions corresponding to the Motion Tokens and the spawn tokens, while other positions (Agent Tokens, Map Tokens, Layout Tokens) are ignored using padding labels. Compared to the multi-stage training (pretraining + fine-tuning, totaling 35 epochs) required by SceneStreamer, RosettaSim achieves superior performance in only 7 epochs of training (8 RTX 5090 GPUs, ~10 hours per epoch; reduces to 3 hours/epoch if agent generation is disabled). The learning rate is 3e-4 with cosine decay and a warmup ratio of 0.01.

Key Experimental Results

Main Results

Short-term Simulation (WOSAC 2025 test split, 8s, agent generation module disabled): RosettaSim, using Qwen2.5-0.5B as the backbone, achieves a Composite Score of 0.7833, surpassing all methods that do not use closed-loop RL fine-tuning, and leading by a large margin among all models supporting dynamic agent insertion (SceneStreamer is only 0.7731, GUMP is 0.7431).

Method	LLM Usage	Agent Insertion	Composite ↑	Kinematic ↑	Interactive ↑	Map-based ↑	minADE ↓
SMART	-	✗	0.7814	0.4854	0.8089	0.9153	1.3931
UniMM	-	✗	0.7829	0.4914	0.8089	0.9161	1.2949
GUMP	GPT-2 Medium	✓	0.7431	0.4780	0.7887	0.7404	1.6041
SceneStreamer	Arch. Inspired	✓	0.7731	0.4492	0.8084	0.9127	1.4252
RosettaSim	Qwen2.5-0.5B	✓	0.7833	0.4905	0.8105	0.9167	1.3417

Long-term Simulation (31.1s rollout): RosettaSim achieves an RMM-F1 comprehensive score of 0.7624, significantly outperforming InfGen (0.7290) and other baselines. The improvement in Traffic Flow Realism (0.7846 vs. InfGen 0.7637) validates that structured autoregressive generation effectively maintains global flow consistency. Under Behavior Realism, the Kinematic metric (0.9176) far exceeds all baselines (SMART 0.8619, CAT-K 0.8582), highlighting the advantage of generating physically realistic motion trajectories.

Method	RMM-F1 ↑	Traffic Flow Realism ↑	Behavior Realism ↑	Kinematic ↑	Interactive ↑	Map-based ↑
InfGen	0.7290	0.7637	0.6973	0.7312	0.7445	0.6172
CAT-K	0.7381	0.7191	0.7581	0.8582	0.8256	0.6140
SMART	0.7383	0.7175	0.7602	0.8619	0.8280	0.6150
RosettaSim	0.7624	0.7846	0.7414	0.9176	0.7663	0.6089

Ablation Study

Ablation on Tuneable Layers (varying freezing degrees of the LLM backbone, 2% WOSAC val split): Even when the LLM is completely frozen ('None'), the Composite Score (0.7638) still outperforms the SMART baseline (0.7631), proving that the pretrained internal representation already contains transferable structure awareness. Fine-tuning only the last layer ('Last 1', 0.7648) performs very close to full fine-tuning ('All', 0.7658), whereas fine-tuning the last five layers (0.7647) brings no additional benefit, indicating that the pretrained knowledge is robust and requires minimal alignment.

Tuneable Layers	Composite ↑	Kinematic ↑	Interactive ↑	Map-based ↑
All	0.7658	0.4854	0.8095	0.8697
Last 5	0.7647	0.4854	0.8083	0.8682
Last 1	0.7648	0.4876	0.8084	0.8671
None	0.7638	0.4843	0.8074	0.8674
SMART Baseline	0.7631	0.4829	0.8065	0.8675

Pretraining vs. Random Initialization: Random initialization of Qwen-0.5B contributes 63% of the improvement over the baseline purely through architectural inductive bias, while the statistical prior of pretraining contributes the remaining 37%, showing both are indispensable.

Adaptability of Different LLMs (single epoch, fully frozen): Qwen2.5-0.5B (Composite 0.7620) > Llama-3.2-1B (0.7594) > TinyLlama-1.1B (0.7397) > GPT-2-Small (0.5425). Modern LLMs are far superior to older architectures, and Qwen’s multilingual pretraining (29 languages vs. Llama's 8 languages) yields stronger generalized sequence modeling capabilities.

Model Scaling: Under the frozen setup, scaling Qwen2.5 from 0.5B to 1.5B yields no improvement (both 0.7620), and 3B even drops (0.7589); after fine-tuning the last layer, performance converges across sizes (0.5B: 0.7637, 1.5B: 0.7644, 3B: 0.7641). This indicates that traffic simulation relies on the "grammar" (sequence dependencies and statistical distributions) of language rather than the deep semantic knowledge or reasoning capabilities of large models.

Effectiveness of RTE: Under 63 diverse rollout variants generated (top-48 random sampling on SMART and CAT-K), Pearson correlations between RTE metrics / log-based metrics and standard WOSAC were calculated. RTE is significantly more correlated than log-based methods on all dimensions (overall \(r=0.83\) vs. \(r=0.74\)). In particular, the log-based Kinematic metric negatively correlates with standard WOSAC, directly validating the core motivation that "global distribution matching penalizes realistic behaviors."

Key Findings

• The structural prior of LLM pretraining is the core source of gain, rather than language semantics: freezing the LLM already outperforms the baseline, and fine-tuning only the last layer yields convergence. Larger models are redundant, proving that the "grammar" of traffic dynamics is shallow and can be captured by small models.
• Attention locality similarity is an effective probe for identifying LLM adaptability: Qwen's locality curves on text and motion data highly overlap, whereas TinyLlama and Llama do not, aligning perfectly with their performance rankings.
• The retrieval fidelity of RTE is strictly validated: The accuracy is around 0.90 in equal-scenario retrieval and around 0.85 under unequal scenarios (high interference), proving that the latent space of the pretrained scene VAE indeed encodes the semantic similarity of driving scenes.
• Short-term specialized methods (SMART, CAT-K) artificially inflate behavior realism scores in long-term settings by "not generating new agents → low density → fewer collisions": The design of the harmonic mean RMM-F1 effectively penalizes such speculative behaviors.

Highlights & Insights

• Systematically demonstrating LLM adaptability with probing experiments rather than direct application: The paper provides falsifiable empirical evidence from the perspectives of Zipf's law and attention locality. This research paradigm of "understanding why it works before designing how to do it" is highly exemplary. In contrast, prior works (GUMP, SceneStreamer) directly applied LLMs without justifying the adaptation principles.
• Structured token sequence design balances flexibility and stability: The fixed structure of \([\text{Agent}, \text{Motion}, \text{Map}, \text{Layout}, \text{Spawn}]\) paired with the padding label mechanism allows end-to-end training with standard causal cross-entropy loss, without requiring complex multi-task weighting or multi-stage training—which is highly valuable for actual engineering deployment.
• The harmonic mean design of RTE is highly transferable: For generation tasks that require evaluating multiple potentially conflicting dimensions (such as fidelity vs. diversity in text-to-image generation), the concept of replacing global distribution matching with semantic retrieval + harmonic mean aggregation serves as a general quality evaluation framework.
• Single-stage, single-loss training for 7 epochs outperforms multi-stage training for 35 epochs: Referencing SceneStreamer, RosettaSim's exceptionally high training efficiency stems from its regular token structure, which fully exploits the LLM pretraining priors. This idea of "regularized input → reusing pretraining → reducing training overhead" is applicable to any task adapting LLMs to non-language domains.

Limitations & Future Work

• Agent deletion currently relies on heuristic boundary rules (deletion once exiting map boundaries) rather than a learnable deletion mechanism. The authors suggest that a dedicated head or special token could be added to achieve learnable deletion.
• The initial speed of newly generated agents uses learnable embeddings instead of explicit prediction, which limits the behavioral diversity of newly generated agents in long-term simulations; future work could introduce initial speed attributes for newly generated agents.
• The collision metric is only a binary indicator and does not consider collision frequency, severity, or patterns; refined collision modeling could further improve the reliability of safety assessments.
• Due to omitting a dedicated offset prediction head (using heuristic centerline snapping instead), there is a slight disadvantage on map-based metrics, with new agents occasionally overlapping with road boundaries.
• Layout Tokens rely on grid discretization (3m resolution), which may introduce quantization errors in scenarios demanding precise positioning (e.g., narrow streets, parking lots), but the paper lacks ablation analysis on this.
• The paper only validates on the Waymo Open Dataset; generalization to traffic data from other regions/sensor configurations remains to be validated.

Related Work & Insights

• vs. InfGen / SceneStreamer: Both jointly model scene generation and motion generation. However, InfGen generates the entire scene topology at once (scene-level), while RosettaSim generates new agents autoregressively token-by-token (agent-level), naturally capturing asynchronous interactions while causal attention ensures new agents do not interfere with existing trajectories. SceneStreamer first generates the initial scene and then decodes motion in parallel, requiring mixed attention mechanisms and multi-stage training; RosettaSim's fixed token structure requires only standard causal attention and single-stage training.
• vs. SMART / Trajeglish: These works are short-term specialized next-token-prediction paradigms, assuming a fixed agent set and unable to handle agent entry/exit. RosettaSim adds an autoregressive agent generation module on top of them, expanding the paradigm from "fixed-set motion prediction" to "variable-set scene evolution."
• vs. GUMP: GUMP uses a multimodal autoregressive model for scene generation and simulation, but relies on language priors and instructions while mixing multiple modal tokens. RosettaSim does not rely on any language supervision and unifies all inputs into a homogeneous structured token sequence.
• vs. LPIPS / Perceptual Metrics: RTE's inspiration directly stems from LPIPS in image generation, but changes the retrieval similarity metric to Wasserstein distance (processing Gaussian distribution parameters) instead of cosine similarity, making it more suitable for the Gaussian representation in the VAE latent space. This "retrieval + distribution divergence" evaluation framework can be generalized to other sequence generation tasks like video generation and trajectory prediction.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First to systematically demonstrate the adaptability of LLM structural priors (rather than language semantics) to traffic simulation. The design of structured variable-length token sequences to unify motion and population dynamics is simple yet effective, and RTE addresses the fundamental pain points of long-term evaluation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers both short-term/long-term settings, comparison of 5 LLMs, frozen layer ablations, Zipf distribution disruption experiments, attention locality probes, model scaling, RTE parameter ablations, and retrieval fidelity validation. The experimental design is comprehensive and offers deep causal reasoning.
• Writing Quality: ⭐⭐⭐⭐⭐ The motivational chain is clear (contradiction -> probing experiments -> method -> evaluation). The appendix contains mathematical proofs (NLL paradox, \(D_{enter}/D_{exit}\) flaws) and detailed pseudocode; the overall argumentation is highly self-consistent.
• Value: ⭐⭐⭐⭐⭐ Provides a methodological paradigm (probing experiments + structural prior adaptation) for applying LLMs to non-linguistic structured sequence modeling. RTE can independently serve as a general evaluation framework for long-term generation tasks, and the high efficiency of single-stage training is of practical significance for industrial deployment.

Related Papers

• [ICCV 2025] Long-term Traffic Simulation with Interleaved Autoregressive Motion and Scenario Generation
• [ECCV 2026] LaGen: Towards Autoregressive LiDAR Scene Generation
• [ECCV 2026] Horizon3D: Sparse Radar-Camera Fusion for Long-Range 3D Perception in Autonomous Driving
• [ECCV 2026] Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rollout
• [ECCV 2026] ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving