ECoSim: Data Efficient Fine-Tuning for Controllable Traffic Simulation

Conference: ECCV2026
Paper: Official ECCV page
PDF: Full paper
Project: https://ecosim-web.github.io/
Area: Autonomous Driving
Keywords: Controllable traffic simulation, parameter-efficient fine-tuning, feature modulation, behavior latents, context retrieval

TL;DR

ECoSim freezes existing traffic generators and adds sketch, behavior-latent, or text control through identity-initialized FiLM adapters and closed-loop training, reducing SMART sketch-control mADE from 1.5044 to 0.2561 with only 1% paired control data while using context matching to transfer behaviors into new traffic scenes.

Background & Motivation

Traffic simulation must do more than generate realistic-looking trajectories: it must let a tester specify that a particular vehicle turns, yields, or changes speed, making it possible to test alternative futures in the same environment. Models such as VBD and SMART learn rich traffic distributions, but unconditional sampling does not guarantee the desired behavior. Inference-time gradients or constraints can guide generation, at the cost of designing control objectives and adding sampling overhead. Directly conditioned approaches such as ProSim instead require substantial paired control data, making new modalities expensive to train.

The difficult part is not merely connecting a text vector to a network. It is changing the designated vehicle's behavior with little data while preserving the model's multi-agent interactions. A traffic model feeds its previous predictions into the next step, so even small initial feature perturbations can accumulate through a closed-loop rollout. Moreover, satisfying a condition does not necessarily produce a plausible scene: transferring a turn into an incompatible road environment can degrade driving quality even when trajectory tracking is accurate.

ECoSim separates the learned driving prior from new control capabilities. The frozen backbone retains the former, lightweight adapters learn the latter, and context compatibility is checked before behavior transfer. Core idea: learn control gradually through near-identity feature modulation, use closed-loop training to handle self-generated prediction errors, and constrain behavior transfer through context retrieval instead of relearning the traffic distribution for every control modality.

Method

Overall Architecture

The inputs are a map, multi-agent states, and a control signal for a designated agent; the output is a set of interacting future trajectories. For direct control, the user supplies a sketch or text. For behavior transfer, the system first selects a control source from a similar traffic context and passes its sketch, behavior latent, or text to the modality encoder. The encoded condition modulates intermediate features of the frozen decoder through FiLM. Adapters learn during closed-loop trajectory generation without updating the traffic backbone.

The system connects four designs: context-matched transfer, multimodal control representations, identity-initialized modulation, and closed-loop adapter training. The diagram places the optional transfer entry and direct input in one workflow. Its final training node describes the training procedure; deployment rolls out an already trained adapter without access to the actual future or online parameter updates.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Target traffic scene"] --> B["Context-matched transfer"]
    B --> C["Multimodal control representations"]
    U["Direct sketch or text input"] --> C
    C --> D["Identity-initialized modulation"]
    D --> E["Closed-loop adapter training"]
    E --> F["Controlled multi-agent trajectories"]

Key Designs

1. Context-matched transfer: select an environment that can accommodate the behavior

A behavior from a recorded trajectory cannot be transferred indiscriminately to any agent. ECoSim embeds the target agent and its scene context, then searches for agents with similar embeddings in the database. The retrieval pipeline also includes heuristic filtering for dynamic feasibility before ranking candidates by similarity. Once a match is selected, the system transfers the behavior control signal; surrounding agents in the target scene continue to interact through the original generator rather than being copied wholesale from the source scene. The main text does not specify filtering thresholds or the full retrieval-encoder configuration, so these details are not reconstructed here.

This entry supports two uses. Counterfactual generation retrieves different driving intentions to explore alternative futures within one scene. Long-tail generation first uses SMART trajectory likelihood to identify hard-to-predict scenes and then transfers their behavior latents into compatible environments. Predictive difficulty is only a proxy for selecting rare candidates, not a direct measure of danger. Context matching improves compatibility but provides no formal guarantee of geometric feasibility or safety.

2. Multimodal control representations: connect spatial paths, interactive behavior, and language intent to one interface

Each modality maps to a 256-dimensional control space. A sketch consists of sparse future waypoints in the agent's local frame, aggregated by a lightweight temporal encoder into a control vector. It specifies a spatial trend without requiring other agents to replay recorded trajectories. Text is encoded by DistilBERT with LoRA, using a linear projection of its [CLS] representation. The language encoder's LoRA parameters are optimized jointly with the closed-loop control objective, gradually grounding abstract commands in motion. This does not claim that text identifies a unique precise trajectory.

The third representation comes from a separately trained BehaviorVAE. It jointly reads future agent motions and environmental context to infer a Gaussian latent for each agent. Joint encoding allows these latents to capture relative geometry and interactions rather than only isolated motion curves. Trajectory reconstruction preserves behavioral information, while KL regularization and annealing constrain the latent space and encourage smooth variation. The trained encoder exports paired latents from recorded trajectories in one pass; the paper also describes sampling new behaviors from the prior, although its transfer experiments primarily use retrieved behavior codes. Only designated agents receive their corresponding vectors, while other agents receive zero vectors. A zero condition alone should not be interpreted as proof that every intermediate feature remains exactly unchanged after training.

3. Identity-initialized modulation: preserve the driving prior before altering decoder features

The control vector does not directly replace the backbone's trajectory output. Instead, it predicts channel-wise scales and shifts for decoder features at each layer. The FiLM affine transformation can be written as:

\[ \widetilde{\mathbf h}^{(l)}=\boldsymbol\gamma^{(l)}(z_i)\odot\mathbf h^{(l)}+\boldsymbol\beta^{(l)}(z_i). \]

Here, \(z_i\) is the designated agent's control representation, \(\mathbf h^{(l)}\) is the feature at layer \(l\), and two lightweight MLPs output modulation parameters matching the feature-channel dimension. This expression follows the prose description of the affine transformation; symbols are missing in the cached equation extraction. Initialization uses \(\boldsymbol\gamma\approx1\) and \(\boldsymbol\beta\approx0\), keeping the starting function close to the original model. Training then gradually learns to amplify, suppress, or shift features relevant to the requested behavior.

This addresses closed-loop stability, not merely the number of trainable parameters. If a random conditioning branch substantially changes the first trajectories, subsequent inputs also move away from states familiar to the backbone, allowing errors to compound. The paper applies the same type of modulation at every Transformer decoder-layer output in diffusion-based VBD and autoregressive SMART, freezing the backbone and learning only the control encoders and FiLM layers. Compatibility with different generative architectures does not imply an identical training procedure or shared adapter weights across them.

4. Closed-loop adapter training: correct behavior under self-generated histories

Supervising the next step only under recorded histories does not capture deployment, where the model repeatedly consumes its own predictions. ECoSim conditions and generates trajectories during closed-loop unrolling, then supervises the adapters against recorded futures. The adapters therefore learn both to follow controls and to continue generation after previous predictions have drifted. The framework reuses backbone-specific closed-loop training mechanisms rather than forcing every architecture into one optimizer procedure.

For the diffusion backbone, each replanning step generates 8 candidate joint trajectories and selects the one with the lowest matching loss against the recorded trajectory. That selected rollout supplies the adapter training loss. This is training-time candidate selection, not deployment-time selection using the actual future. For the autoregressive backbone, cross-entropy targets recorded state tokens under dynamically generated multi-agent histories, with sampling biased toward trajectories aligned with the condition. Training consequently addresses the error distribution encountered in a closed loop, rather than only open-loop prediction accuracy.

A Worked Example

A counterfactual simulation can be understood as comparing continuation of the current motion with execution of a different intention for the same target agent, road layout, and surrounding-agent states. The following is a walkthrough of the mechanism, not an additional quantitative case reported by the authors. Their counterfactual experiment selects 250 query agents and, for each, up to 4 distinct intentions with the highest context-match scores. Rather than randomly replacing a path, the pipeline first finds behavior sources that the current environment is more likely to accommodate. The source agent supplies the control signal, while the target agent and surrounding environment remain in their original scene. Retrieval therefore addresses whether a behavior is suitable to try here, while the generator determines how to execute it here.

Suppose a turning intention is selected. The sketch interface compresses the source trajectory into sparse local-frame waypoints to indicate the direction of motion. The latent interface instead uses BehaviorVAE to jointly encode the source motion and scene, compressing the agent's interaction pattern into a behavior representation. These are alternative control modalities, not two mandatory consecutive inputs. The model then predicts FiLM parameters for the designated agent and modulates its representations in the frozen decoder. Other agents do not receive this behavior condition, but continue moving as the jointly generated scene history evolves. The outcome depends on both the control and the target environment's interactions; it is not simply a copy of source-trajectory coordinates.

As simulation advances, the current state increasingly comes from the model's predictions rather than consistently following the recording. If the previous step leaves the target agent slower or at a shifted position, the next step must pursue the intention from that new state. Closed-loop adapter training exposes the adapter to precisely this situation. Dynamic conditioning re-encodes the control at every step, while static conditioning encodes once and incorporates time embeddings. Their tracking differences are measured by ablation, without assuming dynamic conditioning wins on every metric. Evaluation should separate target following from plausibility: Control ADE checks agreement with the transferred behavior, maneuver success checks the high-level intention, and collision and off-road metrics measure adverse outcomes. A strong first metric does not establish reliable performance on the others, which is why the counterfactual results require qualified safety claims.

Loss & Training

BehaviorVAE is trained separately from the control adapters on 10,000 WOMD training scenarios. Its conditional variational objective combines trajectory reconstruction with KL regularization and uses KL annealing. The corresponding cached equation is damaged and its decoder-conditioning notation is incomplete, so only the explicitly stated objective components are retained here rather than an apparently exact reconstructed ELBO.

The main experiments first additionally train VBD and SMART into the closed-loop backbones VBD-CL and SMART-tiny-CLSFT, then freeze them for adapter training. The reported 1% refers to paired control fine-tuning data, excluding backbone pretraining, closed-loop backbone preparation, and BehaviorVAE training. Data scaling covers 50 to 50,000 scenarios, approximately 0.01% to 10%. Its 100% reference uses the same control adapters while also fine-tuning the backbone with LoRA, rather than being a comparison with no additional training cost.

Key Experimental Results

Main Results

Evaluation uses WOMD under the WOSAC closed-loop simulation protocol. Higher Meta indicates better distributional realism across driving characteristics; mADE is the minimum average displacement error of candidate trajectories against the target, with lower values preferred. Relative improvement is computed as \(\mathrm{Gain}=(\mathrm{mADE}_{base}-\mathrm{mADE}_{ctrl})/\mathrm{mADE}_{base}\times100\%\), not as a percentage-point gain.

In standardized control evaluation, sketches and latents are derived from held-out recorded futures, while text comes from ProSim-Instruct-520k. This tests whether a model follows a supplied target, not whether it predicts the actual future without target information. Retrieval-based transfer addresses control without the target agent's actual future. The following results are from Table 1: every conditioned model uses 1% paired control data, and gains are relative to its own unconditional backbone.

Backbone	Control	Meta ↑	mADE ↓	Relative mADE gain ↑
VBD-CL	Unconditional	0.7186	2.7223	—
VBD-CL	Sketch	0.7432	0.9645	64.56%
VBD-CL	Latent	0.7440	0.9221	66.12%
VBD-CL	Text	0.7306	2.0005	26.51%
SMART-tiny-CLSFT	Unconditional	0.7728	1.5044	—
SMART-tiny-CLSFT	Sketch	0.7984	0.2561	82.97%
SMART-tiny-CLSFT	Latent	0.7912	0.3238	78.48%
SMART-tiny-CLSFT	Text	0.7767	1.3537	10.02%

In an additional comparison using the ProSim architecture, the original model trained with 100% paired data achieves sketch mADE of 1.099; FiLM with 1% achieves 1.130, which is close but not better. Text mADE is 2.330 versus 1.824. However, original ProSim uses the official checkpoint, whereas FiLM uses a retrained unconditional backbone, with respective unconditional mADE values of 2.679 and 2.641. Thus, “same backbone” means the same architecture, not a strictly identical starting checkpoint.

Ablation Study

Table 5 compares when the control representation is updated: Dynamic re-encodes it at each step, whereas Static encodes once and adds a time embedding. This section defaults to the autoregressive backbone and approximately 5,000 training scenarios. These are separate ablation configurations and should not be conflated with the main table.

Modality	Config	Meta ↑	mADE ↓
Sketch	Dynamic	0.7799	0.3970
Sketch	Static	0.7719	1.0598
Latent	Dynamic	0.7848	0.3151
Latent	Static	0.7880	0.4842

Dynamic encoding reduces mADE for both modalities, but static latent encoding has a slightly higher Meta score, showing that precise tracking and distributional realism need not move together. The modulation ablation also reports that additive injection has not converged after 50,000 steps, approximately 9 hours on one RTX 4090, whereas identity-initialized FiLM reaches better mADE in fewer than 12,000 steps, approximately 2.2 hours. This is an observation for that ablation setup, not a universal speedup guarantee.

The following long-tail transfer analysis comes from Table 4. SMART first selects the hardest-to-predict 0.1% of scenes; the authors curate 8 behavior categories and retrieve the top 30 compatible candidates for each. Control ADE is the average displacement error against the transferred target. Traj ADE measures departure from the unconditional generated trajectory, not a tracking error that should necessarily be minimized.

Context selection	Control ADE ↓	Traj ADE	Collision rate ↓	Off-road rate ↓	PDMScore ↑
Random	3.8723	16.33	0.3175	0.1553	39.06
Matched	1.6994	11.30	0.1574	0.0661	65.27

Collision and off-road values retain the source's decimal-rate format. PDMScore is an adapted closed-loop driving-quality score; its full aggregation is not provided in the cached main text, so absolute comparability with other PDM benchmarks should not be assumed.

Key Findings

• Spatial control is stronger than abstract text: SMART sketch control improves mADE by 82.97%, compared with 10.02% for text. The specificity of control information matters; the modalities do not impose equally strong trajectory constraints.
• Latent data efficiency does not mean zero cost. The scaling experiment describes performance near the full-data reference with 0.01% paired fine-tuning data, but BehaviorVAE separately uses 10,000 training scenarios, a prerequisite cost that must be counted.
• Context matching substantially improves long-tail transfer, yet its collision rate remains 0.1574. It reduces failures rather than eliminating the problem of enforcing unsuitable behaviors.
• Safety claims for counterfactual Table 3 require qualification. The unconditional collision rate is 0.0563, versus 0.0836 for sketch control, 0.0774 for latent control, and 0.0520 for text. Improved PDMScore and coverage do not establish that every control modality preserves safety.

Highlights & Insights

• Identity initialization matters because errors amplify through closed-loop inputs, not only because optimization becomes easier. Preserving familiar driving dynamics before learning control avoids immediately corrupting the states consumed by later steps.
• Behavior latents turn existing motion data into reusable control supervision, reducing reliance on individual language annotations. Their interaction context also explains why representation-learning cost should be separated from adapter data efficiency.
• The paper distinguishes following a recorded target from generating alternative futures. Counterfactual evaluation combines maneuver success, coverage, and driving quality instead of treating agreement with the recorded future as the sole goal.

Limitations & Future Work

• The authors acknowledge that infeasible or ambiguous conditions can make the model prioritize control satisfaction over plausible interactions. They suggest reinforcement learning fine-tuning or inference-time guidance, but the current results do not guarantee safety constraints.
• Experiments primarily control one designated agent. Although the generator models multi-agent interactions, the results do not establish coordinated control of several conditioned agents.
• Counterfactual Coverage counts grid cells whose trajectory kernel density estimate exceeds a threshold, while maneuver success measures execution of the requested high-level action. Neither captures every aspect of driving plausibility. Full thresholds, maneuver criteria, and closed-loop PDMScore details require supplementary material for cross-implementation reproduction.
• The local cache contains the complete main paper and references but not the cited supplementary material, and its FiLM and VAE equations have extraction damage. This note therefore does not invent supplementary implementation details, read unreported exact numbers from curves, or substitute qualitative stability claims for measured safety outcomes.

Related Work & Insights

• vs ProSim: ProSim directly trains a promptable closed-loop generator, whereas ECoSim adds conditioning interfaces to a frozen driving prior. Lower paired-data requirements do not eliminate the cost of the existing generator or condition encoders.
• vs ControlNet and LoRA: ECoSim follows the principle of initially preserving the pretrained function, implementing layer-wise FiLM with closed-loop training for traffic generation. It does not introduce a new general-purpose low-rank decomposition.
• vs RealGen: Both use retrieval for scenario control. ECoSim combines compatible-context retrieval with sketch, behavior-latent, and text interfaces and evaluates transfer in closed-loop generation. The key is selecting a suitable environment before control, not merely finding a similar trajectory.

Rating

• Novelty: 3.5/5 — FiLM and conditional VAEs are established; the contribution is their combination with cross-backbone closed-loop traffic adaptation and behavior transfer.
• Experimental Thoroughness: 4/5 — Covers multiple modalities, two main backbone families, data scaling, and transfer ablations, but coordinated multi-agent control and safety boundaries remain underexplored.
• Writing Quality: 3.5/5 — The main workflow is clear, but some safety language is more optimistic than the tables, and supplementary implementation details are absent from the cache.
• Value: 4/5 — Provides a practical reference for reusing traffic models and adding control interfaces with limited paired data.