MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learning

Conference: ECCV2026
Paper: ECCV Official Page (4806)
Area: Autonomous Driving
Keywords: vision-language-action models, online reinforcement learning, language-action mapping, closed-loop planning, PPO

TL;DR

MindDrive uses a shared language model with separate LoRA adapters for decision and action experts, restricting online exploration to discrete driving intentions and optimizing decisions through rewards from executed trajectories; its 3B version reaches 80.59 DS and 58.26% SR on Bench2Drive, although validation remains limited to CARLA simulation.

Background & Motivation

An autonomous driving vision-language-action model (VLA) must turn traffic understanding into the vehicle's future motion, rather than merely produce a plausible explanation. Methods such as ORION use a language model's scene understanding to generate continuous trajectories. However, reliance on imitation learning (IL) exposes the model mainly to states in which the expert has already acted correctly, whereas deployment introduces states caused by the model's own mistakes. A model may learn superficial correlates of an action without understanding how that action changes the environment, allowing errors to accumulate in closed-loop driving. Reducing offline trajectory error alone does not guarantee correct timing for stopping, yielding, or overtaking.

Direct reinforcement learning (RL) over trajectories presents another problem. Coordinates define a large continuous action space, and unconstrained exploration can generate infeasible or unsafe trajectories without yielding useful experience from expensive simulator interactions. Offline RL can improve planning using a fixed dataset, but cannot let the current policy experience the consequences of its choices. Optimizing driving decisions only through language questions also lacks a reliable interface for turning intentions such as slowing down or following a lane into executable trajectories. MindDrive aims to retain the feasible driving prior supplied by IL while using online interaction to correct high-level choices, rather than relearn every detail of motion through RL.

MindDrive therefore lets one expert select a discrete intention and another translate it into a continuous trajectory conditioned on the current scene. Online training primarily changes the former's choice distribution, but feedback comes from the latter's actual execution in the environment. Core Idea: first learn a context-dependent language-to-trajectory mapping through imitation, then optimize discrete meta-action selection with closed-loop success and failure rewards, concentrating exploration on what to do instead of arbitrary trajectory coordinates.

Method

Overall Architecture

The inputs are multi-view camera images and navigation or language instructions, with a vision encoder providing scene representations. The Decision Expert produces longitudinal and lateral meta-actions. The Action Expert combines these with the same scene and instructions to generate speed waypoints and geometric path waypoints. Both experts share a base large language model (LLM) but use separate LoRA adapters: they are neither two fully independent large models nor a system that treats decision text as a direct vehicle control signal.

Training has two stages. Driving questions and expert trajectories first establish language-action alignment. The resulting policy then executes in CARLA, collects environmental feedback, and updates the decision policy using PPO. Solid arrows below represent decision and execution flow; dashed arrows represent training supervision or parameter updates. The value network, reward computation, and reference-policy constraint belong to training, rather than additional driving experts required at deployment.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Multi-view images<br/>and navigation instructions"] --> DECISION["Dual-Expert Meta-Action Interface"]
    DECISION --> ACTION["Context-Conditioned Trajectory Decoding"]
    ACTION --> ENV["Execute trajectory<br/>and observe the next state"]
    ENV --> INPUT
    ENV -->|Online training only| RL["Closed-Loop Policy Optimization"]
    DATA["Driving questions<br/>and expert trajectories"] -.->|Imitation learning| DECISION
    DATA -.->|Imitation learning| ACTION
    RL -.->|PPO and KL updates| DECISION

The mapping is dynamic because the same intention produces different trajectories in different scenes, not because each intention is assigned one fixed trajectory. Lane following must adapt to road curvature, while slowing down must account for current speed and surrounding traffic when generating future waypoints. The high-level selection interface is discrete, but the executed trajectory remains continuous.

Key Designs

1. Dual-Expert Meta-Action Interface: explore executable driving intentions instead of coordinates

The Decision Expert interprets the scene and chooses an intention, while the Action Expert realizes it as motion. Longitudinal and lateral control are separated into 7 speed meta-actions and 6 path meta-actions, allowing speed changes and direction to be expressed separately. An LLM generates planning question-answer pairs, which are manually filtered for semantic correspondence with actions and combined with Chat-B2D reasoning data. The decision model thus learns not just to emit valid words but also the driving behavior associated with them. Separate LoRA adapters let decision adaptation and trajectory adaptation serve different purposes while drawing on shared base knowledge.

For online RL, this interface allows sampling from the probabilities of meta-action tokens instead of blindly perturbing a multidimensional waypoint space. The Action Expert supplies human-like motion candidates learned through IL, and the Decision Expert explores when to select each intention. The counts 7 and 6 should not be taken as evidence that every combination was validated as legal, nor should the correspondence between meta-actions and trajectories be read as a globally fixed lookup table. The main text refers to an appendix for full action definitions and data construction details, but the supplied cache contains only the main paper and references; the missing action inventory is not reconstructed here.

2. Context-Conditioned Trajectory Decoding: preserve continuous motion behind discrete intentions

The Action Expert autoregressively integrates visual and linguistic information. Two special tokens, <speed_waypoints> and <path_waypoints>, provide representations for a variational autoencoder (VAE) with a GRU-based decoder to generate trajectories. The longitudinal branch outputs 6 speed waypoints covering the next 3 seconds at 2 Hz. The lateral branch outputs 20 path waypoints at 1-meter intervals along a 20-meter geometric path. These branches separately describe motion over time and direction through space, providing more expressive control than coarse navigation commands alone.

The VAE turns scene- and intention-conditioned latent variables into a motion sequence, while the GRU decodes it sequentially, giving language choices consequences that can be evaluated through execution. Training relies on behavior cloning from expert trajectories, VAE distribution regularization, and auxiliary detection supervision; it does not back-propagate trajectory rewards directly through CARLA. The central online optimization target is the decision policy, while the preceding stage establishes continuous generation capability. This also constrains the system's ceiling: choosing the correct yielding intention does not guarantee correct trajectory decoding in unfamiliar road geometry.

3. Closed-Loop Policy Optimization: update choices from execution outcomes rather than offline answer scores

At each collection step, the system encodes the scene, samples meta-actions from the Decision Expert's logits, passes them to the Action Expert for trajectory generation and execution, and estimates the current state value. Rewards depend on explicit outcomes: +1 for reaching the destination, -1 for a penalty event, and 0 during other normal driving steps. A penalty immediately terminates the rollout. Events include collisions with pedestrians or vehicles, running a red light, leaving the road or deviating more than 30 meters from the route, and violating a stop sign. All penalty-component weights are 1, without an additional dense trajectory-matching reward.

Sparse rewards can work because the model already has basic driving skills and collection can produce both successful and failed experience. The authors focus exploration on scenes the IL model failed to complete and report 44 rollout routes on which action sampling can reach the destination. Temporal-difference estimates and generalized advantage estimation (GAE) assign terminal feedback to earlier decisions. PPO then increases the probabilities of favorable choices while constraining individual updates. This learns a sequential policy with delayed consequences, rather than labeling the last token before a collision as its sole cause. Evidence for improved reasoning mainly comes from changed closed-loop behavior, not an independent causal reasoning test.

To reduce storage and repeated computation during online updates, the buffer stores vision-encoder state embeddings along with values, decisions, and rewards, avoiding repeated encoding of every multi-view image during PPO updates. The value network reuses LLM weights and predicts state values through an MLP head. The statement that only the MLP head is updated concerns the value network, not a frozen Decision Expert. A reference-policy KL constraint separately regularizes the decision distribution, reducing the risk that limited new experience erases useful IL behavior. Unlike entropy regularization, which encourages greater randomness, this constraint keeps improvement anchored to a usable prior.

A Worked Example

Consider unstable speed decisions during lane following. Figure 5 compares IL and RL behavior on the same road segment. Both can retain Lanefollow, while the timing of acceleration, slowing down, or stopping changes. At T=4.00s, the IL model outputs Slow Speed and the vehicle travels at 0.35 m/s; the RL model outputs Moderate Speed and travels at 5.17 m/s. These are observations from that figure, not target speeds that should be used on every road.

During execution, the Decision Expert selects an intention, the Action Expert combines it with the scene to generate future speed and path waypoints, and the vehicle observes the scene again after moving. During training, success or a violation in the rollout combines with value estimates to determine how earlier intention probabilities should change. This example illustrates altered timing along the same geometric path, but higher speed alone does not establish greater safety, and one example does not provide a general safety guarantee.

Loss & Training

IL combines question-answer token cross-entropy, L1 behavior cloning for speed and path waypoints, the VAE KL loss, and auxiliary detection loss. Online training combines the PPO policy objective, mean squared error for value prediction, and weighted reference-policy KL regularization. To make the direction of the last term explicit, Equation (14) gives:

\[ \mathcal{L}_{KL}=D_{KL}\bigl(P_{\mathrm{ref}}(\cdot\mid s)\parallel P_{\theta}(\cdot\mid s)\bigr). \]

This constrains the Decision Expert's meta-action distribution rather than applying a Euclidean penalty to continuous trajectory coordinates. The reference distribution anchors existing behavior, while advantages from new rollouts drive PPO improvements. Several other equations in the cache contain missing symbols, broken lines, or mixed formatting. This note therefore explains their mechanisms from the prose instead of presenting guessed PPO, GAE, or total-loss reconstructions as the authors' exact equations.

The vision encoder is EVA-02-L, with Qwen2-0.5B and Qwen2.5-3B as the language backbones. Both experts use LoRA rank and alpha of 16. The paper reports experiments on 32 A800 GPUs with 80 GB each and online collection through 24 parallel CARLA simulations. The lightweight claim primarily concerns language-backbone size, not necessarily low compute or simulation cost for the full training system.

The default is 2 online RL epochs per route. More training is not consistently better: Figure 4 reports degradation from the best 78.04 DS to 73.69 and from 55.09% SR to 45.12% as training increases. The authors attribute this to overfitting recent experience and forgetting. The main text does not disclose every hyperparameter required for reproduction, and 2 epochs alone cannot determine total interaction steps or wall-clock training time.

Key Experimental Results

Main Results

Bench2Drive is based on CARLA. IL uses the official base set of 1000 clips, with 950 for training and 50 for open-loop validation. Closed-loop evaluation covers 220 short routes across 44 interactive scenario types. DS measures route completion penalized by traffic infractions, and SR is the proportion of routes completed within the time limit. Multi-Ability evaluates merging, overtaking, emergency braking, giving way, and traffic signs separately; Mean averages these five abilities.

The following selection comes from Table 1, and all rows use camera input. Avg. L2 is the open-loop trajectory error averaged over predictions for the next 2 seconds sampled at 2 Hz. Lower is better for Avg. L2 and higher is better for the other metrics. Methods share the nominal benchmark/base-set setting, but this does not establish matched online interaction budgets or total training costs.

Method	LLM	Training	DS	SR (%)	Mean (%)	Avg. L2
ORION	Qwen2-0.5B	IL	72.89	45.83	51.37	0.67
DriveMoE	Paligemma-3B	IL	74.22	48.64	47.91	0.38
AutoVLA	Qwen2.5-3B	Offline RL	78.84	57.73	Not reported	Not reported
MindDrive	Qwen2-0.5B	IL	75.85	49.30	49.44	0.69
MindDrive	Qwen2-0.5B	Online RL	78.04	55.09	56.94	0.69
MindDrive-L	Qwen2.5-3B	IL	75.78	50.71	52.10	0.66
MindDrive-L	Qwen2.5-3B	Online RL	80.59	58.26	60.81	0.66

Within-model comparisons are more direct: online RL adds 2.19 DS and 5.79 percentage points of SR to the 0.5B version, and 4.81 DS and 7.55 percentage points of SR to the 3B version. The 3B MindDrive-L exceeds AutoVLA by 1.75 DS and 0.53 percentage points of SR. Some original statements use a percent sign for differences; this note distinguishes score increments from percentage-point gains rather than recasting them as relative percentage growth.

Ablation Study

All ablations below use the 0.5B backbone, with 2 online RL epochs per route by default. The two groups in Table 3 vary policy regularization and high-level control separately; they are not a full factorial experiment.

Ablation Group	Config	DS	SR (%)	Note
Regularization	PPO-Vanilla	74.73	46.73	Without the compared additional regularizers
Regularization	PPO-Entropy	75.71	49.24	Entropy regularization
Regularization	PPO-KL	78.04	55.09	Reference-policy KL constraint
Control	Navigation Command (IL)	68.11	41.59	Control through navigation commands alone
Control	Meta Action (IL)	75.85	49.30	Establish the meta-action interface first
Control	Meta Action (RL)	78.04	55.09	Then optimize selection online

KL improves over vanilla PPO by 3.31 DS and 8.36 percentage points of SR. Vanilla PPO even falls below the IL results of 75.85 DS and 49.30% SR, showing that adding online updates alone is insufficient. Meta-action IL exceeds navigation-command IL by 7.74 DS and 7.71 percentage points of SR: both the interface itself and subsequent RL contribute.

Table 2 progressively adds penalty events: C denotes collision, TL red-light violation, RD route deviation, and S stop-sign violation. The IL row is a reference baseline, not an RL configuration trained without penalties.

Config	DS	SR (%)	Mean (%)
IL reference	75.85	49.30	49.44
C	75.41	50.70	53.20
C + TL	76.02	51.47	53.32
C + TL + RD	76.95	51.85	54.67
C + TL + RD + S	78.04	55.09	56.94

Key Findings

• Closed-loop gains do not require improved open-loop L2: it remains 0.69 for the 0.5B model and 0.66 for the 3B model despite higher DS/SR, supporting a contribution from action selection and interaction timing.
• Penalties do not improve every metric monotonically: the collision penalty alone increases SR but lowers DS from 75.85 to 75.41. The findings do not support a claim that every safety penalty improves everything.
• Adding the stop-sign penalty raises SR from 51.85% to 55.09% and merging ability from 27.63% to 32.89%. The authors attribute this to the rule's direct connection to the stop meta-action, which provides an effective signal for the selection policy.

Highlights & Insights

• Discrete and continuous control need not be alternatives. Discrete meta-actions make policy exploration manageable, while conditional trajectory decoding preserves continuous execution adapted to the scene.
• IL provides an exploration prior, not just initialization before RL. It excludes many infeasible motions from high-level selection, helping explain why simple terminal rewards can still be useful.
• State-embedding caching concentrates update resources on the decision policy but does not eliminate simulator costs. Transferring this idea should include separate reporting of policy-update and environment-collection costs.

Limitations & Future Work

• Acknowledged by the authors: evaluation is limited to CARLA, without real-road interaction validation. Simulation gains do not establish deployment safety.
• Acknowledged by the authors: difficulty synchronizing multiple CARLA instances from identical initial states limits comparisons of candidate actions and the use of GRPO. The implemented algorithm is PPO, not group-relative optimization.
• This note's assessment: the 44 rollout routes include a successful-sampling condition. The main text does not fully establish their overlap with the 220 evaluation routes, total interaction budget, or uncertainty across random seeds, so small cross-method gains should not be overstated.
• This note's assessment: the discrete vocabulary and Action Expert limit explorable behavior. Broader hazard coverage, map generalization, recovery from failures, and independent reasoning evaluation remain useful directions.
• Evidence boundary: the cache contains the main paper and references, but not the cited appendix. Full meta-action definitions, question-answer filtering details, online training pseudocode, and some hyperparameters are not verified in this note.

Related Work & Insights

• vs ORION: both connect language understanding to continuous trajectory generation. MindDrive additionally routes online environmental rewards through a meta-action interface; with Qwen2-0.5B in both models, Table 1 shows gains of 5.15 DS and 9.26 percentage points of SR.
• vs DriveMoE: DriveMoE's specialized action generation better reduces open-loop fitting error, while MindDrive emphasizes policy choices during interaction. MindDrive has 0.69 L2 versus DriveMoE's 0.38, so closed-loop gains do not imply superiority on every metric.
• vs AutoVLA / RecogDrive: the paper categorizes these as offline RL methods. MindDrive instead executes its policy in simulation and observes the consequences of its own choices. Comparisons still need to account for model size, data, and interaction budgets.
• Transferable insight: a robot could first learn conditional executors for a finite skill set, then optimize skill selection through environmental returns. This requires both a sufficiently expressive skill interface and reliable executors in newly explored states.

Rating

• Novelty: 4/5. The main contribution combines language-action mapping with online closed-loop RL rather than introducing a new PPO algorithm.
• Experimental Thoroughness: 4/5. Experiments cover model scale, control interface, rewards, regularization, and training epochs, but remain simulation-only with incomplete reproduction details.
• Writing Quality: 4/5. The argument and ablation structure are clear, although some reasoning-improvement claims exceed the directly measured evidence.
• Value: 4/5. The work provides a concrete low-dimensional policy-exploration strategy for driving VLAs; real-world safety requires separate validation.