PixelPilot: Scalable Vision-Language-Action Models for End-to-End Autonomous Driving

Conference: ECCV 2026
Paper: ECCV Official
Code: https://pixelpilotvla.github.io/
Area: Robotics & Embodied AI
Keywords: Vision-Language-Action Models, End-to-End Autonomous Driving, Decoupled Planning and Lifting, Data Scaling, Group Relative Policy Optimization

TL;DR

PixelPilot introduces a decoupled planning and lifting paradigm that shifts policy optimization entirely to sensor-agnostic 2D image space with deterministic 3D lifting at inference, eliminating sensor entanglement and ego-status shortcut learning while achieving state-of-the-art open-loop and closed-loop driving performance.

Background & Motivation

Vision-Language Models (VLMs) have catalyzed a major paradigm shift in end-to-end autonomous driving, where Vision-Language-Action Models (VLAs) leverage broad visual grounding and semantic commonsense to resolve long-tail driving scenarios. However, the prevailing paradigm fine-tunes 2D-pretrained VLMs to directly predict and optimize 3D world trajectories. While physically intuitive, directly regressing 3D metric trajectories introduces severe data scalability bottlenecks: learning 2D-to-3D mappings either explicitly via dedicated projection modules or implicitly through token generation tightly couples model weights with camera intrinsics and extrinsics. When aggregating diverse, heterogeneous driving datasets such as nuScenes and Waymo, identical 2D pixel coordinates correspond to completely disparate 3D physical coordinates under different sensor rigs, introducing severe spatial ambiguity.

Furthermore, direct 3D optimization severely compromises the visual reasoning capabilities of driving VLAs. Because 2D-to-3D metric inversion is ill-posed and difficult to optimize, models readily exploit spurious correlations and collapse into trivial solutions. Rather than reasoning about spatial geometry and visual scene dynamics, they learn to rely predominantly on ego-status (such as vehicle velocity and acceleration) to linearly extrapolate future trajectories. Empirical ablations reveal that when ego-status is masked and only input images are provided, state-of-the-art 3D VLAs experience a drastic performance collapse, with average L2 planning error surging from 0.36 m to 1.98 m.

To overcome these structural limitations, PixelPilot draws inspiration from human driving cognition: human drivers plan paths within their 2D visual perspective regardless of the specific vehicle type, and subsequently translate this plan into physical vehicle control based on their familiarity with the vehicle's physical dimensions. Core idea: decouple driving policy learning into sensor-agnostic 2D trajectory planning within the image plane and deterministic 3D lifting at inference, while using multi-task SFT and verifiable intermediate reward GRPO reinforcement learning to enforce a strict causal chain from visual perception to spatial planning.

Method

Overall Architecture

PixelPilot operates in two distinct phases: (1) sensor-agnostic 2D planning, which formulates perception, chain-of-thought reasoning, meta-action decision, and future trajectory waypoints entirely as 2D-to-2D generation tasks within the image plane; and (2) deterministic 3D lifting, which deterministically projects predicted 2D pixel coordinates into 3D metric coordinates during inference via camera parameters and local ground-plane intersection.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-View Images + Ego-Status Prompt"] --> B["Ego-Centric Consistency Preprocessing<br/>Back views rotated 180° stitching"]
    B --> C["2D Object Detection<br/>Predict bounding boxes and categories"]
    C --> D["Reasoning with Bounding Boxes<br/>CoT explicitly conditioned on object RoIs"]
    D --> E["Meta-Action & 2D Trajectory Planning<br/>Lateral/longitudinal actions + pixel waypoints"]
    E -->|Inference only| F["Deterministic 3D Lifting<br/>Geometric ray-plane intersection"]
    F --> G["Vehicle Chassis PID Control"]

Key Designs

1. Decoupled Planning and Lifting Paradigm: Sensor-agnostic image-space planning and deterministic 3D lifting

To eliminate sensor configuration entanglement and prevent shortcut learning from ego-status, PixelPilot optimizes the policy network entirely in the image space \(\mathcal{P}_{\text{img}}\). For short-horizon (3-second) trajectory planning, empirical road height variation is negligible (averaging only 0.16 m in nuScenes), allowing the drivable ground surface to be modeled as a local plane. Under this local plane assumption, a strict bijective projective mapping connects the 3D ground plane and its 2D image projection, ensuring that a smooth continuous 2D planned trajectory strictly maps to a feasible, smooth 3D metric path. The model autoregressively generates future pixel waypoints \((\hat{u}_t, \hat{v}_t)\).

At inference time, given camera intrinsic and extrinsic matrix \(K\) and local ground height \(h\) (ground equation \(Z = -h\)), the deterministic lifting operator \(\Psi\) computes the ray intersection with the local road plane:

\[\hat{T}_{\text{3D}} = \Psi(\hat{T}_{\text{2D}}, K, h)\]

Because absolute vehicle speed cannot be mapped to pixel displacement without contextual scene depth, the network is fundamentally prevented from bypassing visual scene cues through kinematic shortcuts.

2. Ego-Centric Consistency Preprocessing: Resolving multi-camera viewpoint discontinuity

To construct a spatially continuous and topologically coherent 2D planning surface across surrounding views, PixelPilot identifies geometric distortions in standard multi-camera stitching. Simply stacking front and rear camera views into two rows induces lane marking misalignments across view boundaries. While horizontally flipping rear camera views corrects lane marking orientation, it creates discontinuous trajectory propagation. PixelPilot introduces a 180-degree rotation stitching mechanism for rear views, preserving egocentric consistency and establishing a visually seamless road plane across the stitched panorama.

3. Reasoning with Bounding Boxes: Grounding chain-of-thought in explicit visual anchors

Free-form textual explanations generated by prior VLAs often suffer from visual hallucination and loose causal connection to final motion plans. PixelPilot introduces an explicit "Reasoning with Bounding Boxes" mechanism. The model first detects traffic participants and generates 2D bounding boxes \(\hat{B}_{\text{2D}}\). Subsequently, in its <think> reasoning block, it explicitly conditions spatial reasoning on these predicted coordinates (e.g., <think> The car at [363, 239, 402, 274] is directly ahead and appears close, so I need to reduce speed to maintain a safe following distance. </think>). The model then outputs discrete meta-actions \(\hat{A}\) and continuous 2D waypoints \(\hat{T}_{\text{2D}}\), establishing an interpretable causal sequence: visual perception \(\rightarrow\) spatial reasoning \(\rightarrow\) behavioral intention \(\rightarrow\) trajectory planning.

4. Knowledge-Instilled Policy Learning: Multi-task SFT and intermediate verifiable reward GRPO

To resolve the long-horizon credit assignment dilemma in autoregressive reasoning, PixelPilot employs a two-stage training scheme. In Stage 1, multi-task Supervised Fine-Tuning instills fundamental driving competence across 2D detection, box-grounded reasoning, meta-action decision, and trajectory prediction. In Stage 2, starting from the SFT checkpoint, Group Relative Policy Optimization (GRPO) aligns the model using a holistic composite reward \(R_{\text{acc}}\):

\[R_{\text{acc}} = \lambda_{\text{fmt}} R_{\text{fmt}} + \lambda_{\text{percep}} R_{\text{percep}} + \lambda_{\text{action}} R_{\text{action}} + \lambda_{\text{traj}} R_{\text{traj}}\]

Here, \(R_{\text{fmt}}\) enforces structured tags, bounding box completeness, and trajectory formatting; \(R_{\text{percep}}\) computes the Mean IoU via Hungarian bipartite matching against ground-truth boxes; \(R_{\text{action}}\) evaluates the macro F1 score for lateral and longitudinal meta-actions; and \(R_{\text{traj}}\) combines an L1 threshold reward (\(<10\) pixels), a sigmoid-scaled L2 reward, and closed-loop Predictive Driver Model Score (PDMS) \(R_{\text{PDMS}}\). The intermediate reasoning text is left unconstrained by rigid semantic rewards, preserving exploration flexibility while anchoring the final plan to verified visual perception.

Key Experimental Results

Main Results

PixelPilot achieves state-of-the-art results in both nuScenes open-loop trajectory prediction and Bench2Drive (CARLA) closed-loop benchmarks.

Table 1: Open-Loop Trajectory Planning on nuScenes (Original Paper Table 3)

Method	1s L2 (m) ↓	2s L2 (m) ↓	3s L2 (m) ↓	Avg. L2 (m) ↓	Collision (%) ↓	Intersection (%) ↓
DriveVLM	0.18	0.34	0.68	0.40	0.27	-
OmniDrive	0.14	0.29	0.55	0.33	0.30	3.00
OpenDriveVLA	0.15	0.31	0.55	0.33	0.10	-
Imprompt-VLA	0.13	0.27	0.53	0.30	-	-
AutoVLA	0.21	0.38	0.60	0.40	0.20	-
PixelPilot (Ours)	0.13	0.26	0.51	0.30	0.25	1.77

Table 2: Closed-Loop Autonomous Driving on Bench2Drive (CARLA) (Original Paper Table 4)

Method	Driving Score ↑	Success Rate (%) ↑	Efficiency ↑	Comfortness ↑
UniAD-Base	45.81	16.36	129.21	43.58
VAD	42.35	15.00	157.94	46.01
Orion	77.74	54.62	151.48	17.38
AutoVLA	78.84	57.73	146.93	39.33
PixelPilot (Ours)	79.14	58.87	153.26	38.01

Ablation Study

Table 3: Single-Modality Ablation on Ego-Status vs. Input Images (Original Paper Table 9)

Method	Ego-Status Available	Images Available	Avg. L2 Error (m) ↓
OmniDrive	✗	✓	1.98
OmniDrive	✓	✗	0.36
Imprompt-VLA	✗	✓	2.39
Imprompt-VLA	✓	✗	0.36
PixelPilot (Ours)	✗	✓	0.71
PixelPilot (Ours)	✓	✗	0.90

Table 4: Ablations on Training Strategy and Intermediate Rewards (Original Paper Table 5 & Table 7)

Experiment Setting	Configuration Variant	Avg. L2 Error (m) ↓
Learning Strategy	Qwen2.5-VL-7B Zero-Shot	1.92
	Stage 1 SFT Only	0.46
	Stage 2 RL Only	0.49
	SFT + RL Full Model	0.30
RL Intermediate Rewards	w/o Trajectory Reward (\(R_{\text{traj}}\))	0.44
	w/o Perception Reward (\(R_{\text{percep}}\))	0.41
	w/o Action Reward (\(R_{\text{action}}\))	0.40

Key Findings

• Elimination of Ego-Status Shortcut: As highlighted in Table 3, conventional 3D VLAs (OmniDrive and Imprompt-VLA) achieve near-perfect L2 errors of 0.36 m using ego-status alone while collapsing to 1.98 m and 2.39 m when forced to rely purely on images. In contrast, PixelPilot achieves a substantially lower L2 error of 0.71 m using only vision (outperforming its ego-status-only error of 0.90 m), confirming that image-space planning genuinely grounds decisions in visual cues.
• Cross-Dataset Scalability: Joint training on nuScenes and Waymo yields a 0.06 m reduction in average L2 error compared to training on nuScenes alone. In zero-shot evaluation on Waymo without target domain fine-tuning, PixelPilot delivers an average L2 error of 0.40 m.
• Value of Dense Intermediate Rewards: Dissecting RL rewards shows that supervising perception (\(R_{\text{percep}}\)) and meta-action (\(R_{\text{action}}\)) alongside trajectory errors (\(R_{\text{traj}}\)) addresses long-horizon credit assignment, systematically driving average L2 error down from 0.44 m to 0.30 m.

Highlights & Insights

• Paradigm Inversion: Rather than forcing a deep neural network to learn ill-posed 2D-to-3D geometric camera projections end-to-end, PixelPilot offloads metric transformation to an analytical inverse mapping, freeing model capacity entirely for visual perception and cognitive reasoning.
• RoI-Grounded Reasoning Chain: Explicitly citing predicted 2D bounding boxes inside chain-of-thought tokens establishes an auditable causal link between visual perception and motor control, preventing ungrounded linguistic hallucinations.
• Zero-Cost Vehicle Portability: The core policy model is sensor-agnostic; porting the system to a new vehicle with distinct camera focal lengths or mounting positions requires only updating camera projection matrices in the lifting module at inference time.

Limitations & Future Work

• Local Flat-Plane Assumption: The deterministic lifting step assumes a locally planar road surface over a 3-second horizon. Extreme elevation discontinuities, multi-tiered flyovers, and steep parking ramps can introduce metric depth errors.
• Occlusion and Surface Contact Ambiguity: Under severe traffic occlusion where vehicle-to-ground contact patches are masked, ray-casting can intersect non-ground points, requiring heuristic ground approximations.
• Future Directions: Integrating lightweight monocular sparse depth estimation or coarse digital elevation models (DEM) into the deterministic lifting stage could enhance non-planar terrain adaptability without sacrificing sensor-agnostic training scalability.

Related Work & Insights

• vs OmniDrive / Orion / OpenDriveVLA: These architectures employ 3D Q-Formers or 3D queries to explicitly learn camera parameter projections within model weights, tying the model to specific vehicle sensor setups and obstructing cross-dataset scaling. PixelPilot confines neural prediction to 2D image space, enabling seamless cross-dataset aggregation.
• vs AutoVLA / DriveVLM: These models directly regress 3D coordinates from text tokens and optimize using sparse terminal rewards, causing models to collapse into ego-status extrapolation. PixelPilot decouples 2D/3D spaces and introduces intermediate verifiable rewards, restoring visual reasoning integrity.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Decoupling sensor-agnostic 2D planning from deterministic 3D lifting breaks the entrenched 3D projection paradigm in autonomous driving VLAs.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations spanning nuScenes, Waymo cross-dataset scaling, Bench2Drive closed-loop benchmarks, and rigorous single-modality masking ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous geometric justification, lucid architecture diagrams, and well-structured empirical narratives.
• Value: ⭐⭐⭐⭐⭐ Provides an elegant, practical roadmap for large-scale heterogeneous pretraining and zero-cost cross-vehicle deployment of autonomous driving foundation models.

Related Papers

• [ECCV 2026] AeroVLA: A Vision-Language-Action Model for UAV Navigation via Minimalist End-to-End Control
• [NeurIPS 2025] AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
• [ICLR 2026] End-to-end Listen, Look, Speak and Act
• [ICLR 2026] LeRobot: An Open-Source Library for End-to-End Robot Learning
• [CVPR 2026] End-to-End Language-Action Model for Humanoid Whole Body Control