MOJITO: Modal Joint Learning for Unified End-to-End Autonomous Driving

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/mumucc01/MOJITO
Area: Autonomous Driving
Keywords: End-to-End Autonomous Driving, Multimodal Joint Learning, Diffusion Planner, Anchor-Free Design, Unified Transformer Architecture

TL;DR

MOJITO presents a fully unified sensor-to-action end-to-end autonomous driving framework that eliminates the information bottleneck of cascaded architectures via block-wise Modal Joint Attention, achieving state-of-the-art continuous trajectory diffusion generation without predefined trajectory anchors or dense auxiliary supervision.

Background & Motivation

The primary objective of end-to-end autonomous driving is to map raw sensory observations directly to future trajectory waypoints through joint optimization. However, predominant architectures have long adhered to a cascaded two-stage paradigm: a perception backbone first compresses multi-camera images and LiDAR point clouds into a compact latent context or coarse spatial representation, upon which a downstream planning head predicts or regresses future ego trajectories. While recent Vision-Language-Action (VLA) models enrich perception with general-purpose semantic reasoning, they maintain this one-way flow by flattening sensory inputs into discrete tokens and decoding trajectories via separate MLPs or downstream planners.

This cascaded design imposes three fundamental constraints on end-to-end systems. First, it introduces an information bottleneck; the unidirectional compression of sensory inputs irrevocably discards fine-grained geometry and dense spatial details, leaving the planner unable to query raw visual evidence during critical maneuvering. Second, it incurs heavy reliance on manual heuristics and auxiliary tasks; because trajectory supervision is applied exclusively to highly compressed latent variables, the resulting gradients provide insufficient guidance for representation learning in low-level perception backbones. Consequently, existing pipelines must introduce large sets of predefined trajectory anchors (ranging from 20 up to 8,192 clustered anchors) to stabilize action generation, while enforcing costly auxiliary supervision such as 3D bounding box detection and HD map segmentation. Third, it creates an architectural compatibility gap with modern vision foundation models; while Vision Transformers (ViTs) excel at general-purpose representation learning, current methods restrict them to isolated feature extractors, leaving their deep layers and self-attention capacity unexploited by the action planning process.

To overcome these structural limitations, this paper proposes replacing the one-way cascaded interface with an egalitarian, multimodal joint evolution paradigm. Core idea: build a unified architecture, MOJITO, comprising parallel image ViT, LiDAR ViT, and action DiT branches that interact symmetrically via block-wise Modal Joint Attention, enabling anchor-free and auxiliary-supervision-free continuous trajectory diffusion directly conditioned on dense sensory tokens.

Method

Overall Architecture

MOJITO is structured into three parallel, structurally aligned branches: a multi-view image branch built on DINOv3-S+, a LiDAR point cloud branch adapted from Uni3D-S, and an action planning branch implemented via a Diffusion Transformer (DiT). All three branches share an identical depth of 12 Transformer blocks with a hidden dimension of \(D=384\). Rather than separating perception from planning, the three modalities are processed concurrently and fused inside every block through shared multi-head self-attention, predicting an 8-waypoint trajectory \(\tau=\{w_t\}_{t=1}^8\) over a 4-second future horizon (where each waypoint \(w_t=[x_t, y_t, \theta_t]\) defines 2D position and heading angle).

The end-to-end data flow operates as follows: multi-view camera images are split into patches and linearly projected into visual tokens; raw LiDAR point clouds are discretized into metric-preserving spatial tokens via the PillarGroup module; and the action branch initializes trajectory waypoints from pure Gaussian noise, modulated by high-level navigation commands and diffusion step embeddings via adaptive layer normalization (adaLN). Within each Transformer block, tokens from all three modalities are concatenated into a unified sequence and processed through Modal Joint Attention, where planning and perception tokens mutually query and update each other. Finally, the action tokens are decoded into smooth, collision-free trajectories.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Inputs<br/>Multi-view camera images + raw LiDAR point cloud + trajectory Gaussian noise"] --> B["PillarGroup Tokenization<br/>BEV grid patchify preserving absolute metric scale and geometry"]
    B --> C["Block-wise Modal Joint Attention<br/>Bidirectional interaction among image, LiDAR, and action tokens"]
    C --> D["Anchor-Free Diffusion Planning<br/>Continuous denoising and trajectory decoding guided by command conditions"]
    D --> E["Planned Trajectory Output<br/>8 waypoints with 2D coordinates and heading over 4-second horizon"]

Key Designs

1. PillarGroup Tokenization: Absolute Metric Scale Representation for Large-Scale Driving Scenes

Standard 3D point cloud architectures conventionally employ Farthest Point Sampling (FPS) and K-Nearest Neighbors (KNN) for point grouping. While effective for object-level shapes, FPS and KNN rely solely on relative spatial distances, discarding the absolute metric scale and global coordinate layout essential for scene-level driving navigation. Furthermore, the non-uniform density of automotive LiDAR causes FPS/KNN to over-sample nearby surfaces while merging sparse distant obstacles, degrading multimodal spatial alignment. To address this, MOJITO introduces PillarGroup tokenization: the 3D space surrounding the ego vehicle is bounded within a fixed physical metric range and partitioned into a 2D BEV grid (e.g., \(32 \times 32\) physical pillars). The model selects the top-\(K\) (\(K=512\)) non-empty pillars and uniformly samples \(N=64\) points within each pillar. Because each pillar corresponds to a deterministic physical coordinate volume, the resulting tokens retain absolute metric dimensions, precise ground elevation, and consistent spatial semantics, establishing a reliable geometric foundation for joint attention.

2. Block-wise Modal Joint Attention: Bidirectional Cross-Modal Feature Interaction

Conventional end-to-end planners treat perception features as static conditioning inputs through one-way cross-attention, preventing perception features from adapting to the evolving driving trajectory. MOJITO replaces this one-way barrier with full block-wise self-attention across all 12 layers. Let \(X_I \in \mathbb{R}^{N_I \times D}\), \(X_L \in \mathbb{R}^{N_L \times D}\), and \(X_A \in \mathbb{R}^{N_A \times D}\) denote the token sequences from the image, LiDAR, and action branches at a given block, respectively. The three sequences are concatenated along the token length dimension into a unified joint representation:

\[X_{\text{joint}} = [X_I \parallel X_L \parallel X_A] \in \mathbb{R}^{(N_I + N_L + N_A) \times D}\]

After adding learnable modal positional embeddings \(PE\) and applying Layer Normalization, the concatenated sequence passes through a shared Multi-Head Self-Attention (MHSA) layer:

\[Y_{\text{joint}} = \text{MHSA}(\text{LN}(X_{\text{joint}} + PE)) + X_{\text{joint}}\]

This design enables a bidirectional information exchange: action tokens directly attend to fine-grained visual patches and LiDAR pillars to ground the denoising trajectory within drivable boundaries, while perception tokens dynamically re-weight their receptive fields based on the planned trajectory intent. Attention visualizations reveal that early layers focus broadly on forward drivable surfaces, whereas deeper layers concentrate sharply on trajectory-relevant obstacles and lane bifurcations. Following self-attention, \(Y_{\text{joint}}\) is partitioned back into \(Y_I, Y_L, Y_A\) and routed through independent Feed-Forward Networks (FFNs), decoupling task-specific updates while preserving cross-modal synergy.

3. Anchor-Free Diffusion Planning: Unconstrained Trajectory Generation in Continuous Space

Prior generative and regression planners rely heavily on hand-crafted trajectory anchors (e.g., 8,192 mode anchors in VADv2 or 20 cluster anchors in DiffusionDrive) to prevent optimization divergence, which inherently restricts trajectory diversity in rare or long-tail maneuvers. MOJITO models trajectory planning as a continuous-space conditional diffusion process. Given ground-truth future waypoints \(x^{(0)} \in \mathbb{R}^{8 \times 3}\), a forward diffusion process progressively corrupts states into noisy trajectories \(x^{(k)}\). The denoising network \(\epsilon_\theta\) processes action tokens initialized from Gaussian noise, conditioned on diffusion step \(k\) and high-level navigation commands (e.g., turn left, go straight) injected via adaptive layer normalization (adaLN). Because block-wise joint attention provides continuous fine-grained physical constraints at every network depth, the planner completely dispenses with predefined anchors, generating smooth, collision-free paths in just 2 denoising steps while supporting multi-modal trajectory synthesis with fine-grained turning angles.

Loss & Training

MOJITO is optimized end-to-end exclusively via the trajectory diffusion denoising objective, eliminating auxiliary tasks such as 3D bounding box detection, semantic segmentation, or cost volume construction. The training loss is formulated as:

\[\mathcal{L}_{\theta} = \mathbb{E}_{x^{(0)}, k \sim \mathcal{U}(0, 1), x^{(k)} \sim q(x^{(k)}|x^{(0)})} \left[ \|\epsilon_\theta(x^{(k)}, k, C) - x^{(0)}\|^2 \right]\]

where condition \(C\) encompasses multi-view visual tokens, LiDAR tokens, and high-level driving commands. The image branch and LiDAR branch are initialized from pretrained DINOv3-S+ and Uni3D-S weights and fine-tuned, while the action DiT is trained from scratch. Training is executed using AdamW across 8 NVIDIA H200 GPUs with a global batch size of 512 and a learning rate of \(6 \times 10^{-4}\). Inputs consist of three camera perspectives (front, left 60°, right 60° concatenated to \(1024 \times 256\)) alongside raw LiDAR points, predicting 8 future waypoints spanning 4 seconds.

Key Experimental Results

Main Results

Evaluation is performed on the closed-loop autonomous driving benchmarks NAVSIM-v1 and NAVSIM-v2. Primary metrics include the Predictive Driver Model Score (PDMS, evaluating no at-fault collisions NC, drivable area compliance DAC, time-to-collision TTC, comfort Comf, and ego progress EP) and Extended PDMS (EPDMS, incorporating lane keeping LK, traffic light compliance TLC, driving direction compliance DDC, history comfort HC, and extended comfort EC).

Benchmark & Method	Input Modalities	Anchors	Model Size	NC (%) ↑	DAC (%) ↑	EP (%) ↑	PDMS / EPDMS ↑
NAVSIM-v1 navtest
UniAD	Camera	0	-	97.8	91.9	78.8	83.4
Transfuser	Camera & LiDAR	0	-	97.7	92.8	79.2	84.0
DRAMA	Camera & LiDAR	0	-	98.0	93.1	80.1	85.5
VADv2	Camera & LiDAR	8,192	-	97.2	89.1	76.0	80.9
Hydra-MDP	Camera & LiDAR	8,192	-	98.3	96.0	78.7	86.5
DiffusionDrive	Camera & LiDAR	20	-	98.2	96.2	82.2	88.1
WoTE	Camera & LiDAR	256	-	98.5	96.8	82.6	88.3
ReCogDrive-Base-IL	Camera (InternVL3)	0	2B	-	-	-	86.5
AutoVLA-IL	Camera (Qwen2.5-VL)	0	3B	-	-	-	80.5
MOJITO (Ours)	Camera & LiDAR	0	127M	98.6	96.9	83.5	88.9
NAVSIM-v2 navtest
Transfuser	Camera & LiDAR	0	-	96.9	89.9	87.1	76.7
Hydra-MDP++	Camera & LiDAR	8,192	-	97.2	97.5	83.1	81.4
DriveSuprim	Camera & LiDAR	-	-	97.5	96.5	88.4	83.1
ARTEMIS	Camera & LiDAR	-	-	98.3	95.1	81.5	83.1
DiffusionDriveV2	Camera & LiDAR	20	-	97.7	96.6	88.9	85.5
MOJITO (Ours)	Camera & LiDAR	0	127M	98.2	96.2	87.8	88.4

Ablation Study

Ablations on NAVSIM-v1 navtest isolate the impact of input sensor modalities, LiDAR tokenization strategies, and cross-modal attention formulations:

Config ID	Image	LiDAR	Sampling Strategy	Attention Mechanism	NC (%) ↑	DAC (%) ↑	EP (%) ↑	PDMS ↑	Note
1	✓	✗	-	Self-Attention	98.0	95.3	81.6	86.8	Camera-only unimodal baseline
2	✓	✓	FPS + KNN	Self-Attention	97.8	94.8	81.0	86.1	Shape-level tokenization degrades metric alignment (-0.7)
3	✓	✓	PillarGroup	Cross-Attention	97.8	94.4	80.7	85.7	Unidirectional attention severs gradient feedback (-3.2)
4 (Full)	✓	✓	PillarGroup	Self-Attention	98.6	96.9	83.5	88.9	Full multimodal joint architecture achieves peak score

Key Findings

• Crucial Role of Absolute Metric Scale in PillarGroup: Introducing LiDAR with conventional FPS+KNN tokenization (Config 2) causes PDMS to drop from 86.8 (camera-only) to 86.1. Relative spatial distance sampling lacks physical scale and distorts spatial density in open driving scenes. Switching to PillarGroup grid tokenization (Config 4) restores absolute physical coordinates, boosting PDMS by +2.8 points to 88.9.
• Superiority of Bidirectional Self-Attention over Cross-Attention: Replacing Modal Joint Attention with conventional unidirectional cross-attention (Config 3, where action tokens query sensor tokens but sensor representations remain static) causes performance to drop precipitously from 88.9 to 85.7 (-3.2 PDMS). Dynamic bidirectional adaptation allows sensory tokens to focus on trajectory-critical road boundaries and obstacles as diffusion progresses.
• Strong Depth Scalability and High Parameter Efficiency: Scaling the depth of aligned blocks across the image, LiDAR, and planner branches from 3 blocks (33.4M), 6 blocks (64.4M), 9 blocks (95.5M) to 12 blocks (127M) produces steady performance gains (80.5 → 80.5 → 83.0 → 88.9 PDMS). Compared to 2B-8B parameter VLA architectures requiring complex RL pipelines, MOJITO achieves superior driving performance (88.9 vs 86.5 PDMS for ReCogDrive-Base-IL) with only 127M parameters and an inference latency of 187.65 ms.

Highlights & Insights

• Egalitarian Multimodal Representation without Cascaded Bottlenecks: MOJITO demonstrates that autonomous driving models do not require artificial intermediate representations or explicit perception proxy annotations. By evolving sensory tokens and action tokens jointly in a shared feature space, the planner directly accesses dense geometric and semantic evidence.
• First Truly Anchor-Free Diffusion Driving Planner: Unlike prior diffusion planners constrained by hand-crafted trajectory clusters, MOJITO leverages continuous physical grounding from block-wise joint attention to synthesize diverse, dynamically feasible trajectories from Gaussian noise without discrete anchor priors.
• Direct Coupling of Vision Foundation Models to Control: Rather than employing foundation models as isolated feature extractors or generating low-frequency text tokens via VLMs, MOJITO presents a clean blueprint for aligning standard ViTs with DiTs, establishing a direct vision-to-action paradigm for embodied robotics.

Limitations & Future Work

• Absence of High-Level World Knowledge and Commonsense Reasoning: By omitting large language models (LLMs), MOJITO relies purely on visual-geometric mapping, which may face challenges in complex social interactions requiring commonsense reasoning (e.g., interpreting rare police hand gestures or intricate textual road signage).
• Multi-Step Diffusion Inference Latency: Although 2-step denoising yields an inference latency of 187.65 ms (faster than multi-billion parameter VLAs), it remains slightly slower than single-stage regression models in high-frequency (10-20 Hz) vehicle control loops. Future iterations could adopt Flow Matching or Consistency Distillation for single-step execution.
• Exploration in Reactive Closed-Loop Simulation: Current evaluations focus on non-reactive benchmarks; integrating MOJITO with differentiable world models or reactive simulators will further examine its dynamic multi-agent interaction capabilities.

Related Work & Insights

• vs UniAD / Transfuser / SparseDrive: Traditional end-to-end frameworks construct heavily cascaded perception pipelines with modular proxy heads and compressed bottlenecks; MOJITO unifies perception and planning into a single Transformer, eliminating explicit perception proxy tasks.
• vs DiffusionDrive / VADv2: Prior generative planners rely on 20 to 8,192 predefined trajectory anchors to stabilize action output; MOJITO achieves completely anchor-free diffusion planning through deep joint attention with dense sensory tokens.
• vs ReCogDrive / AutoVLA: VLA frameworks demand 2B-8B parameters and intensive reinforcement learning alignment to reach competitive scores; MOJITO achieves superior performance with a lightweight 127M parameter model trained via pure imitation learning.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Completely eliminates cascaded bottlenecks and anchor dependencies in end-to-end driving, pioneering a symmetrical three-branch multimodal joint learning paradigm.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation across NAVSIM-v1 and NAVSIM-v2 benchmarks, rigorous ablations on tokenization and attention mechanisms, and compelling attention visualizations.
• Writing Quality: ⭐⭐⭐⭐⭐ Cohesive narrative structure, clear mathematical formulations, and insightful architectural comparisons.
• Value: ⭐⭐⭐⭐⭐ Establishes a highly efficient, anchor-free design paradigm that directly connects vision foundation models to robotic action generation.