LEAP-VLA: Latent-Enhanced Action Prototyping via Continuous Residual Latent Spaces for Vision-Language-Action Models

Conference: ECCV 2026
Paper: ECCV Original
Area: Robotics & Embodied AI
Keywords: Vision-Language-Action Model, Continuous Residual Latent Space, Action Prototyping, Embodied AI, Single-Pass Inference

TL;DR

Addressing the reliance of vision-language-action models on massive parameter counts and expensive iterative diffusion decoders, LEAP-VLA proposes a decoupled action representation paradigm: Stage 1 learns a structured continuous action manifold via Multi-level Soft Residual Quantization (MSRQ), and Stage 2 trains a lightweight VLM-external aligner to perform single-pass soft prototype selection, outperforming SOTA models of several times its size using only a 1B backbone.

Background & Motivation

Vision-Language-Action (VLA) models have advanced rapidly, yet most gains still stem from the unconstrained scaling of backbone parameter counts (ranging from 7B to 8.5B parameters), massive pretraining on large-scale embodied demonstration datasets (such as Open X-Embodiment and DROID), or computationally heavy multi-step iterative denoising decoders (e.g., Diffusion or Flow-Matching heads). Behind this wave of scale-driven success, however, action representation itself has long been treated as a secondary architectural detail: direct regression on raw action vectors lacks geometric constraints and manifold regularization, frequently collapsing under complex multimodal distributions; discrete action tokenization (such as VQ-VAE or uniform binning) provides prototype priors but introduces non-differentiable hard assignments that rely on the straight-through estimator (STE) and suffer from severe codebook collapse; and diffusion or flow-matching policies require multi-step iterative sampling during inference, introducing substantial latency bottlenecks for high-frequency closed-loop robot control.

The core tension lies in the fact that conventional VLA paradigms force high-dimensional multimodal features to directly fit an unstructured or discretely fractured action space. Consequently, mapping vision and language inputs to robot actions becomes a blind search over an unconstrained output manifold, forcing models to rely on sheer scale and vast robot datasets to achieve implicit alignment. Without fundamentally establishing the physical continuity and semantic hierarchical structure of the action space, smaller models lacking large-scale embodied pretraining struggle to achieve high precision and generalizability on long-horizon complex manipulation tasks.

The angle of attack in this paper is inspired by latent diffusion models that shift image generation from high-dimensional pixels to compact latent spaces: LEAP-VLA decouples action generation from raw trajectory space into two distinct phases—action structure learning and multimodal alignment. Core idea: learn a continuous, coarse-to-fine action prototype latent space from demonstration trajectories using Multi-level Soft Residual Quantization (MSRQ), and train a lightweight VLM-external aligner to perform soft prototype selection in a single forward pass over the frozen latent manifold, enabling highly accurate action generation without iterative denoising or VLM backbone modification.

Method

Overall Architecture

LEAP-VLA adopts a decoupled two-stage pipeline. In Stage 1 (action structure learning), the model trains an action autoencoder composed of a Bidirectional CNN-Transformer (BiCT) strictly on expert action sequences without vision or language inputs, employing Multi-level Soft Residual Quantization (MSRQ) over learnable continuous prototypes to construct a coarse-to-fine action latent manifold. In Stage 2 (multimodal latent alignment), the learned action prototypes and ActionDecoder are frozen. A lightweight Latent Action Aligner operates externally to the frozen VLM backbone (InternVL3-1B with rank-32 LoRA adapters), extracting multi-layer visual representations, compressed text tokens, and proprioceptive state embeddings to predict latent codes via Aligner Attention and a Prototype-Aware Output Head. Finally, the frozen Stage 1 decoder reconstructs executable action chunks in a single forward pass.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Observations: Images + Instruction + Robot State"] --> B["Multi-level Soft Residual Quantization (MSRQ)<br/>Action-only pretraining constructs coarse-to-fine prototype manifold"]
    B --> C["Bidirectional CNN-Transformer (BiCT)<br/>Hybrid modeling of local motion dynamics and global trajectory context"]
    C --> D["External Latent Aligner & Prototype-Aware Output Head<br/>Three-way joint attention + single-pass soft prototype selection"]
    D --> E["Executable Action Chunks: Stage 1 frozen decoder reconstruction"]

Key Designs

1. Multi-level Soft Residual Quantization: continuous coarse-to-fine action prototype learning via squared Euclidean distance

Traditional discrete Vector Quantization (VQ) relies on non-differentiable argmin assignments, leading to severe codebook collapse where many prototypes remain inactive. MSRQ eliminates hard assignment by maintaining learnable prototype sets \(P^{(l)} \in \mathbb{R}^{K \times D_c}\) across \(L\) residual levels (\(K=256, D_c=32\)). Unlike image tokenizers that use cosine similarity and consequently discard physical action magnitude, MSRQ adopts squared \(\ell_2\) distance to preserve motor command scales. At each residual level \(l\), the residual signal \(r^{(l-1)}\) is compared against prototypes under a learnable temperature \(\tau^{(l)}\) via softmax:

\[\boldsymbol{\alpha}^{(l)} = \mathrm{softmax}\left(-\frac{\|r^{(l-1)} - P^{(l)}\|_2^2}{\tau^{(l)}}\right)\]

The level's output is calculated as the prototype matrix weighted by the soft assignment vector and modulated by a learnable scaling factor \(s^{(l)}\): \(\Delta \mathbf{z}^{(l)} = s^{(l)} (\boldsymbol{\alpha}^{(l)} P^{(l)})\), with the residual updated as \(r^{(l)} = r^{(l-1)} - \Delta \mathbf{z}^{(l)}\). The overall latent code aggregates contributions across all levels: \(\mathbf{z}_q = \sum_{l=1}^L \Delta \mathbf{z}^{(l)}\). During optimization, the learned temperatures \(\tau^{(l)}\) and scale factors \(s^{(l)}\) converge monotonically downward across levels, naturally reflecting a coarse-to-fine hierarchy from global motion primitives down to fine-grained positional adjustments. This formulation is fully differentiable end-to-end, dispensing with the straight-through estimator and eliminating codebook collapse by construction.

2. Bidirectional CNN-Transformer architecture: harmonizing local smoothness with global trajectory coherence

Robot trajectory execution requires balancing high-frequency local dynamics between adjacent control steps with long-horizon procedural dependencies (such as reaching before grasping). Unconstrained MLPs lack temporal inductive bias, causal Transformers produce jagged step transitions because unidirectional attention ignores future horizon context, and pure CNNs suffer from restricted receptive fields. LEAP-VLA introduces a Bidirectional CNN-Transformer (BiCT) autoencoder: the encoder first processes action inputs of dimension \(D_a\) through two 1D convolution blocks (kernel size 3 with GroupNorm and GELU) into hidden dimension \(D_h\) to capture local dynamic transitions, followed by a 4-layer 8-head bidirectional Transformer modeling global dependencies across the full horizon \(T=8\). The decoder mirrors this topology in reverse order, reconstructing global temporal context through the bidirectional Transformer before mapping back to control commands via 1D convolutions without activation functions. This architecture cuts reconstruction Mean Absolute Error (MAE) by nearly \(50\times\) compared to causal Transformer baselines.

3. External latent aligner and prototype-aware output head: drift-free single-pass manifold prediction

To avoid corrupting the pretrained multimodal foundation during downstream adaptation, LEAP-VLA maintains an aligner entirely external to the VLM, tuning only lightweight rank-32 LoRA adapters on InternVL3-1B. The aligner features a Text Compressor (condensing variable-length instructions into \(k\) fixed query tokens via cross-attention), an Action Initializer (mean-pooling multi-layer visual features to seed \(T\) action tokens), and stacked Aligner Attention blocks. Aligner Attention executes joint 3-way attention across self, language-proprioception, and visual key-value streams within a single softmax operation, employing a learnable gate initialized at \(\tanh(0)=0\) to prevent initial gradient dominance by the large volume of visual tokens. Rather than using an unconstrained linear projection for output regression, the Prototype-Aware Output Head maps refined representations \(\mathbf{h}\) directly onto the frozen prototype manifold:

\[\Delta \hat{\mathbf{z}}_q^{(l)} = s^{(l)} \cdot \mathrm{softmax}\left(\frac{\mathrm{Linear}_l(\mathbf{h})}{\tau^{(l)}}\right) P^{(l)}\]

Because predictions are formed as convex combinations over frozen Stage 1 prototypes, the output \(\hat{\mathbf{z}}_q\) is guaranteed to lie within the valid learned action manifold, completely eliminating physically impossible out-of-distribution commands. The decoded action \(\hat{\mathbf{a}} = \mathcal{D}(\hat{\mathbf{z}}_q)\) is then generated in a single pass without iterative diffusion steps.

Loss & Training

Stage 1 jointly optimizes reconstruction fidelity and prototype diversity via \(\mathcal{L}_{\mathrm{Stage1}} = \mathcal{L}_{\mathrm{recon}} + \lambda_{\mathrm{ent}} \mathcal{L}_{\mathrm{entropy}}\). Reconstruction adopts an L1 loss to prevent over-smoothing large-magnitude commands (e.g., rapid gripper toggles). The entropy regularization loss penalizes deviations from uniform prototype usage across the batch:

\[\mathcal{L}_{\mathrm{entropy}} = 1 - \frac{1}{L} \sum_{l=1}^L \frac{\mathcal{H}(\bar{\boldsymbol{\alpha}}^{(l)})}{\log K}\]

Stage 2 applies dual supervision across both latent and action spaces: \(\mathcal{L}_{\mathrm{Stage2}} = \mathcal{L}_{\mathrm{align}} + \mathcal{L}_{\mathrm{action}}\). The alignment loss \(\mathcal{L}_{\mathrm{align}} = \mathcal{L}_{\mathrm{prox}} + \lambda_{\mathrm{dir}} \mathcal{L}_{\mathrm{dir}}\) combines MSE Euclidean proximity with cosine directional consistency (\(\lambda_{\mathrm{dir}}=0.5\)), while \(\mathcal{L}_{\mathrm{action}}\) enforces L1 fidelity between decoded actions and expert demonstrations.

Key Experimental Results

Main Results

On the LIBERO benchmark (evaluating 4 diverse task suites with 50 evaluation rollouts per task) and CALVIN ABC→D (evaluating cross-environment generalization under zero-shot lighting and scene shifts), LEAP-VLA outperforms existing SOTA methods across all suites:

Model	Venue	Params (B)	Robot PT	LIBERO-Spatial	LIBERO-Object	LIBERO-Goal	LIBERO-Long	LIBERO-Avg
UD-VLA	ICLR'26	8.5	△ (Video)	94.1%	95.7%	91.2%	89.6%	92.7%
OpenVLA	CoRL'24	7.0	✓ (OXE etc.)	84.7%	88.4%	79.2%	53.7%	76.5%
UniVLA	RSS'25	7.0	✓ (OXE etc.)	96.5%	96.8%	95.6%	92.0%	95.2%
CoT-VLA	CVPR'25	7.0	✓ (OXE etc.)	87.5%	91.6%	87.6%	69.0%	81.1%
ThinkAct	NeurIPS'25	7.0	✓ (OXE etc.)	88.3%	91.4%	87.1%	70.9%	84.4%
\(\pi_0\) (Flow-Matching)	RSS'25	3.0	✓ (OXE etc.)	96.8%	98.8%	95.8%	85.2%	94.2%
Fast-ThinkAct	CVPR'26	3.0	✓ (OXE etc.)	92.0%	97.2%	90.2%	79.4%	89.7%
LEAP-VLA (Ours)	ECCV 2026	1.0	✗ (VLM only)	98.4%	99.4%	97.6%	93.0%	97.1%

On CALVIN ABC→D, LEAP-VLA reaches an average completed sequence length of 4.28 consecutive subtasks (completing 5 chained tasks in a row at a 73.0% success rate), outperforming both the un-pretrained VLM baseline VLM4VLA (4.06, 8.3B params) and robot-pretrained policies such as OpenVLA-OFT (4.10, 7B params).

Ablation Study

The ablation study on action prediction targets and output head architectures on LIBERO-Spatial highlights the contribution of each design component:

Prediction Target	Output Head	Spatial Success (%)	Note
Raw action \(a\)	Linear Head	92.4%	Unconstrained baseline direct regression
Latent code \(z_q\)	Linear Head	94.6%	+2.2% gain from introducing MSRQ latent targets
Latent code \(z_q\)	Prototype-Aware Output Head	98.4%	+3.8% gain by constraining predictions to prototype convex hull

Quantization performance and codebook utilization across different quantization variants (all using BiCT, \(K=256\), 10K steps):

Quantization Method	Level Config	Entropy Ratio (Ent.)	Effective Count (Eff., max=256)	Reconstruction MAE (\(\times 10^{-3}\))
VQ-VAE (grad.)	Single-Level	58.3%	25.3	86.6
OptVQ	Single-Level	98.6%	237.3	79.4
SoftVQ	Single-Level	90.6%	151.6	4.9
RVQ-Commit	4 Levels (4L)	81.5%	129.6	9.0
SoftRVQ	4 Levels (4L)	90.4%	151.5	6.0
MSRQ (Ours)	4 Levels (4L)	99.6%	250.8	1.5

Key Findings

• Action space structure outweighs raw model capacity: Shifting prediction targets from unconstrained raw actions to the continuous MSRQ latent space brings an immediate +2.2% improvement, while incorporating the Prototype-Aware Output Head yields an additional +3.8% jump. Explicit prototype navigation provides stronger performance dividends than scaling backbone size.
• Uniform codebook utilization across residual hierarchy: While discrete residual quantization suffers from severe shallow-level collapse (RVQ-Commit drops to 51.9% utilization at Level 0), MSRQ maintains \(\ge 99.5\%\) utilization across all 4 levels through smooth \(\ell_2\) softmax weighting and entropy regularization.
• Bidirectional context prevents trajectory disruption: Ablations on temporal architectures demonstrate that replacing BiCT with a causal Transformer increases MAE by nearly 50 times (\(1.5 \times 10^{-3} \to 71.9 \times 10^{-3}\)), proving that bidirectional attention is crucial for action chunk modeling.

Highlights & Insights

• Geometric manifold constraints embedded into the prediction head: The Prototype-Aware Output Head enforces that generated action latents are formed exclusively via convex combinations over validated Stage 1 prototypes, preventing policy drift into physically invalid action regimes.
• External alignment preserves generalist multimodal representations: By isolating multimodal fusion and action decoding into an external aligner, the VLM backbone remains completely untouched, fully preserving rich semantic reasoning capabilities without task-specific interference.
• Fast single-pass execution: Producing action chunks through a single forward pass eliminates the computational and temporal overhead of iterative diffusion sampling, offering high control frequencies for real-world robotic systems.

Limitations & Future Work

• Absence of real-world hardware validation: Experiments are currently confined to simulation benchmarks (LIBERO and CALVIN); deployment onto physical robotic platforms may introduce distribution shifts caused by sensor calibration inaccuracies and contact dynamics.
• Semantic upper bound dictated by compact VLM capacity: While a 1B model performs remarkably well on standard benchmarks, tasks requiring extensive commonsense reasoning or fine-grained visual parsing may still be bottlenecked by backbone size.
• Cross-embodiment transfer unexplored: Future investigations should evaluate whether a unified MSRQ latent prototype space can represent diverse robot kinematic morphologies across heterogeneous embodiments.

Related Work & Insights

• vs OpenVLA / UniVLA: Prior models rely on discrete action tokenization and autoregressive prediction, facing quantization errors and codebook collapse; LEAP-VLA implements continuous soft residual quantization with smooth gradient propagation, reaching higher success rates without embodied pretraining.
• vs \(\pi_0\) / Flow-Matching: Diffusion-based policies require iterative denoising steps that increase inference latency; LEAP-VLA learns a structured prototype space to enable single-pass soft selection, significantly accelerating closed-loop control.
• vs ACT (Action Chunking Transformer): ACT assumes an unstructured Gaussian prior via CVAE, suffering from a train-test mismatch by taking \(z=0\) during inference; LEAP-VLA provides genuine geometric grounding through learnable multi-level prototypes.

Rating

• Novelty: ⭐⭐⭐⭐☆ Introduces MSRQ continuous soft residual quantization and external single-pass prototype alignment, moving beyond discrete tokens and diffusion decoders.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across LIBERO and CALVIN, including detailed codebook utilization, temporal architecture, and output head ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear conceptual motivation, rigorous mathematical formulation, and well-structured empirical validation.
• Value: ⭐⭐⭐⭐⭐ Demonstrates that structured continuous action spaces allow lightweight 1B models to surpass 7B+ systems, providing an efficient path for embodied AI deployment.