Towards More Efficient Decoding for Autoregressive Vision-language-action Models¶
Conference: ECCV 2026
Paper: ECCV 2026 Poster Page
Area: Robotics & Embodied AI
Keywords: Vision-Language-Action Models / Jacobi Parallel Decoding / Consistency Distillation / Adaptive Early-exit Decoding / Robotic High-Frequency Control
TL;DR¶
Addressing the severe inference speed bottleneck of autoregressive Vision-Language-Action (VLA) models in robotic control, CEED-VLA introduces consistency distillation on Jacobi trajectories to break token dependencies and pairs it with adaptive early-exit decoding to cut tail redundant iterations, achieving over 4× acceleration across diverse architectures while preserving high manipulation success rates.
Background & Motivation¶
End-to-end Vision-Language-Action (VLA) models have emerged as a dominant foundation for embodied intelligence and robotic manipulation. While recent works introduce flow matching and continuous diffusion policies to achieve fine-grained trajectory generation, autoregressive (AR) architectures—exemplified by OpenVLA, π0-FAST, and UniVLA—remain highly compelling due to their superior sample efficiency during training, robust instruction-following capabilities, and seamless unified modeling across vision, language, and action tokens. However, the standard autoregressive paradigm mandates sequential next-token prediction, requiring multiple forward passes to generate an action chunk. This serial decoding mechanism fundamentally limits control frequency, creating a prohibitive bottleneck when deploying VLAs on high-frequency, dexterous real-world manipulation tasks such as pouring liquids or folding cloth.
To overcome this sequential limitation, recent studies have explored Jacobi decoding, reframing sequential generation into a system of nonlinear fixed-point equations solved concurrently. While promising, directly executing Jacobi decoding on standard autoregressive VLAs yields marginal empirical speedup (only around 1.28×). The root cause lies in training exposure: vanilla VLAs are trained under strict teacher-forcing on ground-truth prefixes. Consequently, when preceding tokens in intermediate Jacobi iterations are imperfect—a natural condition during early iterations—the model fails to predict subsequent tokens correctly. In practice, the model still only resolves one correct token per iteration. Furthermore, the strict convergence criterion of Jacobi decoding (\(Y^{(k)} = Y^{(k-1)}\)) induces lengthy tail iterations where token updates barely alter the physical action, severely dragging down average throughput and causing latency spikes (minimum decoding speed plummeting from 30.87 to 19.50 tokens/s).
Addressing these sequential dependencies and convergence bottlenecks requires transforming parallel decoding from a test-time heuristic into a joint train-test co-optimization framework. Core idea: propose CEED-VLA, which performs consistency distillation on teacher Jacobi trajectories to empower the student model to map arbitrary intermediate states directly to converged fixed points, and couples this with confidence-guided adaptive early-exit decoding and prefix KV reuse to eliminate redundant iterations, achieving up to 4.1× inference speedup and 4× real-world control frequency.
Method¶
Overall Architecture¶
CEED-VLA restructures the decoding dynamics of autoregressive VLAs through an end-to-end four-stage pipeline: teacher model initialization, offline Jacobi trajectory harvesting, consistency distillation training, and inference acceleration via adaptive early-exit decoding with prefix key-value reuse.
The workflow begins with a pretrained autoregressive VLA teacher model \(P_\gamma\). During offline data preparation, the teacher model solves action prediction tasks on robot demonstration datasets via standard Jacobi fixed-point iteration, recording the entire sequence of states from random initialization to final convergence as trajectory \(\mathcal{J}\). A student model \(Q_\theta\) with identical architecture is then initialized from \(P_\gamma\) and optimized via a consistency objective, learning to jump from any perturbed intermediate state along the trajectory directly to the fixed point, regularized by an auxiliary autoregressive loss to retain action precision. Finally, during runtime inference, the student model decodes action tokens in parallel, terminating early once the minimum token confidence meets a threshold, while reusing cached attention representations of static visual and linguistic prompts.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Multi-view Observations + Language Instruction"] --> B["Jacobi Trajectory Collection<br/>Teacher solves fixed-point iterations and logs trajectories"]
B --> C["Consistency Distillation Learning<br/>Student maps arbitrary intermediate states to fixed points"]
C --> D["Adaptive Early-exit Decoding<br/>Monitors minimum confidence to relax strict convergence"]
D --> E["Prefix Attention State Reuse<br/>Caches static visual and textual KV embeddings"]
E --> F["Output: High-Frequency Smooth Action Token Sequence"]
Key Designs¶
1. Jacobi Trajectory Collection: Capturing Multi-Token Parallel Dynamics To overcome the inability of vanilla autoregressive VLAs to correct errors under noisy preceding contexts, this design systematically records the iteration dynamics of Jacobi decoding. For a prompt \(x\) comprising visual observation \(s_t\) and instruction \(l\), standard autoregressive generation produces \(n\) action tokens sequentially via \(y_i = \arg\max_y p(y \mid \mathcal{Y}_{i-1}, x)\). Jacobi decoding reformulates this as a system of nonlinear equations: $\(y_i^{(j+1)} = \operatorname*{arg\,max}_{y}\;p\left(y \mid \mathcal{Y}_{i-1}^{(j)}, x\right), \quad i=1,\dots,n\)$ Starting from a randomly initialized token sequence \(Y^{(0)} = \{y_1^{(0)}, \dots, y_n^{(0)}\}\), every token variable is updated simultaneously in each forward pass until the sequence no longer changes between successive iterations, satisfying \(Y^{(k)} = Y^{(k-1)}\) to reach fixed point \(Y^* := Y^{(k)}\). By archiving the sequence of iterations into a full trajectory \(\mathcal{J} = \{Y^{(0)}, Y^{(1)}, \dots, Y^*\}\), the method builds a distillation dataset \(\mathcal{D}\) exposing intermediate non-converged states.
2. Consistency Distillation Learning: One-Step Mapping from Intermediate States to Fixed Points Slow convergence in traditional Jacobi decoding stems from the model never encountering noisy intermediate states during training. Drawing inspiration from Consistency Models, this design enforces self-consistency within the student VLA \(Q_\theta\): the model must directly map any intermediate state \(Y\) along trajectory \(\mathcal{J}\) to the converged fixed point \(Y^*\), bypassing iterative steps. The algorithm samples an intermediate iteration \(Y \sim \mathcal{J}\) and minimizes the forward Kullback-Leibler (KL) divergence between the student distribution and the stop-gradient teacher distribution at the fixed point: $\(\mathcal{L}_{\text{C}} = \mathbb{E}_{(s_t, l, \mathcal{J})\sim \mathcal{D}, Y\sim \mathcal{J}}\left[\sum_{i=1}^n \mathrm{KL}\left(Q_{\theta^-}(\cdot \mid \mathcal{Y}_i^*, s_t, l) \,\parallel\, Q_\theta(\cdot \mid \mathcal{Y}_i, s_t, l)\right)\right]\)$ where \(\theta^- = \text{stopgrad}(\theta)\). This objective cultivates "fixed tokens"—tokens correctly predicted in a single pass despite errors in preceding positions—greatly accelerating convergence across the action chunk.
3. Adaptive Early-exit Decoding: Confidence-Guided Relaxation of Strict Convergence The rigid convergence rule (\(Y^{(k)} = Y^{(k-1)}\)) generates severe tail latency, even though minor token fluctuations in final iterations carry negligible physical impact on robot trajectory execution. This design introduces an adaptive early-exit mechanism driven by output token confidence. For token \(i\) at iteration \(j\), the confidence score is defined as the maximum candidate probability over vocabulary \(r\): $\(c_{j,i} = \max_r Q_\theta\left(r \mid \mathcal{Y}_{<i}^{(j)}, s_t, l\right)\)$ The global sequence quality is measured by the minimum token confidence \(c_{\min}^{(j)} = \min_{0 \le i \le n} c_{j,i}\). Given threshold \(\sigma\) (optimally set to \(0.5\)), whenever \(c_{\min}^{(j)} \ge \sigma\), the model halts decoding immediately and emits intermediate state \(Y^{(j)}\) as the final action. This truncates long-tail iterations, lifting the minimum inference speed from 19.50 tokens/s to 62.07 tokens/s.
4. Prefix Attention State Reuse: Eliminating Multi-Round Visual and Linguistic Redundancy Throughout iterative Jacobi updates, the prompt prefix tokens—comprising multi-view camera inputs and linguistic instructions—remain identical across iterations. Repeating forward feature extraction and attention projections over this static context wastes substantial memory bandwidth and computation. CEED-VLA locks and caches the Key and Value attention states of prefix tokens on the initial pass, updating only the action token representations in subsequent Jacobi steps.
Loss & Training¶
To prevent the student model from diverging from valid manipulation policies during consistency training, the framework incorporates an auxiliary autoregressive cross-entropy loss: $\(\mathcal{L}_{\text{AR}} = \mathbb{E}_{(s_t, l, \mathcal{J})\sim \mathcal{D}}\left[-\sum_{i=1}^n \log Q_\theta\left(\mathcal{Y}_i^* \mid \mathcal{Y}_{<i}^*, s_t, l\right)\right]\)$ The composite loss function is balanced by weighting factor \(\omega\): $\(\mathcal{L}(\theta) = \mathcal{L}_{\text{C}} + \omega \mathcal{L}_{\text{AR}}\)$ In practice, \(\omega = 1.0\) yields optimal trade-offs. Profiling on 8 NVIDIA A100 GPUs indicates that while base VLA fine-tuning takes 48 hours, CEED-VLA requires only 2 hours for trajectory collection and 12 hours for consistency distillation. Moreover, pre-collected trajectories can be reused across training runs, ensuring computational lightweightness.
Key Experimental Results¶
Main Results¶
Evaluation was conducted on simulation benchmarks CALVIN (ABC→D long-horizon multi-subtask completion) and LIBERO (Long 10 multi-step manipulation suite) across LLaVA-VLA, OpenVLA, π0-FAST, and UniVLA.
| Benchmark & Base Model | Chunk Size | Decoding Method | Speed (Tokens/s) | Task Metric (Avg Len / SR %) | Frequency (Hz) |
|---|---|---|---|---|---|
| CALVIN (LLaVA-VLA) | - (1-step) | Autoregressive (AR) | 39.6 (1.0×) | 2.01 | 2.23 (1.0×) |
| CALVIN (LLaVA-VLA) | Chunk 5 | Autoregressive (AR) | 39.6 (1.0×) | 3.70 | 4.33 (1.9×) |
| CALVIN (LLaVA-VLA) | Chunk 5 | FastV (Token Pruning) | 28.7 (0.7×) | 2.54 | 1.87 (0.8×) |
| CALVIN (LLaVA-VLA) | Chunk 5 | SparseVLM (Sparsification) | 32.4 (0.8×) | 2.83 | 2.01 (0.9×) |
| CALVIN (LLaVA-VLA) | Chunk 5 | PD-VLA (Parallel Decoding) | 52.8 (1.3×) | 3.69 | 5.43 (2.4×) |
| CALVIN (CEED-VLA, Ours) | Chunk 5 | Adaptive Early-exit | 79.0 (2.0×) | 3.68 | 7.26 (3.3×) |
| LIBERO-Long (OpenVLA) | - (1-step) | Autoregressive (AR) | 54.4 (1.0×) | 53.2% | 5.95 (1.0×) |
| LIBERO-Long (OpenVLA) | Chunk 3 | Autoregressive (AR) | 54.4 (1.0×) | 60.4% | 7.08 (1.2×) |
| LIBERO-Long (OpenVLA) | Chunk 3 | PD-VLA (Parallel Decoding) | 85.0 (1.6×) | 62.4% | 10.79 (2.4×) |
| LIBERO-Long (CEED-VLA, Ours) | Chunk 3 | Adaptive Early-exit | 225.2 (4.1×) | 62.3% | 25.62 (4.3×) |
Generalization across diverse action tokenization schemes and unified world-modeling backbones on LIBERO:
| Model & Decoding Setup | Spatial SR% / Speed | Object SR% / Speed | Goal SR% / Speed | Long SR% / Speed | Average SR% / Speed |
|---|---|---|---|---|---|
| π0-FAST (Baseline) | 96.4% / 105.3 | 96.8% / 108.1 | 88.6% / 109.2 | 60.2% / 107.5 | 85.5% / 107.5 (1.0×) |
| π0-FAST + CEED-VLA | 96.6% / 419.4 | 96.2% / 422.1 | 89.4% / 421.6 | 59.9% / 419.3 | 85.5% / 420.6 (3.9×) |
| UniVLA (Baseline) | 97.0% / 50.2 | 99.0% / 50.6 | 92.6% / 50.4 | 90.8% / 51.1 | 94.8% / 50.6 (1.0×) |
| UniVLA + CEED-VLA | 96.8% / 200.4 | 98.2% / 199.6 | 92.0% / 198.7 | 90.6% / 202.4 | 94.4% / 200.3 (4.0×) |
Real-world deployment on dual AgileX PiPer robotic arms (20 rollouts/task, 80 total rollouts): - Basic tasks: Push button success reached 85% (vs. 65% for LLaVA-VLA); Lift block reached 70% (vs. 40%). - High-frequency dexterous tasks: Pour water reached 80% (vs. 15%); Fold towel reached 75% (vs. 5%). Average success surged from 33.25% to 77.50%, with control frequency rising from 3.3 Hz to 13.0 Hz (3.9× speedup).
Ablation Study¶
Component ablation on OpenVLA across LIBERO-Long:
| Jacobi Decoding | Consistency Training | Adaptive Early-exit | Speedup | Success Rate (%) |
|---|---|---|---|---|
| ✓ | ✓ | ✓ | 4.1× | 62.3% |
| ✓ | ✓ | ✗ | 2.5× | 61.2% |
| ✓ | AR Fine-tuning only | ✗ | 2.3× | 54.2% |
| ✓ | ✗ | ✗ | 1.6× | 62.4% |
| ✗ | ✗ | ✗ (Standard AR) | 1.0× | 60.4% |
Ablation on confidence threshold \(\sigma\) across benchmarks:
| Threshold \(\sigma\) | CALVIN ABC→D (Speed / Avg Len) | LIBERO-10 (Speed / SR%) | Real-world (Speed / SR%) | Observation & Analysis |
|---|---|---|---|---|
| \(\sigma = 0.7\) | 1.3× / 3.72 | 1.5× / 61.3% | 1.8× / 81.0% | Conservative exit; constrained acceleration |
| \(\sigma = 0.6\) | 1.5× / 3.68 | 2.4× / 60.5% | 2.5× / 79.0% | Moderate acceleration with no performance drop |
| \(\sigma = 0.5\) (Default) | 2.0× / 3.67 | 3.9× / 59.9% | 3.9× / 77.5% | Optimal Pareto frontier between speed and accuracy |
| \(\sigma = 0.4\) | 3.2× / 3.63 | 4.6× / 58.3% | 4.4× / 72.5% | Aggressive speedup; slight accuracy degradation on dexterous tasks |
Extreme decoding speed profiling on CEED-VLA: - AR Decoding: Average speed 39.64 tokens/s, Minimum 30.87, Maximum 49.10. - Standard Jacobi: Average speed 57.57 tokens/s, Minimum 19.50, Maximum 82.64 (highlighting performance drag from tail iterations). - Adaptive Early-exit (Ours): Average speed 79.24 tokens/s, Minimum 62.07, Maximum 93.53 (eliminating slowest iterations).
Key Findings¶
- Consistency Distillation Drives Fixed Token Emergence: Native LLaVA-VLA yields 0 fixed tokens per step, while PD-VLA produces 8.75 and CEED-VLA achieves 13.5. Truncating intermediate Jacobi iterations without consistency training collapses performance (CALVIN completion length drops to 1.44 at 8 steps), whereas consistency distillation preserves performance above 3.34.
- High Control Frequency Unlocks Dexterous Manipulation: On lightweight robotic arms, latency from slow AR decoding causes pauses and physical jitters, causing towels to slip and liquid to spill. Boosting execution to 13.0 Hz produces smooth, continuous motions that drastically elevate success on dexterous manipulation.
- High Data and Compute Efficiency: Trajectory scaling experiments show performance saturates at 60K trajectories, requiring only 12 hours of distillation on 8 A100s.
Highlights & Insights¶
- Reimagining Parallel Decoding via Consistency Models: Instead of treating parallel decoding purely as a post-hoc inference trick, CEED-VLA trains the model to anticipate noisy trajectories, turning intermediate states into direct jumps toward convergence.
- Pragmatic Confidence-Guided Termination: Exploiting the physical insight that tail-iteration token flips barely alter manipulator trajectory, the minimum confidence threshold eliminates tail latency spikes and stabilizes runtime control frequency.
- Universal, Orthogonal Applicability: CEED-VLA requires no modifications to parameter counts or network layers, integrating seamlessly with action chunking, frequency-domain tokenizers (FAST), and VQ-based world models (UniVLA).
Limitations & Future Work¶
- Bottlenecked by Teacher Policy Competence: The consistency objective distills teacher trajectories; sub-optimal trajectories or perceptual hallucinations in the teacher policy will be transferred to the student model.
- Scope Confined to Discrete Action Spaces: The formulation targets discrete autoregressive token generation and cannot be directly applied to continuous flow matching or diffusion heads (e.g., standard π0 or GR00T).
- Uniform Confidence Threshold: The early-exit threshold \(\sigma\) is currently scalar and uniform across all degrees of freedom. Dynamically tuning thresholds based on actuator sensitivity (e.g., coarse arm translation vs. delicate gripper state) could further optimize efficiency.
Related Work & Insights¶
- vs PD-VLA (IROS 2025): PD-VLA introduced Jacobi decoding to action-chunked VLAs without fine-tuning, resulting in few fixed tokens (8.75 vs. 13.5) and sluggish convergence under strict criteria; CEED-VLA doubles the speedup factor from 1.3~1.6× to 3.9~4.1× via distillation and early-exit.
- vs FastV / SparseVLM: Visual token pruning reduces prompt length but damages spatial geometry critical for manipulation, causing severe drops on CALVIN (2.54~2.83 vs. CEED-VLA's 3.68); CEED-VLA preserves full visual fidelity.
- vs Diffusion / Flow Matching Models: While continuous diffusion policies generate smooth paths, they require multi-step denoising loops; CEED-VLA demonstrates that discrete autoregressive VLAs can match or exceed their frequency (25 Hz+) through efficient parallel decoding.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ [Creative transfer of consistency distillation to parallel action decoding dynamics]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across simulated benchmarks, diverse architectures, and real-world dual-arm platforms]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, mathematically grounded formulation, and rigorous experimental presentation]
- Value: ⭐⭐⭐⭐⭐ [Resolves a foundational deployment bottleneck for autoregressive embodied foundation models]