Fast-dVLA: Accelerating Discrete Diffusion VLA to Real-Time Performance

Conference: ECCV2026
Paper: ECCV Paper
Area: Robotics & Embodied AI; VLM Inference Efficiency
Keywords: vision-language-action models, discrete diffusion, block-wise causal attention, diffusion forcing, asymmetric distillation

TL;DR

Fast-dVLA adapts discrete diffusion vision-language-action models to cache completed prefixes and denoise multiple blocks concurrently through lightweight asymmetric distillation, increasing Dream-VLA decoding throughput from 98.8 to 313.1 tokens/s on LIBERO and reporting 30 Hz execution in real-world tasks.

Background & Motivation

Vision-language-action (VLA) models turn camera observations and language instructions into robot controls. Discrete diffusion VLAs encode actions as discrete tokens and repeatedly predict masked positions, avoiding both token-by-token autoregression and a separate continuous flow-matching action head. This supports unified representations and generation across vision, language, and actions. However, predicting several tokens in one pass does not necessarily make the complete sequence fast to generate: bidirectional attention lets already generated tokens depend on changing future content, so their key-value (KV) states change across denoising iterations and cannot be cached as stable autoregressive prefixes.

Ordinary block diffusion restores caching but makes different action blocks strictly sequential: the next block cannot start until the current one is complete. The authors observe that Dream-VLA tends to unmask earlier action timesteps before later ones even under bidirectional attention. They attribute this to both residual autoregressive behavior from initialization and temporal dependencies between actions. This is an empirical observation, not a theorem about every discrete diffusion model. It suggests preserving temporal order at the block level without forcing later actions to wait until earlier actions are fully determined.

The goal is therefore not only to reduce the number of forward passes, but also to avoid recomputing stable content within each pass while letting unfinished blocks share computation. Core Idea: use block-wise causal attention to stabilize completed-prefix KV states, diffusion forcing to learn from partially denoised predecessors, and asymmetric distillation plus staged pipeline activation to turn these dependencies into efficient inference.

Method

Overall Architecture

The inputs are visual observations, a language instruction, and the action-token sequence to generate; UD-VLA additionally includes discrete tokens for future visual prediction. The model partitions outputs into temporal blocks, predicts within each block in parallel, and only reads prefix information across blocks. Training adapts attention and noise distributions, whereas inference maintains a growing pipeline of active blocks before decoding the resulting action tokens into executable controls.

The solid arrows below distinguish training data flow from online decoding flow; the dashed arrow indicates only the transfer of distilled parameters. Online inference neither invokes the teacher nor receives ground-truth future actions. The teacher's global noisy sequence is used only for training supervision.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Training action sequence"] --> B["Block-wise Causal Attention"]
    B --> C["Diffusion Forcing"]
    C --> D["Asymmetric Distillation"]
    T["Bidirectional teacher<br/>Global noisy context"] -->|Distribution supervision at masked positions| D
    D -.->|Student parameters| E["Pipelined Parallel Decoding"]
    I["Online observations and instruction<br/>Fully masked output blocks"] --> E
    E --> O["Action tokens and controls"]
    E -->|Cache completed prefixes and continue denoising| E

Key Designs

1. Block-wise Causal Attention

The problem with ordinary bidirectional attention is not whether token values have been fixed, but whether their representations still read changing future content. Fast-dVLA lets each block attend to the prompt prefix, preceding blocks, and itself, but not later blocks; interactions within the block remain bidirectional. A completed prefix is consequently unaffected by subsequent denoising, so its stable KV states can be retained and later forward passes compute only unfinished content. More precisely, safe reuse concerns completed prefixes whose predecessors are also stable, not an arbitrary later block that happens to finish first.

Block boundaries are not arbitrary either. Dream-VLA and DD-VLA use a default block size of 7, matching the number of action tokens for one timestep. Keeping the dimensions of a simultaneous action together allows within-block coordination instead of imposing additional causal restrictions inside a control vector. For UD-VLA, whose output reaches 625 tokens, the authors instead use multiples of 32. Action-dimension alignment is therefore a representation-dependent principle, not a requirement that every model use 7-token blocks.

2. Diffusion Forcing

With block-level causality alone, the straightforward decoder still completes one block after another. Fast-dVLA additionally changes the noise states of different blocks during training: earlier blocks receive lower masking ratios and later blocks receive higher ratios, creating monotonically increasing corruption. The student thus learns to predict a block from incompletely denoised predecessors, rather than only from fully determined preceding actions. Masked positions are predicted while tokens already revealed remain unchanged.

Inter-block parallelism then becomes compatible with the training distribution. As an earlier block approaches completion, a later block can start denoising from the information already available; one forward pass both advances the earlier block and accumulates reliable predictions for the later one. This does not restore global bidirectional attention: later blocks still cannot alter earlier ones. Diffusion forcing teaches the model to handle different noise levels across blocks, while block-wise causal attention makes stable prefixes cacheable. The two components address different bottlenecks.

3. Asymmetric Distillation

The authors do not train a new robotic foundation model from the beginning. Instead, they start with a discrete diffusion VLA already fine-tuned on the task. The teacher retains bidirectional attention and the student uses block-wise causal attention; both share the same architecture and receive the same monotonic noise schedule. At a given masked position, the teacher can inspect the entire noisy sequence, whereas the student can inspect only the current block and its prefix. A KL-divergence constraint on predictions at these positions teaches the student to approximate the teacher with restricted context.

The asymmetry concerns visible context, not teacher assistance with future completion during inference. It reuses the action knowledge in the fine-tuned model and concentrates additional training on the new attention and noise combination. Equations (1)-(4) are visibly damaged in the cached PDF extraction, so this note does not reconstruct them as exact author formulas or infer the implementation's KL direction from fragments. The supported description is a distributional discrepancy aggregated over masked positions, with asymmetric distillation as the default adaptation objective.

4. Pipelined Parallel Decoding

Inference does not forcibly reveal all blocks at once. It increases parallelism according to predecessor completion. The completion ratio is the number of unmasked tokens in a block divided by its total token count. Once the predecessor exceeds the addition threshold, the next block becomes semi-activated and accepts only tokens whose prediction confidence exceeds a threshold. After the predecessor exceeds the activation threshold, the successor becomes fully activated: besides confidence filtering, confidence ranking guarantees that at least a fraction of remaining tokens is decoded at every step. The paper writes this fraction as \(1/n\), but the local main text does not sufficiently specify this parameter, so no default value is invented here.

Semi-activation commits only relatively reliable early predictions, avoiding forced decisions while predecessor context remains uncertain; full activation prevents stalling when too few predictions exceed the confidence threshold. Completed stable prefixes become cached while subsequent blocks continue sharing forward passes. This neither terminates actions early nor reduces the number of actions the robot must execute; it reorganizes computation for the same output task. The threshold fractions in Figure 6 are merged by text extraction, so they are not transcribed as executable numerical settings.

A Worked Example

Consider the default 7-token action blocks and an instruction to place an object in a tray. All action blocks begin masked. The first active block predicts several action dimensions; before it is complete, enough reliable information accumulates to meet the addition condition, and the second block joins the same forward pass. At this point, the second block commits only high-confidence predictions rather than being forced to determine the entire action at once.

As the first block becomes more complete, the second becomes fully activated and can in turn provide progressively clearer predecessor context to the third. Once the first block and its prefix are stable, its KV states are cached and computation continues only over active blocks. This example illustrates state transitions rather than a measured token trajectory. It also explains why inter-block parallelism and causal caching are compatible: simultaneous processing does not require earlier blocks to attend to later ones.

Loss & Training

The original dVLA reconstructs masked action positions using cross-entropy, while Fast-dVLA uses the asymmetric distribution-distillation objective for its default post-training. The paper also compares continued block-diffusion training from fine-tuned weights and block-diffusion adaptation from a pretrained dVLA. The latter comparison, described as training from scratch, still involves a pretrained starting point and should not be interpreted as training the whole VLA from random initialization.

In the experimental setup, Dream-VLA and DD-VLA each receive 4k distillation steps, approximately 1/5 and 1/8 of their respective original task fine-tuning budgets. UD-VLA receives 3k steps with a batch size of 12, approximately 1/8 of its original fine-tuning steps; other training hyperparameters follow the original models. A separate Dream-VLA/LIBERO convergence analysis finds convergence after approximately 2,000 distillation steps. This is the efficiency observation in Figure 8, not the same configuration as the 4k steps used in the experimental setup.

The semi-activated confidence threshold selected for UD-VLA is 0.5. It controls whether a token is accepted early, not task success or attention causality, and must be considered together with the addition and activation conditions at deployment.

Key Experimental Results

Main Results

Table 1 summarizes LIBERO results from the paper's Table 1. Success is averaged across Spatial, Goal, Object, and Long and converted to percentages below. Speed is discrete-output decoding throughput in tokens/s, not closed-loop control frequency. The cached main text does not specify the GPU model, complete timing boundary, or inference batch size for these measurements; they are comparisons under the authors' setup, not cross-hardware guarantees.

Base Model	Decoding Strategy	Average Success Rate (%)	Speed (tokens/s)
Dream-VLA	Original discrete diffusion	85.6	98.8
Dream-VLA	Fast-dLLM	82.8	183.2
Dream-VLA	Block Diffusion	85.8	181.7
Dream-VLA	Fast-dVLA	87.0	313.1
DD-VLA	Original discrete diffusion	96.3	152.1
DD-VLA	Fast-dLLM	93.5	312.5
DD-VLA	Block Diffusion	96.7	322.1
DD-VLA	Fast-dVLA	96.6	402.7

Speedup denominators matter: Dream-VLA improves by 313.1/98.8, approximately 3.17x, relative to itself; DD-VLA improves by 402.7/152.1, approximately 2.65x, relative to itself. The last entry is labeled 4.1x in the paper's Table 1, corresponding to 402.7/98.8 against the common Dream-VLA reference. It should not be described as a 4.1x improvement over unaccelerated DD-VLA. The full DD-VLA variant also does not achieve the highest success in the table: its 96.6% is slightly below Block Diffusion's 96.7%.

On CALVIN ABCD-to-D in the paper's Table 2, UD-VLA throughput rises from 67.3 to 186.7 tokens/s, approximately 2.8x. Average consecutive tasks completed falls from 4.64 to 4.54, and the rate of completing all five tasks falls from 0.840 to 0.812. Average length here refers to consecutive robotic tasks, not linguistic sentence length, and acceleration is not lossless.

Across the four SimplerEnv WidowX tasks in the paper's Table 4, Dream-VLA average task success rises from 51.0% to 59.3% and throughput from 100.1 to 366.4 tokens/s. Grasp success and final task success are distinct metrics; these numbers refer to the latter, not an assumption that grasping alone completes placement or stacking.

Ablation Study

Table 2 corresponds to the paper's Table 5, using Dream-VLA on LIBERO-Long. The authors compare and average several block-size choices between 7 and 14. Multiples of the action dimension mean alignment with the existing action representation, not a new action encoder.

Block-size Choice	Success Rate (%)	Reported Speedup
Multiples of the action dimension	74.7	4.01x
Random block sizes	73.3	3.95x

The 1.4-percentage-point difference supports action-aligned blocks. However, the denominator for speedup is not explicitly restated in the paper's Table 5, so it should not be directly reconciled by dividing the average throughput values in Table 1. Figure 9 also shows that lowering the semi-activation confidence threshold increases speed but hurts performance; the authors choose 0.5. The text's approximately 2% performance decline does not clearly identify its metric convention and is therefore not converted into success-rate percentage points here.

Key Findings

• In Figure 8, distillation converges after approximately 2,000 steps, about 5x faster than continued block-diffusion training from fine-tuned weights and about an order of magnitude fewer steps than adaptation from the pretrained starting point. These are convergence-step comparisons, not measured total training times.
• Real-world tests use an AgileX bimanual platform with 6 degrees of freedom and a gripper per arm, an overhead camera, and two wrist cameras. Each of three tasks has 100 expert demonstrations and 40 evaluation trials; Figure 7/Section 5.4 reports 30 Hz execution.
• Conveyor picking is evaluated by successful grasps per minute, with the authors describing nearly doubled efficiency. Exact bar values are not preserved in the text cache and are not guessed here. The two vegetable tasks take less time but incur slight success-rate losses relative to the baseline.

Highlights & Insights

• The acceleration target is the combination of attention dependencies and decoding schedules. Caching alone risks stale states, whereas strictly sequential blocks lose parallelism; training must jointly accommodate both requirements.
• Action timing gives block boundaries a more concrete meaning than arbitrary segmentation of language. Keeping simultaneous control dimensions together creates a semantic boundary between local bidirectional interaction and global temporal order.
• Transferring global-teacher distributions to a context-restricted student offers a relatively inexpensive way to change generation dependencies. It is not simply removing bidirectional attention and reusing the original weights without adaptation.

Limitations & Future Work

• Costs reported by the authors: UD-VLA loses some long-horizon CALVIN performance, and two semantic real-world tasks also show small success-rate losses. The results do not support lossless acceleration in every setting.
• Reading assessment: the main text does not fully disclose inference hardware or end-to-end timing boundaries. Tokens/s and robot execution Hz are not interchangeable; 30 Hz alone does not establish perception, planning, or communication tail latency.
• Reading assessment: block size, semi-activation thresholds, and action discretization interact. Different robots, action dimensions, or longer visual sequences require tuning and evaluation; this is not training-free, plug-and-play caching.
• Future direction: report end-to-end latency percentiles across hardware, sensitivity to action-block length, and dynamic-disturbance evaluations to better establish real-time control benefits and safety boundaries.

Related Work & Insights

• Compared with Block Diffusion: both use block-level causality to make KV caching valid. Fast-dVLA additionally starts later blocks before predecessors are fully clean and adapts training to this parallel schedule.
• Compared with Fast-dLLM: this paper emphasizes state bias when directly reusing KV under bidirectional dVLA attention. Fast-dLLM speeds up decoding but loses success in Table 1; Fast-dVLA changes dependencies to establish stable-cache conditions.
• Compared with continuous flow-matching VLAs: Fast-dVLA retains discrete outputs and unified visual/action generation rather than adding a continuous action head. Cross-architecture advantages still need comparison under matched data, hardware, and end-to-end control metrics.

Rating

• Novelty: 4/5. The combination of action structure, diffusion forcing, and cached pipelining has a clear adaptation target, although its components build on prior work.
• Experimental Thoroughness: 4/5. Three dVLA families, three simulation benchmarks, and real-world tasks are covered, but hardware and tail-latency disclosure remain limited.
• Writing Quality: 3/5. The mechanisms are clear, but some speedup denominators and performance-decline conventions are easy to confuse.
• Value: 4/5. The method provides a robot-tested acceleration path for policies that retain discrete unified generation.