DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Models¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/hustvl/DiffusionVL
Area: Multimodal VLM
Keywords: autoregressive model conversion, block diffusion, visual instruction tuning, parallel decoding, KV cache

TL;DR¶

DiffusionVL preserves a strong autoregressive backbone and generates visual question-answering text through block diffusion fine-tuning and cached block decoding, reaching 25.0 on MMMU-Pro vision with only 738K additional fine-tuning samples for its 7B variant, although speed and quality gains depend on the task and decoding configuration.

Background & Motivation¶

Vision-language models (VLMs) typically encode an image into visual embeddings and then generate an answer token by token from the image and question. Autoregressive generation can reuse the KV cache of a completed prefix, but each token waits for its predecessors, making long descriptions and reasoning chains vulnerable to sequential decoding costs. Masked diffusion offers another generation procedure: reserve multiple unknown positions and predict some of them simultaneously using the available context. The challenge is not whether this procedure can represent text, but that existing diffusion language backbones remain less mature and capable than strong autoregressive ones. Methods such as LLaDA-V and LaViDa usually perform visual instruction tuning from diffusion language models, so their multimodal performance can inherit limitations of the language backbone.

DiffusionVL consequently separates parameter initialization from the training and generation paradigm. The Transformer architecture does not require exclusively left-to-right generation; attention visibility and the training objective determine how context is used. Given an already aligned Qwen2.5VL, learning multimodal capabilities again from a weaker diffusion language backbone may be unnecessary. Starting from a language-only backbone such as Qwen2.5 additionally requires aligning the visual connector, so the conversion procedure for an existing VLM cannot simply be reused unchanged. This distinction explains why the paper studies both AR-VLM conversion and AR-LM conversion instead of reporting only a decoder replacement experiment.

Successful conversion does not automatically make inference practical: full-sequence diffusion commonly requires a predefined generation length, while bidirectional dependencies complicate stable reuse of historical KV states. The paper combines intra-block diffusion with inter-block autoregression, restricting parallel updates to the current block and treating completed blocks as fixed conditions. Diffusion operates on discrete answer tokens; images provide conditioning, with no pixel denoising or image generation module. Core Idea: inherit the knowledge and visual capabilities of a strong autoregressive backbone, then learn a new generation paradigm through masking supervision matched to block decoding instead of first building a separate diffusion language backbone.

Method¶

Overall Architecture¶

The input is an image and a textual question; the output is a textual answer that can grow block by block. A vision encoder and projector turn the image into embeddings accepted by the language model and concatenate them with text embeddings. Model construction involves entry-point conversion, blockwise denoising supervision, and cached block decoding; the first two determine training, whereas the last governs deployed generation. An existing AR-VLM retains its architecture and is fine-tuned directly; the AR-LM route first adds a vision encoder and connector, then performs multimodal alignment and paradigm conversion. Thus, preserving the architecture applies most directly to an existing VLM and does not mean that a language-only model needs no visual input pathway.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["AR-VLM or AR-LM"] --> B["Entry-point conversion"]
    B -->|Training| C["Blockwise denoising<br/>supervision"]
    T["Image, question, and ground-truth answer"] -->|Ground truth used only in training| C
    C -->|Trained model| D["Cached block decoding"]
    I["New image and question"] -->|Inference input| D
    D --> O["Textual answer"]

Key Designs¶

1. Entry-point conversion: distinguish visual alignment from generation paradigm conversion

For Qwen2.5VL-3B-Instruct and Qwen2.5VL-7B-Instruct, image and text information already work together within the model. The paper directly applies full-parameter diffusion fine-tuning, without training a separate diffusion teacher or first converting the language component into a dLLM. The model retains its existing vision-language capabilities while learning to predict missing tokens in partially masked answers. This route uses 738K additional instruction samples from LLaVA-Next. Its data savings concern the conversion stage and do not erase the backbone's earlier large-scale vision-language training from the total cost. Comparisons with the original backbone are necessary to identify which capabilities survive and which deteriorate after conversion.

For Qwen2.5-7B-Instruct, visual inputs have not yet been aligned with the language space. The paper adds a SigLip2-400M vision encoder and a randomly initialized two-layer MLP projector, initially training only the connector. This pretraining stage retains the autoregressive objective because changing the input modality and prediction paradigm simultaneously while the connector is random would make optimization harder. Once visual embeddings become usable by the language model, all components are trained jointly through diffusion fine-tuning. This route uses the 580K pretraining samples and 738K instruction samples listed in Section 4.1, rather than the single-stage cost of directly converting a VLM. Comparisons with a LLaDA backbone under matched data test whether a diffusion VLM genuinely requires a diffusion language model.

2. Blockwise denoising supervision: predict from clean history and a noisy current block

Training sequences are padded with <EOS> to a multiple of block size \(D\) and partitioned into non-overlapping blocks. Images and questions remain unmasked as known conditions, while noise is applied to response and padding content. Each block requiring corruption independently samples a noise level \(t_b\sim U(0,1)\), and its tokens are independently replaced by [MASK] with that probability. Blockwise noise means independently choosing a masking probability for each block, not deleting a whole block or forcing every block to contain the same number of masked tokens. Compared with sharing one noise level across a sequence, this exposes different blocks to different uncertainty levels and better matches blockwise denoising training. The default training block size is \(D=8\), although block size itself controls a speed-quality trade-off.

To provide appropriate conditions in one training forward pass, the authors concatenate noisy and clean sequences along the sequence dimension and restrict information flow through an attention mask. A noisy block can attend to all positions inside itself, so revealed tokens on either side can help prediction. It can also attend to earlier blocks in the clean sequence, but cannot use that copy to access its own clean answers or future clean blocks. The clean copy is not an additional inference input; it supplies ground-truth history to each block during training. Cross-entropy is computed only at masked positions, rather than repeatedly training the model to copy every already visible answer token. Using <EOS> for padding also brings termination tokens into the training objective, allowing block decoding to learn when to stop.

3. Cached block decoding: fix the history and complete only the current block in parallel

Inference first encodes the image and question to establish the context cache, then initializes the current answer block with \(D\) [MASK] tokens. Each denoising step predicts tokens at unfilled positions from the fixed history and currently visible block tokens, using the model probability of each selected token as its confidence. The static strategy commits a subset of the highest-confidence positions per step; unselected candidates remain masked and can be predicted again later. This is not a separate visual correctness verifier: high confidence indicates a preferred prediction, not guaranteed image-grounded evidence. Algorithm 1 specifies the number of positions committed per step as:

\[ k=\min\left(\left\lceil D/S\right\rceil,\left|\mathcal{M}^{(s-1)}\right|\right). \]

Here, \(S\) is the number of denoising steps per block and \(\mathcal{M}^{(s-1)}\) is the set of positions that remain masked. The dynamic strategy augments the static top-k set with every position whose confidence exceeds threshold \(\tau\), potentially completing simple content earlier. Committed tokens remain available as context in subsequent steps; the procedure does not erase already settled blocks at every iteration. Once a block is complete, it joins the historical KV cache and decoding advances to the next block; the current block has no future-block dependency, enabling reuse of past cache entries. Generation stops at <EOS> or the maximum length: arbitrary-length generation means avoiding a single fixed answer canvas, not unlimited context or computation. The default static configuration sets \(S=D\) and normally commits one token per step; committing multiple tokens requires fewer steps or the dynamic strategy. The potential for parallel decoding must therefore be distinguished from the default decoding behavior; diffusion does not by itself imply that the default emits multiple words per step.

A Worked Example¶

The following illustrates the algorithm rather than a new measured result: consider a street image and the question, "Describe the traffic conditions." The image and question establish the fixed prefix, and the model allocates \(D=8\) unknown token positions for the first answer block. With static decoding at \(S=4\), each step commits up to \(k=2\) predictions among the remaining positions, whereas the default \(S=8\) usually commits one. A step can determine a high-confidence token in the middle of the block before using it to fill neighboring positions, so intra-block generation need not proceed left to right. The completed first block becomes fixed history for the second; the second block cannot revise the first. Generation ends if the second block produces <EOS>; otherwise, it allocates another block until termination or the length limit. Training provides ground-truth history, whereas deployment supplies generated history, so prefix errors can still propagate forward.

Loss & Training¶

The central supervision predicts ground-truth tokens at masked positions, conditioned on earlier clean blocks and the current noisy block. Equations (1) through (5) in the local full-text extraction are visibly corrupted, particularly logarithms, loss weights, and piecewise notation; this note does not reconstruct them as exact author equations. The objective and attention descriptions above follow Sections 3.2 through 3.4 and Figure 3, while the commitment-count formula follows the legible Algorithm 1. Training uses AdamW with cosine learning-rate decay. Connector pretraining uses \(1\times10^{-3}\); joint fine-tuning uses \(1\times10^{-5}\) for the language model and projector and \(2\times10^{-6}\) for the vision encoder. For long reasoning chains, the authors randomly sample 100K chain-of-thought (CoT) examples from OpenMMReasoner-SFT and continue fine-tuning with the original data, producing DiffusionVL-CoT. The CoT variant receives additional training rather than a simple inference parameter change, so its throughput and accuracy must be reported separately.

Key Experimental Results¶

Main Results¶

The following selects results from Table 1, page 9, using LMMS-Eval default prompts; column names preserve evaluation subsets, and higher values are better. MME cognition reports benchmark scores rather than accuracy percentages; the sample column follows the original table and does not represent complete lifetime training data for each model.

Model	Size	Samples in source table	MMBench en-dev	MMMU val	MMMU-Pro std.	MMMU-Pro vision	MME cognition
Qwen2.5VL	7B	>9M	83.5	51.1	36.7	33.4	646
LLaDA-V	8B	16.5M	82.9	48.6	35.2	18.6	491
DiffusionVL	3B	738K	80.1	47.2	31.2	20.2	594
DiffusionVL	7B	738K	83.5	49.3	36.9	25.0	675

Against LLaDA-V, the 7B model increases MMMU-Pro vision from 18.6 to 25.0, a gain of 6.4 points or approximately 34.4% relative improvement, not 34.4 percentage points. MME cognition rises from 491 to 675, approximately 37.5% relative improvement; however, MMMU val and MMMU-Pro vision remain below the original Qwen2.5VL-7B. The primary achievement is a stronger diffusion VLM, not universal superiority over its autoregressive backbone.

Ablation Study¶

Table 5, page 13, fixes Qwen2.5VL-3B initialization, data, and compute to compare continued AR fine-tuning against diffusion fine-tuning. For DetailCaps, BERT denotes BERTScore and TPS denotes generated tokens per second, which should not be interpreted directly as end-to-end request latency.

Config	MMMU val	MMStar test	MME cognition	ChartQA	AI2D	DetailCaps BERT	DetailCaps TPS
Original Qwen2.5VL-3B	46.8	55.9	620	84.0	81.6	29.6	36.08
+ AR fine-tuning	46.8	58.6	651	84.7	80.7	32.1	34.17
+ Diffusion fine-tuning	47.2	55.9	594	79.9	78.4	30.9	46.37

Diffusion fine-tuning reaches 46.37 DetailCaps TPS versus 34.17 for matched AR fine-tuning, but BERTScore decreases from 32.1 to 30.9. This controlled experiment establishes a clearer causal boundary than cross-backbone comparisons: much downstream capability comes from the strong initialization, while diffusion conversion supplies another generation procedure with some quality loss.

The following selects Table 4, page 12, to examine mathematical reasoning speed and quality; higher Score and TPS are better. For this comparison, the paper uses fast-dLLM caching and generation length 64 for LLaDA-V, while DiffusionVL-CoT uses maximum length 1024, so budgets are not identical.

Model	MathVista Score	MathVista TPS	MathVerse Score	MathVerse TPS	MathVision Score	MathVision TPS
DiffusionVL-7B	64.30	5.21	36.29	14.38	19.08	17.74
LLaDA-V-8B	55.10	23.07	31.35	23.18	17.11	22.75
DiffusionVL-CoT-7B	63.70	39.47	32.74	42.31	18.75	39.26

Key Findings¶

On MathVista, the CoT variant exceeds LLaDA-V by 8.60 points with 39.47 versus 23.07 TPS, approximately 1.7 times faster; the base variant reaches only 5.21 TPS, so mathematical tasks are not universally accelerated.
Figure 4's approximately 2.0-fold captioning speedup concerns DetailCaps with at most 512 tokens and matched parallelism; speeds were collected on 8 GPUs and averaged per device, not established for arbitrary requests.
In Table 8, page 14, block versus full diffusion scores 84.20 versus 52.00 on ChartQA but 1519 versus 1607 on MME perception; block diffusion does not win every metric.

Highlights & Insights¶

Separating initialization knowledge from generation paradigm is the most transferable insight. A new decoding paradigm need not rebuild all capabilities from a weaker new backbone.
A clean historical copy and restricted attention jointly enable blockwise supervision. The key is preventing access to the current block's clean answers, not merely adding bidirectional attention.
Organizing both training and inference around blocks enables stable cache reuse. Length management and historical state reuse matter alongside parallel sampling in practical deployment.

Limitations & Future Work¶

The less-than-5% data claim compares 738K with LLaDA-V's 16.5M and does not reduce total from-scratch training cost to 5%; Qwen2.5VL's inherited capabilities are a substantial prior investment.
"Any" in the title expresses methodological applicability, not proof for every architecture. Table 6 validates Qwen2.5VL, Qwen3VL, and InternVL3.5, leaving further scales and modalities to test.
Throughput comparisons jointly depend on visual token counts, output lengths, CoT data, and cache implementations. Table 9's end-to-end FLOPs setting uses 315 versus 3654 visual tokens, so the entire difference cannot be attributed to decoding alone.
Inter-block dependencies remain autoregressive, preventing later blocks from correcting completed history. Lower thresholds accelerate generation, but Table 7(b) drops from 30.2 to 18.6 BERTScore as the threshold falls from 1.0 to 0.2.
The authors propose extending conversion to pretraining and reinforcement learning; these are future directions, not validated benefits. Corrupted equation extraction also limits verification of exact loss weighting in this note.

LLaDA-V / LaViDa: learn multimodal capabilities from diffusion language backbones; this paper primarily changes initialization while adopting cache-compatible block diffusion.
BD3-LM / SDAR / Fast-dLLM-V2: establish foundations for intra-block diffusion and inter-block autoregression; this paper applies related ideas to VLM conversion rather than originating block diffusion itself.
A2D-VL: identified as concurrent work in the paper, it also adapts AR-VLMs to diffusion decoding; DiffusionVL emphasizes a simpler fine-tuning recipe and extends conversion to AR-LMs and different VLM families.
Research implication: future comparisons could fix visual encoding, output budgets, and training data to separate gains from initialization, block structure, and CoT data through speed-quality curves; this is a reader proposal.

Rating¶

The following is the note author's subjective assessment, not a conference rating. - Novelty: 4/5. Unifies existing AR-VLMs and AR-LMs as starting points for diffusion VLMs, although block diffusion has precedents. - Experimental Thoroughness: 4/5. Includes matched AR fine-tuning, cross-family tests, and several ablations, while speed comparisons retain budget and implementation differences. - Writing Quality: 4/5. Clearly explains both conversion routes and decoding, but headline acceleration claims require task-specific reading. - Value: 4/5. Provides a practical route to diffusion decoding from strong VLM backbones while revealing genuine quality-throughput trade-offs.