UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformer¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/MCG-NJU/UniDDT
Area: Multimodal VLM
Keywords: Unified Multimodal Model, Decoupled Diffusion Transformer, Unified Visual Space, Flow Matching, Understanding-Generation Duality

TL;DR¶

UniDDT unifies the semantic encoding of understanding and generation behind a single Noisy ViT encoder plus an LLM backbone, decouples diffusion decoding from text decoding with a separate diffusion decoder, and turns the same image-text pairs into dual-format training data for joint training and duality-based post-training — reaching 1699.5 on MME, 0.87 on GenEval and 86.9 on DPG.

Background & Motivation¶

Unified Multimodal Models (UMMs) try to fit understanding and generation into one framework, and the mainstream form today is the AR-diffusion hybrid: autoregressive next-token prediction on the text side, diffusion on the image side. But the two tasks differ far more than they appear to, and existing solutions have not settled any of three layers. On modeling, early attempts used adapters to assemble a understanding model and a generative model together — a superficial integration; the Native-UMMs that followed place both objectives in one network, but typically as parallel branches, which produces a see-saw performance trade-off rather than the hypothesised mutual promotion. The field's response was to decouple the parameters per task. On the visual space, understanding models thrive on high-dimensional semantic representations while generative models simply cannot train in them, so most UMMs end up giving each task its own space — semantics for understanding, a VAE space for generation. This fragmentation complicates the whole pipeline and obstructs large-scale scaling. On training data, although Mogao argued that interleaved multimodal data is the key to true unification, most UMMs still train each component on task-specific data and merely stitch the pieces together, never exploiting the duality inherent in image-text pairs.

The root of all three is the same tension: the semantic richness understanding demands and the trainability generation demands are in direct conflict inside one representation space and one set of parameters. Since sharing everything drags both sides down, the angle here is not "share more" but to work out precisely what must be shared and what must be separated: semantic encoding can be shared (understanding is already doing semantic extraction, and generation needs a semantic condition), whereas the decoding processes must not be (text uses causal token decoding, images use iterative denoising; their output structure and training objective are incompatible). Following this idea, UniDDT is organised around treating visual understanding as a preceding task to visual generation.

Core idea: let the Noisy ViT encoder and the LLM backbone share one semantic pathway (noisy and clean inputs go through the same semantic encoding), pull diffusion decoding out into a separate decoder that consumes only the semantic condition, and thereby achieve "shared semantics, decoupled decoding" inside a single unified visual space — with the dual-format construction of the same image-text pairs tying the understanding and generation objectives together.

Method¶

Overall Architecture¶

UniDDT is a rewrite of the decoupling idea from DDT (Decoupled Diffusion Transformer) for the combined understanding-plus-generation setting. The original DDT splits a diffusion model into a heavy condition encoder and a light velocity decoder: the encoder consumes the prompt, the noisy input and the timestep to produce a self-conditioning feature, and the decoder estimates the velocity field from it. UniDDT keeps that skeleton but splits the condition encoder in two: the Noisy ViT encoder extracts semantics from the noisy input, and the LLM backbone fuses those semantics with the text condition. Together they play the role of DDT's condition encoder; the diffusion decoder remains the dedicated velocity decoder that does the denoising.

Concretely: given an image-text pair \((y, x)\), the image is encoded into the unified visual space and noised to yield the noisy input \(x_t\); the Noisy ViT encoder takes \(x_t\) and the timestep \(t\) and outputs a semantic feature \(z_t\). The backbone then takes one of two routes depending on the task. For understanding, the LLM causally encodes \(z_t\) together with the understanding prefix and autoregressively decodes the answer text \(y^*\). For generation, the LLM causally encodes the prompt \(y\) together with the visual semantics \(z_t\), performing semantic injection to obtain a prompt-refined feature \(\hat z_t\), which is handed to the separate diffusion decoder; the decoder combines \(x_t\), \(t\) and \(\hat z_t\) to estimate the velocity \(v_t\) and integrates an image. Finally, a duality-based post-training step sends the estimated state \(x_s\) at an intermediate time \(s\) back into the understanding branch, constraining the diffusion decoder with the likelihood of recovering the prompt from \(x_s\).

What matters is that only semantics are shared; decoding is fully separated. Understanding goes through the LLM's causal text decoding head, generation through an entirely new diffusion decoder, and the two share no decoding parameters — so there is no head that must simultaneously classify tokens and regress a velocity field.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image-text pair (y, x)"] --> B["Noisy ViT encoder + unified visual space<br/>noisy x_t + timestep t → semantics z_t"]
    B --> C["LLM backbone<br/>one causal backbone, two read-outs"]
    C -->|understanding: causal text decoding| D["Text y*"]
    C -->|generation: injection yields ẑ_t| E["Separate diffusion decoder<br/>estimates velocity v_t from x_t, t, ẑ_t"]
    E --> F["Generated image x"]
    E -.->|intermediate state x_s sent back| G["Duality-based post-training<br/>maximize log p(y | x_s, s)"]

Key Designs¶

1. Noisy ViT encoder and unified visual space: make semantic encoding swallow both clean and noisy inputs, then pick the latent space as the representation both tasks share

The first obstacle to unification is that a understanding model's visual encoder has never seen noise, whereas every generation step operates on a noisy input; reusing a pretrained VLM encoder verbatim makes it fail outright at high noise levels. UniDDT takes the structure of DDT's condition encoder (interleaved Attention and FFN blocks) as its Noisy ViT and injects the timestep \(t\) through AdaLN-Zero, exactly as DiT/SiT do, so the output becomes a function of \(t\): \(z_t = \mathrm{NoisyViT}(x_t, t)\). At small \(t\) (clean) the encoder degenerates into an ordinary semantic encoder, while at large \(t\) (heavy noise) it still returns usable semantics. To give it this ability from the start, most parameters are initialised from a pretrained visual foundation model (SigLIP2 for Native-UniDDT, Qwen-NaViT for VLM-UniDDT) with only the timestep-related AdaLN-Zero modules left as new parameters, and REPA-style representation alignment pulls the encoder's intermediate features towards the teacher's features. The key difference from Show-o2 is that this Noisy ViT genuinely treats \(t\) as an extra condition for semantic extraction, rather than encoding only clean inputs and delegating noise handling elsewhere.

The "unified visual space" question is then which representation this pathway runs on. The authors compare raw pixel space against the latent space of Flux-VAE (16 channels, 8× downsampling) and conclude: pixel space is marginally better for understanding but the gap is negligible, while generation is markedly worse and shows no better scaling. UniDDT therefore adopts the latent space as its unified visual space. The benefit is that understanding and generation share one representation, so no space conversion is needed between the semantic encoder and the VAE, which is what lets the whole pipeline scale cleanly. The choice is not made on intuition: Sec. 4.1 quantifies it with cosine similarity, MME and GenEval curves.

2. LLM backbone: understanding decoding and generative semantic injection share one causal backbone

Giving understanding and generation separate backbones doubles the parameters and forfeits any transfer of understanding ability into generation; yet forcing one backbone to emit both text tokens and image velocities brings the old conflict right back. UniDDT's answer is to let the LLM backbone handle only two semantic-level jobs and hand off "how this becomes the final output" entirely. Following Qwen-VL, distinct chat templates are built for understanding and generation and the image placeholder token is replaced by the semantic features \(z_t\) from the Noisy ViT: for understanding the backbone causally encodes \(z_t\) with the understanding prefix and then autoregressively decodes text; for generation it causally encodes the prompt \(y\) with \(z_t\) and injects the prompt's semantics into the visual features to obtain \(\hat z_t\). In both cases the backbone's output is described uniformly as "visual semantics corrected by the text condition", differing only in how it is read out — as text tokens in one case, fed to the diffusion decoder in the other — and the parameters are fully shared. There is a bonus: because the timestep is randomly sampled during understanding training, the backbone also learns to reason over noisy semantics, so the model can "read" what it is drawing mid-generation (Fig. 5 shows understanding barely degrades at acceptable noise levels).

One easily overlooked but crucial detail: the diffusion decoder consumes only \(\hat z_t\), never the hidden states of the text tokens. This departs from the usual diffusion-transformer practice of conditioning on the LLM's last-layer text hidden states, effectively pushing the responsibility for "has the prompt been compressed into the visual semantics" onto the backbone — and the ablation explicitly verifies that this works (see below).

3. Separate diffusion decoder: decouple diffusion decoding from text decoding

This is where the word "decoupled" in the title lands. The final output on the generation side is an image, requiring continuous velocity-field regression and iterative denoising, which is incompatible with discrete causal text decoding in output structure, loss form and inference procedure; cramming both into one head is the most direct source of mutual interference. UniDDT therefore opens a dedicated diffusion decoder, architecturally homologous to the Noisy ViT (interleaved Attention and FFN blocks, close to DiT/SiT), which takes \(x_t\) and \(t\) as input, uses \(\hat z_t\) as the condition, and outputs the velocity \(v_t\), trained with a flow-matching objective. Its only interface with the conditioning side is the single tensor \(\hat z_t\), so it can be trained with the Noisy ViT and the LLM fully frozen — which the authors treat as evidence of a sound design: if that interface carries enough information, the denoiser can be learned with the conditioning side frozen.

The injection mechanism was also changed: DDT injects the condition via AdaLN-Zero, whereas UniDDT uses attention to inject \(\hat z_t\) into the decoder features (the block-structure equations, Eq. 8-9, are corrupted in the cached text, so only the mechanism is described here; ⚠️ refer to the original paper). For training stability the decoder is also followed by several full-attention refinement blocks (the refiner design borrowed from Fluid/PixNerd) that polish the hidden states supplied by the backbone. Overall the decoder is "heavy conditioning side plus a dedicated light decoding side": the conditioning side carries semantics, the decoding side only turns semantics into pixels.

4. Duality-based post-training: send generative intermediate states back to the understanding branch so the likelihood repairs generation

The first three steps already give a model that both generates and understands; this step makes the understanding judgement act directly as a judge for generation. Since the model can understand noisy inputs, the intermediate states it produces along the denoising trajectory need not be mere half-finished artefacts — they can be self-audited for "is this state still faithful to the prompt". Concretely: freeze the Noisy ViT and the LLM backbone and train only the diffusion decoder; given \(x_t\), \(t\) and the prompt \(y\), the decoder produces \(v_t\) and a first-order extrapolation estimates the state at another time \(s\),

\[x_s = x_t + v_t\,(s - t)\]

and \((x_s, s)\) is fed into the understanding branch to estimate the likelihood \(\log p(y \mid x_s, s)\); maximising this likelihood is what optimises the decoder. In other words, the generator is required to make "the intermediate frame it produced" still look to the understanding branch like the image the prompt describes. Compared with optimising only the distance between the final image and the ground truth, this hews much closer to semantic consistency, and it reuses existing branches with no extra parameters — the most direct payoff of framing understanding as a prerequisite for generation.

Loss & Training¶

Three-stage training. The authors found that starting joint training from random initialisation easily collapses the language model, so they split it into three stages:

Noisy ViT warm-up: SigLIP2 / Qwen-NaViT serves as the teacher and representations are distilled into the Noisy ViT (intermediate-feature alignment, a REPA-style cosine loss) for 40K steps with a constant learning rate of 2e-4 and EMA 0.9999. One detail is worth noting: the large time-shift values common in diffusion (corresponding to more high-noise timesteps) significantly impair understanding, especially OCR, so a small time-shift value is used instead, which generalises better across noise levels.
Diffusion decoder warm-up: freeze the Noisy ViT and the LLM backbone, add a projection layer to align dimensions, and jointly train the projection layer and the diffusion decoder for 100K steps at a maximum sequence length of 16384, with the flow-matching loss. That equation is corrupted in the cache; its standard form regresses the target velocity \((x_{\text{data}} - \epsilon)\) against the predicted \(v_t(\cdot)\) for \(t \sim \mathcal{U}[0,1]\), ⚠️ refer to the original paper.
Joint training: unfreeze everything. On the data side comes the paper's second "beyond decoupling" design — two formats constructed from the same image-text pair \((y, x)\): the generation format <user>generate.y<user><assistant>x<assistant> and the understanding format <user>describe.x<user><assistant>y<assistant>, one sampled at random per step. Understanding samples apply cross-entropy only to the text \(y\), generation samples apply the diffusion loss to the image \(x\), and the two sum into the joint loss (Eq. 10 is corrupted in the cache; restated from the prose as \(\mathcal{L}_{\text{joint}} = \mathbb{E}[\mathcal{L}_{\text{diff}}(x \mid y) + \mathcal{L}_{\text{ce}}(y \mid x)]\), ⚠️ refer to the original paper). Native-UniDDT trains for 120K steps and VLM-UniDDT for 10K, at a maximum sequence length of 8192.

Duality-based post-training: as in design 4, train only the diffusion decoder by maximising \(\log p(y \mid x_s, s)\). 4o-data fine-tuning: finally, 8K steps of fine-tuning on OpenAI GPT-4o-style synthetic data (a step that contributes a great deal to generation quality, see the ablation).

Other settings: the data is a mixed set of roughly 70M publicly available images, and every image is re-captioned with Qwen2.5-VL-7B with captions of varying length; native-aspect-ratio training avoids the image-text mismatch caused by centre cropping; FSDP shards the parameters; the hardware is 16×A100. The four configurations are Native-UniDDT-B (Qwen3-0.6B backbone, 12-layer Noisy ViT, 16+4-layer diffusion decoder), Native-UniDDT-L (Qwen3-1.7B, 24-layer Noisy ViT, decoder with 24 heads / 1536 dim), Native-UniDDT-XL (the same as L but with the decoder dimension scaled to 2560) and VLM-UniDDT (Qwen3-VL-4B backbone, otherwise identical to L).

Key Experimental Results¶

Main Results¶

Understanding is measured on MME (perception), GQA, SEED-Bench, MMB(en), MMMU(val), MMStar and AI2D; generation on GenEval (six sub-metrics plus overall) and DPGBench. All understanding evaluations fix \(t=1.0\).

Model	Params	MME(p)↑	SEED↑	MMB(en)↑	MMMU(val)↑	MMStar↑	AI2D↑
LLaVA-OV (understanding-only)	7B	1580.0	-	80.8	48.8	57.5	81.4
Show-o2	7B	1620.5	69.8	79.3	48.9	56.6	78.6
BAGEL	14B	1687.0	-	85.0	55.3	-	-
Mogao	7B	1592.0	74.6	75.0	44.2	-	-
Janus-Pro	7B	1567.1	72.1	79.2	41.0	-	-
VLM-UniDDT	4B+1B	1699.5	76.5	82.2	52.6	57.7	78.1

Method	GenEval Single	Two	Counting	Colors	Position	Color Attri.	Overall↑	DPG↑
SD3-Medium (generation-only)	0.99	0.94	0.72	0.89	0.33	0.60	0.74	84.08
DALL-E 3 (generation-only)	0.96	0.87	0.47	0.83	0.43	0.45	0.67	83.50
Show-o2-7B	1.00	0.87	0.58	0.92	0.52	0.62	0.76†	86.14
Janus-Pro-7B	0.99	0.89	0.59	0.90	0.79	0.66	0.80	84.19
BLIP3-o-8B	-	-	-	-	-	-	0.84	81.60
BAGEL-7B	0.98	0.95	0.84	0.95	0.78	0.77	0.88†	-
VLM-UniDDT (512×512)	0.99	0.93	0.71	0.92	0.85	0.80	0.87	86.9

The two other variants are also reported: Native-UniDDT-L reaches GenEval 0.88 / DPG 86.6 and Native-UniDDT-XL 0.89 / 87.1. ⚠️ Note the abstract states "0.86 GenEval", which disagrees with the 0.87 in the body and tables of the paper; this note follows the tables.

Ablation Study¶

Much of the ablation is a decomposition of the training stages plus a space comparison. The most informative piece is the per-stage GenEval progression (VLM-UniDDT, before 4o fine-tuning):

Stage	Two Obj.	Counting	Position	Color	Overall
Warm-up only	0.63	0.32	0.20	0.27	0.52
Joint training	0.69	0.45	0.41	0.37	0.60
Duality post-training	0.84	0.56	0.50	0.61	0.72
+ 4o data fine-tuning	0.93	0.71	0.85	0.80	0.87

Ablation	Setting	Observation
Time shift of the Noisy ViT	large vs small vs a clean-only ViT	A large time shift badly hurts understanding (especially OCR); a clean-only ViT generalises poorly to noisy inputs; a small time shift stays stable at high noise
Diffusion decoder warm-up	freeze Noisy ViT and LLM, train only the projection layer + decoder	Learns effectively even from \(\hat z_t\) alone (no text tokens), with performance rising steadily with compute
Joint training without the understanding loss	unfreeze everything but zero the understanding loss weight	The latent variant improves only marginally and the pixel variant even degrades; with duality-aware joint training the latent variant jumps markedly
Visual-space scaling	pixel vs latent across warm-up / joint / post-training	The latent level is uniformly higher: warm-up \(y=7.81\times10^{-5}x^{0.34}\) vs pixel \(1.17\times10^{-4}x^{0.31}\); joint 0.178·\(x^{0.05}\) vs 0.088·\(x^{0.07}\); post-training 0.209·\(x^{0.05}\) vs 0.044·\(x^{0.11}\). In the latter two stages the pixel exponent reads slightly larger, but its absolute level is far lower and it shows no better scaling overall

Key Findings¶

Dual training is one of the biggest contributors on the generation side, and its benefit depends on the visual space: zeroing the understanding loss leaves the latent variant almost flat and makes the pixel variant degrade outright — the understanding signal acts as a stabiliser in latent space, whereas in pixel space the model is unstable to begin with.
Fine-tuning on 4o-style data produces the steepest single jump (overall 0.72 → 0.87), with Position leaping from 0.50 to 0.85 and Color from 0.61 to 0.80 — semantic-binding abilities (attributes, position) are extremely sensitive to high-quality captions. The architecture buys trainability; it does not buy data quality.
Noise robustness on the understanding side is tunable: a small time shift flattens the cosine-similarity curve of the Noisy ViT across the whole noise range at little cost, and evaluation at \(t=1.0\) (cleanest) already gives the best result, so no per-task timestep tuning is required.
Pixel space's "slightly better understanding" is real but tiny, whereas its generation deficit is large — that asymmetry is the entire reason for choosing latent space.

Highlights & Insights¶

Reframing understanding as a prerequisite for generation is a clever move: it narrows the shared surface between the two tasks from "both must produce something" to "both must read the input". Once the shared surface narrows, the conflict disappears on its own — and the shared part (semantic encoding) is exactly what both sides need. The idea transfers to any setting where one model must be both discriminative and generative, e.g. detection + generation or retrieval + generation.
Feeding only \(\hat z_t\) and no text hidden states is a strong assumption that was empirically validated: it amounts to requiring the LLM backbone to compress the prompt fully into the visual-semantic slot. The information bottleneck looks narrow, yet the experiments show it works — evidence that the conditioning channel need not be the LLM's last-layer hidden state.
Duality-based post-training is a free lunch with zero parameters: no structural change, no new parameters, just sending generative intermediate states back to the understanding branch for likelihood maximisation, which improves semantic consistency. Any setting where a model is both generative and discriminative can try this self-consistency loop.
Unfavourable results are reported honestly: the authors explicitly state that pixel space is slightly better for understanding and that a large time shift hurts OCR. Ablations like these are far more valuable than a page of good numbers only.

Limitations & Future Work¶

Native-UniDDT can only caption images and cannot follow instructions. Because the text in the training data comes almost entirely from captions generated by other models, instruction data is missing, and the authors simply do not report understanding metrics for the Native variants. This is the direct cost of the "use dual data, skip task-specific data" design — duality solves data efficiency but not data diversity.
The generation evaluation resolution is low (GenEval is measured at 512×512) while the showcased samples are 1024×1024, so the two do not fully correspond.
No controlled comparison against generation-only models at equal data scale: roughly 70M images plus 4o synthetic data is not comparable to the SD3/BAGEL recipes, so the GenEval lead should not be read as a quantitative measure of architectural advantage.
Concurrent work has already pointed to a stronger VAE (RepFusion uses the stronger RAE), whereas this paper fixes Flux-VAE; the authors also note that their experiments predate JiT, so the pixel-space prediction parameterisation was not considered — meaning the "latent beats pixel" conclusion has a time boundary.
An observation of my own: post-training freezes the understanding branch and trains only the decoder, which assumes the understanding branch's likelihood is a reliable judge; if that branch is itself biased on noisy inputs, the loop could pull generation in the wrong direction. No lower-bound experiment on this risk is reported.

vs DDT (Decoupled Diffusion Transformer): DDT is generation-only and splits a diffusion model into a heavy condition encoder and a light velocity decoder; UniDDT keeps that skeleton but replaces the condition encoder with "Noisy ViT + LLM", so one semantic encoding serves both understanding and generation, and it swaps DDT's AdaLN-Zero conditioning for attention-based injection.
vs Show-o2: Show-o2 is a single transformer carrying both tasks with mixed causal and full attention, and it also uses latent space; the biggest differences are that UniDDT's Noisy ViT explicitly conditions on the timestep for semantic extraction and that UniDDT splits the diffusion decoder off entirely instead of making one backbone do both jobs.
vs Mogao / BAGEL: these use parallel understanding and generation branches with decoupled parameters and are much larger (14B scale); BAGEL's 0.88 GenEval also involves prompt rewriting († in the table) and is not directly comparable with UniDDT's 0.87. UniDDT at 4B+1B overtakes BAGEL-14B on MME but still trails on MMB(en) (82.2 vs 85.0).
vs BLIP3-o / MetaQuery: these adapt a specialised MLLM to a diffusion model with adapters — assembly in essence. UniDDT belongs to the native line with a wider sharing surface (semantic encoding fully shared), at the cost of instruction ability being limited by the training data.
vs RAE / RepFusion: both belong to the "denoise in representation space" family, and RepFusion shares a similar architecture but additionally swaps in the stronger RAE; UniDDT's distinguishing point is that it folds the understanding branch into the training loop (duality-based post-training), rather than merely using the representation as a condition.

Rating¶

Novelty: ⭐⭐⭐⭐ The combination of a decoupled diffusion decoder with unified semantic encoding is a clear architectural claim, and "understanding as a prerequisite for generation" plus duality-based post-training is genuinely new, though each individual part can be traced to DDT / REPA / Show-o2.
Experimental Thoroughness: ⭐⭐⭐⭐ Both understanding and generation have main results, and the per-stage ablation, space comparison and three scaling curves are solid; points deducted for the low generation resolution, the incomparable data scale against generation-only models, and the complete absence of understanding results for the Native variants.
Writing Quality: ⭐⭐⭐⭐ The motivation is laid out clearly across all three layers and formulas and implementation details are mostly complete; points deducted for the many corrupted equations in the cache and the GenEval discrepancy between abstract and body.
Value: ⭐⭐⭐⭐ It gives a reproducible answer to "what should a unified multimodal model share and what should it separate", and both the three-stage recipe and the duality post-training loop transfer directly.