Discrete Diffusion Models with MLLMs for Unified Medical Multimodal Generation

Conference: ECCV2026
Paper: ECCV Paper
Project: MeDiM
Area: Multimodal VLM / Medical Imaging
Keywords: discrete diffusion, bidirectional attention, timestep conditioning, medical report generation, joint image-report generation

TL;DR

MeDiM adapts pretrained Liquid into a discrete diffusion model with bidirectional attention and timestep conditioning, using one denoising network for medical report-to-image, image-to-report, and joint image-report generation, with image-generation FID scores of 16.60 on MIMIC-CXR and 24.19 on PathGen.

Background & Motivation

Medical images and reports describe the same case through different modalities, yet existing generative systems often learn only one direction. A report generator reads an image and produces text, while an image generator synthesizes an image from text; chaining them does not automatically align lesion location, appearance, and description. PairAug and MedM2G explore medical multimodal generation, but the authors argue that external model composition or modality-specific components still constrain task unification and extensibility. The objective here is therefore not merely to render better chest X-rays, but to let one model complete missing modalities and generate mutually corresponding images and reports from scratch.

Unified models such as Liquid, UniDisc, and MMaDA in the natural-image domain provide a modeling foundation for this objective. Liquid places images and text in a unified token sequence but retains token-by-token autoregressive generation; discrete diffusion can repeatedly update multiple unknown positions. For paired medical data, this allows a report to disambiguate an image while the recovering image also constrains the report. The difficulty is that a pretrained multimodal large language model (MLLM) uses causal attention, restricting each position to its left context rather than enabling this bidirectional denoising. Furthermore, a masked position has different amounts of reliable context early and late in diffusion, so the model must know its current recovery stage.

MeDiM retains the visual-language knowledge of a pretrained model while changing how it organizes context and receives information about the noise stage. Quantized image tokens and report tokens are jointly corrupted, Liquid restores them, and removing the causal mask allows the modalities to access each other. Timestep embeddings enter intermediate layers through adaptive layer normalization, distinguishing widespread missing content from local refinement instead of treating every corruption state as the same completion problem. Core Idea: turn a pretrained multimodal model into a timestep-aware bidirectional discrete denoiser, using a shared joint distribution rather than task-specific generators to unify medical image and report synthesis.

Method

Overall Architecture

The modeling unit is a sequence containing both report and image tokens, not a continuous noisy image. Reports use the LLaMA tokenizer, images are quantized by Chameleon's VQGAN encoder in the implementation, and the denoising backbone is pretrained Liquid. The method starts with shared discrete modeling, predicts missing symbols through a bidirectional multimodal backbone, and controls intermediate recovery behavior through timestep conditioning. During training, inputs are corrupted versions of real image-report pairs, and the supervision target is the original clean tokens. At inference time, known conditions remain visible, only the target portion is initialized with [MASK], and MaskGIT-style iterative recovery follows.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Medical images and reports<br/>or retained conditions"] --> TOKEN["Shared Discrete Modeling"]
    TOKEN --> BACKBONE["Bidirectional<br/>Multimodal Backbone"]
    TIME["Current diffusion timestep"] --> ADAPT["Timestep Conditioning"]
    BACKBONE --> ADAPT
    ADAPT --> PREDICT["Predict tokens to recover"]
    PREDICT -->|Inference: update and denoise again| BACKBONE
    PREDICT --> OUTPUT["Image, report,<br/>or image-report pair"]
    CLEAN["Training: clean tokens"] -.->|Recovery supervision| PREDICT

Timestep conditioning in the diagram denotes modulation inside the backbone, not a separate generator appended after a complete Liquid forward pass. The dashed supervision connection exists only during training; the loop represents inference repeatedly using the updated sequence. Recovered image codes must also pass through the corresponding image decoding process, while report tokens are decoded as text. Here, unification primarily concerns joint modeling and the denoising backbone; it does not eliminate modality-specific tokenizers.

Key Designs

1. Shared Discrete Modeling: express cross-modal generation as missing-content recovery in one sequence

Text is already discrete, while vector quantization converts images into codebook indices, allowing both modalities to be predicted through categorical distributions rather than pixel regression. Section 2.1 represents a clean sample by concatenating its report and image portions; its readable structural relationship is retained below:

\[ x_0=[x_{r0},x_{i0}]. \]

Here, \(x_{r0}\) denotes report tokens and \(x_{i0}\) denotes quantized image tokens; these remain different symbol types, without forcing each word to correspond to an image patch. The base shared vocabulary size is the sum of the text vocabulary and image codebook sizes, with one additional absorbing [MASK] state. Forward diffusion progressively replaces original symbols with this state rather than adding Gaussian noise to discrete indices. Once a position enters the absorbing state, subsequent forward steps cannot restore it; at sufficiently large timesteps, the corrupted sequence approaches a fully masked state. The preceding generic reference to an approximately uniform distribution in the paper should not be treated as the terminal noise distribution of this absorbing implementation.

This representation directly determines how the three tasks share a network. Report-to-image generation retains the report and masks the image; image-to-report generation retains the image and recovers the report; joint generation recovers both portions. Switching tasks therefore primarily changes which positions serve as conditions instead of loading another task-specific model. Joint generation may start unconditionally or retain prompts specifying modality, anatomical region, or a pathological condition. The practical benefit is that image-text relationships participate in every recovery round, instead of attaching a potentially mismatched description after image generation finishes. However, a shared sequence alone does not guarantee medical correctness: both generated modalities could consistently describe a case that is not medically valid.

2. Bidirectional Multimodal Backbone: preserve pretrained knowledge while removing generation-order restrictions

MeDiM does not learn medical image-text relationships from a randomly initialized generic Transformer; it reuses Liquid's pretrained multimodal weights. The authors interpret this advantage as a distribution-alignment prior acquired through visual-language pretraining, subsequently adapted to medical content. With conventional causal attention, a report token cannot freely read image tokens appearing later in the sequence, even when those positions have already been recovered in the current iteration. This ordering constraint suits left-to-right prediction but blocks the cross-modal exchange required for joint denoising. Removing the causal mask allows positions to access context on both sides of the current sequence, enabling image evidence and report descriptions to constrain each other within a forward pass. Bidirectional attention does not mean leaking supervision targets into the input: training still receives a corrupted sequence, with target tokens supplied through the recovery loss.

This design combines two distinct questions: which initialization to use and which attention structure to adopt. Table 4 separately compares DiT and UniDisc backbones, removal of pretrained weights, and retention of the causal mask, exposing different forms of degradation. Keeping the causal mask causes the largest performance loss, indicating that simply attaching a pretrained MLLM to a diffusion task is insufficient. Mechanistically, pretrained weights supply existing knowledge, while bidirectional connections let that knowledge support recovery at arbitrary missing positions. This also explains why medically fine-tuned Liquid is not equivalent to the structurally adapted MeDiM. The alignment-prior explanation is supported by ablations, but those experiments do not completely disentangle semantic knowledge, optimization difficulty, and model capacity.

3. Timestep Conditioning: tell intermediate layers whether to perform coarse recovery or refinement

Discrete diffusion inputs can all contain [MASK] while requiring different recovery behavior at different noise stages. Early on, many positions are unknown and the network depends more on known conditions and its learned joint distribution; later, more image-text context supports finer local consistency. MeDiM maps the current timestep to a continuous embedding, explicitly providing the backbone with transition-schedule information. Continuous here describes the timestep embedding vector, not a return to continuous Gaussian diffusion for images and text. The embedding further enters adaptive layer normalization, or AdaLN, dynamically predicting normalization affine parameters instead of always using fixed scales and shifts. The paper also discusses the AdaLN-Zero variant, treating it together with AdaLN as a design for stable training and feature modulation.

Intermediate features from both modalities thus receive a common time condition instead of each layer inferring the recovery stage solely from the visible masking pattern. This is conditioning inside the denoising network, without an external diagnostic model or additional modality expert. In the no-timestep-embedding ablation, the authors also replace AdaLN with ordinary LayerNorm, removing the entire temporal conditioning pathway. By contrast, the no-AdaLN ablation still injects timestep embeddings into token representations through weighted addition, but no longer uses adaptive normalization. The two rows therefore address different questions, namely whether to provide temporal conditioning and how to inject it, rather than independently deleting equivalent small modules. The main text does not fully specify the layer-by-layer AdaLN-Zero wiring, so this note does not supply unverified gating equations.

A Worked Example

Suppose a user requests a chest X-ray and report pair with a prompt specifying only the chest X-ray modality and a pathological condition; this is an illustrative walkthrough of the paper's inference setup, not an additional experimental case. The model retains the prompt and initializes both target image and report regions with [MASK], without a real image as a hidden input. The first denoising round reads the prompt and timestep and predicts missing symbols in both modalities; image-code recovery and text recovery belong to the same sequence task. Later rounds read the updated sequence, using bidirectional attention to let recovered visual information influence text and existing descriptions constrain image refinement. After iteration ends, image codes decode into an image and text tokens decode into a report, forming a synthetic pair that can support subsequent training. For report-to-image generation, the full report instead remains visible and only the image portion is recovered; the reverse direction retains the image. The paper names the MaskGIT inference strategy, but the supplied main text does not specify the sampling-round count, confidence thresholds, or per-round token counts; none are invented here.

Loss & Training

Training learns to recover original discrete symbols from corrupted sequences, with an objective described as expected negative log-likelihood weighted by the transition schedule. The timestep determines the corruption distribution and recovery weighting, so the network must cover inputs ranging from heavy to light masking. Equations (1), (2), and (6) in the local full text have obvious extraction damage; in particular, the logarithmic term and weighting in Equation (6) cannot be reliably recovered from that text. This note therefore retains the verbal definition rather than presenting a familiar diffusion loss as the authors' exact equation. The data setup includes 368,960 MIMIC-CXR chest X-ray/report pairs and 736,188 PathGen pathology pairs; the text also states that pathology data are subsampled to balance the distribution. These are the reported dataset counts, not enough information to infer an exact per-epoch mixing ratio. Images have resolution \(512\times512\), text is capped at 256 tokens, and the initial learning rate is \(1\times10^{-5}\). Training uses a warmup cosine schedule with restarts; the authors report 1M steps, 8 A100 GPUs, and 160 GPU hours. The main text does not provide the batch size and throughput needed to reconcile that compute figure, so 160 GPU hours should not be silently interpreted as 160 hours of wall-clock time. Evaluation uses the MIMIC-CXR test set and 8,000 pathology pairs, while PathGen report evaluation further selects 5,000 high-quality reference reports using Qwen2-VL.

Key Experimental Results

Main Results

The following selection comes from Tables 1 and 2 on page 10, evaluating report-conditioned image generation; lower FID and higher IS are better. Models marked SFT are medically fine-tuned, and comparisons apply to the paper's settings rather than every publicly available configuration of those methods.

Dataset	Method	FID	IS
MIMIC-CXR	SDM (SFT)	78.97	2.91
MIMIC-CXR	UniDisc	82.54	2.82
MIMIC-CXR	Diff-CXR	19.50	3.24
MIMIC-CXR	MeDiM	16.60	2.87
PathGen	SDM (SFT)	55.76	4.03
PathGen	UniDisc	80.99	4.15
PathGen	PixCell	98.54	4.27
PathGen	MeDiM	24.19	4.28

Chest X-ray FID decreases from Diff-CXR's 19.50 to 16.60, an absolute reduction of 2.90, but MeDiM's IS is 2.87 compared with 3.24, so it does not lead both metrics. The quantitative evidence for fidelity here is primarily FID; distributional similarity should not be equated with diagnostic accuracy for lesions.

The next selection comes from Table 3 on page 11, with higher values better throughout; PathGen uses the 5,000 filtered reference reports described above.

Dataset	Method	BLEU-1	BLEU-2	BLEU-3	METEOR	ROUGE-L
MIMIC-CXR	R2Gen	0.305	0.179	0.104	0.233	0.395
MIMIC-CXR	MeDiM	0.328	0.185	0.109	0.265	0.297
PathGen	R2Gen	0.160	0.090	0.055	0.251	0.278
PathGen	MeDiM	0.185	0.084	0.037	0.258	0.226

MeDiM exceeds R2Gen on METEOR in both domains, but scores lower on PathGen BLEU-2, BLEU-3, and ROUGE-L, and on chest X-ray ROUGE-L. The supported conclusion is competitive report generation from a unified model, not a universal advantage over specialized report generators.

Ablation Study

The following selection comes from Table 4 on page 13, reporting cross-domain means over MIMIC-CXR and PathGen rather than scores on either individual test set. The causal-mask row restores the autoregressive restriction; the no-AdaLN row still retains additively injected timestep embeddings.

Config	Mean BLEU-1	Mean METEOR	Mean FID	Mean IS
Full MeDiM	0.256	0.262	20.40	3.57
DiT backbone	0.195	0.214	63.22	2.81
UniDisc backbone	0.223	0.255	51.59	3.05
Without pretrained MLLM weights	0.205	0.229	68.27	2.83
Without timestep embeddings	0.221	0.246	40.03	3.13
Without AdaLN designs	0.232	0.247	32.68	3.16
With causal mask	0.152	0.142	143.72	2.02

Key Findings

• Bidirectional access provides the strongest local evidence: retaining the causal mask worsens mean FID from 20.40 to 143.72 and lowers mean METEOR from 0.262 to 0.142.
• Pretraining matters: removing pretrained MLLM weights results in mean FID of 68.27, supporting the value of transferring existing image-text knowledge.
• Paired-consistency evaluation uses 8,000 generated pathology pairs and 5,159 generated chest X-ray pairs; human evaluation samples only 100 pairs, split 1:1 between the domains (Section 3.5, page 13).
• Downstream augmentation combines 200k real pairs and 200k synthetic pairs, with each source balanced 1:1 across the two domains; the gains concern this low-data setting rather than training on all real data (Section 3.5, page 13).
• On pages 2 and 4, the authors report relative PathGen downstream gains of 6.43%, 18.57%, 31.58%, and 4.80% for BLEU-1/2/3 and METEOR; the bar values in Figures 5 and 6 are not fully preserved by text extraction, so absolute baseline scores are not reverse-engineered.

Highlights & Insights

• Task switching is determined by conditioning positions. Marking different parts of one image-text sequence as known reuses the denoiser instead of maintaining three separate generation systems.
• Attention constraints deserve early scrutiny when adapting pretrained models. Table 4 shows that a strong backbone cannot automatically translate its knowledge advantage into joint generation if it cannot access the required context.
• Joint generation is also tested through downstream models. Whether synthetic pairs improve an independent report generator is more relevant to practical augmentation than a few visually appealing paired examples alone.

Limitations & Future Work

• The main validation covers chest X-rays and pathology; CT, electronic health records, and broader clinical reasoning mentioned in the introduction are not tasks already validated by the supplied main text.
• FID, IS, and text-overlap metrics do not establish clinical factual correctness; Qwen2-VL judgments and 100 human-evaluated pairs are also insufficient to establish deployment reliability.
• Qwen2-VL filters PathGen reference reports and also evaluates paired consistency. Evaluation preferences and sample-selection effects deserve attention rather than treating these as fully independent clinical evidence.
• Internally consistent synthetic pairs need not reflect real pathological mechanisms. Future evaluation should include disease labels, localization consistency, external institutions, and independent clinical experts.
• The local main text lacks readable bar-chart values, an exact inference-step count, and a complete training-loss equation; these are reproducibility boundaries, not gaps to fill by guessing.

Related Work & Insights

• Liquid, source reference [32]: supplies a unified autoregressive image-text backbone; MeDiM reuses its weights but changes attention and temporal conditioning rather than merely performing medical instruction tuning.
• UniDisc, source reference [27]: is also unified multimodal discrete diffusion and a direct joint-generation baseline; this paper emphasizes adapting pretrained MLLMs to the medical domain.
• MMaDA, source reference [35]: is associated by the authors with diffusion-language backbones, whereas MeDiM demonstrates structural conversion from an autoregressive MLLM; this comparison concerns the versions discussed in the paper.
• MaskGIT, source reference [5]: supplies the inference idea of iteratively recovering masked tokens; MeDiM applies it to sequences containing two medical modalities rather than natural images alone.
• Research direction: independent consistency evaluation between image lesion regions and report entities could test whether current metrics miss fluent but medically incorrect content, without changing the generation objective.

Rating

• Novelty: 4/5. Adapting a pretrained autoregressive MLLM into a medical joint discrete denoiser is a clear integration, although its core components have precedents.
• Experimental Thoroughness: 3/5. Three tasks, two domains, and key ablations are covered, but clinical factual evaluation and external generalization evidence remain limited.
• Writing Quality: 3/5. The task narrative is understandable, but some scope claims are broad and training and sampling details remain insufficient for independent reproduction.
• Value: 4/5. The approach has clear research value for unified medical generation and synthetic image-text augmentation, without establishing readiness for direct clinical use.