Flexible Control of 3D CT Generation via Text and Semantically-Defined Segmentation Prompts

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/batmanlab/MedSynV2
Area: Medical Imaging
Keywords: 3D CT Generation, Controllable Diffusion, Diffusion Transformer, Flow Matching, Multimodal Guidance

TL;DR

MedSynV2 introduces a flexible multimodal framework that tokenizes volumetric CT and arbitrary single-lesion/anatomy binary masks into a shared video latent space, combining compound text prompts with a gated-attention DiT to achieve high-resolution 3D CT generation without requiring full-organ annotations.

Background & Motivation

Generative models for volumetric medical imaging play an indispensable role in contemporary healthcare AI, serving pivotal functions such as privacy-preserving synthetic data augmentation, anatomical prior modeling for solving ill-posed inverse reconstruction problems, and standardized training simulations for radiology residents. However, synthesizing high-resolution, anatomically coherent three-dimensional computed tomography (3D CT) scans under precise user controllability remains an open and formidable challenge. Existing paradigms predominantly bifurcate into two distinct control modalities: text-driven generation conditioned on free-form radiology reports (e.g., GenerateCT, MedSyn, Text2CT) and layout-driven generation conditioned on semantic segmentation maps (e.g., MAISI, 3D MedDiffusion). While radiology reports offer natural clinical flexibility, their prose descriptions are inherently qualitative and approximate, lacking the spatial precision required to delineate the exact boundary, 3D coordinate, or infiltrating geometry of focal lesions. Conversely, segmentation-guided approaches provide explicit spatial boundaries but enforce an onerous dependency on full-organ annotations spanning dozens or even hundreds of anatomical classes, which drastically inflates annotation and inference overhead while stifling the generative diversity of unannotated regions.

The core tension underlying these approaches is that clinical workflows rarely demand or provide exhaustive whole-body organ segmentation; clinicians and researchers typically care about a single target pathology—such as an isolated pulmonary nodule, a unilateral pleural effusion, or specific lobar changes. Yet existing generative frameworks can neither synthesize coherent global anatomical context from an isolated binary mask nor seamlessly integrate localized spatial constraints with the nuanced pathological details present in lengthy clinical reports. Retraining full-organ generators whenever a novel abnormality is encountered is computationally prohibitive and practically intractable.

To bridge this divide between rigid full-mask dependencies and unconstrained text prompting, this work decouples spatial mask geometry from semantic identity. Core idea: MedSynV2 projects arbitrary single-lesion or anatomical binary masks into a shared video tokenization latent space via channel-wise early concatenation, couples mask semantic descriptors with long radiology reports into compound text prompts, and trains a gated-attention diffusion transformer under rectified flow matching to achieve arbitrary combinations of text and localized spatial control.

Method

Overall Architecture

The end-to-end generative pipeline of MedSynV2 is formulated as a latent rectified flow system where a modified diffusion transformer (DiT) directly predicts clean data latents (\(x_0\)-prediction formulation) from noisy volumetric representations. The framework gracefully supports four conditioning configurations at inference: text report only, single-lesion segmentation mask only, joint text and mask guidance, or unconditional synthesis. Raw CT volumes and binary segmentation masks are first affine-registered to a common anatomical template to minimize inter-patient spatial variance. A 3D video tokenizer fine-tuned from Open-Sora compresses the \(448 \times 448 \times 448\) volumes into compact latent tensors. The semantic identity of the binary mask is encoded as a textual prefix (\(c_m\)) and prepended to the clinical report (\(c_r\)) to form a compound prompt processed by CXR-BERT. Inside the DiT backbone, features are updated through multi-head self-attention (MHSA), multi-head cross-attention (MHCA) with gated modulation, and depthwise separable convolutions before a Dormand-Prince ODE solver computes the clean trajectory.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input: 3D CT Volume x and Optional Binary Mask m<br/>along with Optional Radiology Report"] --> T1["Shared Video Tokenizer & Latent Projection<br/>Open-Sora 3D VAE compresses to unified latent space"]
    T1 --> T2["Early Channel Fusion & Compound Prompting<br/>Concatenate [zt; zm] Patchify and merge mask text with report"]
    T2 --> T3["Gated Attention DiT Backbone<br/>Multi-head Self-Attention + Text Cross-Attention + Depth-Sep-Conv"]
    T3 --> T4["Conditional Rectified Flow Velocity Field & ODE Solver<br/>Train via v-loss and sample with Dormand-Prince in 50 steps"]
    T4 --> Out["Output: High-Resolution 3D CT Volume x_hat_0"]

Key Designs

1. Shared Video Tokenizer & Latent Projection: Unifying Heterogeneous 3D Geometry and Image Spaces

Directly modeling high-resolution volumetric images (\(448 \times 448 \times 448\)) in voxel space incurs prohibitive GPU memory consumption, and training specialized segmentation encoders introduces redundant parameters. MedSynV2 interprets 3D CT scans as video slice sequences along the axial plane and adapts the Open-Sora video autoencoder fine-tuned exclusively on CT-Rate images. A CT volume \(x\) is encoded into continuous latent tokens \(z_0 = \text{Enc}(x) \in \mathbb{R}^{(n_d \times n_h \times n_w) \times n_c}\) with dimensions \((112 \times 56 \times 56) \times 4\). Crucially, the identical frozen encoder is applied to the binary segmentation mask \(m\), yielding \(z_m = \text{Enc}(m)\). Because both spatial masks and image textures are projected onto a shared latent manifold, the noisy latent \(z_t\) and mask latent \(z_m\) can be directly concatenated along the channel dimension prior to patchification:

\[h_t = \text{Patchify}([z_t; z_m])\]

This early-fusion strategy requires zero additional token length (retaining exactly 43,904 patches with patch size \(2 \times 2 \times 2\)), injecting crisp geometric priors into the transformer at negligible extra computational cost.

2. Compound Prompting & Gated Attention: Grounding Arbitrary Binary Masks in Long-Form Clinical Text

A single binary mask is inherently devoid of categorical semantics; an identical blob-like region could represent a benign nodule, consolidation, or fluid accumulation. To resolve this ambiguity without relying on rigid one-hot label channels, MedSynV2 introduces a compound text prompt \(c_{\text{text}}\). A descriptive prefix \(c_m\) detailing the mask's identity (e.g., "I give you segmentation of <The Heart>", replaced with a null descriptor when absent) is prepended to the optional clinical report \(c_r\) (up to 512 tokens spanning Findings and Impressions). In addition, because standard attention architectures suffer from attention sinks that over-index early tokens while neglecting critical diagnostic findings situated later in long reports, gated attention is integrated across every transformer block:

\[Q_h = h_j W_Q, \quad K_h = c_{\text{text}} W_K, \quad V_h = c_{\text{text}} W_V, \quad G_h = h_j W_G\]

\[A_h = \omega(G_h) \odot \left( \text{Softmax}\left(\frac{Q_h K_h^\top}{\sqrt{d_h}}\right) V_h \right)\]

The gating vector \(G_h\) introduces non-linear feature-wise filtering, enabling the model to dynamically balance localized spatial guidance from the pre-prompt against global pathological cues throughout the full clinical narrative.

3. Depthwise Separable Convolution: Preserving Inter-Patch Continuity and 3D Anatomical Fidelity

While standard Vision Transformers excel at capturing long-range contextual relationships, patch-based tokenization tends to disrupt fine-grained local anatomical smoothness, frequently inducing blocking artifacts and orthogonal slice discontinuities across sagittal and coronal planes. To reinforce local inductive biases, MedSynV2 incorporates a lightweight 3D depthwise separable convolution (Depth-Sep-Conv) layer following the feedforward network of each DiT block:

\[h_{j+1} = \text{Conv}(h_j) + v_j\]

By independently executing spatial filtering across contiguous patch tokens followed by channel-wise mixing, this component enforces boundary smoothness and anatomical consistency across neighboring voxels. Qualitatively, it eliminates grid-like interpolation artifacts along vessel bifurcations, bronchial pathways, and pleural interfaces with minimal parameter overhead.

Loss & Training

The framework is trained using a conditional rectified flow objective under the \(x\)-prediction formulation. Given clean latent \(z_0\) and standard Gaussian noise \(\omega \sim \mathcal{N}(0, I)\), the linear trajectory is defined by \(z_t = (1-t)z_0 + t\omega\) for \(t \in [0, 1]\). The neural network predicts clean latent states \(z_\theta(z_t, t, c)\), defining the velocity vector field and conditional flow matching loss as:

\[v_\theta(z_t, t, c) := \frac{z_\theta - z_t}{1 - t}\]

\[\mathcal{L} = \mathbb{E}_{t, z_0, \omega} \left\| v_\theta(z_t, t, c) - \frac{\omega - z_0}{1} \right\|_2^2\]

Training incorporates modality randomization across each batch: samples are stochastically partitioned into text-only, mask-only, dual-condition, or unconditional modes (\(z_m\) is set to zero when masks are omitted). Optimization is conducted on a single NVIDIA A100 GPU using AdamW (learning rate \(10^{-4}\), batch size 1 with 4 gradient accumulation steps). At inference time, deterministic sampling is executed using a Dormand-Prince ODE solver in 50 integration steps.

Key Experimental Results

Main Results

Quantitative comparisons were conducted on the CT-Rate dataset across four clinical intensity windows (Lung, Vessel, Soft Tissue, and Bone). FID scores were evaluated across all three orthogonal planes (XY: Axial, YZ: Sagittal, ZX: Coronal) under the text-report-only setting.

Model	Lung Mean FID ↓	Vessel Mean FID ↓	Soft Tissue Mean FID ↓	Bone Mean FID ↓
GenerateCT (ECCV 2024)	14.27	10.37	12.54	8.71
MedSyn (TMI 2024)	12.83	15.72	17.52	7.44
Text2CT (arXiv 2025)	12.58	9.35	11.96	7.36
MedSynV2 (Ours)	9.59	6.46	8.47	4.58

Semantic alignment was evaluated using Pillar-0 (11 intensity windows) for text-to-image (T2I) CLIP-Score and zero-shot CT-CLIP classification across 18 pathology classes:

Source / Model	Pillar-0 T2I (%) ↑	Pillar-0 I2I (%) ↑	Classification AUROC (%) ↑	Classification F1 (%) ↑
Real (Ground Truth)	26.36	—	67.34	58.95
GenerateCT	22.10	32.11	50.57	26.33
MedSyn	22.75	44.59	51.32	40.12
Text2CT	25.80	49.38	50.96	37.43
MedSynV2 (Ours)	27.61	46.79	52.24	48.40

Ablation Study

The utility of synthetic data was benchmarked on the SemiSeg Ground-Glass Opacity (GGO) segmentation challenge. A standard UNet was trained under three data regimes: 50 real cases, 416 synthetic cases, and their combined set (466 cases).

Training Data Regime	Sample Size	Dice (%) ↑	Jaccard (%) ↑	ASD (pixel) ↓	HD95 (pixel) ↓
Real Data Only	50	58.78	43.49	21.65	65.33
Synthetic Data Only	416	48.82	32.30	56.32	10.61
Combined Training (Real + Synthetic)	466	68.13	54.27	43.32	4.69
Inf-Net (Baseline)	—	62.41	45.35	32.67	20.63
MedPSeg (Domain SOTA)	—	65.20	48.13	28.45	5.51

Key Findings

• Orthogonal Plane Coherence: Baseline models relying on 2D slice super-resolution (GenerateCT) exhibited severe degradation along non-axial views (e.g., sagittal FID 14.07, coronal FID 21.71 in lung window). MedSynV2 achieved isotropic structural fidelity (coronal FID 11.40 vs. Text2CT's 17.69), driving a 24% relative improvement in overall mean FID.
• Downstream Data Efficiency: Augmenting the limited real training set with 416 synthetic cases elevated UNet GGO segmentation Dice from 58.78% to 68.13% (+9.35% absolute gain) and reduced HD95 from 65.33 to 4.69 pixels, surpassing dedicated multi-task architectures like MedPSeg (65.20%).
• Expert Double-Blind Concordance: In a double-blind evaluation of 150 cases assessed by two board-certified radiologists (4 and 17 years of experience), synthetic volumes showed distribution patterns comparable to real scans in the "Match" (40/36) and "Partial Match" (18/30) tiers, with negligible disagreement counts (17/9).

Highlights & Insights

• Decoupled Mask Geometry and Linguistic Semantics: By mapping binary spatial masks into a shared video VAE space and shifting semantic classification entirely into the text pre-prompt, the model sidesteps the need for fixed categorical segmentation channels, achieving plug-and-play spatial conditioning.
• Resource-Efficient 3D Volumetric Synthesis: Compressing \(448^3\) CT scans into \((112, 56, 56) \times 4\) latents and fusing masks via channel stacking preserves a constant token length of 43,904, allowing full flow-matching training on a single commodity A100 GPU.
• Progressive Disease Staging: Integrating SDEdit stochastic inversion enables sequential multi-stage synthesis—transitioning from healthy anatomy to progressive pleural effusion and metastatic nodule seeding—providing an effective tool for pedagogical resident training.

Limitations & Future Work

• Multi-Center Domain Shift: The model was primarily trained on affine-aligned CT-Rate volumes; robustness across varying reconstruction kernels, slice thicknesses, and contrast-enhanced imaging protocols remains to be established.
• Spatial Collision in Multi-Mask Conditioning: When multiple closely abutting or overlapping pathological masks are presented simultaneously, concatenating them into a single binary input channel may introduce semantic ambiguity, warranting future exploration into multi-channel instance encoding.

Related Work & Insights

• vs MedSyn: MedSyn operates directly in voxel space using multi-resolution stages and exhibits loose mask adherence where synthesized organs spill over boundaries; MedSynV2 transitions to latent flow matching with early channel fusion, achieving strict mask adherence and lower FID across tissue windows.
• vs MAISI-v2: MAISI relies on full-organ segmentations covering over 100 classes and lacks natural language report prompting; MedSynV2 enables flexible conditioning on arbitrary individual binary masks, long-form reports, or both.

Rating

• Novelty: ⭐⭐⭐⭐ [Decoupling binary mask geometry from textual semantics eliminates full-organ annotation constraints]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated across four CT contrast windows, dual CLIP metrics, downstream segmentation augmentation, and expert double-blind reviews]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear technical motivation, well-structured mathematical formulation, and thorough ablation evidence]
• Value: ⭐⭐⭐⭐⭐ [Provides a practical, highly controllable, open-source 3D CT generative foundation for medical image synthesis]