MedRegion-CT: Region-Aware Multimodal Large Language Model via SlowFast Tokenization and Pseudo-Mask Guidance for 3D CT Report Generation¶

Conference: ECCV 2026
arXiv: 2506.23102
Code: https://github.com/babbu3682/MedRegion-CT
Area: Multimodal VLM
Keywords: CT Report Generation, Region-Aware Visual Encoding, SlowFast Representation, Mask Guidance, Multimodal Large Language Model

TL;DR¶

MedRegion-CT proposes a region-aware 3D CT report generation framework. By employing three modules—Region-based SlowFast Tokenizer, Mask-Driven Visual Extractor, and Lesion Attribute Extractor—it explicitly models the hierarchical diagnostic workflow of radiologists into an MLLM, achieving SOTA performance on structured chest CT report generation benchmarks.

Background & Motivation¶

3D CT report generation has consistently been a key challenge in medical image analysis. A comprehensive radiology report needs to cover abnormal findings across multiple anatomical regions, and the pathological characteristics (size, morphology, location) in each region directly influence clinical diagnosis. Recently, 3D medical multimodal large language models (such as RadFM, M3D, Med3DVLM, etc.) have demonstrated the capability of joint reasoning over CT images and clinical language, achieving significant progress in report generation and visual question answering. However, existing methods almost exclusively rely on global volume-level feature representations: they either use 3D ViT with token compression to reduce computational complexity, or apply slice-level 2D encoding coupled with inter-frame aggregation. Essentially, both approaches compress the entire CT volume into a "flat" global feature, making it difficult to capture pathological details in specific anatomical regions. Processing slices through global average pooling or uniform sampling may discard or dilute fine-grained information of key organ regions within a vast background of empty slices.

In clinical practice, radiologists follow a clear hierarchical workflow: they first quickly scan through all slices to obtain a global overview, then conduct a detailed analysis of key organ regions, and finally make diagnostic judgments by combining quantitative attributes (volume, maximum diameter, spatial location) of the lesions. This process is inherently coarse-to-fine—transitioning from global to local and then to quantitative, with clear anatomical anchors at each step. The "globally flattened" representation of current MLLMs fundamentally conflicts with this workflow: the models lack mechanisms to focus on specific anatomical regions being described, nor do they possess means to perceive concrete geometric attributes of lesions. Some works attempt to introduce region-aware capabilities using object detection or bounding box annotations, but the annotation cost is extremely high for 3D CT, and these methods rely heavily on manual fine-grained localization signals.

The core idea of this paper is that since universal segmentation models (such as TotalSegmentator) can produce pseudo-masks of major organs and common lesions in a zero-shot manner, these masks can serve as "regional guidance signals" to help the model focus on clinically relevant anatomical locations during the visual encoding stage. Core Idea: To build a complete region-aware pipeline. First, use a Region-based SlowFast Tokenizer to anchor attention on organ regions (Fast path provides global structure, Slow path preserves local details). Next, use a Mask-Driven Visual Extractor to generate compact organ-level representations via multi-level mask pooling. Finally, use a Lesion Attribute Extractor to encode quantitative attributes such as volume, maximum diameter, and spatial position of lesions into structured text prompts injected into the LLM, thereby achieving both region localization accuracy and clinical quantitative precision in report generation.

Method¶

The overall architecture of MedRegion-CT is shown in Figure 2. The core mechanism is to decompose the traditional "coarse-to-fine" diagnostic workflow of radiologists into three parallel visual/attribute extraction pathways, which are then fed into the LLM to generate structured reports.

Overall Architecture¶

The input is a 3D CT volume (448×448×128). First, a universal segmentation model, TotalSegmentator, generates pseudo-binary masks for six major anatomical regions (lung parenchyma, airway, mediastinum, heart and great vessels, abdominal organs, musculoskeletal system) and four common lesions (lung nodules, pleural effusion, pericardial effusion, renal cysts). Concurrently, the CT slices are processed frame-by-frame through a pre-trained 2D visual encoder, MedSigLIP, to extract slice-level features.

The visual features and organ/lesion masks then enter three parallel processing pathways: 1. Region-based SlowFast Tokenizer — Guided by the organ masks, it extracts global Fast tokens and local Slow tokens in parallel from slice features, merging them into a region-aware 3D visual representation \(T_{vision}\); 2. Mask-Driven Visual Extractor — Extracts compact representations \(T_{mask}\) for each organ via differentiable mask pooling across multi-level features from MedSigLIP; 3. Lesion Attribute Extractor — Performs deterministic geometric computations (volume, maximum diameter, position) on the lesion pseudo-masks, encoding the results into structured text prompts \(T_{attr}\).

Finally, these three representations, along with instruction prompts, are concatenated and fed into LLaMA-3.1-8B (fine-tuned using LoRA) to generate a structured radiology report organized by the six major anatomical regions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["3D CT Scan<br/>448×448×128"] --> B["Universal Segmentation Model<br/>TotalSegmentator"]
    A --> C["2D Visual Encoder<br/>MedSigLIP"]
    B --> D["Six Organ Pseudo-Masks<br/>+ Lesion Pseudo-Masks"]
    C --> E["Slice-Level Visual Features<br/>D × T"]
    D --> F["Region-based SlowFast Tokenizer<br/>Fast(Global)+Slow(Local)"]
    D --> G["Mask-Driven Visual Extractor<br/>Multi-Level Differentiable Mask Pooling"]
    D --> H["Lesion Attribute Extractor<br/>Volume/Diameter/Position"]
    E --> F
    E --> G
    F --> I["T_vision"]
    G --> J["T_mask"]
    H --> K["T_attr"]
    I --> L["LLaMA-3.1-8B<br/>LoRA Fine-tuning"]
    J --> L
    K --> L
    L --> M["Structured Radiology Report<br/>6 Anatomical Regions"]

Key Designs¶

1. Region-Aware SlowFast Tokenizer: A Dual-Pathway Strategy Anchoring Visual Encoding with Organ Masks

Existing 3D MLLMs' visual encoding either utilizes 3D ViT with token compression (sacrificing fine-grained details) or performs inter-frame aggregation on slice-level 2D features (losing spatial relations in the depth dimension). More importantly, conventional SlowFast strategies [LITA, SlowFast-LLaVA] uniformly sample all slices regardless of content—but a large number of slices in a CT scan contain no organs of interest (e.g., scan ranges extending beyond the body boundaries), meaning features of these empty slices only dilute useful information.

The proposed improvement is to directly embed organ pseudo-masks into the tokenization process. Given the \(j\)-th spatial token feature \(f(i,j)\) of slice \(i\) and an indicator variable \(m(i,j) \in \{0,1\}\) from the organ mask, the Fast pathway preserves only mask-activated slices (\(\sum_j m(i,j) > 0\)), performing global average pooling over all spatial tokens of that slice (kernel = T) to obtain a global profile representation with one token per slice. The Slow pathway goes further: it extracts tokens only from within the organ regions defined by the masks and applies a 2×2 spatial pooling kernel to retain local textural details—essentially performing an "organ attention mask" naturally during token compression. The outputs of both pathways are concatenated and fed into the LLM. By discarding background empty slices and non-organ regions, the sequence length is reduced from \(D \times T\) to a manageable few hundred tokens, with each token carrying explicit anatomical location semantics.

2. Mask-Driven Visual Extractor: Organ Encoding with Multi-Level Cross-Scale Mask Pooling

Simple SlowFast tokens provide sparse slice-level sampling, which struggle to cover entire organ regions comprehensively. Inspired by the Osprey framework, this paper designs a volume mask pooling mechanism for 3D CT. It extracts multi-scale features from layers 6, 12, 18, and 24 of MedSigLIP, applying differentiable weight average pooling with the organ pseudo-mask on each layer (aligning the masks to the feature map resolution via trilinear interpolation). For each organ \(o\) at layer \(l\), this yields a mask token \(k_l(o)\). The \(k_l(o)\) from each layer is processed by an independent linear projector \(\phi_l\), element-wise summed, and finally mapped to the LLM embedding space via a shared projector \(\psi\), generating a compact representation \(T_{mask}\) of one token per organ. Key strengths of this design are: (1) multi-level features capture both fine-grained organ boundaries and high-level semantics; (2) each organ is represented using only one token (6 organ tokens in total), imposing almost no burden on the LLM's context length; (3) these tokens are inserted into fixed positions in the LLM via <region> special tokens, allowing the model to directly "see" the visual information of each organ when generating the corresponding paragraph.

3. Lesion Attribute Textualization: Deterministic Geometric Computation and Structured Text Injection

3D CT scans often undergo substantial downsampling or cropping during the encoding process, resulting in the loss of fine-grained morphological details. Even if visual tokens contain regional information, they can hardly express precise quantitative clinical characteristics such as "this lung nodule has a volume of 3.2 mL, a maximum diameter of 18 mm, and is located in the right upper lobe". Instead of relying on implicit learning, this work extracts the lesion pseudo-masks from TotalSegmentator and performs deterministic algorithms in the voxel space: first, it computes the intersection of the lesion mask and the anatomical guidance region (e.g., right upper lobe), filters segmentation noise using the largest connected component, and then calculates the absolute volume \(V = (\sum \mathcal{M}_{target}) \times (s_x s_y s_z) / 10^3\) and the bounding box maximum diameter \(d_{max} = \max_{i \in \{x,y,z\}} (L_i \times s_i)\). These attributes are assembled into structured natural language, such as "A nodule with a diameter of 18.0 mm and a volume of 3.2 mL is identified in the right upper lobe.", which is input to the LLM as attribute tokens \(T_{attr}\). This "deterministic geometric computation + textualization" paradigm is more reliable and interpretable than implicitly predicting lesion attributes from visual features, and it is naturally immune to the resolution lower bound of the visual encoder—as long as the segmentation model detects the lesion, the geometric calculations are performed in the full-resolution voxel space.

Loss & Training¶

Training adopts a two-phase LoRA fine-tuning strategy. In the first phase, the visual encoder and LLM are frozen, and only the multimodal connector and mask projection layers are trained. They are pre-aligned on raw image-report pairs for 6 epochs (lr=1e-3, batch=48). In the second phase, the LLM is unfrozen, and all trainable modules are fine-tuned for 6 epochs targeting structured regional reports (lr=2e-5, batch=48). Training employs the AdamW optimizer with DeepSpeed ZeRO stage 3 distributed strategy on A100 80GB GPUs. The visual encoder is MedSigLIP (pre-trained on millions of CT slice-report pairs), the LLM is LLaMA-3.1-8B, with LoRA parameters set to rank=128 and alpha=256.

Key Experimental Results¶

Main Results¶

Dataset	Metric	RadFM	M3D	MedM-VL	Med3DVLM	CT-CHAT	MedRegion-CT
RadGenome	BLEU	0.3153	0.3177	0.3026	0.3269	0.3081	0.3435
(Internal)	ROUGE	0.3732	0.3973	0.3854	0.3991	0.3900	0.4225
	METEOR	0.4894	0.4938	0.4811	0.5029	0.4831	0.5205
	GREEN	0.3030	0.4013	0.3970	0.3881	0.3966	0.4555
	CRG	0.3477	0.3717	0.3426	0.3866	0.3425	0.3959
	GPT-CA	0.2852	0.3586	0.3168	0.3733	0.3265	0.4003
AMC	GREEN	0.0977	0.1102	0.1115	0.1138	0.1138	0.1572
(Zero-shot)	CRG	0.3476	0.3725	0.3472	0.3791	0.3487	0.3822
	GPT-CA	0.2018	0.2243	0.1963	0.2366	0.2053	0.2642

Ablation Study¶

Configuration	BLEU	GREEN	CRG	Description
Full (RR+Attr+Mask)	0.3273	0.4025	0.3861	Full model
w/o Mask (RR+Attr)	0.3271	0.4010	0.3846	Removing the mask extractor has minimal impact
w/o Attr (RR+Mask)	0.3248	0.3937	0.3699	Clinical metrics drop significantly when removing attribute encoding
RR only	0.3152	0.3930	0.3613	SlowFast Tokenizer only
M3D encoder replacement	0.3148	0.4009	0.3487	Replace RR with pre-trained 3D ViT
LITA encoder replacement	0.3106	0.3880	0.3562	Replace RR with conventional SlowFast

Key Findings¶

All three modules contribute positively to the final performance, where Mask and Attr offer complementary benefits: Attr provides more noticeable improvements on clinical metrics (GREEN, CRG, GPT-CA), while Mask shows a stronger advantage in zero-shot generalization.
Compared to directly using a pre-trained 3D ViT (M3D encoder) or conventional SlowFast (LITA encoder), the proposed Region-aware SlowFast Tokenizer (RR) performs significantly better on clinical metrics, suggesting that the organ-mask-guided tokenization strategy indeed helps in preserving critical diagnostic information.
Segmentation noise stress testing reveals that distorting organ masks leads to a larger performance drop than distorting lesion masks. This is because organ masks serve both the SlowFast Tokenizer and the Mask-Driven Visual Extractor, whereas lesion masks are only utilized for attribute extraction.
Stratified analysis by lesion size shows that the proposed method significantly outperforms baselines in the detection rate of small-to-medium-sized nodules (\(>5\) mm³), but falls below baselines for extremely small nodules (\(\le 5\) mm³). This is because the upstream TotalSegmentator missed 110 out of 160 micro-nodules, meaning the performance lower bound remains limited by the segmentation model.
In a radiologist blind review (50 randomly sampled cases, scored on a 1-10 scale), the proposed model scored \(6.98 \pm 1.41\), greatly exceeding the runner-up Med3DVLM (\(4.44 \pm 2.01\)), validating its practical clinical trust under cross-institutional scenarios.

Highlights & Insights¶

Explicit Modeling of Radiologist Workflow: The boldest and most elegant aspect of this work is that it does not attempt to let the MLLM "implicitly learn" region-aware capabilities from global features. Instead, it directly mimics the hierarchical clinical workflow of radiologists (global overview \(\rightarrow\) regional detail analysis \(\rightarrow\) quantitative attributes) using three parallel modules. This "human-like workflow modeling" naturally mitigates the learning burden on the MLLM.
Multi-Role Reuse of Pseudo-Masks: A single set of organ/lesion pseudo-masks output by TotalSegmentator is reused three times across the three modules: guiding token selection in the SlowFast Tokenizer, serving as weights for Mask Pooling, and acting as inputs for attribute extraction. A single segmentation footprint drives the entire pipeline, making the design clean and highly efficient.
Deterministic Geometric Computation Over Implicit Learning: Lesion volume and maximum diameter are computed using deterministic algorithms in voxel space instead of being learned by the network, and are then injected as text. This "compute, do not learn" approach guarantees the precision and interpretability of quantitative information, and is completely free from the resolution limits of visual encoders—as long as the segmentation model detects the lesion, calculations are performed at full resolution.
Structuring Reports is a Contribution in Itself: The authors restructured the unformatted RadGenome reports into a standardized six-region format using DSPy + Llama-3.3-70B and released it as a benchmark. This NLP pipeline holds lasting value for subsequent 3D CT report generation research.

Limitations & Future Work¶

The performance ceiling is bounded by the upstream segmentation model: the missed detection of extremely small nodules and the quality of organ masks directly impact the overall outcome. This suggests that future research could focus on joint training or end-to-end optimization of the segmentation model and the report generation model, rather than relying on a two-stage pipeline.
Currently validated only on chest CTs, the organ and lesion types are restricted by the label space of TotalSegmentator. Although the framework is model-agnostic and theoretically generalizable to whole-body CT or MRI by replacing the segmentation backbone, this remains to be validated.
The stride-2 pooling in the Slow path of the SlowFast Tokenizer might still lose ultra-fine-grained textural details in local regions, especially for clinical scenarios requiring high-resolution observation such as interstitial lung disease.
The LLM component only relies on an 8B parameter model. With the emergence of larger medical MLLMs, there remains potential for further performance improvements.

vs. M3D / RadFM / Med3DVLM: These methods employ global volume-level encoding (3D ViT + token compression or slice-level 2D encoding + inter-frame aggregation), treating all spatial locations uniformly with unified pooling/compression. The key difference in this work is the introduction of organ pseudo-masks as spatial guidance, allowing the model to allocate visual capacity selectively to clinically relevant areas, fundamentally abandoning the "treating all voxels equal" paradigm.
vs. Conventional SlowFast (LITA / SlowFast-LLaVA): Conventional SlowFast uniformly samples all slices, with the Fast path performing global average pooling over all slices and the Slow path densely sampling a uniform subset. The proposed variant dynamically decides which slices to keep in the Fast path and which spatial locations to sample in the Slow path based on organ masks, enabling patient-specific rather than fixed-stride tokenization.
vs. MAIRA-SEG / RGRG / Reg2RG: These chest X-ray/CT region-aware approaches also utilize segmentation masks or detectors to extract regional features, but they are primarily designed for 2D or coarse-grained organ-level guidance. This work extends region awareness on 3D CTs into a three-channel fusion (region-aware visual tokens + mask pooling tokens + quantitative attribute text), yielding substantially deeper integration and complexity.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Decomposes the hierarchical radiology workflow into three learnable visual/attribute extraction modules and drives the entire pipeline with a single set of pseudo-masks. The concept is refreshingly elegant and carefully engineered.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Conducts evaluations across both internal and external benchmarks against 5 SOTA baselines. Shows comprehensive ablations, stratified analysis, segmentation noise stress tests, and a blind review by radiologists, leaving no stone unturned in the evaluation.
Writing Quality: ⭐⭐⭐⭐⭐ Highly clear motivation (each design is driven by clinical workflows), detailed description of methods (replete with formulas and pseudo-code), and rich dimensions of experimental analysis.
Value: ⭐⭐⭐⭐⭐ Provides a reproducible and high-quality framework for region-aware modeling in 3D medical MLLMs, while the structured report benchmark itself addresses a gap in evaluation scales for CT report generation.