Comprehensive language–image pre-training for 3D medical image understanding

Conference: ECCV2026
Paper: Official paper page
PDF: Full paper
Code: https://huggingface.co/microsoft/colipri
Area: Medical Imaging
Keywords: Vision-language pretraining, opposite sentence loss, report generation, masked autoencoders, 3D CT

TL;DR

COLIPRI combines chest CT image–text alignment, affirmative/negative short-statement discrimination, report generation, and multiresolution masked reconstruction so that one visual encoder captures both global semantics and local structure; the full model reaches 73.23% zero-shot AUROC on RAD-ChestCT, although not every task requires the report-generation branch.

Background & Motivation

CT scans usually come with radiology reports, making CLIP-style image–text contrastive learning an attractive approach: aligning examinations with their reports enables natural-language abnormality queries and similar-case retrieval. However, publicly available paired data remain scarce, and volumetric inputs require far more GPU memory than 2D images. Small crops may exclude an organ or lesion described in the report, while aggressively downsampling the whole volume can erase subtle abnormalities. Both choices can weaken the correspondence between the image and its supervision.

There is also an easily overlooked mismatch on the text side. Training reports contain multiple organs, normal observations, and abnormal findings, whereas zero-shot evaluation often asks only whether a particular abnormality is present. The CT-RATE Findings section averages 243 tokens under the CXR-BERT tokenizer. Learning to match an entire report does not necessarily teach an encoder to distinguish short affirmative and negative statements. Meanwhile, global image–text matching alone does not ensure that visual tokens preserve the spatial details needed for segmentation.

Rather than treating these limitations as a problem of model scale alone, the paper addresses the form, density, and sources of supervision. Core idea: use opposite sentence loss to train the abnormality-presence decisions required at inference, report generation to extract denser supervision from existing pairs, and vision-only masked reconstruction to incorporate unpaired CT scans while preserving local structure.

Method

Overall Architecture

COLIPRI is an encoder-pretraining recipe, not a complete diagnostic system that directly delivers clinical conclusions. Its inputs include CT–report pairs from CT-RATE and unpaired images from sources such as NLST. It produces global embeddings for image–text matching and dense visual tokens for downstream tasks. Primus-M is the image encoder, and CXR-BERT is the text encoder.

Paired data support radiology image–text alignment, supplemented by opposite sentence loss and parallel report generation. Unpaired images train the same visual encoder through multiresolution masked reconstruction. The diagram shows how training signals converge, rather than a sequence of modules executed at inference. Masked reconstruction also initializes the visual encoder, so its branch should not be interpreted as something enabled only at the end of training.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    I["CT and paired reports"] --> A["Radiology image–text alignment"]
    A --> B["Opposite sentence loss"]
    A --> C["Parallel report generation"]
    U["Paired or unpaired CT"] --> D["Multiresolution masked reconstruction"]
    A --> E["Shared visual encoder"]
    B --> E
    C --> E
    D --> E
    E --> F["Global embeddings and<br/>dense visual tokens"]

Key Designs

1. Radiology image–text alignment: match the report to a sufficiently broad 3D field of view

The basic image–text branch receives isotropic CT inputs of size \(160\times160\times160\) with 2 mm voxel spacing. A broad physical field of view helps include the multiple organs described in a report, while limiting voxel count keeps the 3D token sequence manageable. Image and text tokens undergo separate multihead attention pooling to obtain global representations. The image representation is then projected into the lower-dimensional text space, and a CLIP contrastive objective aligns examinations with reports. Standard CLIP loss outperformed sigmoid loss in the authors' setting; this is not presented here as a universal result.

Report preprocessing provides a common foundation for the two additional language-supervision branches. A large language model organizes each report into eight clinical subsections, assigns sentences to subsections, produces concise versions, and labels findings as abnormality present or normal. For image–text alignment, sentence order is randomly shuffled to reduce dependence on writing conventions, and concise statements replace the original wording with a specified probability to soften the long-report/short-prompt mismatch. These augmentations discourage stylistic shortcuts in pairing, but do not directly teach the distinction between an affirmative statement and its negation.

2. Opposite sentence loss: train the two-way decision needed at inference

Opposite Sentence Loss (OSL) does more than bring an image close to one description: it allocates probability between two opposing descriptions. For an abnormal finding actually stated in the current report, the method constructs a negated version and labels the pair as abnormality present. Another sample type comes from clinical subsections with no positive findings in the current report. An abnormal statement is sampled from the same subsection of another report, treated as a description that should not match the current image, and paired with its negation. The model therefore sees both real abnormalities and subsection-conditioned examples of absent abnormalities.

Let \(v\) denote the image embedding, \(e^+\) and \(e^-\) the affirmative and negative statement embeddings, and \(y\) whether the affirmative statement applies to the image. The source's two-class softmax and cross-entropy can be written for one sample as:

\[ p^+=\frac{\exp(\operatorname{sim}(v,e^+)/\tau)}{\exp(\operatorname{sim}(v,e^+)/\tau)+\exp(\operatorname{sim}(v,e^-)/\tau)},\qquad \ell_{\mathrm{OSL}}=-y\log p^+-(1-y)\log(1-p^+). \]

Here, \(\operatorname{sim}\) is cosine similarity and \(\tau\) is the temperature; training aggregates the loss over samples. The important detail is not merely adding an arbitrary negative example, but making the two text candidates differ in abnormality presence so that the encoder must learn this distinction. If a report explicitly describes a lung nodule, the image should favor the affirmative statement over its negation. If a subsection contains no positive finding, a borrowed abnormal statement instead acts as a negative example. This illustrates sampling rather than adding an experimental result; its reliability still depends on abnormalities not being omitted from reports or subsection labels.

3. Parallel report generation: recover clinical content rather than just pairing cues

Examination–report contrastive learning only needs to identify the correct pairing. A few salient abnormalities may suffice, leaving other clinical information unrepresented. The paper therefore trains a lightweight EVA-02 Transformer decoder to recover reports by attending to dense visual tokens through cross-attention. The authors compare autoregressive and parallel generation, ultimately selecting the latter: the target body tokens are masked and predicted simultaneously, rather than each word depending on previously generated text. This makes it harder for the decoder to solve the task from language context alone and encourages greater use of the image.

Parallel prediction is ambiguous when the same examination permits multiple narrative orders. The method therefore uses reports organized into eight clinical subsections, retaining unmasked subsection headers to constrain where content belongs. This does not contradict sentence shuffling in the alignment branch: alignment should avoid ordering shortcuts, whereas generation requires a stable output structure. The pretraining decoder is also distinct from the downstream reporting system. In the downstream experiment, the visual encoder is frozen, a two-layer MLP connects it to Qwen2.5, and the language model is trained to generate the Findings section.

4. Multiresolution masked reconstruction: use unpaired images to supply spatial supervision

Image–text-only training is limited by paired-data availability and can subordinate local textures and anatomical boundaries to global semantics. A masked autoencoder (MAE) instead masks volumetric patches and uses a lightweight decoder to reconstruct missing voxels from context with a mean-squared-error objective. Because reports are unnecessary, datasets such as NLST can be included. The authors first pretrain Primus-M with MAE using nnSSL and then perform joint multimodal training, rather than learning every objective from random initialization.

The MAE branch samples subvolumes at 1 mm and 2 mm isotropic resolutions. Local voxel reconstruction does not require the global field of view needed for report matching, allowing finer spatial details within a manageable input size and reducing the gap to high-resolution segmentation. During joint training, masked vision-only batches alternate with image–text batches to update the shared visual encoder. This avoids matching partially masked images directly against reports describing the whole examination. The resulting gains involve additional data, initialization, and local supervision together; comparisons between final models cannot attribute all improvements to just one factor.

Loss & Training

Model suffixes reflect the objectives added to the recipe: C includes CLIP and OSL; CR adds report generation; CM combines C with MAE; and CRM includes all objectives. Paired image–text batches optimize:

\[ \mathcal{L}_{\mathrm{VLM}}=0.5\mathcal{L}_{\mathrm{CLIP}}+0.5\mathcal{L}_{\mathrm{OSL}}+\lambda_{\mathrm{RRG}}\mathcal{L}_{\mathrm{RRG}}. \]

The last term is omitted when report generation is not used; vision-only batches optimize a weighted MAE reconstruction loss. CT-RATE supplies both batch types, while NLST supplies vision-only batches. This is alternating supervision across batches, not simultaneous masked reconstruction and complete image–text alignment for every CT scan.

The local main-paper cache does not contain the cited supplementary material, so unverified masking ratios, report-generation loss weights, and pretraining step counts are not supplied. Parallel report recovery is the pretraining objective; downstream autoregressive generation and segmentation fine-tuning are separate ways of evaluating the learned encoder.

Key Experimental Results

Main Results

All values below are macro-averaged percentages; higher AUPRC and AUROC are better. Following Table 2, zero-shot evaluation excludes Medical Material and Lymphadenopathy to match fVLM's 16-class protocol. The fVLM numbers are cited results, and its inference requires segmentation masks. Probing selects the pooling method and learning-rate configuration with the best validation performance on frozen features. Some probes use attention pooling, so the evaluation should not be described uniformly as strictly linear probing.

Task / Dataset	Metric	COLIPRI-CRM	Baseline	Baseline Value	Gain (percentage points)
Zero-shot / CT-RATE	AUROC ↑	79.81	fVLM	77.8	+2.01
Zero-shot / RAD-ChestCT	AUROC ↑	73.23	fVLM	68.0	+5.23
Classification probing / CT-RATE	AUPRC ↑	61.28	Merlin	54.81	+6.47
Classification probing / RAD-ChestCT	AUPRC ↑	52.55	Merlin	45.30	+7.25

CRM is not the best variant in every column. On CT-RATE, CM achieves 80.52 zero-shot AUROC versus CRM's 79.81, whereas CRM performs better on RAD-ChestCT. For report-to-image retrieval over 1493 test samples after report deduplication, CRM achieves 35.10% Recall@5 and CM achieves 34.16%. The table's CT-CLIP value of 2.90% comes from its original paper, so protocol differences remain relevant to cross-paper comparisons.

Report generation should not be assessed using BLEU or ROUGE alone. The main text summarizes CRM's RadBERT MacroF1 as approximately 45% and its positive-finding-only RadFact-CT (+) F1 as approximately 22%. These are the approximate values stated in the text, not exact values inferred from the unavailable Table S15. RadBERT uses a text classifier to evaluate abnormality-label agreement; RadFact-CT (+) uses model judgments of the logical correctness of positive-finding statements and summarizes logical precision and recall. Neither metric establishes clinical readiness.

Ablation Study

The OSL ablation comes from validation-set Table 3 and should not be mixed with the test results above. Native prompts average the embeddings of 50 randomly sampled reports with each abnormality and 50 without it; Short prompts use brief presence/absence statements.

Config	Prompt Style	AUPRC ↑ (%)	AUROC ↑ (%)
COLIPRI-CRM	Native	51.17	81.91
Without OSL	Native	47.48	80.48
COLIPRI-CRM	Short	50.70	81.66
Without OSL	Short	39.08	74.09

Removing OSL reduces short-prompt AUROC by 7.57 percentage points, compared with only 1.43 points for native prompts. This supports the explanation that OSL primarily addresses the long-report/short-prompt mismatch more directly than an overall leaderboard gain would.

The segmentation comparison holds the Primus-M architecture and 37.5k-step fine-tuning schedule fixed. The metric is foreground-class mean Dice similarity coefficient (DSC, higher is better) under five-fold cross-validation, rather than a collection of best results from different training budgets.

Dataset	From Scratch	Pure MAE	COLIPRI-C	COLIPRI-CRM
LiTS	74.39	80.27	76.77	80.46
Lung	62.74	67.12	65.89	68.98
HVS	64.67	67.10	64.64	67.05
KiTS23	78.90	85.34	81.02	85.79

Key Findings

• OSL provides the clearest targeted evidence: short-prompt AUROC approaches native-prompt AUROC, rather than improving only through longer queries.
• MAE matters for both local and global tasks, but CRM does not lead every segmentation comparison. Its short-schedule HVS result is slightly below pure MAE, and C is even slightly below training from scratch.
• RRG behaves more like auxiliary supervision, with clearer retrieval benefits. The authors acknowledge that CM is a similarly performing, simpler option when retrieval is not a priority.
• Normal statements affect report evaluation. RadFact-CT (+/−), which includes both normal and abnormal findings, can be dominated by statements that no abnormality is present and should not be interpreted as proof of better abnormality detection.

Highlights & Insights

• Align the inference question, not just text length. Shortening reports changes their form, while OSL explicitly trains abnormality-presence decisions, enabling a targeted test of its benefit for short prompts.
• Preserve two levels of representation in one encoder. Global image–text supervision supplies clinical semantics, while local reconstruction preserves voxel structure. The branches use different field-of-view and masking rules rather than sharing unsuitable preprocessing.
• Separate representation pretraining from downstream generation. Parallel report recovery shapes visual features but is not the final autoregressive reporting system, helping explain why RRG does not necessarily produce the largest downstream reporting gain.

Limitations & Future Work

• The authors explicitly state that report-generation clinical metrics remain insufficient for clinical use. Larger and more diverse data, along with evaluation stratified by findings and patient characteristics, are still needed.
• OSL negatives depend on the report-level judgment that a subsection contains no positive finding. Unrecorded abnormalities or preprocessing omissions may introduce noisy supervision. This is a potential limitation of the sampling mechanism, not a failure rate quantified by the paper.
• MAE is introduced together with additional unpaired data, so the tabulated gains are not a single-objective causal effect under strictly matched data volumes. The cache lacks supplementary material, leaving detailed training settings to be checked in the original attachments.
• Segmentation claims should remain bounded by the comparisons. Longer fine-tuning is not consistently better, and Primus-M does not comprehensively surpass the strong ResEnc-L CNN reference. Better general representations do not imply the strongest results on every segmentation task.

Related Work & Insights

• vs CT-CLIP / Merlin: These models establish foundations for 3D medical image–text representation. COLIPRI primarily adds short-statement presence supervision, report-content recovery, and vision-only reconstruction rather than merely substituting an encoder name.
• vs fVLM: fVLM emphasizes fine-grained anatomical information, and the table's protocol excludes two nonlocalizable abnormality categories. COLIPRI matches short prompts directly to global embeddings without segmentation masks at inference, but cited results must still be interpreted within their respective protocols.
• vs MAE / nnSSL: Pure MAE is a strong 3D segmentation-pretraining baseline but lacks image–text alignment. COLIPRI adds a language interface while seeking to preserve its spatial representation quality. The main benefit is task coverage, not a substantial improvement over MAE in every segmentation cell.

Rating

• Novelty: 4/5 — OSL specifically targets the mismatch between long medical reports and short prompts; the other contributions largely integrate existing objectives effectively.
• Experimental Thoroughness: 4/5 — Classification, retrieval, generation, and segmentation are covered, but baseline protocols and data-scale effects require careful interpretation.
• Writing Quality: 4/5 — The model family and objective relationships are clear, although some broad leadership claims extend beyond the tables' support.
• Value: 4/5 — A reusable training recipe for 3D medical image–text encoders, not a clinically validated diagnostic tool.