DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/ArielTYH/DiffVP
Area: Medical Imaging
Keywords: 3D CT Report Generation / Vision-Language Models / Semantic Discrepancy Modeling / Normal Reference Prior / Differential Visual Prompting

TL;DR

Inspired by radiological cognitive subtraction, DiffVP introduces normal chest CT scans as a visual reference prior to model complementary global and local semantic discrepancies in an aligned latent space, which are projected into learnable visual prefix prompts for an LLM to steer report generation without requiring explicit lesion segmentation.

Background & Motivation

Computed tomography (CT) is one of the most indispensable 3D imaging modalities in clinical oncology and pulmonology, playing a central role in screening, differential diagnosis, and treatment planning. However, interpreting 3D chest CT volumes imposes a heavy cognitive burden on radiologists, who must scrutinize hundreds of axial slices slice-by-slice to produce comprehensive medical reports. In response, automated radiology report generation has gained significant momentum. Recent advances have shifted from classical CNN-RNN pipelines to large language model (LLM)-based architectures, where a 3D visual encoder transforms volumetric data into a sequence of visual tokens and passes them into an autoregressive language decoder (e.g., in R2GenGPT and Reg2RG).

Nevertheless, this prevailing uniform token-feeding paradigm deviates fundamentally from the diagnostic reasoning of human radiologists. In clinical reading rooms, radiologists summarize imaging findings by mentally contrasting the patient scan against an internal prior of normal human anatomy—a process termed "cognitive subtraction." Crucially, this comparison is performed at an abstract semantic level rather than at the raw pixel scale: due to non-rigid respiratory deformation, inter-subject morphological variations, and inevitable registration errors, direct voxel subtraction introduces substantial spatial noise. Furthermore, abnormalities in 3D CT scans are inherently sparse, while the vast majority of volumetric tokens are dominated by redundant, invariant anatomical structures such as ribs and musculature. When an LLM is fed undifferentiated visual tokens without any comparative guidance, critical pathological cues are heavily diluted by normal background, causing the model to produce generic, templated narratives that miss subtle, fine-grained diagnostic details such as nodule diameters and subsegmental atelectasis.

To resolve this limitation, this paper dispenses with fragile pixel-level subtraction and tedious manual lesion contouring, proposing to inject paired normal scans as reference priors within a shared, compact latent space to guide LLM attention. The core idea is to introduce normal CT scans as a reference prior, extract complementary multi-scale discrepancies through global Transformer interaction and metric-weighted local residual aggregation in an aligned token space, and project them into continuous visual prefix prompts that implicitly suppress invariant anatomical background while amplifying diagnostically relevant evidence for LLM decoding.

Method

Overall Architecture

The DiffVP pipeline consists of three sequential components: dual-volume latent encoding with shared-query resampling, hierarchical difference extraction, and difference-to-prompt projection for hybrid LLM conditioning. Given an input target CT volume \(X\) and a normal reference volume \(X_{\text{ref}}\) retrieved from an uncorrupted reference pool, both volumes are first encoded by a frozen 3D ResNet-18 backbone into dense volumetric representations. A Q-Former-style visual resampler equipped with a shared set of learnable latent queries compresses these dense features into fixed-budget, structurally aligned latent token sequences \(I\) and \(I^r\). Next, the Hierarchical Difference Extractor (HDE) computes macroscopic structural shifts \(\Delta_{\text{global}}\) via a global Transformer operator and fine-grained localized deviations \(\Delta_{\text{local}}\) via a metric-distance aggregation operator. The Difference-to-Prompt Generator (DPG) then fuses and projects these discrepancies into a continuous soft visual prefix \(\Delta_{\text{prompt}}\). Finally, this prefix is concatenated with the target visual tokens \(I\), frozen diagnostic classification predictions \(E\), and instructional text prompts \(T\), and fed into a LoRA-fine-tuned LLaMA-2-7B model to autoregressively generate the diagnostic report.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Target CT Volume X and Normal Reference Volume X_ref"] --> B["3D ResNet-18 Encoding & Shared Latent Resampling<br/>Produce Aligned Tokens I and I^r"]
    B --> C["Hierarchical Difference Extractor<br/>Global Transformer Operator & Local Distance-Weighted Aggregator"]
    C --> D["Difference-to-Prompt Generator<br/>MLP Fusion & Sequence Expansion to 16-Token Visual Soft Prefix"]
    D --> E["Diagnostic Semantic Anchoring & Hybrid Sequence Construction<br/>Concatenate Difference Prefix, Visual Tokens, Class Predictions & Instruction"]
    E --> F["LLaMA-2 Decoder with LoRA Adaptation<br/>Autoregressive Generation of Structured Clinical Radiology Report"]

Key Designs

1. Hierarchical Difference Extractor: Decoupling Holistic Structural Shifts from Localized Deviations

Pathological manifestations in 3D CT volumes span heterogeneous spatial scales, ranging from extensive mediastinal shifts and lobar consolidation to minute subsegmental nodules. To comprehensively capture this spectrum without suffering from voxel-level misalignment, the Hierarchical Difference Extractor (HDE) operates on resampled latent tokens via two complementary operators: - Global Delta Operator: A learnable global difference query vector \(\delta \in \mathbb{R}^d\) is introduced and concatenated with the target tokens \(I\) and reference tokens \(I^r\). A single Transformer layer models all-to-all cross-attention across the unified sequence, after which the state corresponding to \(\delta\) is extracted: $\(\Delta_{\text{global}} = \mathcal{D}_{\text{diff}}(I, I^r) = \operatorname{Transformer}_\theta([\delta; I; I^r])_\delta\)$ This operator captures holistic morphological reconfigurations and overall semantic discrepancies across the entire scan. - Local Delta Operator: To quantify localized pathological anomalies while preserving topological continuity, the operator functions directly on paired aligned tokens \((I_i, I^r_i)\). It assigns a normalized, scale-invariant importance weight \(w_i\) based on squared Euclidean distance in the latent metric space: $\(w_i = \frac{\|I_i - I^r_i\|_2^2}{\sum_{j=1}^N \|I_j - I^r_j\|_2^2 + \epsilon}\)$ where \(\epsilon = 1 \times 10^{-5}\) avoids zero division. The localized deviation descriptor is computed via residual aggregation: \(\Delta_{\text{local}} = \sum_{i=1}^N w_i (I_i - I^r_i)\). Because this formulation is completely parameter-free, tokens exhibiting substantial pathological deviation naturally dominate the representation, ensuring robustness against minor anatomical shape variations.

2. Difference-to-Prompt Generator: Projecting Latent Discrepancies into Continuous Soft Prefixes

Extracted visual discrepancy vectors cannot be directly ingested by a text-based autoregressive model. The Difference-to-Prompt Generator (DPG) first concatenates the global and local descriptors \([\Delta_{\text{global}}; \Delta_{\text{local}}]\), applies a multi-layer perceptron (MLP) projector, and reshapes the output into a sequence of \(p\) learnable prefix tokens matching the LLM hidden dimension: $\(\Delta_{\text{prompt}} = \operatorname{Projector}([\Delta_{\text{global}}; \Delta_{\text{local}}]) \in \mathbb{R}^{p \times d_{\text{LLM}}}\)$ Empirical validation identifies \(p=16\) as the optimal prompt length. Operating as continuous soft prefix embeddings, \(\Delta_{\text{prompt}}\) prepends the LLM sequence, directly steering the self-attention weights of all downstream layers to prioritize diagnostically distinct visual features over invariant physiological backgrounds.

3. Diagnostic Semantic Guidance & Hybrid Input Assembly: Preserving Anatomical Context with Explicit Priors

Relying solely on discrepancy prompts risks discarding the overall anatomical context of the patient. DiffVP assembles a hybrid conditioning sequence combining four complementary components: \(\mathcal{X}_{\text{in}} = [\Delta_{\text{prompt}}; I; E; T]\): - \(\Delta_{\text{prompt}}\): The 16 learnable differential visual prefix tokens carrying discrepancy cues; - \(I\): The original resampled visual tokens from the target CT, preserving comprehensive spatial and physiological context; - \(E\): Diagnostic semantic anchors generated by a pre-trained, frozen 3D ResNet-18 classifier across 18 report-independent thoracic disease classes (e.g., cardiomegaly, emphysema, nodules), converted into textual tokens to anchor disease identification without label leakage; - \(T\): Structured task instructions and prompting templates that guide the LLM's autoregressive syntax.

Loss & Training

The network is optimized end-to-end via a multi-task learning objective combining conditional report generation with auxiliary diagnostic classification: $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{gen}} + \mathcal{L}_{\text{cls}}\)$ The generation loss \(\mathcal{L}_{\text{gen}}\) computes standard Negative Log-Likelihood (NLL) over ground-truth report tokens \(Y = \{y_1, \dots, y_L\}\): $\(\mathcal{L}_{\text{gen}} = -\sum_{t=1}^L \log P(y_t \mid y_{<t}, \mathcal{X}_{\text{in}}; \theta)\)$ The auxiliary classification loss \(\mathcal{L}_{\text{cls}}\) applies binary cross-entropy (BCE) across the \(K=18\) diagnostic categories to regularize the visual latent space and enforce diagnostic semantic consistency. The 3D ResNet-18 visual encoder and auxiliary classifier remain frozen throughout training; only the resampler, HDE, DPG, and LoRA adapters on LLaMA-2-7B are trained across 10 epochs using a learning rate of \(5 \times 10^{-5}\) on 4 NVIDIA H20 GPUs.

Key Experimental Results

Main Results

DiffVP was comprehensively evaluated on two large-scale 3D chest CT benchmarks: RadGenome-ChestCT (25,692 pairs; official split of 24,128 training and 1,564 test volumes) and CTRG-Chest-548K (1,804 pairs). Baselines include generalist medical vision-language foundation models (RadFM, M3D) and specialized 3D CT report generation systems (R2GenGPT, MedVInT, CT2Rep, Reg2RG). NLG metrics include BLEU-1 to BLEU-4, ROUGE-L, and METEOR. Clinical Efficacy (CE) metrics (Precision, Recall, F1) were extracted using a pre-trained RadBERT classifier over 18 major thoracic conditions on RadGenome-ChestCT.

Dataset	Method	Year	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-L	METEOR
RadGenome-ChestCT	RadFM†	2023	44.20	34.49	28.06	23.65	31.53	39.94
	M3D†	2023	43.57	34.48	28.54	24.49	32.61	39.95
	R2GenGPT	2023	43.28	34.11	28.16	24.16	32.26	39.85
	MedVInT	2024	44.28	34.91	28.75	24.60	32.58	40.39
	CT2Rep	2024	44.42	34.43	27.94	23.56	30.99	40.16
	Reg2RG	2025	47.25	36.49	29.57	24.87	36.65	44.07
	DiffVP (Ours)	2026	58.16	48.74	40.93	34.28	32.32	47.40
CTRG-Chest-548K	RadFM†	2023	48.66	40.28	34.73	30.89	49.08	49.18
	M3D†	2023	46.27	39.02	34.23	30.86	50.24	49.26
	R2GenGPT	2023	41.82	36.37	32.70	30.10	50.93	47.05
	MedVInT	2024	47.38	39.60	34.28	30.68	49.53	49.32
	CT2Rep	2024	42.28	36.16	32.08	29.19	50.17	47.00
	Reg2RG	2025	49.63	41.43	35.91	32.04	47.76	49.71
	DiffVP (Ours)	2026	50.37	46.57	41.94	37.57	45.00	50.44

Clinical Efficacy (CE) Performance on RadGenome-ChestCT:

Method	Model Type	Precision	Recall	F1-Score
RadFM†	Foundation Model	0.382	0.131	0.195
M3D†	Foundation Model	0.407	0.090	0.148
R2GenGPT	Specialized Report Model	0.340	0.066	0.110
MedVInT	Specialized Report Model	0.377	0.148	0.212
CT2Rep	Specialized Report Model	0.317	0.089	0.139
Reg2RG	Specialized Report Model	0.423	0.181	0.253
DiffVP (Ours)	Specialized Report Model	0.366	0.496	0.421

Ablation Study

A systematic component breakdown on RadGenome-ChestCT demonstrates the incremental value of each proposed module:

Config ID	\(\Delta_{\text{global}}\)	\(\Delta_{\text{local}}\)	Diagnostic Anchor \(E\)	BLEU-1	BLEU-4	METEOR	ROUGE-L	Precision	Recall	F1-Score
Baseline	-	-	-	55.29	33.17	44.11	31.94	0.295	0.109	0.159
#1	-	-	✓	55.62	33.44	44.80	32.35	0.311	0.217	0.256
#2	✓	-	✓	57.46	33.20	46.63	32.08	0.373	0.459	0.411
#3	-	✓	✓	57.68	33.89	47.67	32.76	0.360	0.464	0.406
Ours (Full)	✓	✓	✓	58.16	34.28	47.40	32.32	0.366	0.496	0.421

Visual Prefix Length (\(p\)) Sensitivity on RadGenome-ChestCT:

Prefix Length \(p\)	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE-1	ROUGE-2	ROUGE-L
\(p=4\)	53.97	44.29	36.37	29.72	43.37	59.63	38.41	31.46
\(p=8\)	57.98	48.60	40.78	34.13	47.96	63.48	42.60	32.43
\(p=16\) (Default)	58.16	48.74	40.93	34.28	47.40	63.52	42.63	32.32
\(p=24\)	56.89	47.46	39.79	33.32	47.33	62.33	41.49	32.27
\(p=32\)	56.24	46.51	38.43	31.53	46.19	61.82	40.38	31.23

Key Findings

• Substantial Breakthrough in Clinical Recall: DiffVP boosts clinical disease recall to 0.496 on RadGenome-ChestCT, representing a 174% improvement over Reg2RG (0.181). The overall clinical F1-score rises from 0.253 to 0.421, verifying that normal reference comparison prevents the LLM from glossing over subtle or multiple co-occurring pathologies.
• Complementary Synergy of Global and Local Operators: Incorporating only \(\Delta_{\text{global}}\) (#2) elevates precision to 0.373 and reinforces holistic anatomical consistency, whereas adding only \(\Delta_{\text{local}}\) (#3) maximizes localized lexical metrics (METEOR 47.67, ROUGE-L 32.76). Integrating both yields the highest long-range n-gram match (BLEU-4 of 34.28) and the optimal diagnostic F1 balance.
• Superiority of Semantic-Level Discrepancy over Pixel Subtraction: Direct pixel-level image subtraction (Pixel-level Diff) degrades performance due to non-rigid thoracic deformation and registration noise. Computing differences in the downsampled latent token space provides natural invariance to physiological misalignment while isolating pathological changes.
• Induction of Attention Sparsity: Analysis of geometric token discrepancy \(S_i = \|I_i - I^r_i\|_2^2\) reveals that baseline models exhibit flat, near-uniform token importance across the volume. Injecting \(\Delta_{\text{prompt}}\) induces sharp attention sparsity, where the top 8 latent tokens capture 45% of the total importance mass, concentrating decoding capacity on active lesion loci.
• LLM-as-a-Judge Clinical Dominance: Pairwise blind evaluation conducted by GPT-5 across five clinical axes reveals overwhelming preferences for DiffVP in descriptive granularity (74.7% win-rate), anatomical localization (74.0%), and quantitative measurement descriptions (73.3%), corroborating qualitative findings like exact millimeter nodule measurements (e.g., 4.6 mm).

Highlights & Insights

• Radiological Cognitive Subtraction Reified as Continuous Soft Prompts: Seamlessly instantiates the radiologist's mental workflow—contrasting against expected normal anatomy—as a learnable prefix prompt without introducing complex multi-stage anatomical segmentation pipelines.
• Parameter-Free Latent Metric Aggregator: The local difference operator relies purely on normalized squared Euclidean distance to dynamically weight token deviations, requiring zero trainable parameters while naturally ensuring that large pathological variations govern the differential vector.
• High Portability to Other Volumetric Modalities: The normal-reference sampling and latent differential prompting paradigm can be readily transferred to other 3D medical tasks, such as brain MRI tumor progression monitoring, stroke lesion segmentation, and longitudinal follow-up CT analysis.

Limitations & Future Work

• Reliance on Explicit Normal Reference Cohorts: The method currently requires building a dedicated pool of verified normal scans from existing training sets. In specialized subfields or low-resource pediatric imaging where guaranteed-normal volumes are sparse, constructing such a bank remains challenging.
• Lack of Conditioned Patient Matching: Normal references are currently sampled uniformly at random from the pool, without conditioning on patient demographic factors such as age, sex, body mass index, or contrast-enhancement phase, which could occasionally introduce physiological morphology mismatches.
• Future Directions: Exploring unconditional generative models to synthesize patient-specific pseudo-normal reference volumes, or incorporating adaptive semantic retrieval to dynamically fetch the most morphologically matched normal reference from the database.

Related Work & Insights

• vs Reg2RG (TMI 2025): Reg2RG enforces region-guided grounding using complex pre-trained anatomical segmentors, which fail when abnormalities occur outside pre-defined organ boundaries. DiffVP requires no segmentation masks, yet outperforms Reg2RG by +9.41 in BLEU-4 (34.28 vs 24.87) and nearly triples clinical recall (0.496 vs 0.181).
• vs DiffRGenNet (MIDL 2025) & Prior Difference Methods: Previous difference-aware techniques were restricted to 2D X-rays and relied on naive feature vector subtractions or isolated patches, disrupting cross-slice continuity. DiffVP models hierarchical multi-scale discrepancies across the entire 3D voxel sequence and converts them into structured prompt prefixes for an LLM rather than passive auxiliary features.

Rating

• Novelty: ⭐⭐⭐⭐⭐ (Pioneering formulation translating radiological cognitive subtraction into continuous visual prefix prompts for 3D CT LLMs)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Rigorous evaluation across two large datasets, covering NLG metrics, RadBERT clinical efficacy, parameter-free distance distributions, and GPT-5 clinical judge)
• Writing Quality: ⭐⭐⭐⭐⭐ (Crystal-clear motivation, mathematically elegant formulation, and highly convincing visualizations)
• Value: ⭐⭐⭐⭐⭐ (Substantially resolves the long-standing diagnostic sensitivity and detail-omission bottlenecks in automated 3D volumetric report generation)