Generative Vision-Language Multiple Instance Learning for Weakly Supervised Neonatal Fundus Screening and Reporting

Conference: CVPR 2026
Paper: CVF Open Access
Code: https://github.com/CVIU-CSU/GVL-MIL
Area: Medical Imaging
Keywords: neonatal fundus screening, multiple instance learning, generative vision-language models, clinical descriptions, entropy-guided fusion

TL;DR

GVL-MIL adapts a generative vision-language model with a small set of expert image descriptions, separately aggregates intermediate visual evidence and generated-text semantics, and fuses their predictions by entropy, achieving 92.14 ± 0.72% patient-level test AUC on NFSD while producing per-image clinical descriptions.

Background & Motivation

Neonatal fundus screening rarely reduces to examining a single photograph: a patient has multiple views, lesions may appear only in some peripheral retinal images, and other images may be normal or low quality. Consequently, patient diagnoses are easier to obtain than lesion labels for every image. Multiple instance learning (MIL) groups a patient's images into a bag and learns screening from bag labels, matching this annotation setting. However, a conventional visual encoder may lose subtle hemorrhages or demarcation lines, and a bag-level category does not explain what was observed.

Vision-language approaches such as ViLa-MIL introduce semantics through category descriptions, generally to support contrastive classification rather than to describe findings in the current image. Relying directly on generated reports is also risky: a normal view can belong to a positive patient, and a generator can omit lesions or misstate their locations. The goal here is not simply to attach text output to MIL. It is to improve instance representations with limited expert descriptions, then exploit complementary textual and visual information for patient-level decisions without allowing unreliable text to corrupt visual features.

Core Idea: first adapt a generative VLM using per-image findings and diagnostic proxy labels, then extract visual tokens and textual semantics from different model locations, preserve their evidence in independent MIL branches, and fuse patient-level predictions according to predictive uncertainty.

Method

Overall Architecture

The input is a set of fundus images from one neonate. There are two types of output: a clinical description for each image and a four-class prediction for the patient bag, covering NEG, retinal hemorrhage RH, retinopathy of prematurity ROP, and white spots WS. The image-level generator also emits a diagnostic term, but MIL produces the final bag prediction; these classification levels must not be conflated.

Training has two stages. Offline preparation performs diagnosis-guided description generation on the expert-described subset, selects a layer by matrix entropy, and freezes the adapted VLM. This model uses a fixed Layer 23 rather than selecting a layer anew for each patient. Stage 2 trains the attention-based feature aggregator and dual-branch MIL; inference reuses the trained modules and fixed feature paths. The visual stream directly reads Layer 23 image tokens, while the text stream uses the full generator to autoregressively produce descriptions and diagnostic terms, then reads the end-token hidden state. Each branch first produces bag-level logits, followed by predictive-entropy fusion.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
  subgraph Prep["Offline: Stage 1 Adaptation and Layer Selection"]
    Supervision["Expert descriptions<br/>and proxy diagnoses"] -.->|Stage 1 supervision| Gen["Diagnosis-Guided<br/>Description Generation"]
    Gen --> Select["Matrix-Entropy Layer Selection<br/>Fixed Layer 23"]
  end
  Select -->|Configure model and layer| Frozen
  subgraph Runtime["Fixed Flow: Stage 2 Training / Inference"]
    Images["Multiple views of one patient"] --> Frozen["Frozen adapted VLM"]
    Frozen -->|Layer 23 image tokens| Aggregate["Attention-Based<br/>Feature Aggregation"]
    Frozen --> Text["Full autoregressive generation<br/>Descriptions and diagnostic terms"]
    Aggregate -->|Per-image visual vectors| Dual["Dual-Branch MIL and<br/>Predictive-Entropy Fusion"]
    Text -->|End-token hidden states| Dual
    Text --> Reports["Per-image clinical descriptions"]
    Labels["Patient bag labels"] -.->|Stage 2 supervision only| Aggregate
    Labels -.->|Stage 2 supervision only| Dual
    Dual --> Result["Patient-level four-class prediction"]
  end

Dashed edges denote training supervision. The solid edge between subgraphs delivers the fixed model configuration; solid edges inside them denote preparation steps or inference data flow. In Stage 2, bag-level losses update the aggregator through the visual branch and train both MIL branches, without back-propagating into the frozen VLM. Inference removes all supervision inputs. The visual stream need not wait for text generation, but the full dual-branch version still needs the text stream's end token; only the image-only version can omit autoregressive generation.

Key Designs

1. Diagnosis-Guided Description Generation: learn clinical findings as instance representations rather than copying bag labels

The generator is LLaVA-OneVision-7B: a SigLip vision encoder extracts image tokens, a two-layer MLP projects them into the embedding space of Qwen2-7B-Instruct, and the LLM generates Chinese fundus descriptions. The supplement specifies visual and language hidden dimensions of 1152 and 3584. Adaptation updates only the projector and LoRA parameters within the LLM, keeping the vision encoder and original LLM backbone frozen; it does not train a specialized fundus foundation model from scratch.

The supervision target is central. Annotated images become single-turn conversations whose input only asks about possible fundus abnormalities. The target answer places the expert description inside <think>, followed by a diagnosis inside <answer>. Images with visible pathological findings inherit the bag diagnosis as a proxy label, whereas images without obvious abnormalities receive a negative label. Thus, the model is not forced to describe every view from a positive patient as diseased; the supplement indeed contains many normal images within positive patient categories. Here, <think> is a formatting container for expert descriptions, not a separately collected reasoning trace.

The diagnostic term belongs to the training response, not to ground truth leaked into the test input. Through generative training, it encourages semantic consistency between the description and diagnosis. Training images without descriptions do not thereby acquire new expert reports; they still participate in Stage 2 MIL through bag labels. Weak supervision therefore refers to the patient-screening setting, not the complete absence of instance-level human supervision.

2. Matrix-Entropy Layer Selection: find intermediate representations that retain rich visual structure

Visual tokens evolve as they pass through the language model, whose final layer also serves vocabulary prediction and need not be optimal for distinguishing retinal lesions. The method extracts only hidden states at image-token positions in each layer, forming an image-token-count by hidden-dimension matrix. It computes the Gram matrix and uses the von Neumann entropy of its normalized eigenvalues to measure spectral dispersion. Concentration in a few directions produces a more concentrated spectrum; a dispersed spectrum indicates richer representational directions, but does not inherently guarantee better diagnosis.

Let \(G^{(l)}\) denote the Gram matrix of image tokens at layer \(l\). Its normalized spectral entropy is:

\[ q_k^{(l)}=\frac{\lambda_k(G^{(l)})}{\operatorname{tr}(G^{(l)})},\qquad S(G^{(l)})=-\sum_k q_k^{(l)}\log q_k^{(l)}. \]

The authors compare average matrix entropy across candidate layers, exclude the final layer, and select the non-final entropy peak. For the fine-tuned 7B model, this is Layer 23; the paper indexes projector output as -1 and decoder layers as 0 through 27. LoRA weights are then merged and the model is frozen, with the vision encoder, projector, and early LLM layers supplying visual features. Layer selection configures the extractor rather than choosing a diagnostic branch anew for each patient; text generation still proceeds through later layers.

This entropy measures spectral diversity in token representations, not the predictive entropy of the four-class probabilities used later. Layer selection favors higher matrix entropy, whereas fusion gives more weight to lower predictive entropy. The directions differ because the measured objects and purposes differ. The layer ablation evaluates this heuristic on NFSD.

3. Attention-Based Feature Aggregation: collect evidence within an image before combining patient views

The selected layer still outputs many regional tokens, which cannot directly serve as one instance vector in a patient bag. The aggregator applies layer normalization and key/value projections, then uses a learnable query to read the regions through multi-head cross-attention. A linear mapping and normalization produce a compact per-image visual representation. This lets the model weight diagnostically relevant regions rather than uniformly averaging background and lesions.

Two aggregation levels must be distinguished: this module compresses regional tokens within one image, whereas V-MIL later aggregates different images from the same patient. The former asks which regions to retain within an image; the latter asks which views better support the patient diagnosis. This explains why the model can display regional attention maps as well as assign different weights to different views. The aggregator is trained through the Stage 2 screening objective using bag labels rather than pixel-level lesion masks.

4. Dual-Branch MIL and Predictive-Entropy Fusion: preserve visual evidence while limiting textual interference

V-MIL receives each image's visual vector; T-MIL receives the hidden state at the end of each generated sequence. Because the final token follows the entire generated content, the authors use it as a proxy for overall description semantics rather than invoking a separate text encoder. Both branches use pooling and classification structures resembling DSMIL and independently produce patient-level logits. Text hidden states come from the generative VLM, not from a new clinical measurement independent of the image, so the branches are not statistically independent evidence sources.

Fusion operates on logits rather than features. Each branch's bag logits are softmax-normalized to compute Shannon entropy over the four classes; applying softmax to the two negative entropies gives fusion weights:

\[ p_m=\operatorname{softmax}(z_m),\quad H_m=-\sum_c p_{m,c}\log p_{m,c},\quad (w_V,w_T)=\operatorname{softmax}(-H_V,-H_T),\quad z=w_Vz_V+w_Tz_T. \]

Here \(m\in\{V,T\}\), and the final operation combines \(z_V,z_T\), rather than directly averaging probability vectors. A flatter text-branch distribution for a patient receives less weight, allowing descriptive semantics to contribute without mixing textual noise into the visual feature space. The weights measure model confidence.

Both branches also use a consistency calibration loss, CCL. Within each bag, the method selects the top-k images by maximum-class instance confidence and penalizes KL divergence in both directions between selected instance distributions and the bag distribution. It does not select known true-positive images or require every image in the bag to have the same class. Defaults are \(k=2\) and loss weight \(\lambda=0.05\). This lets confident instances and the overall decision constrain each other during training.

A Worked Example

Suppose a patient bag contains 4 images: a nasal peripheral view shows a demarcation line, 2 other views show no obvious abnormality, and the fourth is too blurred to assess confidently. This is a teaching scenario, not a case or measured result from the paper. Classifying each image and taking a majority vote could overwhelm the only clearly pathological view with normal views. Yet an image not capturing a lesion is different from a patient not having one.

GVL-MIL first aggregates regional tokens within the nasal image, retaining evidence near the demarcation line in its visual vector; V-MIL then combines the 4 views. MIL learns to use instances according to diagnostic relevance, allowing a few informative views to dominate the bag decision rather than assigning one vote per image. This provides a way to learn patient-level ROP from lesions visible in only a few images, without guaranteeing that the right view is always selected.

The text stream generates descriptions for individual images, and T-MIL aggregates end-token semantics. If the pathological image's description omits the demarcation line and the text branch's bag prediction is flatter, predictive-entropy fusion reduces its weight; the visual stream retains evidence extracted directly from the images. The outputs are one patient diagnosis and 4 per-image descriptions. Normal-view descriptions need not be rewritten as ROP, and per-image reports are not automatically combined into a complete patient report.

Loss & Training

Stage 1 uses supervised generative instruction tuning for 3 epochs, with LoRA rank 32, alpha 128, and learning rate \(10^{-4}\). Supplementary Table 9 additionally specifies batch size 1, gradient accumulation 8, bfloat16, AnyRes, grids from 1 × 1 to 3 × 3, warmup 0.03, and weight decay 0. The mapping between stages and settings is discussed under reproducibility boundaries below.

Stage 2 freezes the entire adapted VLM, resizes input images to 384 × 384, uses embedding dimension 512 for the aggregator and MIL, and trains at learning rate \(2\times10^{-5}\). Each modality branch uses classification loss plus CCL:

\[ \mathcal{L}_m=\mathcal{L}_{\mathrm{cls}}+0.05\mathcal{L}_{\mathrm{ccl}}. \]

CCL first computes symmetric KL between each selected instance and the bag distribution, then averages over selected instances. Classification uses weighted cross-entropy with weights [0.48, 0.59, 0.90, 1.00] for NEG, RH, ROP, and WS. Each epoch randomly retains 25% of negative samples and all other categories. The supplement specifies effective MIL batch size 64, dropout 0.25, and early-stopping patience 10; Table 10 lists 20 epochs for GVL-MIL.

Key Experimental Results

Main Results

NFSD is reported to contain 9,260 neonates and 121,801 images acquired with RetCam3 during 2015-2019 at an original resolution of 1600 × 1200. Expert descriptions cover 5,180 images from 526 neonates. Main Table 2 reports training/validation/test patient counts of 5,558/1,851/1,851 and image counts of 73,222/24,295/24,284.

Keep the protocols separate: main results train on training plus validation data and evaluate on the test set; ablations train only on the training set and evaluate on validation data. The following excerpts from main Table 3 are patient-bag-level, four-class macro-averaged metrics in %, reported as mean ± standard deviation over 5 fixed-seed runs. They are not per-image classification or report-generation scores.

Method	AUC	Sensitivity	Specificity	F1	ACC	Precision
DSMIL	89.33 ± 0.46	71.16 ± 1.20	90.88 ± 0.52	65.03 ± 1.99	89.31 ± 1.62	62.14 ± 2.97
LD2GMIL	89.73 ± 0.56	69.72 ± 0.93	90.27 ± 0.63	65.10 ± 2.35	89.95 ± 1.50	62.78 ± 3.40
ViLa-MIL	89.69 ± 0.48	70.15 ± 1.07	91.18 ± 0.26	67.27 ± 0.96	91.17 ± 0.63	65.74 ± 1.94
GVL-MIL, image-only branch	91.60 ± 0.25	74.92 ± 0.22	92.51 ± 0.08	70.22 ± 0.19	91.83 ± 0.06	66.97 ± 0.36
GVL-MIL, full dual branch	92.14 ± 0.72	76.26 ± 1.50	93.11 ± 0.64	72.03 ± 3.55	92.46 ± 1.49	69.16 ± 5.16

The image-only branch is already strong; dual-branch gains should be read as differences in means. In Table 3, the highest non-GVL-MIL AUC is LD2GMIL's 89.73, which the full model exceeds by 2.41 percentage points. All six image-only results are marked p > 0.05 relative to the full model, showing no significant differences. The narrative names ViLa-MIL's 89.69 as the strongest baseline AUC, contrary to the table; this note follows the table. ACC retains the paper's macro-averaged definition.

The overall dataset scale follows the authors' report; conflicting component counts are collected under "Reading & Reproducibility Boundaries."

Ablation Study

Main Table 6 tests the fusion location. All entries below are validation-set patient-level results; its 92.53 AUC and the test table's 92.14 must not be treated as a controlled difference.

Config	AUC (%)	F1 (%)	ACC (%)	Note
image	91.67 ± 0.61	70.79 ± 0.54	92.90 ± 0.66	Visual-token features only
eot	88.63 ± 0.83	59.67 ± 2.42	86.35 ± 1.82	Generated end-token features only
feature	90.77 ± 0.04	69.20 ± 0.44	91.59 ± 0.02	Feature-level fusion
logits	92.53 ± 0.21	73.73 ± 3.34	93.05 ± 1.65	Dual-branch logit fusion

Main Table 4 instead evaluates per-image description generation as a validation-set ablation. With diagnostic supervision, only the description inside <think> is extracted for scoring, excluding the diagnostic answer. Scores retain the original scale rather than being converted to percentages; no unreported variance or BLEU order is added.

Stage 1 supervision target	BLEU	ROUGE-L	METEOR	CIDEr
Description only	0.7121	0.7794	0.7789	0.6323
Description + diagnosis	0.7201	0.7914	0.7895	0.6365

Key Findings

• Feature fusion is not automatically beneficial: relative to image, feature reduces validation AUC by 0.90 percentage points, whereas logits improves it by 0.86 percentage points. This supports preserving modality branches in this setting, not the claim that adding text always helps.
• In Table 5's same-seed layer ablation, Layer 23 achieves AUC/F1 of 93.84/74.08, versus 92.45/66.41 for the final layer. However, Layer 26 has F1 74.50, and Layer 15 has sensitivity 77.45, above Layer 23's 75.98. The entropy-peak layer is not optimal on every metric.
• Table 7's CCL ablation has its own baseline: without CCL, AUC/F1/ACC are 90.23 ± 0.78/63.77 ± 4.78/87.94 ± 3.95; with \(k=2\), they are 89.88 ± 0.54/65.44 ± 2.06/89.47 ± 1.12. Mean F1 and ACC improve, while mean AUC decreases slightly. These results must not be presented as Table 3's full-model results.
• The supplement observes greater visual-branch weights for ROP and WS; the authors hypothesize that subtle lesions and more normal views may reduce description quality.

Highlights & Insights

• The image-only branch's 91.60 ± 0.25% test AUC is especially informative: language can participate in representation learning during offline adaptation without always serving as a decision input at inference time. Table 3 supports strong performance after removing the text stream, but does not separately isolate the gains from language fine-tuning, intermediate-layer selection, and the aggregator; the entire improvement cannot be attributed to language fine-tuning.
• Separating which intermediate layer to read, how to compress within-image tokens, and how to combine patient views makes the level of each improvement explicit. This is more reusable than describing everything as multimodal attention.
• For noisy generated text, forming separate predictions before fusion works better than mixing features directly. Other clinical imaging applications should revalidate this finding and test whether confidently wrong predictions are amplified.

Limitations & Future Work

Study Limitations

• The authors acknowledge reliance on expert-described images, so annotation costs remain. Although baselines share SigLip, they do not thereby receive the same generative backbone and per-image descriptive supervision; the comparison cannot fully isolate individual architectural gains.
• Supplementary Table 11 contains concrete inaccuracies: the RH description changes an inferotemporal location to nasal and adds yellow-white deposits; the ROP description changes nasal to temporal and omits vascular tortuosity and dilation. Better text similarity does not establish clinical factuality or justify calling the reports validated expert-level explanations.
• Experiments focus on NFSD four-class classification without an independent external cohort or prospective clinical validation. Bag classification, per-image lesion localization, disease staging, and complete patient reports are distinct tasks; the results do not automatically cover the latter tasks.
• Low predictive entropy indicates confidence only: a confidently wrong branch can still receive greater weight, and CCL may reinforce incorrect instances. The supplement also reports near-one-hot instance attention after softmax, motivating tests of vulnerability to individual highly weighted views.
• Priorities include anatomical-location errors, omitted and fabricated findings, cross-device generalization, confidence calibration, and human review workflows, not only higher text-overlap scores.

Reading & Reproducibility Boundaries

• Count conflicts: Main Table 2 gives 1,678/587 ROP descriptions for training/validation; supplementary Table 8 gives 1,676/557, with 553 for testing in both. These sum to 2,818 and 2,786, respectively, against the reported total of 2,786. RH patient counts satisfy 708＋236＋236＝1,180, versus 1,181 in the total column; category-level patient totals sum to 9,261, versus 9,260 overall. These differences are retained rather than corrected on the authors' behalf.
• Configuration scope: Supplementary Table 9 specifies Stage 1 warmup/weight decay as 0.03/0, while supplementary §8.2 gives 0.05/0.01 for MIL; the main text's general description does not clearly distinguish stages. The main text gives 200 epochs for all baselines, but supplementary Table 10 lists 25 and 100 for LD2GMIL and ViLa-MIL. Early stopping after combining training and validation data also needs clarification. This note uses the main and supplementary texts and has not checked executed code configurations.
• Visualization scope: The supplement displays L1-normalized raw instance attention rather than actual softmax weights; regional attention has not been validated as precise segmentation either. Cached text allows checking captions and Table 11 descriptions, but cannot replace visual inspection of overlays on the original images.

Related Work & Insights

• vs DSMIL: DSMIL supplies the foundation for instance- and bag-level modeling. GVL-MIL mainly adds generative representation adaptation, within-image token aggregation, and visual/textual branches rather than inventing MIL from scratch.
• vs ViLa-MIL: ViLa-MIL uses category descriptions for contrastive classification; this method generates descriptions for individual fundus images and reads generated-sequence semantics. The comparison adapts ViLa-MIL with SigLip, 16 pseudo prototypes, and duplicated category prompts, rather than reproducing its original dual-scale pathology configuration unchanged.
• vs LD2GMIL: Both target neonatal fundus MIL. LD2GMIL strengthens visual instance representations, whereas this work adds per-image language supervision and generative intermediate-layer features. A useful next comparison would match supervision budgets and isolate the remaining contributions of language adaptation, layer selection, and fusion.

Rating

These are the reader's subjective assessments of the paper: 3 means adequate, 4 strong, and 5 outstanding; half points are allowed. Writing Quality refers to the original paper, and Value concerns research contributions and potential, not clinical readiness.

• Novelty: 4.5/5. Connects per-image language supervision, intermediate-layer selection, and patient-level MIL into a distinctive screening and reporting design, although the components have established foundations rather than constituting a new learning paradigm.
• Experimental Thoroughness: 4.5/5. Includes multiple baselines, layer, fusion, and CCL ablations, report evaluation, repeated runs, and significance testing; external cohorts and report-factuality validation remain limited, and the dual branch does not significantly outperform the image-only branch.
• Writing Quality: 4/5. The two-stage pipeline and supplementary material are substantial, but discrepancies in dataset counts, the strongest-baseline narrative, and training settings increase the effort required to understand the reproduction protocol.
• Value: 5/5. NFSD's pairing of patient bags with expert descriptions and the combined screening/reporting pipeline provide strong support for neonatal fundus research under limited annotation; clinical applications still require independent validation.