BioMedVR: Confusion-Aware Mixture-of-Prompt Experts for Biomedical Visual Reprogramming

Conference: ECCV2026
arXiv: 2606.24740
Area: Medical Image
Keywords: Visual Reprogramming, Confusion Suppression, Mixture-of-Prompt Experts, CLIP Adaptation, Few-Shot Learning

TL;DR

BioMedVR introduces visual reprogramming (VR) to the biomedical imaging field for the first time. Through a dual-expert decoupled confusion-aware MoPE architecture—where a positive expert handles main class discrimination and a negative expert suppresses easily confused classes with the help of LLM-generated confusion attributes, coupled with a margin-based confusion suppression loss (CS Loss) to explicitly widen the gap between positive and negative scores—it consistently outperforms existing VR and prompt learning methods across 11 medical datasets (spanning 9 imaging modalities) and 7 natural image benchmarks, using only 1/500 of the parameters of full-model fine-tuning.

Background & Motivation

Pre-trained vision-language models (such as CLIP) have demonstrated powerful zero-shot generalization capabilities on natural images, but their direct application in the biomedical imaging field faces severe challenges. Medical data is typically characterized by scarce labeling and diverse imaging modalities (ultrasound, MRI, CT, pathology slices, etc.). Moreover, extremely subtle visual differences exist among many disease subtypes—for example, cataracts vs. glaucoma, or renal cysts vs. renal tumors have highly similar appearances in images, making them prone to misdiagnosis even by inexperienced radiologists. Full-parameter fine-tuning of large models like CLIP on medical data is not only computationally expensive but also prone to overfitting when data is highly limited.

Parameter-efficient transfer learning methods have thus become a natural choice. Prompt learning adapts frozen backbones by inserting learnable tokens at the text or vision token level, but its core dependency is the requirement to access the internal structure of the model. In contrast, visual reprogramming (VR) offers a completely different path: it learns a differentiable perturbation in the input space to overlay onto the image without modifying any model parameters or internal architecture, making it architecture-agnostic and highly attractive for data-privacy-sensitive medical scenarios (e.g., data remaining in hospitals, black-box model API calls). Existing VR methods (such as AttrVR) perform well on natural images, but they all use a single shared visual perturbation to adapt to all classes. In medical imaging scenarios with high inter-class similarity, a single perturbation struggles to simultaneously capture heterogeneous semantics across multiple classes. When the text embeddings of two disease subtypes almost overlap, optimization gradients decay rapidly, and the model tends to amplify confusion between categories.

The Key Challenge: biomedical fine-grained classification not only needs to know "what a category is", but also "which categories it is easily confused with and what the differences are"—which is the core concept of clinical differential diagnosis but is completely neglected in existing VR frameworks. The Key Insight of this paper is: since LLMs possess rich medical knowledge, they can be utilized to automatically generate confusion attribute descriptions of "what other diseases a class looks most like" for each category, thereby explicitly modeling the confusion relationships between categories. Core Idea: Decouple visual reprogramming from a single-perturbation paradigm into a dual-expert architecture—where the positive expert reinforces main class recognition using discriminative attributes, and the negative expert suppresses easily confused classes leveraging LLM-generated confusion-aware attributes. Their outputs are adaptively fused through a learnable gating mechanism, which is further optimized by a confusion suppression loss that explicitly widens the margin between positive and negative scores to sharpen decision boundaries.

Method

Overall Architecture

The core architecture of BioMedVR is the Confusion-aware Mixture-of-Prompt Experts (MoPE), which explicitly decomposes input-space visual reprogramming into two complementary paths. For an input image, the positive expert \(\delta^+\) generates a reprogramming perturbation oriented toward main class recognition, while the negative expert \(\delta^-\) generates a reprogramming perturbation oriented toward confusion suppression. After overlaying the two perturbations onto the original image, they are passed through the same frozen CLIP vision encoder to extract feature embeddings, which are then aggregated with corresponding category attribute texts via Top-\(k\) similarity to obtain positive and negative logit scores, respectively. The two-way logits are fused via a shared learnable gating vector using a soft weighting scheme, and the final classification loss is optimized by combining cross-entropy and the confusion suppression loss.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Input Image x"] --> B["Positive Expert δ⁺"]
    A --> C["Negative Expert δ⁻"]
    B --> D["x̃⁺ = x + δ⁺"]
    C --> E["x̃⁻ = x + δ⁻"]
    D --> F["CLIP Vision Encoder (Frozen)"]
    E --> F
    F --> G["Positive Score s⁺<br/>= Top-k similarity<br/>with positive attributes"]
    F --> H["Negative Score s⁻<br/>= Top-k similarity<br/>with confusion attributes"]
    G --> I["Adaptive Gating<br/>[g⁺, g⁻] = softmax(w)"]
    H --> I
    I --> J["Fused logit<br/>s = g⁺·s⁺ + g⁻·s⁻"]
    J --> K["Cross Entropy + CS Loss"]

Key Designs

1. Dual-Expert Visual Reprogramming (MoPE): Explicitly Decoupling Discrimination and Confusion Suppression Objectives

Existing VR methods utilize a single, shared input perturbation \(\delta\) to fit all categories. In medical imaging scenarios with high inter-class similarity, this leads to conflicting optimization goals—the same perturbation must pull positive classes closer while pushing multiple confusable classes away, which causes mutual gradient cancellation. BioMedVR decouples perturbations into two independent experts: the positive expert \(\delta^+\) learns alignment with discriminative positive attribute texts (e.g., "diffuse lens opacity" for cataracts), while the negative expert \(\delta^-\) aligns with LLM-generated confusion attributes (e.g., "thinned optic cup/disc" — a classic feature of glaucoma, which represents a confusion signal that should be suppressed for cataracts). The two experts' feature embeddings after CLIP encoding are aggregated with their corresponding attribute sets via Top-\(k\) similarity to yield two independent logits \(s^+_y\) and \(s^-_y\). They are fused via soft weighting using a shared learnable gating vector \(w\):

\[s_y = g^+ s^+_y + g^- s^-_y, \quad [g^+, g^-] = \text{softmax}(w)\]

This gating mechanism is shared across the entire dataset and automatically learned, naturally balancing during training—the weight of the negative expert increases for easily confused categories, while the positive expert dominates for clearly separable categories.

2. LLM-Driven Confusion-Aware Attribute Generation: Injecting Clinical Differential Diagnosis Knowledge into the VR Framework

The exact direction of confusion features that the negative expert should suppress is not known a priori. BioMedVR leverages the medical knowledge of LLMs (GPT-4.1 by default) to automatically generate confusion-aware descriptions for each disease category. For each class \(y\), it prompts the LLM with "generate negative descriptions that are visually most easily confused with \([class\ name]\)" to yield \(N_c\) confusion-aware text attributes \(\mathcal{T}^-_y = \{t^-_1, ..., t^-_{N_c}\}\). Taking renal tumors as an example, the positive attributes describe "heterogeneously enhancing solid mass," while the LLM-generated confusion attributes describe "well-circumscribed, cystic, internally homogeneous"—which are typical features of renal cysts. By encoding such "cyst-like" cues as alignment targets for the negative expert, the model learns to actively suppress the tendency to misclassify an image as a renal tumor when observing cyst-like textures. LLM attribute generation runs offline and is not needed during inference, thus incurring zero inference overhead.

3. Confusion Suppression Loss (CS Loss): Explicitly Widening the Gap Between Positive and Negative Scores to Sharpen Decision Boundaries

Relying solely on the dual-expert architecture and standard cross-entropy is still insufficient to guarantee a sufficiently large safety margin between positive and negative logits. BioMedVR designs a margin-based confusion suppression loss to bridge this gap:

\[\mathcal{L}_{cs} = \mathbb{E}_{(x,y)}\left[\max(0, \max_{c \neq y} s^-_c - s^+_y + m)\right]\]

where \(m\) is a margin hyperparameter (default is 0.5). The core operation of this loss is as follows: for each sample, it identifies the highest score assigned by the negative expert across all incorrect categories \(\max_{c \neq y} s^-_c\)—representing "what other disease it looks most like"—and then forces this maximum confusion score to be at least \(m\) lower than the target class score of the positive expert \(s^+_y\). A gradient is generated only when \(s^+_y - \max_{c \neq y} s^-_c < m\), and no penalty is applied otherwise, avoiding redundant suppression on samples that are already clearly differentiated. This mechanism directly mimics the clinical differential diagnosis process: ruling out the most competitive differential diagnoses before confirming a diagnosis.

Loss & Training

The final loss function is a weighted combination of the cross-entropy loss and the confusion suppression loss:

\[\mathcal{L} = \mathcal{L}_{CE} + \beta \, \mathcal{L}_{cs}\]

where \(\beta=0.3\) controls the intensity of confusion suppression. Only the three learnable components \(\{\delta^+, \delta^-, w\}\) (approx. 0.30M parameters, only 1/500 of the 150M parameters of the full CLIP model) are optimized during training, with both the CLIP vision and text encoders completely frozen. Optimization is performed using SGD (initial lr=40, momentum=0.9) with a cosine annealing schedule for 400 epochs and a batch size of 512.

Key Experimental Results

Main Results

Evaluated under a 16-shot setting across 11 medical datasets (covering 9 imaging modalities and 10 organs), with ViT-B/16 CLIP as the backbone:

Dataset	Modality	Baseline (AttrVR)	Ours (BioMedVR)	Gain
BUSI	Ultrasound	78.0%	82.6%	+4.6%
Knee X-ray	X-ray	33.5%	45.7%	+12.2%
Kvasir	Endoscopy	79.4%	80.2%	+0.8%
LungColon	Pathology	93.8%	94.7%	+0.9%
OCTMNIST	OCT	80.4%	80.3%	-0.1%
BTMRI	MRI	76.5%	81.7%	+5.2%
CHMNIST	Pathology	85.3%	84.5%	-0.8%
COVID-19	X-ray	71.0%	77.4%	+6.4%
CT-Kidney	CT	71.6%	74.0%	+2.4%
DermaMNIST	Dermoscopy	61.6%	65.3%	+3.7%
Retina	Fundus	71.5%	74.1%	+2.6%
Average	—	73.0%	76.4%	+3.4%

Using general-domain CLIP (ViT-B/16) as the backbone, BioMedVR achieves an average accuracy of 76.4%, surpassing all compared VR methods (AttrVR 73.0%, AR 72.6%, VP 68.3%) and prompt learning methods (BioMedCoOp 71.2%, CoCoOp 66.6%, CoOp 63.9%), with a parameter size of only 0.30M (about half of BioMedCoOp). Under the zero-shot setting, BioMedVR (ZS) improves upon the original CLIP by +3.5% (28.1% \(\rightarrow\) 31.6%), and reaches 47.6% when built on the BioMedCLIP backbone.

Ablation Study

Configuration	Average Accuracy	Relative Change	Description
Full Model	76.4%	—	All components
w/o Confusion-Aware Attributes (CA)	71.4%	-5.0%	Negative expert degrades to NaN supervision
w/o Dual-Expert (MoPE)	75.6%	-0.8%	Degrades to AttrVR-style single perturbation
w/o CS Loss	73.6%	-2.8%	Unstable training, increased variance across datasets

Key Findings

• Confusion-aware attributes are the most critical component: Eliminating them leads to an average accuracy drop of 5.0%, which is the largest single performance impact among all ablations, demonstrating that the effectiveness of the negative expert depends heavily on high-fidelity confusion descriptions generated by LLMs rather than random text.
• Specialization outclasses quantity: MoE architectural analysis indicates that increasing the number of generic experts yields limited performance gains, whereas exactly two semantically specialized experts (a positive expert + a confused negative expert) yield the best results. This confirms that semantic specialization, rather than the raw number of experts, is the key design variable for VR.
• Robustness across backbones: The method achieves stable improvements of +1.5% and +2.6% on ViT-B/32 and RN50 backbones, respectively, verifying that the approach is architecture-agnostic and generalizes well to different types of backbones.
• Outstanding zero-shot capabilities: Combining ONLY the LLM-generated confusion attributes with positive attributes (without any training) drastically outperforms the zero-shot CLIP baseline, notably on Knee X-ray (+10.9%) and LungColon (+28.9%).

Highlights & Insights

• Introducing VR to biomedical imaging for the first time. Previously, VR was primarily validated on natural images. This work demonstrates its feasibility across nine medical modalities without sacrificing architecture agnosticism and privacy protection—which holds practical deployment value for medical scenarios demanding data localism and black-box model usage.
• A tightly coupled trilogy of confusion attribute generation, dual-expert decoupling, and CS Loss: Without confusion attributes, the negative expert degrades to noise (-5.0%); without dual-experts, positive and negative objectives cannot be separately optimized (-0.8%); without the CS Loss, the margin cannot be effectively enforced (-2.8%). The ablation studies clearly validate the self-consistency of the overall design.
• Highly lightweight gating design: Using only a 2D softmax vector, it achieves adaptive weighting of positive/negative experts without complex routing networks, demonstrating highly effective decoupled fusion under an extremely parameter-light footprint of 0.30M.
• Inherent value of the differential diagnosis perspective for classification: Rather than simply "learning category prototypes", the model actively learns "which other categories are easily confused and suppresses them". This paradigm can be widely applied to various fine-grained classification scenarios (such as species identification, mineral classification, and industrial defect detection).

Limitations & Future Work

• High reliance on LLM attribute quality. The confusion attributes are fully generated offline by LLMs (GPT-4.1 performs best, other LLMs show slightly inferior results). The paper has not explored whether clinical experts (e.g., radiologists) providing structured differential diagnosis lists would outperform LLM generation. In real-world clinical deployments, expert clinical knowledge might serve as a more reliable semantic source.
• Limitation to image-level classification. Currently, BioMedVR focuses on disease diagnostic classification and has not been validated on pixel-level dense prediction tasks such as segmentation and detection—which happen to be more common in medical image analysis.
• Lack of sample-level adaptability for globally shared gating. Currently, the gating vector is shared across the entire dataset. For inherently clear samples, continuous computation of the negative expert might be wasteful, whereas extremely ambiguous samples might require stronger modulation from the negative expert. Introducing sample-conditioned gating is a natural extension direction.
• Training computation is approximately twice that of single perturbation (\(9.10\text{ s/epoch}\) vs AttrVR \(6.55\text{ s/epoch}\)). Although inference overhead can be mitigated by performing a single encoder forward pass and executing computations in the embedding space, the efficiency of the training phase still has room for optimization.

Related Work & Insights

• vs AttrVR: The most direct baseline, which also uses attribute-guided VR but relies on a single shared perturbation. BioMedVR decomposes it into positive and negative dual experts combined with CS Loss, boosting average performance by +3.4% across 11 medical datasets.
• vs BioMedCoOp: The first work to apply prompt learning to medical VLM adaptation, which requires accessing the internal token layers of the model. As an input-space VR method, BioMedVR is architecture-agnostic and optimal for privacy-sensitive scenarios, and achieves higher accuracy with fewer parameters on a general CLIP backbone (76.4% vs 71.2%).
• vs CoOp / CoCoOp: These classic prompt learning methods suffer from limited token-level representational capacity on medical data—for instance, CoOp achieves only 27.1% on Knee X-ray, while BioMedVR reaches 45.7%. Input-space VR learns pixel-level perturbations, allowing it to capture microstructural textures and other pixel-level features, which are fine-grained differences that token-level prompts struggle to express.

Rating

• Novelty: ⭐⭐⭐⭐ Introducing VR to medical imaging for the first time and designing the confusion-aware dual-expert mechanism represents a solid paradigm contribution. Nonetheless, the individual components (VR, attribute guidance, MoE, LLM generation) are clever integrations of existing technologies.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extremely comprehensive evaluation spanning 11 medical + 7 natural datasets, ablation of 4 LLM backbones, exploration of multiple vision backbones, t-SNE visualizations, hyperparameter analysis, and MoE size experiments.
• Writing Quality: ⭐⭐⭐⭐ Clear motivation and complete Method description, though some details (such as the Top-\(k\) selection mechanism) rely on citations of AttrVR, requiring readers to refer to the original paper for full comprehension.
• Value: ⭐⭐⭐⭐ As a foundational work applying VR to the medical domain, the design of input space + architecture-agnostic + confusion-awareness is highly practical for low-resource biomedical AI deployment. Notable potential for extensions beyond simple classification.

Related Papers

• [ICML 2025] I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts
• [ICML 2026] EEG-Based Multimodal Learning via Hyperbolic Mixture-of-Curvature Experts
• [CVPR 2026] SegMoTE: Token-Level Mixture of Experts for Medical Image Segmentation
• [NeurIPS 2025] MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding
• [ICLR 2026] Mixture of Mini Experts: Overcoming the Linear Layer Bottleneck in Multiple Instance Learning