Enhancing Interpretability in CLIP with Optimal Transport-based Submodular Optimization for Ophthalmic Imaging¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/SihongLu/OTSMDL
Area: Medical Imaging
Keywords: Ophthalmic Imaging, CLIP Interpretability, Submodular Optimization, Unbalanced Optimal Transport, Superpixel Attribution

TL;DR¶

To address the issues of fragmented activations and noisy backgrounds in ophthalmic CLIP explanations, OTSMDL couples multi-dimensional submodular scoring for candidate initialization with unbalanced optimal transport for global consensus re-ranking, accurately grounding diagnostic lesions and anatomical structures.

Background & Motivation¶

Contrastive Language-Image Pretraining (CLIP) models have demonstrated impressive zero-shot transfer capabilities across diverse ophthalmic diagnostic benchmarks, including color fundus photography and optical coherence tomography (OCT). However, in safety-critical medical contexts, outputting prediction probabilities alone is insufficient to cultivate clinical trust. Clinicians and patients require transparent, inspectable visual evidence demonstrating whether predictions are grounded in valid anatomical landmarks or pathological lesions. Existing explainability paradigms—including gradient/CAM-based attributions, attention/relevance rollouts, and dense CLIP localization adapters—predominantly cater to object-centric natural images. When transferred to pathology- and anatomy-centric ophthalmic imaging, these methods frequently produce noisy, highly fragmented, or semantically misaligned attribution heatmaps that fail to delineate key diagnostic structures such as the optic disc/cup complex or subtle microaneurysms.

The root cause of this failure lies in the combination of subtle pathological hallmarks, vast non-pathological background areas, and the exacerbated vision-language semantic gap in specialized medical contexts. While submodular optimization frameworks (such as Less-is-More) exploit diminishing marginal returns to enforce coverage and curb redundancy, they rely heavily on greedy, step-wise sequential additions. Purely greedy heuristics make irreversible choices driven strictly by local marginal gains, frequently getting trapped in local optima and producing scattered superpixel clusters that lack global structural cohesion.

To overcome both greedy myopic biases and background noise contamination, this paper reframes explainable evidence identification into a collaborative paradigm: submodular subset initialization followed by optimal transport global consensus. The core idea is to employ submodular-inspired utility metrics not merely for selecting a greedy subset, but as a score-guided source mass distribution prior within an unbalanced optimal transport framework, enabling global competition and semantic alignment across all candidate superpixels under unified visual-textual cost constraints.

Method¶

Overall Architecture¶

OTSMDL operates across three integrated stages: boundary-aware superpixel over-segmentation, multi-dimensional submodular utility initialization, and unbalanced optimal transport (UOT) global re-ranking. First, the Simple Linear Iterative Clustering (SLIC) algorithm partitions the retinal image into superpixels adhering to natural anatomical contours. Next, candidates are evaluated along three utility dimensions—contrast, importance, and uniqueness—to construct an initial subset under diminishing marginal return constraints while formulating the source mass distribution. Finally, an unbalanced Sinkhorn transport plan is iteratively solved over a joint visual-textual cost matrix, aggregating column-wise transported mass to re-rank all candidate regions until convergence.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Ophthalmic Image & Target Text"] --> B["Superpixel Segmentation<br/>SLIC preserves anatomical boundaries"]
    B --> C["Multi-dimensional Submodular Scoring<br/>Importance, contrast, and uniqueness"]
    C --> D["Diminishing Marginal Return Selection<br/>Initial candidate subset & source distribution prior"]
    D --> E["Dual Cost Matrix Construction<br/>Visual cohesion & text semantic alignment"]
    E --> F["Unbalanced Optimal Transport Re-ranking<br/>Sinkhorn soft matching & column mass aggregation"]
    F -->|Iterative update until convergence| G["Faithful Coherent Evidence Heatmap"]

Key Designs¶

1. Superpixel Segmentation: Preserving Anatomical and Lesion Boundaries

Standard patch-level partitioning in Vision Transformers slices arbitrarily across delicate vascular structures and irregular retinal lesions. To preserve morphological integrity, OTSMDL adopts the SLIC algorithm to partition the input image \(I \in \mathbb{R}^{H \times W \times 3}\) into \(N\) disjoint, boundary-adhering superpixel regions \(R = \{R_i\}_{i=1}^N\). Each region defines a binary spatial mask \(m_i \in \{0, 1\}^{H \times W}\) yielding a candidate masked sub-image \(I_i = I \odot m_i\). This ensures that irregular anatomical features, such as exudates and the cup-to-disc boundary, remain intact within individual units of the candidate pool \(\mathcal{M} = \{I_i\}_{i=1}^N\).

2. Multi-dimensional Submodular Scoring: Balancing Specificity, Relevance, and Rarity

To measure the diagnostic contribution of each candidate \(m_i\) for a target diagnostic class \(y\) with text embedding \(t_y\) and visual feature \(f_i = F(I_i)\), the framework formulates three complementary utility metrics: - Contrast Score (\(s_i^{\text{con}}\)): Isolates class-discriminative evidence and suppresses ambiguous inter-class overlaps by measuring the margin between the target class alignment and the strongest competing class \(c \neq y\): \(s_i^{\text{con}} = \left[ \cos(f_i, t_y) - \max_{c \neq y} \cos(f_i, t_c) \right]_+\). - Importance Score (\(s_i^{\text{imp}}\)): Combines direct local alignment with a deletion-based perturbation gap \(g_i = \max(0, \cos(F(I), t_y) - \cos(F(I \setminus m_i), t_y))\), computing \(s_i^{\text{imp}} = \cos(f_i, t_y) \cdot g_i\) to identify regions whose removal drops model confidence the most. - Uniqueness Score (\(s_i^{\text{uni}}\)): Leverages the clinical prior that pathological lesions form spatial anomalies deviating from healthy retinal backgrounds. It computes the squared Euclidean distance between \(f_i\) and the mean global visual embedding \(\bar{f} = \frac{1}{N} \sum_{j=1}^N f_j\): \(s_i^{\text{uni}} = \| f_i - \bar{f} \|_2^2\).

These metrics are weighted, summed, and min-max normalized into a base utility \(\hat{s}_i \in [0, 1]\). To penalize redundancy among visually similar regions, a pairwise kernel \(K_{ij} = \frac{1 + \cos(f_i, f_j)}{2}\) (with \(K_{ii} = 0\)) guides greedy subset selection via marginal gain:

\[\Delta(i \mid S) = \hat{s}_i - \lambda_{\text{dr}} \sum_{j \in S} K_{ij}\]

Iteratively picking regions with maximal \(\Delta(i \mid S)\) collects an initial subset \(S_0\) of budget \(K\).

3. Dual Cost Matrix Construction: Coordinating Visual Cohesion and Semantic Alignment

Rather than treating the greedy subset \(S_0\) as the final explanation, OTSMDL employs it as the initial state \(S^{(r-1)}\) for global optimal transport. Across the current selected set \(S^{(r-1)}\) and all candidate regions \(\{1, \dots, N\}\), a rectangular cost matrix \(C^{(r)} = \alpha C_{\text{vis}}^{(r)} + \beta C_{\text{text}}^{(r)} \in \mathbb{R}^{K \times N}\) is formulated: - Visual Transport Cost: \(C_{\text{vis}}^{(r)}(u, j) = \| f_u - f_j \|_2 + \| f_{u \cup j} - f_{S^{(r-1)}} \|_2\), where \(f_{u \cup j}\) represents the merged feature of superpixels \(u\) and \(j\), and \(f_{S^{(r-1)}}\) is the pooled representation of the current selected set. This term aligns visually concordant regions while penalizing deviation from the subset's global structural consensus. - Text-Guided Transport Cost: With region similarities \(\text{sim}_i = \cos(f_i, t_y)\) and merged similarity \(\text{sim}_{u \cup j} = \cos(f_{u \cup j}, t_y)\), the cross-modal cost is defined as \(C_{\text{text}}^{(r)}(u, j) = (1 - \text{sim}_u) + (1 - \text{sim}_j) + (1 - \text{sim}_{u \cup j}) + |\text{sim}_u - \text{sim}_j|\), penalizing low diagnostic relevance and inter-region semantic disparity.

4. Unbalanced Optimal Transport Re-ranking: Mass Relaxation and Global Consensus Aggregation

Standard balanced OT enforces strict mass conservation, inadvertently dispersing transported mass into healthy background patches. OTSMDL adopts Unbalanced Optimal Transport (UOT), defining the source distribution with normalized base utility weights \(a_i^{(r)} = \hat{s}_{u_i} / \sum_{j=1}^K \hat{s}_{u_j}\) and setting a uniform target prior \(b = \frac{1}{N} \mathbf{1}\). Relaxing marginal constraints allows an unbalanced Sinkhorn solver to match diagnostic evidence flexibly. The accumulated column-wise mass is converted into global importance weights:

\[w_i^{(r)} = \frac{\sum_u \hat{s}_u \Pi_{u, i}^{(r)}}{\sum_j \sum_u \Pi_{u, j}^{(r)}}\]

The top-\(K\) scoring candidates form the updated set \(S^{(r)}\). This process repeats until the set stabilizes for \(p=5\) consecutive rounds, yielding the final coherent explanation set \(S^\star\).

Key Experimental Results¶

Main Results¶

Evaluation spans seven ophthalmic datasets covering Diabetic Retinopathy (IDRiD, APTOS2019, EyePACS), multi-disease OCT / macular holes (OCTDL, OCTID, OIMHS), and glaucoma assessment (PAPILA). Faithfulness is quantified using Deletion AUC (lower is better) and Insertion AUC (higher is better).

Table 1: Explanation performance on three Diabetic Retinopathy datasets (ground-truth target class)

Methods	IDRiD Deletion (↓)	IDRiD Insertion (↑)	APTOS2019 Deletion (↓)	APTOS2019 Insertion (↑)	EyePACS Deletion (↓)	EyePACS Insertion (↑)
Grad-ECLIP	0.397	0.427	0.361	0.399	0.159	0.237
GAME	0.357	0.305	0.363	0.226	0.151	0.123
Grad-CAM	0.493	0.300	0.471	0.287	0.243	0.128
M2IB	0.346	0.326	0.321	0.292	0.159	0.137
MaskCLIP	0.465	0.323	0.479	0.291	0.223	0.139
RISE	0.244	0.583	0.306	0.444	0.192	0.229
Rollout	0.356	0.286	0.347	0.236	0.169	0.113
Less-is-More	0.224	0.450	0.209	0.380	0.113	0.179
CLIPSurgery	0.329	0.413	0.337	0.281	0.138	0.182
LeGrad	0.350	0.393	0.402	0.365	0.139	0.245
LibraGrad	0.311	0.400	0.351	0.411	0.148	0.216
MDA	0.256	0.452	0.305	0.428	0.128	0.239
OTSMDL (Ours)	0.226	0.624	0.232	0.484	0.122	0.253

Table 2: Explanation performance on OCT and glaucoma datasets (ground-truth target class)

Methods	OCTDL Deletion (↓)	OCTDL Insertion (↑)	OCTID Deletion (↓)	OCTID Insertion (↑)	PAPILA Deletion (↓)	PAPILA Insertion (↑)
Grad-ECLIP	0.307	0.208	0.433	0.511	0.230	0.702
GAME	0.235	0.208	0.400	0.265	0.280	0.577
Grad-CAM	0.339	0.192	0.449	0.480	0.256	0.573
M2IB	0.259	0.364	0.425	0.317	0.208	0.622
RISE	0.184	0.500	0.339	0.236	0.158	0.573
Less-is-More	0.221	0.337	0.379	0.506	0.089	0.678
LeGrad	0.245	0.365	0.402	0.433	0.190	0.721
LibraGrad	0.291	0.263	0.378	0.440	0.250	0.714
MDA	0.186	0.365	0.406	0.476	0.149	0.690
OTSMDL (Ours)	0.164	0.530	0.334	0.482	0.148	0.725

Ablation Study¶

Ablations on IDRiD and OCTDL dissect the impact of the cost matrices, scoring metrics, and OT configurations.

Table 5: Ablation study of visual and text transport cost components

Visual Cost (\(\alpha\))	Text Cost (\(\beta\))	IDRiD Del. (↓)	IDRiD Ins. (↑)	OCTDL Del. (↓)	OCTDL Ins. (↑)
\(\times\)	\(\checkmark\)	0.226	0.601	0.168	0.528
\(\checkmark\)	\(\times\)	0.274	0.487	0.203	0.359
\(\times\) (Pure submodular greedy)	\(\times\)	0.289	0.406	0.248	0.336
\(\checkmark\)	\(\checkmark\)	0.226	0.624	0.164	0.530

Table 6: Ablation study of submodular utility components and OT variants

Config / Variant	IDRiD Del. (↓)	IDRiD Ins. (↑)	OCTDL Del. (↓)	OCTDL Ins. (↑)	Note
Full Model (Default Settings)	0.226	0.624	0.164	0.530	All 3 scores + UOT + score-guided source
Balanced OT (full scores)	0.313	0.426	0.293	0.346	Strict mass constraint disperses into background
Uniform Source (\(a = \text{uniform}\))	0.225	0.603	0.167	0.529	Loss of base utility prior degrades insertion
No Scoring Guidance (\(\times \times \times\))	0.233	0.566	0.173	0.506	Pure OT optimization remains competitive
Importance Only (\(s^{\text{imp}}\))	0.221	0.524	0.165	0.510	Favors deletion drop at the cost of insertion
Uniqueness Only (\(s^{\text{uni}}\))	0.231	0.612	0.173	0.521	Captures outlier lesions, boosting insertion

Key Findings¶

Unbalanced formulation is vital for medical imaging: Replacing Unbalanced OT with Balanced OT causes a catastrophic performance drop in Insertion AUC on both IDRiD (0.624 \(\rightarrow\) 0.426) and OCTDL (0.530 \(\rightarrow\) 0.346). Forcing mass conservation artificially penalizes sparse lesions and spreads attribution across extensive healthy retina regions.
Text transport cost drives semantic steering: Dropping text transport cost reduces IDRiD Insertion AUC by 13.7 points (0.624 \(\rightarrow\) 0.487), whereas dropping visual cost causes only a minor drop to 0.601, demonstrating that cross-modal semantic guidance is the dominant anchor.
Clinical feasibility and computational efficiency: With global feature pre-caching and incremental feature updating during OT loops, OTSMDL achieves an average latency of 5.28 s/image with a peak memory footprint of \(\approx 6.4\) GB on an RTX 5080, substantially outperforming Less-is-More in execution speed while operating within practical clinical bounds.

Highlights & Insights¶

Dual-role bridging of submodular utility and transport theory: Instead of using greedy selection as a final decision, OTSMDL creatively uses submodular scores as the initialization pool and source mass prior for Optimal Transport, successfully reconciling local non-redundant utility with global structural consensus.
Sparse-aware unbalanced transport modeling: By recognizing the inherent sparsity of ophthalmic biomarkers, relaxing marginal constraints via unbalanced Sinkhorn enables soft matching that naturally concentrates evidence on salient lesion clusters while suppressing healthy backgrounds.
Post-hoc, architecture-agnostic interpretability: The entire framework operates post-hoc without fine-tuning backbone weights, making it directly transferable to multimodal vision-language models in other clinical domains.

Limitations & Future Work¶

Fixed superpixel granularity and region budget: SLIC superpixel compactness and the budget parameter \(K\) are statically defined. This limits adaptability when encountering multi-scale co-occurring lesions, such as tiny microaneurysms alongside large geographic retinal atrophy.
Static convergence thresholds: The framework relies on fixed iteration limits (\(R_{\max}=5\)) and stability patience (\(p=5\)) rather than condition-adaptive stopping criteria.
Future directions: Exploring multi-scale adaptive superpixels and extending the formulation toward end-to-end differentiable weakly supervised lesion segmentation.

vs Less-is-More (Chen et al., ICLR 2024): Less-is-More relies strictly on greedy submodular optimization, which suffers from irreversible local greediness and scattered heatmaps in retinal images. OTSMDL uses greedy picks only as an initialization and refines them globally via UOT, outperforming Less-is-More in IDRiD Insertion AUC by 17.4 points (0.624 vs. 0.450).
vs Attention Rollout / GAME: Attention propagation methods often generate diffused background noise across Vision Transformer heads. OTSMDL enforces anatomical boundary constraints via superpixels and semantic cost filtering, producing tight evidence localization around the optic disc and lesions.
vs Gradient-based Attributions (Grad-CAM / Grad-ECLIP / LeGrad): Gradient signals often suffer from saturation and high-frequency noise. OTSMDL optimizes over discrete superpixels using perturbation-based deletion gaps and global transport costs, yielding higher spatial Dice/IoU alignment with ground-truth lesion masks.

Rating¶

Novelty: ⭐⭐⭐⭐☆ (Ingenious integration of submodular utility prior with unbalanced optimal transport for post-hoc vision attribution)
Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Tested across 7 multimodal ophthalmic datasets against 13 baselines with mask IoU/Dice and granular ablations)
Writing Quality: ⭐⭐⭐⭐⭐ (Mathematically sound formulation, cohesive narrative, and transparent analysis)
Value: ⭐⭐⭐⭐☆ (Highly practical post-hoc interpretability solution for clinical ophthalmic vision-language models)