Unsupervised Source-Free Ranking of Biomedical Segmentation Models Under Distribution Shift¶
Conference: ECCV 2026
Paper: ECCV Official Page
Code: https://github.com/kreshuklab/model_ranking
Area: Medical Imaging
Keywords: Model Ranking, Distribution Shift, Biomedical Segmentation, Consistency Evaluation, Unsupervised Source-Free
TL;DR¶
Addressing the acute shortage of biomedical annotations and pronounced domain shifts, this paper introduces CMR, the first black-box, unsupervised, source-free ranking framework for semantic and instance segmentation models, achieving high correlation with ground-truth rankings via only two inference passes measuring prediction consistency under perturbations.
Background & Motivation¶
Deep learning has achieved remarkable milestones in biomedical image segmentation, yet in practical clinical and biological workflows, the labor-intensive curation and expert annotation of datasets remain the primary bottleneck. With the rapid expansion of model repositories such as the BioImage Model Zoo, public challenges like ToothFairy, and emerging foundation models (e.g., µSAM, CellposeSAM), practitioners can now access dozens of pre-trained models for a given task. However, because variations in imaging modalities, staining protocols, and anatomical targets introduce severe distribution shifts, candidate models display drastically divergent transfer capabilities on new, unlabelled target datasets. Selecting the top-performing model under strictly unsupervised and source-free conditions without target annotations or access to source training data remains an unresolved roadblock to scalable model reuse.
Existing transferability estimation metrics and unsupervised domain adaptation (UDA) validation schemes were predominantly designed for image classification tasks. They invariably depend on target domain labels, require white-box feature-space extraction, or enforce rigid geometric assumptions such as intra-class compactness and linear inter-class separability. When extended naively to biomedical segmentation, their correlation with actual segmentation performance breaks down entirely. More critically, instance segmentation produces permutation-invariant masks derived from non-differentiable post-processing (such as connected components analysis), lacking any fixed-dimensional representation in feature space; consequently, unsupervised ranking for instance segmentation has remained virtually unaddressed. Furthermore, conventional uncertainty estimation and quality control methods necessitate Bayesian training modifications or dozens of Monte Carlo samples, rendering them unsuitable for black-box repository environments.
To resolve these challenges, the authors shift away from feature-space assumptions toward probing the decision boundary margin directly in the output space. On distribution-shifted target data, models that transfer effectively maintain wider decision margins, producing stable predictions resistant to slight input or latent perturbations; conversely, poorly adapted models exhibit cramped margins where minor noise triggers widespread pixel misclassifications. Core idea: an unsupervised, source-free Consistency-based Model Ranking (CMR) framework that quantifies pixel-wise output consistency between unperturbed and slightly perturbed inputs across candidate models, recovering accurate target performance rankings with only two forward passes.
Method¶
Overall Architecture¶
Consistency-based Model Ranking (CMR) is tailored for source-free, unsupervised model selection across heterogeneous model repositories. Given an unlabelled target dataset \(X_T = \{x_n\}_{n=1}^{N_T}\) and a candidate pool of pre-trained segmentation models \(\mathcal{M} = \{M_m\}_{m=1}^{N_M}\), CMR treats candidate models as black boxes and operates strictly on their output space.
For each target image, CMR obtains an unperturbed baseline prediction alongside a perturbed prediction generated via minor input transformations or test-time latent feature dropout. Predictions are routed to dedicated task-specific metrics: semantic segmentation uses foreground-restricted soft consistency (CMR-EI) or hard label-flip ratios (CMR-NHD), whereas instance segmentation employs an Adapted Rand Score (CMR-ARS) tracking pairwise pixel co-clustering. Individual scores are aggregated across perturbations and target images using a robust median operator to generate the final descending model rank. The entire workflow requires zero manual annotations and zero source data.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Unlabelled Target Domain Image"] --> B["Prediction Consistency Boundary Probing<br/>Input TTA or Feature TTD Yields Perturbed Prediction"]
B --> C["Foreground-Restricted Semantic Consistency<br/>CMR-EI Soft Confidence / CMR-NHD Hard Bit-Flips"]
B --> D["Pairwise Pixel Adapted Rand Metric<br/>CMR-ARS Evaluates Instance Segmentation Consistency"]
C --> E["Robust Median Aggregation Ranking<br/>Mean Over Perturbations and Median Across Samples"]
D --> E
Key Designs¶
1. Prediction Consistency Boundary Probing: Feature-Agnostic Black-Box Output Evaluation Prior transferability metrics rely on evaluating cluster separability or nuclear norms within latent feature spaces. However, biomedical segmentation networks trained with Dice and cross-entropy losses do not enforce feature compactness, and heterogeneous model repositories exhibit disparate feature dimensions. CMR shifts the evaluation focus to the output space, treating perturbation-induced pixel label flips as a direct proxy for decision boundary margins on the target manifold. Perturbations need not reflect realistic domain shifts; their purpose is strictly to probe boundary sensitivity. CMR introduces two perturbation mechanisms: - Input-Space Test-Time Augmentation (TTA): Injects additive Gaussian noise \(\mathcal{N}(0, \vartheta^2)\), gamma adjustments, or subtle brightness/contrast shifts into the input image, suitable for closed-source APIs and containerized inference; - Feature-Space Test-Time Dropout (TTD): Applies dropout to intermediate layers (such as the bottleneck where representations are highly redundant) during inference while keeping network weights strictly frozen, circumventing the need for specialized Bayesian training required by MC-Dropout.
2. Foreground-Restricted Semantic Consistency: Soft and Hard Metrics for Extreme Class Imbalance In biomedical semantic segmentation, background pixels typically comprise more than 90% of the field of view. Unrestricted consistency calculations would allow stable background regions to dominate the score, obscuring critical failures on minute foreground structures. CMR restricts evaluation to the union of foreground pixels predicted under unperturbed and perturbed states, \(I^{(c)}_n = \{i : \tilde{y}^{(c)}_{n,i} = 1 \text{ or } \hat{y}^{(c)}_{n,i} = 1\}\), formulating both soft and hard scoring schemes: - Soft Consistency (CMR-EI): Extends Effective Invariance to dense segmentation by incorporating continuous softmax probabilities alongside label agreement: $\(\text{CMR-EI}^{(c)}_n = \frac{1}{|I^{(c)}_n|} \sum_{i \in I^{(c)}_n} \hat{p}^{(c)}_{n,i} \cdot \tilde{p}^{(c)}_{n,i} \cdot [\tilde{y}^{(c)}_{n,i} = \hat{y}^{(c)}_{n,i}]\)$ - Hard Consistency (CMR-NHD): Mitigates probability miscalibration under domain shift by computing the foreground-restricted pixel label agreement using Normalized Hamming Distance: $\(\text{CMR-NHD}^{(c)}_n = 1 - \frac{1}{|I^{(c)}_n|} \sum_{i \in I^{(c)}_n} [\tilde{y}^{(c)}_{n,i} \neq \hat{y}^{(c)}_{n,i}]\)$ This formulation is equivalent to the Intersection over Union (IoU) between perturbed and unperturbed masks, offering fast computation unaffected by uncalibrated confidence drifts.
3. Pairwise Pixel Adapted Rand Metric: Instance Segmentation Ranking via Co-clustering Instance segmentation models output permutation-invariant sets of arbitrary instance IDs, rendering direct pixel-to-pixel label matching impossible across perturbed iterations. Prior heuristic metrics (e.g., SEG) reduce instances to dilated centroids, ignoring intricate pixel-level shapes. Drawing upon cluster validation theory, CMR adapts the Adapted Rand Score (ARS) to assess whether pixel pairs consistently co-occur within the same instance: $\(\text{CMR-ARS}_n = \frac{\sum_{k,\omega} w_{k\omega}^2}{\omega \sum_k \tilde{s}_k^2 + (1-\omega) \sum_\omega \hat{s}_\omega^2}\)$ where \(w_{k\omega}\) denotes the fraction of pixel pairs assigned to instance \(k\) in \(\tilde{y}_n\) and instance \(\omega\) in \(\hat{y}_n\) restricted to the joint foreground union; \(\tilde{s}_k = \sum_\omega w_{k\omega}\) and \(\hat{s}_\omega = \sum_k w_{k\omega}\) represent marginal sums. CMR-ARS represents the weighted harmonic mean between over-segmentation penalty (\(\text{ARS}_{\text{split}}\)) and under-segmentation penalty (\(\text{ARS}_{\text{merge}}\)), using \(\omega=0.5\) by default. This establishes the first pixel-precise consistency metric applicable across arbitrary instance segmentation pipelines (including Mask R-CNN, Cellpose, and SAM).
4. Robust Median Aggregation Ranking: Two-Pass Inference for Practical Repositories To prevent pathological outliers or mode-collapsed models (e.g., trivial all-foreground or all-background predictions) from distorting ranks, CMR executes two-level aggregation: $\(\text{CMR}_{M_m}(X_T) = \text{median}_{n=1,\dots,N_T} \left( \frac{1}{N_{\text{pert}}} \sum_{j=1}^{N_{\text{pert}}} \text{consis}\left(\hat{y}_{n,m}, \tilde{y}_{n,m}^{(j)}\right) \right)\)$ While classical uncertainty methods require 10 to 50 stochastic forward passes, empirical results demonstrate that setting \(N_{\text{pert}}=1\) (exactly one unperturbed pass plus one perturbed pass, totaling two inferences per image) achieves fully saturated rank correlation, delivering exceptional efficiency on large-scale 3D volumes.
A Worked Example¶
Consider a practitioner evaluating 3 candidate architectures (2D U-Net, Residual U-Net, and a fine-tuned µSAM) on an unlabelled target dataset of pancreatic tissue slices. 1. Perturbation: The input slice is normalized, and a perturbed counterpart is produced by adding mild Gaussian noise (\(\vartheta=0.05\)). 2. Dual Inference: Each candidate performs inference on both the original and perturbed slices. 3. Consistency Evaluation: - The well-generalized Residual U-Net maintains sharp, confident boundaries; only a tiny fraction of edge pixels fluctuate, yielding a high CMR-NHD of 0.88; - The overfitted 2D U-Net exhibits thin decision margins; noise induces extensive false-positive speckling, dropping its CMR-NHD to 0.54; - Computing median scores across all slices yields the ranking: Residual U-Net > µSAM > 2D U-Net. 4. Validation: This unsupervised ranking matches the exact order of true F1 scores measured subsequently against gold-standard ground truth.
Key Experimental Results¶
Main Results¶
The authors evaluated ranking correlation against true target F1 performance across electron microscopy (EM mitochondria segmentation, 4 target datasets, \(|M|=15\) models) and light microscopy (LM nuclei segmentation, 4 target datasets, \(|M|=7\) models). Correlation was quantified using Kendall's tau (\(K_\tau\)), Spearman's rank correlation (\(S_\rho\)), and Pearson's correlation coefficient (\(P_r\)).
| Setting | Method | Mitochondria \(K_\tau\) | Mitochondria \(S_\rho\) | Mitochondria \(P_r\) | Nuclei \(K_\tau\) | Nuclei \(S_\rho\) | Nuclei \(P_r\) |
|---|---|---|---|---|---|---|---|
| Unsupervised (Ours) | CMR-EI (Gauss) | 0.77 ±0.1 | 0.85 ±0.1 | 0.83 ±0.1 | 0.62 ±0.1 | 0.77 ±0.1 | 0.98 ±0.0 |
| Unsupervised (Ours) | CMR-NHD (Gauss) | 0.71 ±0.1 | 0.83 ±0.1 | 0.79 ±0.1 | 0.69 ±0.1 | 0.82 ±0.1 | 0.97 ±0.0 |
| Unsupervised (Ours) | CMR-EI (DropOut) | 0.73 ±0.1 | 0.86 ±0.1 | 0.86 ±0.1 | 0.74 ±0.2 | 0.85 ±0.1 | 0.90 ±0.1 |
| Unsupervised (Ours) | CMR-NHD (DropOut) | 0.69 ±0.1 | 0.85 ±0.1 | 0.84 ±0.1 | 0.71 ±0.2 | 0.84 ±0.1 | 0.62 ±0.4 |
| Unsupervised Baseline | Transfer Score (TS) | 0.25 ±0.2 | 0.30 ±0.3 | 0.23 ±0.3 | 0.02 ±0.3 | 0.10 ±0.4 | -0.09 ±0.4 |
| Unsupervised Baseline | Nuclear Norm (NuNo) | 0.17 ±0.2 | 0.20 ±0.3 | 0.13 ±0.3 | 0.09 ±0.5 | 0.17 ±0.5 | 0.08 ±0.5 |
| Unsupervised Baseline | Dispersion | -0.03 ±0.0 | -0.07 ±0.1 | -0.17 ±0.1 | -0.18 ±0.3 | -0.29 ±0.4 | -0.19 ±0.5 |
| Supervised Benchmark | LEEP (Needs Labels) | 0.89 ±0.1 | 0.96 ±0.0 | 0.97 ±0.0 | 0.67 ±0.4 | 0.77 ±0.3 | 0.94 ±0.0 |
| Supervised Benchmark | NLEEP (Needs Labels) | 0.41 ±0.4 | 0.49 ±0.5 | 0.39 ±0.3 | 0.11 ±0.1 | 0.12 ±0.1 | 0.32 ±0.2 |
| Supervised Benchmark | LogME (Needs Labels) | 0.08 ±0.2 | 0.10 ±0.3 | 0.06 ±0.3 | -0.20 ±0.3 | -0.28 ±0.4 | -0.49 ±0.4 |
| Supervised Benchmark | CCFV (Needs Labels) | -0.11 ±0.1 | -0.16 ±0.1 | -0.18 ±0.1 | -0.03 ±0.3 | -0.05 ±0.4 | -0.16 ±0.5 |
Ablation Study¶
The framework was further validated on the ToothFairy2 challenge (MICCAI 2024, 42-class 3D CBCT volume segmentation, 8 top challenge models) and on instance segmentation benchmarks (measured against ground truth mAP@[0.5:0.95]).
1. 3D Multi-Class Segmentation on ToothFairy2 (|M| = 8)¶
| Anatomical Structure | Metric | CMR-EI (Gauss) | CMR-NHD (Gauss) | CMR-NHD (Gamma) | TS (SOTA UDA) | NuNo | Dispersion |
|---|---|---|---|---|---|---|---|
| Inferior Alveolar Canals (IACs) | Kendall \(K_\tau\) | 0.90 | 0.81 | 0.90 | 0.21 | 0.14 | 0.07 |
| Teeth | Kendall \(K_\tau\) | 0.81 | 1.00 | 0.90 | 0.36 | 0.07 | 0.43 |
| Maxillary Sinuses (Sinus) | Kendall \(K_\tau\) | 0.43 | 0.81 | 0.90 | 0.57 | -0.07 | 0.34 |
| Overall (All 42 Classes) | Kendall \(K_\tau\) | 0.71 | 0.90 | 1.00 | 0.33 | 0.25 | 0.28 |
| Overall (All 42 Classes) | Spearman \(S_\rho\) | 0.86 | 0.96 | 1.00 | 0.49 | 0.28 | 0.50 |
| Overall (All 42 Classes) | Pearson \(P_r\) | 0.93 | 0.98 | 0.96 | 0.55 | 0.26 | 0.44 |
2. Instance Segmentation Model Ranking (Mean Correlation vs. mAP@[0.5:0.95])¶
| Target Modality | Metric | CMR-ARS (Gauss) | CMR-ARS (DropOut) | SEG (Centroid Dilation Baseline) |
|---|---|---|---|---|
| Cells (3 datasets, $ | M | =8$) | Kendall \(K_\tau\) | 0.69 ±0.15 |
| Cells | Spearman \(S_\rho\) | 0.83 ±0.09 | 0.83 ±0.06 | 0.16 ±0.68 |
| Cells | Pearson \(P_r\) | 0.90 ±0.04 | 0.91 ±0.04 | 0.31 ±1.02 |
| Nuclei (4 datasets, $ | M | =5$) | Kendall \(K_\tau\) | 0.72 ±0.25 |
| Nuclei | Spearman \(S_\rho\) | 0.83 ±0.21 | 0.85 ±0.24 | 0.53 ±0.49 |
| Nuclei | Pearson \(P_r\) | 0.79 ±0.17 | 0.84 ±0.11 | 0.13 ±0.72 |
Key Findings¶
- Failure of Classification Transferability Metrics: Conventional transfer metrics (LogME, NCTI, RegHscore, TS, NuNo) lean heavily on linear separability in latent embeddings. In dense biomedical segmentation, their Spearman correlation with ground truth drops below 0.3 or turns negative.
- Superiority in Instance Segmentation: CMR-ARS achieves robust Spearman correlations (\(S_\rho \ge 0.83\)), whereas the centroid-based SEG baseline fluctuates drastically (Pearson variance reaching ±1.02), confirming the essential role of pairwise pixel clustering for irregular cellular morphologies.
- Drastic Memory and Compute Efficiency: Competing metrics (TS, NuNo, Dispersion) require storing complete softmax distributions across 3D volumes, exceeding 200 GB for ToothFairy2. CMR-EI and CMR-NHD require only predicted class indices and maximal confidences, consuming minimal memory and saturating rank correlation with just \(N_{\text{pert}}=1\).
Highlights & Insights¶
- Black-Box Decision Margin Philosophy: By assessing perturbation sensitivity in prediction space rather than imposing handcrafted geometric assumptions on latent features, CMR completely decouples ranking from model architectures and training loss formulations.
- Adapted Rand Metric for Instance Ranking: Framing instance consistency via the Adapted Rand Score elegantly circumvents the permutation-invariance dilemma of arbitrary instance identifiers.
- Unsupervised Challenge Replication: On the competitive ToothFairy2 multi-class benchmark, CMR-NHD (Gamma) achieved a perfect Kendall tau (\(K_\tau = 1.00\)), demonstrating that challenges and model hubs can establish automated validation without revealing test ground truths.
Limitations & Future Work¶
- Task Alignment Assumption: CMR presumes candidate models share the same semantic definition as the target task. Generalist models (e.g., prompt-free SAM variants) may consistently segment non-target structures, producing artificially elevated consistency scores despite low task relevance.
- Susceptibility to Mode Collapse: Pathological models that collapse to constant foreground or constant background yield deceptively high consistency. Incorporating prediction entropy or diversity regularization would suppress these degenerate edge cases.
- Boundary-Aware Weighting: Current formulations weight all foreground pixels equally. Incorporating geodesic distance transforms or boundary-sensitive weight maps could enhance correlation with boundary metrics like Hausdorff distance and Boundary IoU.
Related Work & Insights¶
- vs. Supervised Transferability Metrics (LEEP, NLEEP, LogME): Prior methods require annotated target validation sets and fine-tuning assumptions; CMR operates in a fully unsupervised, source-free, zero-shot regime.
- vs. UDA Validation (Transfer Score): Transfer Score struggles on dense 3D volumes and consumes over 200 GB of RAM; CMR scales gracefully to multi-class 3D CT while delivering substantially higher rank correlation.
- vs. Unlabelled Instance QC (SEG): SEG simplifies instances to dilated centroids, breaking down on non-spherical cellular geometries; CMR-ARS retains full pixel-level topological sensitivity.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering black-box, source-free, unsupervised ranking framework unifying semantic and instance segmentation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Validated across 2D/3D modalities, light/electron microscopy, and CT, encompassing 15+ datasets and 20+ architectures.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous exposition with crisp conceptual intuition and mathematically grounded formulations.
- Value: ⭐⭐⭐⭐⭐ Highly practical tool for model repositories (e.g., BioImage Model Zoo) and clinical zero-shot model selection.