Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation¶
Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://lingrayy.github.io/MSSA/
Area: Medical Imaging
Keywords: Test-Time Adaptation, Medical Image Segmentation, Vision-Language Models, Training-Free, Prototype Matching
TL;DR¶
Addressing the vulnerability of test-time fine-tuning in vision-language models where noisy updates degrade pretrained representations, this paper proposes MSSA, a training-free synergistic adaptation framework that stabilizes coarse predictions, curates a noise-aware dynamic memory bank, and conducts cross-image prototype matching for structural guidance.
Background & Motivation¶
Recent breakthroughs in vision-language models (VLMs) and universal foundation models (such as BiomedCLIP and SAM) have revolutionized prompt-driven medical image segmentation. By aligning visual and textual anatomical representations via attention mechanisms, these models exhibit impressive zero-shot generalization across diverse tasks. Nevertheless, in clinical real-world deployments, severe domain shifts—originating from heterogeneous scanner hardware, multi-institutional acquisition protocols, and pathological variations—remain ubiquitous. When target test distributions deviate significantly from the pretraining distribution, zero-shot segmentation suffers from severe localization degradation, resulting in coarse attention boundaries and false-positive artifacts.
To mitigate such distribution shifts, test-time adaptation (TTA) has garnered substantial attention by adapting models using unlabeled target streams on-the-fly. However, conventional TTA paradigms are predominantly designed for unimodal vision foundation models, and directly porting them to multimodal VLM segmentation faces fundamental hurdles. Mainstream methods rely on self-supervised optimization—such as entropy minimization or pseudo-label consistency regularization—and update model parameters online (e.g., via Batch Normalization layers or LoRA adapters). In dense medical segmentation without ground-truth supervision, pseudo-label noise invariably corrupts the delicately pre-aligned vision-language representation space, triggering catastrophic error accumulation and irreversible performance collapse.
The key insight of this work stems from observing the complementary strengths of cross-modal and cross-image paradigms: VLM image-text grounding provides rich semantic priors for anatomical localization but produces structurally coarse masks under domain shifts; conversely, self-supervised visual features (e.g., DINOv2) preserve cross-image structural and boundary consistency across samples yet lack semantic anchoring to pinpoint target categories. Core idea: abandon parameter updates altogether and introduce MSSA, a training-free synergistic TTA framework that dynamically curates reliable image-text predictions into an online memory bank and leverages relevance-driven prototype alignment to transfer structural consistency onto query images.
Method¶
Overall Architecture¶
MSSA operates via a synergistic, non-parametric two-stage pipeline without gradient backpropagation: First, in the Image-Text Coarse Grounding stage, an anatomical descriptive prompt generated by an LLM is paired with BiomedCLIP to produce a saliency map, which is stabilized using Gaussian point sampling and dual majority voting to yield robust candidate segmentation masks. Second, in the Test-Time Adaptation stage, a noise-aware memory construction module screens candidate segmentations based on semantic alignment and spatial smoothness scores under a monotonically non-decreasing adaptive threshold, progressively populating a dynamic FIFO memory bank. Finally, for each query test image, the most feature-similar anchor sample is retrieved from memory to extract localized foreground and background prototypes, decoding a sharp and structurally consistent final segmentation mask via cosine similarity.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Test Image Stream & Text Prompt"] --> B["Image-Text Coarse Grounding Stabilization<br/>Gaussian Point Selection + Dual Majority Voting"]
B --> C["Noise-Aware Memory Construction<br/>Dual Semantic & Smoothness Quality Auditing"]
C -->|Pass Monotonic Adaptive Threshold| D["Dynamic Feature-Mask Memory Bank"]
A --> E["DINOv2 Visual Feature Extraction"]
E --> F["Relevance-Driven Non-Parametric Adaptation<br/>Nearest Anchor Retrieval + Local Prototype Matching"]
D --> F
F --> G["Final Accurate Segmentation Mask"]
Key Designs¶
1. Image-Text Coarse Grounding Stabilization: suppressing prompt randomness and attention instability
Conventional promptable segmentation pipelines (such as MedCLIP-SAMv2) draw point prompts uniformly or randomly from thresholded VLM saliency maps, making them vulnerable to noisy boundary outliers and resulting in inaccurate SAM mask predictions. To enforce reliable visual prompts, the authors propose Gaussian Point Selection. By computing the distance transform of the coarse saliency mask, points are sampled based on a 2D Gaussian kernel centered at the mask centroid \((x_0, y_0)\): $\(G_M(x, y) = \exp\left( -\frac{(x - x_0)^2 + (y - y_0)^2}{2\sigma^2} \right)\)$ where \(\sigma\) controls the spatial spread. This concentrates point prompts within the anatomical center of mass, yielding consistent SAM refinements. Furthermore, to combat cross-modal attention flickering under domain shifts, a dual majority voting scheme is applied: photometric augmentations (CLAHE, Gamma correction) are aggregated via voting for saliency generation, while geometric augmentations (flipping, scaling, rotation) are applied to SAM inputs, producing a stabilized candidate mask \(Y_{\text{cand}}\).
2. Noise-Aware Memory Construction: dual-criterion scoring with monotonic adaptive thresholding
Directly storing unverified pseudo-labels into a memory bank risks polluting reference prototypes with false-positive artifacts. The authors design a dual-criterion scoring function \(\mathcal{Q}_{\text{total}} = \mathcal{Q}_{\text{sem}} \cdot \mathcal{Q}_{\text{smo}}\) to evaluate candidates. The semantic alignment score \(\mathcal{Q}_{\text{sem}}\) measures the cosine similarity between the masked foreground image and the text prompt using BiomedCLIP embeddings to filter semantic mismatches. The spatial smoothness score \(\mathcal{Q}_{\text{smo}} = R_s + B_s\) integrates shape regularity \(R_s\) (area-to-perimeter compactness) with boundary smoothness \(B_s\) (derived from contour polygon vertex density), heavily penalizing fragmented, noisy, or jagged predictions.
To eliminate manual dataset-specific threshold tuning under varying domain shifts, an adaptive percentile thresholding rule is maintained: after a brief warm-up buffer of \(N_{\text{warm}} = 10\) samples, the threshold updates as \(\tau_t = \max(\tau_{t-1}, \operatorname{Percentile}(\mathcal{H}_t, P))\) with \(P=80\). The non-decreasing constraint ensures that the quality bar strictly ascends as superior targets appear, preventing transient distribution dips from admitting degraded pseudo-labels. Retained candidates are stored in a FIFO memory bank \(\mathcal{B} = \{(F_j, Y_j)\}\) of capacity 15, keyed by their DINOv2 visual features \(F_j\).
3. Relevance-Driven Non-Parametric Adaptation: parameter-free cross-image prototype matching
Unlike conventional TTA methods that fine-tune network weights and induce representation drift, MSSA harnesses DINOv2 representations to perform non-parametric structural adaptation. For an incoming query image, its feature \(F_t\) is matched against all stored keys via cosine similarity to retrieve the most visually aligned anchor pair \((F_a, Y_a)\). To handle anatomical heterogeneity across image regions, local adaptive prototypes (adapted from ALPNet) are extracted. Average pooling with a sliding window \(\Omega^{(m,n)}\) is applied over regions defined by anchor mask \(Y_a\) for both foreground (\(c=1\)) and background (\(c=0\)): $\(P_c^{(m,n)} = \frac{1}{|\Omega^{(m,n)}|} \sum_{(u,v) \in \Omega^{(m,n)}} F_a(u,v) \cdot \mathbb{I}[Y_a(u,v) == c]\)$ Cosine similarity maps \(M_l^c\) are computed between query features \(F_t\) and each local prototype \(P_l^c\), aggregated via element-wise maximum into class similarity maps \(\tilde{M}^1, \tilde{M}^0\), and normalized via Softmax to obtain the final prediction \(\hat{Y}_t = \operatorname{argmax} L\). This inference pipeline requires zero backward passes, eradicating catastrophic drift at test time.
Key Experimental Results¶
Main Results¶
MSSA was thoroughly evaluated across five public fundus datasets for optic disc segmentation (Domain A: RIM-ONE-r3, Domain B: REFUGE, Domain C: ORIGA, Domain D: REFUGE-Valid, Domain E: Drishti-GS) and three chest X-ray lung datasets (COVID-QU-Ex, MC, SZ) against leading fine-tuning TTA methods and zero-shot VLM baselines.
| Dataset / Task | Metric | Ours (MSSA) | Prev. SOTA TTA (TTCS) | Gain (vs TTCS) | Zero-Shot Baseline (MedCLIPv2) |
|---|---|---|---|---|---|
| Optic Disc (Domain A) | DSC (%) / mIoU (%) | 76.6 / 63.1 | 65.3 / 52.4 | +11.3 / +10.7 | 59.8 / 44.5 |
| Optic Disc (Domain B) | DSC (%) / mIoU (%) | 87.0 / 77.1 | 86.7 / 79.1 | +0.3 / -2.0 | 82.2 / 71.0 |
| Optic Disc (Domain C) | DSC (%) / mIoU (%) | 84.6 / 73.8 | 65.2 / 54.1 | +19.4 / +19.7 | 64.9 / 49.8 |
| Optic Disc (Domain D) | DSC (%) / mIoU (%) | 81.5 / 69.5 | 64.7 / 54.0 | +16.8 / +15.5 | 71.1 / 57.2 |
| Optic Disc (Domain E) | DSC (%) / mIoU (%) | 89.9 / 82.1 | 76.4 / 67.4 | +13.5 / +14.7 | 79.0 / 67.2 |
| Optic Disc (5-Domain Avg) | DSC (%) / mIoU (%) | 83.9 / 73.1 | 71.7 / 61.4 | +12.2 / +11.7 | 71.4 / 58.0 |
| Lung (COVID-QU-Ex) | DSC (%) / mIoU (%) | 73.2 / 58.7 | 57.8 / 44.6 | +15.4 / +14.1 | 54.6 / 40.7 |
| Lung (MC) | DSC (%) / mIoU (%) | 79.2 / 69.4 | 65.2 / 43.1 | +14.0 / +26.3 | 62.1 / 54.6 |
| Lung (SZ) | DSC (%) / mIoU (%) | 82.9 / 72.1 | 65.2 / 43.1 | +17.7 / +29.0 | 60.2 / 50.5 |
| Lung (3-Dataset Avg) | DSC (%) / mIoU (%) | 78.4 / 66.7 | 62.8 / 43.6 | +15.6 / +23.1 | 59.0 / 48.6 |
Ablation Study¶
Ablation experiments on the optic disc segmentation task demonstrate the cumulative contribution of each individual design component.
| Row | Component Config (SAM-MV / ATN-MV / GP / NMC / Similar) | Optic Disc Avg DSC (%) | Optic Disc Avg mIoU (%) | Note |
|---|---|---|---|---|
| 1 | Baseline (MedCLIPv2) | 71.4 | 58.0 | Standard zero-shot VLM model |
| 2 | + SAM-MV (SAM geometric majority voting) | 72.8 | 61.8 | Enforces spatial geometric robustness (+3.8% mIoU) |
| 3 | + ATN-MV (photometric majority voting) | 73.9 | 62.9 | Stabilizes initial cross-modal saliency map |
| 4 | + GP (Gaussian point selection) | 75.7 | 64.7 | Mitigates boundary noise, centers visual prompts |
| 5 | Unimodal image-image memory (+Bank) | 79.2 | 67.0 | Uses single warm-up anchor without VLM stream |
| 6 | Unimodal memory + selection (+Select) | 71.2 | 59.7 | Lacks cross-modal diversity; collapses on Domain D (18.6% mIoU) |
| 7 | Full MSSA w/ random anchor selection | 83.2 | 70.8 | Validates synergistic grounding and memory filtering |
| 8 | MSSA Full Model (w/ nearest feature similarity) | 83.9 | 73.1 | Best full pipeline achieving top metrics across all domains |
Key Findings¶
- Fine-tuning TTA fails or triggers negative transfer on dense VLMs: Mainstream gradient-based TTA methods (TENT, EATA, CoTTA, DELTA, SVDP) struggle when deployed on VLM segmentation backbones, delivering marginal improvements (+0.0% to +0.4% DSC) or severe drops (UniVPT plunged to 12.4% mIoU on lung X-rays). Gradient updates disrupt delicate image-text alignments, whereas training-free adaptation completely circumvents this collapse.
- Unimodal self-referential adaptation risks error cascading: As highlighted in Row 6 of the ablation table, relying solely on image-image self-iteration without continuous multimodal semantic grounding leads to catastrophic failure on Domain D (collapsing to 18.6% mIoU), proving that multimodal semantic anchoring and unimodal geometric alignment must act in tandem.
- Dual-faceted memory scoring provides essential complementarity: In memory scoring ablations, relying solely on \(\mathcal{Q}_{\text{sem}}\) admitted fragmented masks, while using only \(\mathcal{Q}_{\text{smo}}\) selected well-shaped but semantically erroneous false-positive regions. The fused score \(\mathcal{Q}_{\text{total}}\) boosted lung average DSC from ~69.8% to 78.4%.
Highlights & Insights¶
- Training-free paradigm eliminates parameter drift: By substituting gradient descent with dynamic memory curation and non-parametric prototype matching, the framework preserves foundational representations while dynamically adapting to unseen target shifts.
- Harmonious coupling of cross-modal and unimodal representations: Leverages VLM embeddings for what they do best (semantic class grounding) while relying on self-supervised DINOv2 features for what they excel at (dense structural correspondence), achieving a robust 1+1 > 2 synergy.
- Monotonic percentile thresholding: The adaptive non-decreasing threshold calibrates candidate filtering to the intrinsic hardness of the target domain without hyperparameter hand-tuning, preventing early low-quality predictions from corrupting the memory pool.
Limitations & Future Work¶
- Inference latency and computational overhead: Due to dual-augmentation majority voting and online sliding-window prototype extraction, MSSA requires approximately 12.1 seconds per test image on an RTX A6000 GPU, compared to 1.7–5.9 seconds for lightweight fine-tuning methods. Optimizing this pipeline is necessary for time-critical emergency applications.
- Restricted applicability to intricate tubular structures: Current validations center on organs with relatively regular geometries (such as optic disc and lungs). For highly branched, tortuous structures like micro-vascular networks, the spatial smoothness criterion may penalize true anatomical complexity, warranting future exploration of fine-grained topological priors.
Related Work & Insights¶
- vs TTCS [BIBM 2024]: While TTCS pioneered test-time adaptation for medical VLMs, it relies on online LoRA parameter updates that are susceptible to error accumulation. MSSA demonstrates that a training-free prototype memory mechanism not only prevents drift but also outperforms TTCS by 12.2% DSC on average.
- vs Traditional Source-Free / TTA (TENT / CoTTA / DELTA): Traditional approaches depend on BN statistics updates or entropy minimization, which easily degrade on dense medical prediction. MSSA establishes an effective retrieval-based, training-free alternative for dense adaptation.
- vs Few-Shot Prototype Segmentation (ALPNet / ProtoSAM): ALPNet requires an annotated source-domain support set. MSSA removes this prerequisite by dynamically self-curating a pseudo-support set from unlabeled test streams via noise-aware multimodal auditing.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneers a training-free TTA paradigm for dense medical VLM segmentation, resolving the parameter drift bottleneck]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 5 fundus domains and 3 cross-site/pathology lung benchmarks with thorough ablations and runtime profiling]
- Writing Quality: ⭐⭐⭐⭐⭐ [Cohesive narrative, rigorous formulation of dual-quality scoring, and clear structural motivation]
- Value: ⭐⭐⭐⭐⭐ [Offers a robust, parameter-freeze solution for real-world clinical deployment under severe domain shifts]