title: >- [Paper Note] SPDA: Efficient Online Test-Time Adaptation for Promptable Medical Segmentation description: >- [ECCV 2026][Medical Imaging][Test-Time Adaptation] SPDA introduces a lightweight online test-time adaptation framework for promptable medical segmentation foundation models. By coupling domain-adaptive low-frequency radial-spectrum perturbation with prompt under-sampling perturbation, it generates reliable consistency signals and updates only a lightweight post-encoder adapter, eliminating encoder backpropagation overhead. tags: - "ECCV 2026" - "Medical Imaging" - "Online Test-Time Adaptation" - "Foundation Models" - "Promptable Segmentation" - "Frequency Domain Perturbation" - "SAM2" date: 2026-09-19 content_hash: c74970bd195098f6

SPDA: Efficient Online Test-Time Adaptation for Promptable Medical Segmentation¶

Conference: ECCV 2026
Paper: ECCV Official
Code: To be released
Area: Medical Imaging
Keywords: Promptable Foundation Model, Online Test-Time Adaptation, Medical Image Segmentation, Dual-Perturbation Consistency, Lightweight Post-Encoder Adapter

TL;DR¶

Addressing performance degradation of general and medical foundation segmentation models under clinical sub-domain shifts, SPDA presents an efficient online test-time adaptation framework that constructs reliable self-supervised learning signals via domain-adaptive low-frequency radial-spectrum perturbation and prompt under-sampling, while updating only a lightweight post-encoder adapter to bypass heavy encoder backpropagation.

Background & Motivation¶

Promptable foundation segmentation models such as SAM and SAM2 have exhibited exceptional zero-shot segmentation capabilities across natural visual scenes. Their medical adaptations, including MedSAM and MedSAM2, have substantially alleviated natural-to-medical domain discrepancy through extensive medical pre-training. Nevertheless, during actual clinical deployment, even domain-specialized models suffer severe performance degradation when evaluated on unseen clinical data from diverse scanners, hospitals, imaging protocols, and anatomical structures due to substantial sub-domain shifts.

Online test-time adaptation (OTTA) provides a viable paradigm to dynamically refine models on incoming unannotated test streams without accessing source domain data. Existing medical OTTA approaches, however, encounter two fundamental bottlenecks in clinical practice. First, constructing dependable unsupervised learning signals remains challenging. Feature-alignment methods require accessing source domain statistics, which introduces patient privacy concerns and hinders adaptation to target-specific distributions. Consistency-based techniques relying on traditional high-frequency data augmentations often violate anatomical semantics, as radially averaged power spectral density (RAPSD) reveals that intra-sub-domain variations in medical images predominantly lie in low-frequency bands. Conversely, student-teacher inference perturbations suffer from low perturbation diversity and error accumulation from confirmation bias. Second, edge deployment in clinical hardware imposes severe computational and memory constraints. Conventional fine-tuning and popular OTTA algorithms backpropagate through the entire vision encoder, incurring prohibitive activation caching memory and latency overhead, an issue that internal PEFT modules like LoRA fail to resolve.

To resolve the dual dilemma between supervision signal fidelity and hardware efficiency, this paper proposes Semantic-Preserving Dual-Perturbation Adaptation (SPDA). Core idea: exploit the spectral prior that medical sub-domain variations concentrate in low frequencies to introduce domain-adaptive low-frequency radial-spectrum perturbation along with prompt under-sampling for robust semantic-preserving consistency, while keeping the entire foundation model frozen and updating solely a lightweight post-encoder adapter to completely circumvent backpropagation through the heavy vision encoder.

Method¶

Overall Architecture¶

SPDA is tailored for promptable segmentation foundation architectures comprising an image encoder, prompt encoder, mask decoder, and memory modules. For a streaming sequence of unlabeled test samples, all original parameters of the pre-trained foundation model are frozen. A lightweight bottleneck adapter placed immediately after the image encoder is the only trainable module. During test-time inference on each incoming slice, SPDA constructs three concurrent forward pathways via dual semantic-preserving perturbations: the primary pathway producing anchor predictions, the input-perturbed pathway producing low-frequency perturbed features and masks, and the inference-perturbed pathway generating prompt-undersampled masks.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Streaming Test Inputs<br/>Image Slices and Interactive Prompts"] --> B["Domain-Adaptive Low-Frequency Perturbation<br/>EWMA Statistics and Spectrum Scaling"]
    A --> C["Prompt Under-Sampling Perturbation<br/>Box Expansion and Point Subset Sampling"]
    A --> D["Frozen Large Image Encoder<br/>Original Feature Extraction"]
    B --> E["Frozen Large Image Encoder<br/>Perturbed Feature Extraction"]
    D --> F["Lightweight Post-Encoder Adapter<br/>Trainable Parameters Only"]
    E --> F
    F --> G["Feature Consistency Constraint<br/>Multi-Level Cosine Dissimilarity"]
    F --> H["Frozen Prompt Encoder & Mask Decoder<br/>Multi-View Segmentation Prediction"]
    C --> H
    H --> I["Prediction Consistency Constraint<br/>Dice Loss Between Anchor and Perturbed Views"]
    G --> J["Joint Optimization Backward Pass<br/>Updates Only Post-Encoder Adapter"]
    I --> J
    J --> K["Adapted Final Segmentation Mask"]

Key Designs¶

1. Domain-Adaptive Low-Frequency Radial-Spectrum Perturbation: Simulating Sub-Domain Shifts Without Semantic Distortions

Conventional spatial augmentations perturb fine-grained high-frequency details, causing out-of-domain distortions and gradient misalignment in medical segmentation. Motivated by radially averaged power spectral density (RAPSD) findings showing that clinical sub-domain variations reside primarily in low frequencies, SPDA performs a 2D Fourier transform on each image slice and partitions the frequency plane into $K$ concentric radial bands. By tracking target-domain statistics via Exponentially Weighted Moving Average (EWMA) and Variance (EWVar), it dynamically samples perturbation factors from an adaptive log-normal distribution combining intra-sample and inter-sample variance. Perturbations are strictly confined to the lowest $K_{low}$ radial bands by modulating the real Fourier coefficients before inverse transformation, thereby capturing style variations while preserving critical pathological and anatomical boundaries.

2. Prompt Under-Sampling Perturbation: Generating Diverse Semantic-Invariant Inference Views

Rather than perturbing visual pixels alone, the interactive prompt interface provides a separate dimension for constructing unbiased self-supervised learning signals. Standard additive noise can easily displace bounding boxes or points away from the region of interest. SPDA introduces a semantic-preserving prompt under-sampling mechanism: for bounding boxes, it isotropically enlarges the area by a scaling factor $\zeta > 1$, maintaining complete coverage of the target structure while diminishing localization specificity; for sets of $N \ge 2$ point prompts, it randomly samples a strict subset $N' < N$ points. Because under-sampled prompts unambiguously refer to the identical anatomical target, predictions from both views should achieve strong mutual consensus without adding parameter overhead.

3. Lightweight Post-Encoder Adapter: Bypassing Image Encoder Backpropagation for Real-Time Execution

In standard test-time adaptation and internal PEFT methods such as LoRA or series adapters, gradient computation still requires backpropagating through deep vision Transformer layers, necessitating the storage of massive intermediate activation maps. SPDA designs a lightweight bottleneck convolutional adapter placed exclusively at the output of the image encoder, leaving the pre-trained image encoder, prompt encoder, and mask decoder entirely frozen. Gradients terminate at the adapter, entirely eliminating the storage and backpropagation burden of the encoder backbone and reducing per-slice compute from several hundred TFLOPs to 39.88 TFLOPs. Furthermore, to prevent catastrophic forgetting during continuous streaming adaptation, SPDA stochastically restores 1% of the adapter weights to their initial pre-adaptation state on each sample.

Loss & Training¶

During the online test phase, SPDA receives sequential mini-batches of batch size 1 and performs a single optimization iteration per batch. The overall objective comprises two semantic-preserving consistency terms:

The multi-level feature consistency loss $\mathcal{L}_f$ encourages the representations produced by the adapter to remain robust against low-frequency style fluctuations across all feature levels $l \in \{1, \dots, L\}$: $$ \mathcal{L}f = \frac{1}{L} \sum\right) \right) $$}^{L} \left( 1 - \cos\left(\mathbf{z}_d^{(l)}, \widetilde{\mathbf{z}}_d^{(l)

The multi-perturbation prediction consistency loss $\mathcal{L}_p$ aligns the anchor mask $\mathbf{y}_d$ with the input-perturbed prediction $\widetilde{\mathbf{y}}_d$ and prompt-undersampled prediction $\bar{\mathbf{y}}_d$ using soft Dice loss: $$ \mathcal{L}p = \frac{1}{2} \left( \mathcal{L}}}(\mathbf{yd, \bar{\mathbf{y}}_d) + \mathcal{L}_d) \right) $$}}(\mathbf{y}_d, \widetilde{\mathbf{y}

The total adaptation objective is formulated as: $$ \mathcal{L} = \lambda_f \mathcal{L}_f + \lambda_p \mathcal{L}_p $$ Default hyperparameter values across all experiments: total radial bands $K = 24$, low-frequency cutoff $K_{low} = 8$, resolution parameter $\gamma = 1.7$, EWMA momentum $\eta = 0.1$, variance weights $\lambda_{intra} = 0.3, \lambda_{inter} = 0.7$, box scaling factor $\zeta = 1.005$, under-sampled points $N' = 1$, and loss balancing weights $\lambda_f = 0.1, \lambda_p = 1.0$. The adapter is updated using the AdamW optimizer with a single gradient step.

Key Experimental Results¶

Main Results¶

Experiments were conducted on the benchmark dataset from the CVPR 2024 Segment Anything in Medical Images Challenge, comprising 11 imaging modalities (CT, Dermoscopy, Endoscopy, Fundus, Mammography, Microscopy, MR, OCT, PET, Ultrasound, and XRay). Models were evaluated under SAM2 and MedSAM2 backbones with box and point prompts. The primary evaluation metric is Dice Similarity Coefficient (DSC, %).

Backbone / Prompt Setup	Metric	Source-Only	InTEnt (CVPRW'24)	VPTTA (CVPR'24)	GraTa (AAAI'25)	SAM-TTA (arXiv'25)	SPDA (Ours)
SAM2 (Box Prompt)	Avg. DSC (%)	67.11	67.49	72.49	72.78	72.16	74.18 (+1.40)
SAM2 (3-Click Point)	Avg. DSC (%)	51.48	51.41	52.88	55.61	55.93	61.44 (+5.51)
MedSAM2 (Box Prompt)	Avg. DSC (%)	83.36	83.07	85.92	86.26	85.69	88.05 (+1.79)
MedSAM2 (Point Prompt)	Avg. DSC (%)	78.02	77.74	78.72	79.05	79.01	81.11 (+2.06)

Resource and computational efficiency comparison under the MedSAM2 configuration:

Method	Computational Cost (TFLOPs) ↓	Trainable Parameters (M) ↓
InTEnt	204.56	38.97
VPTTA	233.20	38.98
GraTa	497.18	38.99
SAM-TTA	63.65	78.04
SPDA (Ours)	39.88	39.16

Ablation Study¶

Ablation analysis on component contributions and optimization strategies using MedSAM2 with box prompts:

Optimization Strategy / Components	Input Perturbation	Prompt Perturbation	Avg. DSC (%) ↑	Cost (TFLOPs) ↓	Note
Baseline (No Adaptation)	None	None	80.07	-	Raw MedSAM2 zero-shot inference
Prompt Perturbation Only	None	Enabled	81.45	39.88	Consistent +1.38% gain
Traditional Aug. + Prompt Pert.	RandAugment	Enabled	83.78	39.88	High-frequency distortion hurts boundaries
Input Perturbation Only	Domain-Adaptive Low-Freq	None	83.91	39.88	Low-frequency perturbation drives core gain
SPDA Full Model	Domain-Adaptive Low-Freq	Enabled	85.04	39.88	Dual perturbations provide optimal synergy
LayerNorm Tuning	LayerNorm within Encoder	Enabled	80.85	77.61	Full encoder backward pass required
LoRA Tuning	LoRA inserted in Encoder	Enabled	85.11	78.90	Comparable DSC (+0.07%) at double compute

Key Findings¶

Crucial Role of Low-Frequency Perturbation: Injecting domain-adaptive low-frequency radial-spectrum perturbation alone lifts the DSC from 80.07% to 83.91%, whereas traditional high-frequency data augmentations achieve only 83.78%, confirming that preserving high-frequency anatomical structures is vital in medical TTA.
Halving Computation at Zero Performance Drop: The post-encoder adapter consumes only 39.88 TFLOPs per sample, approximately half the compute of LoRA (78.90 TFLOPs) and LayerNorm optimization (77.61 TFLOPs), while matching LoRA's segmentation performance (85.04% vs. 85.11%).
Modality-Specific Frequency Tuning: While a global setting of $K_{low} = 8, \gamma = 1.7$ provides strong general performance, exploring modality-specific spectral parameters unlocks further accuracy. For instance, MR modalities favor narrower low-frequency bands ($K_{low} = 4, \gamma = 1.5$), yielding an impressive +5.05% DSC gain.

Highlights & Insights¶

Grounding Test-Time Adaptation in Frequency Spectrum Priors: By recognizing that medical domain variations reside primarily within the low-frequency spectrum, the method replaces blind spatial augmentations with principled radial Fourier perturbations that protect delicate structural boundaries.
Architectural Decoupling for Lightweight Adaptation: Placing the adapter downstream of the vision encoder resolves the fundamental efficiency bottleneck of backpropagation-based OTTA, slashing memory footprint and computational overhead for resource-constrained edge clinical environments.
Exploiting Prompt Redundancy for Self-Supervision: The under-sampling strategy provides semantic-invariant prediction views via simple bounding box expansion and point subset selection, establishing effective pseudo-supervision without extra architectural complexity.

Limitations & Future Work¶

Dependency on Modular Architectural Separation: The framework assumes separate image encoder, prompt encoder, and mask decoder modules. Adapting it to fully fused, unified end-to-end vision-language architectures remains an open problem.
Limited Modeling of Severe Geometric Deformations: Low-frequency spectrum scaling predominantly adjusts contrast and intensity distributions; severe organ deformations and non-linear anatomical dislocations may require complementary deformation modeling.
Future Directions: Exploring instance-adaptive frequency band selection via lightweight gating to dynamically tune radial perturbation boundaries for challenging clinical anomalies.

vs. VPTTA: VPTTA requires accessing source-domain feature statistics to optimize a visual prompt, raising privacy risks; SPDA operates entirely source-free via dual-perturbation consistency, simultaneously improving adaptability to target distributions.
vs. GraTa: GraTa enforces implicit gradient alignment between entropy and consistency objectives, incurring heavy backpropagation across the entire encoder (497.18 TFLOPs); SPDA reduces compute to 39.88 TFLOPs through its post-encoder bottleneck adapter.
vs. SAM-TTA: SAM-TTA adopts a student-teacher design that increases parameter count to 78.04M and suffers from confirmation bias; SPDA achieves higher stability and superior segmentation accuracy with only 39.16M parameters.

Rating¶

Novelty: 4.5/5.0 (Principled integration of low-frequency radial spectral perturbations with prompt under-sampling and a post-encoder bottleneck adapter)
Experimental Thoroughness: 5.0/5.0 (Extensive validation across 11 imaging modalities, 2D/3D settings, box and point prompts, with comprehensive ablations)
Writing Quality: 4.5/5.0 (Clear motivation, rigorous mathematical formulation, and well-structured empirical validation)
Value: 4.5/5.0 (Addresses real-time computational bottlenecks of foundation models on clinical edge devices with practical significance)