Training-free Cross-domain Few-shot Segmentation via Robust Semantic Representation and Matching¶

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/Sparkling-Water/RSRM
Area: Segmentation
Keywords: cross-domain few-shot segmentation, training-free, semantic representation, support enhancement, hybrid prototype matching

TL;DR¶

By recombining intermediate features from frozen DINOv3, enhancing support representations with reliable query regions, and fusing prototype matching at multiple granularities, this method achieves average 1-shot / 5-shot mIoU of 68.39 / 73.08 across four target domains without task-specific training or target-domain fine-tuning.

Background & Motivation¶

Few-shot segmentation identifies regions of a target class in a query image using a small number of support images with pixel-level annotations. Conventional methods often assume that training and test classes differ but their image distributions remain similar; cross-domain few-shot segmentation also has to cross appearance gaps between natural, satellite, and medical images. Prototype methods compress support regions into representative vectors, affinity methods preserve finer pixel correspondences, and source-domain training or target-domain fine-tuning adapts the representations to the task. These approaches can all allow limited task data to alter otherwise general features, so a stronger pretrained encoder does not necessarily deliver stronger cross-domain performance. The paper provides a direct example: LoEC achieves average 1-shot mIoU of 65.01 with a standard ViT-B, but 63.37 after replacing it with DINOv3-B, as shown in Table 1.

The proposed approach therefore preserves pretrained knowledge and computes representations and matching rules within each test episode, without learning new adaptation parameters. Freezing alone is insufficient, however: the final DINOv3 features may emphasize local positional consistency rather than the semantic information most useful for separating foreground from background. Even when support and query images contain the same class, instance shape and background composition can differ, making direct comparison with a single mean support prototype unreliable. Furthermore, a global prototype loses regional variation, whereas matching every pixel can be influenced by incidental local similarities; different target domains need different matching granularities.

Instead of delegating all these problems to a SAM mask decoder, the authors construct an interpretable test-time pipeline around a single vision encoder. The pipeline answers three concrete questions in sequence: which layers to read out, which query information to trust, and which granularity should represent the support class. Here, training-free refers to the downstream segmentation procedure; it neither negates DINOv3's large-scale pretraining nor removes the need for pixel annotations on support images. Core Idea: select semantically discriminative intermediate layers, reduce support-query discrepancies through class-restricted query aggregation, and complete cross-domain segmentation with adaptive multi-granularity matching.

Method¶

Overall Architecture¶

Each test episode contains a support set of \(K\) annotated images and a query image whose ground-truth mask is not provided to the method. The output is a binary foreground mask for the query image, with support annotations specifying the class to segment. Section 3.1 states that no additional source-domain data is used for training; query ground truth is used for evaluation, not for the layer or threshold selection below. Frozen DINOv3 extracts multi-layer patch tokens from support and query images, and these tokens are reshaped into spatial feature maps. Semantic-aware Feature Re-fusion (SAFR) first creates a consistent feature readout for both sides. Adaptive Support Enhancement (ASE) then lets support features absorb reliable same-class information from the query image. Hybrid Prototype Matching (HPM) finally compares global, regional, and pixel-level prototypes and combines their query predictions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Support images and masks<br/>Query image"] --> B["Frozen DINOv3<br/>Multi-layer features"]
    B --> C["Semantic-aware Feature Re-fusion<br/>SAFR"]
    C --> D["Adaptive Support Enhancement<br/>ASE"]
    C -->|Query features| E["Hybrid Prototype Matching<br/>HPM"]
    D -->|Enhanced support features| E
    E --> F["Query foreground mask"]

Key Designs¶

1. Semantic-aware Feature Re-fusion: change the feature readout, not the frozen network's internal computation

SAFR starts from the feature analysis on pages 6-7 and in Figure 3. The authors select the highest-variance channels in the final layer and observe that they often exhibit position-related patterns rather than clear foreground-background separation. These channels form a position-sensitive channel set of size \(N_c=\operatorname{round}(C/\gamma)\), where \(C\) is the channel count. For the raw output of each attention module, the method computes \(r^l_{attn}\), the sum of variances in this channel set divided by the sum across all channels. This measures the relative contribution of selected position-sensitive channels to the current feature, not attention entropy or query prediction confidence. It also computes the Fisher discriminant ratio \(f^l_{attn}\), the trace ratio of between-class to within-class scatter matrices, to measure foreground-background separability. In Figure 3, larger position-sensitive variance proportions generally accompany lower Fisher discriminant ratios and segmentation performance, but this is an empirical association rather than a guarantee for arbitrary data.

Layers whose variance proportions exceed the across-layer mean are marked as position-aware layers and used to partition the candidate intervals. The layer with the smallest variance proportion between adjacent position-aware layers is selected as a representative semantic-aware layer; the paper also appends \(L+1\) as the terminal boundary. This avoids both indiscriminately averaging all layers and changing the input distribution received by subsequent layers. Selected layers are weighted by their Fisher discriminant ratios divided by their position-sensitive variance proportions, favoring readouts with stronger separation and weaker positional interference. The core fusion rule is Equation (4) on page 8:

\[ \bar{\mathbf F}=\sum_{l\in\mathcal L_{sem}}\omega^l\mathbf F^l_{attn},\qquad \omega^l=\frac{f^l_{attn}/r^l_{attn}}{\sum_{l'\in\mathcal L_{sem}}f^{l'}_{attn}/r^{l'}_{attn}}. \]

The layer set and fusion weights are computed from support features and then applied to both support and query features, placing them in the same matching space. Support annotations provide the basis for assessing semantic separability; this stage should not be interpreted as choosing layers using query ground truth. Re-fusion changes the feature readout, while DINOv3 continues to execute its original frozen forward pass. Table 3 supports this distinction: reducing residual feature weights or the variance of related channels hurts performance, whereas removing position-sensitive channels from the final output helps but remains weaker than SAFR.

2. Adaptive Support Enhancement: absorb information only from reliable same-class query regions

Instance differences may remain after SAFR, so ASE does not simply use the support prototypes to produce the final mask. It first applies masked average pooling to support foreground and background separately, producing two prototypes, then computes cosine similarities with support and query features followed by softmax. The support similarity map can be compared with known support ground truth, enabling episode-specific threshold selection. Specifically, \(N_t\) thresholds are sampled uniformly within \((0,1)\) and used to binarize the support foreground similarity map. The threshold \(t_f^*\) giving the highest support prediction mIoU is selected, but this support-optimal threshold is not applied directly to the query image. The authors observe that support features are usually more similar to their own prototypes than query features are, motivating a further robust-region threshold construction. Equation (5) averages all candidate thresholds satisfying \(t>\alpha t_f^*\) to obtain \(t_f^{rob}\); the same procedure is applied independently to the background. This is an average over a candidate set, not simply the optimal threshold multiplied by \(\alpha\), and the resulting threshold is not guaranteed to be lower than the optimum.

After generating reliable foreground and background masks, support features act as attention queries, while query-image features act as keys and values. The query image and the query in attention terminology are therefore different concepts: the representation being updated belongs to the support image. The module uses no trainable projection layers and adds a class-matching mask to the attention matrix. Only support-foreground to reliable-query-foreground and support-background to reliable-query-background connections are allowed; other connections are blocked with negative infinity. The aggregation rule is Equation (7) on page 9:

\[ \mathbf A_{sq}=\operatorname{softmax}\left(\frac{\mathbf Q\mathbf K^T}{\sqrt C}+\mathbf M_{sq}\right)\mathbf V. \]

The mask \(\mathbf M_{sq}\) is zero at allowed positions and negative infinity elsewhere; softmax normalizes over the query-image positions available for aggregation. The aggregated result is reshaped into a spatial feature map and averaged with the original support features using equal weights to produce enhanced support features. This brings support prototypes closer to the appearance of the current query while keeping clearly irrelevant query regions out of the foreground representation. Reliable regions still come from model similarities rather than ground truth, so ASE reduces incorrect interactions without guaranteeing perfectly correct pseudo-labels.

3. Hybrid Prototype Matching: let multiple semantic granularities jointly determine the foreground

HPM constructs three kinds of prototypes from enhanced support features, while the query side continues to use the query features produced by SAFR. Global prototypes average all support foreground and background features separately, giving stable but coarse class representatives. Regional prototypes use k-means++ clustering with \(N_r\) foreground clusters and \(2N_r\) background clusters. More centers are allocated to background because it usually contains more semantic categories, rather than assuming that foreground and background are equally complex feature groups. Pixel-level prototypes retain every support foreground or background feature, providing more candidates for fine-grained correspondence. For each granularity and class, HPM computes cosine similarities between all prototypes and each query pixel, then takes the maximum across prototypes. The regional and pixel branches therefore retain the strongest match rather than averaging similarities across all candidates.

Each granularity first compares its own foreground and background similarities to obtain a temporary query mask. It then computes the mean of its foreground similarity map inside the predicted foreground region and inside the predicted background region separately. Their difference, \(\omega_m=\mu_{m,f}-\mu_{m,b}\), measures whether this granularity makes predicted foreground distinct from predicted background. Both means come from the foreground similarity map of that granularity, not from the foreground and background maps respectively. The three differences are normalized with softmax into fusion weights, which separately combine the foreground and background similarity maps. The final prediction compares the two fused maps pixel by pixel and labels the higher-foreground-score positions as the target, as described by Equations (8)-(9) on page 10. This weight reflects predicted separation on the current query; it is neither accuracy calibrated with labels nor a fixed set of coefficients learned for each dataset. The different granularity preferences across domains in Table 5 provide empirical motivation for fusion rather than imposing a single prototype type everywhere.

Loss & Training¶

The method introduces no segmentation training loss, optimizer iterations, or target-domain gradient updates; support annotations participate in statistical computations and threshold selection. Fisher discriminant ratios, threshold search, clustering, and attention aggregation are all test-time computations, so no trainable parameters does not mean no adaptation cost. Page 10 specifies DINOv3 ViT-B/16 with images resized to \(480\times480\). The main hyperparameters are \(\gamma=200\), \(\alpha=0.75\), \(N_t=20\), and \(N_r=9\), giving 18 background clusters for regional prototypes. The paper reports both 1-shot and 5-shot support sizes, but the available main-paper cache does not fully detail multi-support aggregation or every implementation edge case. Reproduction should check the authors' code for empty reliable regions and insufficient clustering samples rather than treating missing details as rules established by the paper.

Key Experimental Results¶

Main Results¶

The following results come from Table 1 on page 11 of the paper PDF; the metric is mIoU, with higher values indicating better performance. Results are averaged over 5 random seeds; each seed samples 1200 episodes from each of Deepglobe, ISIC, and Chest X-ray, and 2400 from FSS-1000. This table uses the original ViT-B version of LoEC as the main comparator, whereas the proposed method uses DINOv3-B, so the entire difference cannot be attributed to the proposed modules.

Target domain	LoEC 1-shot	Ours 1-shot	LoEC 5-shot	Ours 5-shot
Deepglobe	42.12	49.71	51.48	58.88
ISIC	52.91	55.73	62.43	62.55
Chest X-ray	83.94	85.44	84.12	87.01
FSS-1000	81.05	82.67	83.69	83.89
Four-domain average	65.01	68.39	70.43	73.08

The average gains are 3.38 and 2.65 mIoU points, respectively, rather than relative percentage improvements. Matcher with DINOv3-B+SAM-B averages 61.20 / 67.27 in Table 1, giving corresponding gains of 7.19 / 5.81 points for the proposed method. However, Matcher with DINOv2-B+SAM-B averages 61.75 in 1-shot, so 7.19 points is not the gain over the best 1-shot result among all Matcher variants. Best four-domain average also does not mean best in every domain: on FSS-1000 in 5-shot, Matcher with DINOv2-B+SAM-B reaches 88.52, exceeding the proposed method's 83.89. DFN also slightly exceeds the proposed method on Chest X-ray in 5-shot, at 87.14 versus 87.01.

Ablation Study¶

The component ablation is Table 2 on page 12, reporting four-domain average mIoU; it provides more direct evidence of module contributions than comparisons across backbones.

Config	1-shot	5-shot
Baseline	58.74	64.54
Baseline + SAFR	62.53	68.64
Baseline + SAFR + ASE	66.16	70.86
Baseline + SAFR + HPM	65.99	71.61
Baseline + SAFR + ASE + HPM	68.39	73.08

SAFR alone adds 3.79 / 4.10 points; adding ASE to SAFR gives 3.63 / 2.22 points, while adding HPM gives 3.46 / 2.97 points. The latter two gains share the SAFR configuration as their reference and must not be mistaken for additive gains along a single sequential module-addition path. The further prototype analysis comes from Table 5 on page 13, entirely under 1-shot, using representations that already include SAFR and ASE.

Prototype type	Deepglobe	ISIC	Chest X-ray	FSS-1000	Average
Global	49.48	55.98	80.14	79.03	66.16
Regional	47.98	54.06	85.48	82.80	67.58
Pixel	47.65	51.03	84.55	83.64	66.72
HPM	49.71	55.73	85.44	82.67	68.39

Key Findings¶

In Table 5, global prototypes perform best on ISIC, regional prototypes on Chest X-ray, and pixel prototypes on FSS-1000; HPM improves the average without achieving every per-domain maximum.
Table 4 on page 13 reports averages of 63.76, 64.53, 65.95, and 66.16 for no mask, a fixed threshold, the best support threshold, and the robust threshold, respectively, supporting class-restricted interaction and threshold adaptation.
Table 8 on page 14 reports 161.9 GFLOPs, 47.62 FPS, and zero trainable parameters for the proposed method; LoEC uses 150.1 GFLOPs and reaches 52.91 FPS, so the proposed method is not faster than LoEC at inference.
The same table reports 727.6 GFLOPs and 0.47 FPS for Matcher, showing that training-free methods can still differ greatly in inference cost; experiments use an RTX 4090.

Highlights & Insights¶

SAFR separates freezing an encoder from being restricted to its final-layer output. The reusable idea is to select feature readouts by task separability without disturbing pretrained inter-layer distributions.
ASE uses support ground truth to calibrate the entrance to feature interaction rather than only thresholding the final output. This restricts potential foreground-background confusion before aggregation.
HPM does not assume that a single representational granularity is optimal across domains. Its fusion criterion comes from separation on the current query, offering a concrete alternative to learning a fusion network.

Limitations & Future Work¶

The authors attribute limited gains from stronger backbones in existing methods to overfitting; backbone-replacement results support this interpretation but do not rule out mismatched adaptation strategies or hyperparameters.
Training-free removes the proposed downstream parameter-fitting process, not pretraining bias, episode-selection bias, or possible hyperparameter dependence on evaluation domains.
Reliable query regions and granularity weights are both derived from the method's own predictions, so incorrect but well-separated predictions may still be reinforced; the main paper does not establish systematic robustness boundaries for these cases.
Experiments cover four common cross-domain datasets, which is insufficient to guarantee equivalent performance across arbitrary imaging protocols, very small objects, or complex multi-class settings; this limits the evidence rather than describing reported failure results.
The available cache contains main text and references on pages 1-18, not the supplementary material repeatedly cited in the paper; finer dataset splits, complete hyperparameter sensitivity, and multi-support aggregation details cannot be confirmed from it.

Compared with LoEC: LoEC trains its backbone on the source domain and emphasizes low-level features, whereas this method preserves a frozen backbone and selects semantic feature readouts; the backbone replacements in Table 1 and internal ablations in Table 2 should be interpreted separately.
Compared with PATNet and ABCDFSS: freezing a backbone is not equivalent to being training-free, since the former still trains mapping or decoding components and the latter fine-tunes adaptation layers; the proposed method replaces downstream adaptation with support-driven statistical computation.
Compared with Matcher and GF-SAM: these approaches combine cross-image matching with SAM segmentation, whereas the proposed method directly produces masks through prototype matching; Table 6 on page 13 shows that incorporation into a SAM-based system does not guarantee improvement over using the proposed similarities alone.
Research direction: support-label-driven feature readout selection and reliable interaction could be studied in other dense matching tasks, but whether position-sensitive channels cause the same semantic interference must be revalidated rather than assumed from this paper.

Rating¶

Novelty: 4/5. Three designs form a coherent frozen-encoder framework for cross-domain matching, although prototype fusion and support-query interaction have established precedents.
Experimental Thoroughness: 4/5. Four domains, two shot settings, component and mechanism ablations, and efficiency comparisons are covered, but complete variance reporting and supplementary details are absent from the available main paper.
Writing Quality: 4/5. The method is clearly organized, but average versus per-domain superiority, comparator variants, and gain references require careful distinction.
Value: 4/5. A useful cross-domain segmentation baseline without downstream training and with separable mechanisms; deployment still requires evaluating test-time cost and pseudo-label reliability.