Hierarchical Spatial and Channel Aggregation for Cross-domain Few-shot Segmentation¶

Conference: ECCV2026
arXiv: 2606.24296
Code: To be confirmed
Area: Semantic Segmentation
Keywords: Cross-domain few-shot segmentation, semantic over-alignment, attribute over-alignment, hierarchical aggregation, slot attention

TL;DR¶

This paper proposes DHANet, which mitigates semantic over-alignment via hierarchical aggregation along the spatial dimension, and alleviates attribute over-alignment via hierarchical aggregation along the channel dimension. Supported by an online probabilistic semantic bank that offers auxiliary support information during inference, the proposed method achieves SOTA performance on four cross-domain few-shot segmentation benchmarks.

Background & Motivation¶

Cross-domain few-shot segmentation (CD-FSS) aims to enable a model, after being trained on a large amount of labeled data in a source domain, to segment completely new categories using only a few labeled samples in a target domain. Existing CD-FSS methods mainly focus on narrowing the feature distribution shift caused by style discrepancies—strategies such as style perturbation, extracting domain-agnostic features, and fine-tuning with small amounts of target domain data are widely adopted. While these methods narrow the gap in global feature distributions between source and target domains to some extent, they overlook a deeper issue: categories across different domains may exhibit fundamental differences in semantic granularity and discriminative attributes.

This neglect leads to two levels of "over-alignment" issues during training. First, semantic over-alignment: during training, models are forced to align all foreground pixel features to a single class prototype at a specific granularity of the source domain. Consequently, the model only learns to distinguish between foreground and background at that particular granularity—for instance, learning to recognize "cat" as foreground and "other classes" as background, but failing to distinguish a cat's head from a cat's body at a finer granularity. When the target domain requires finer or coarser segmentation granularity, the model's foreground-background discriminative capability degrades rapidly. Second, attribute over-alignment: different instances of the same class often exhibit variations in attributes (e.g., white cats and orange cats have different color attributes). These source-domain insensitive attributes are initially activated by different channels. However, channel-level alignment during training forces these channels to gradually degrade, losing their capability to capture differences. When the target domain happens to be sensitive to these attributes (such as dermoscopy images depending on color to identify lesions), the intra-class feature consistency is significantly weakened.

Some state-of-the-art works (such as SDRC and HSL) have attempted to mitigate semantic over-alignment. However, they either rely on superpixel priors provided by external models (which limits generalization) or only perform feature decoupling under a single granularity (offering limited adaptability to granularity variations in different domains), while completely ignoring the problem of attribute over-alignment. The core idea of this paper is to perform hierarchical aggregation along both spatial and channel dimensions: aggregate regional semantic features under multiple granularities along the spatial dimension to mitigate semantic over-alignment, and aggregate attribute features under multiple granularities along the channel dimension to alleviate attribute over-alignment, converting hard single-granularity alignment into soft multi-granularity alignment, supplemented by an online probabilistic semantic bank to provide richer support information during inference.

Method¶

Overall Architecture¶

The overall pipeline of DHANet is as follows: The support and query images first undergo Global Style Randomization (GSR) for style perturbation (used only during training) and are then fed into a weight-shared ViT-B/16 encoder to extract features. The features sequentially pass through the Hierarchical Spatial Aggregation (HSA) and Hierarchical Channel Aggregation (HCA) modules, obtaining representations that are hierarchically enhanced in both semantic and attribute dimensions. The main branch calculates a foreground confidence map through dense support-query feature matching. In the auxiliary branch (enabled only during testing), the Online Probabilistic Semantic Bank (OPSB) maintains class probability distributions and samples multiple pseudo-prototypes from them, providing auxiliary support information to compute auxiliary foreground confidence maps. Finally, the confidence maps from both branches are fused and an adaptive threshold is applied to obtain the final segmentation prediction.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Support Image + Query Image"] --> B["Global Style Randomization GSR<br/>(Training Only)"]
    B --> C["Shared Image Encoder<br/>ViT-B/16"]
    C --> D["HSA: Hierarchical Spatial Aggregation<br/>Multi-granularity Region Aggregation → Semantic Enhancement"]
    D --> E["HCA: Hierarchical Channel Aggregation<br/>Multi-granularity Attribute Aggregation → Attribute Enhancement"]
    E --> F["Main Branch<br/>Dense Support-Query Matching"]
    E --> G["Auxiliary Branch (Testing Only)<br/>OPSB: Online Probabilistic Semantic Bank"]
    F --> H["Foreground Confidence Map M_conf"]
    G --> I["Multi-pseudo-prototype Auxiliary Confidence Map<br/>M_conf_max + M_conf_avg"]
    H --> J["Confidence Map Fusion + Adaptive Thresholding"]
    I --> J
    J --> K["Final Segmentation Prediction"]

Key Designs¶

1. HSA: Hierarchical Spatial Aggregation to Decouple Semantic Granularity

The essence of semantic over-alignment is that all foreground pixels are forced to align to a single class prototype, causing the model to lose the ability to distinguish semantic regions under different granularities. The HSA module addresses this issue: inspired by slot attention, it introduces multi-level learnable spatial slots along the spatial dimension. The number of slots increases layer by layer (the proposed design sets \(L_s=2\), with 10 slots in the first layer and 20 in the second), covering semantic regions of different granularities from coarse to fine. Each slot first aggregates initial information from the encoder's output features \(\mathbf{F}^h\) via cross-attention, and is then iteratively updated into a regional prototype of that layer via slot attention. Concurrently, it produces attention maps to assign each pixel to its corresponding region. The regional prototypes from each layer are mapped back to spatial coordinates via Reverse Masked Average Pooling (RMAP) to generate region feature maps \(\mathbf{F}^i\) at corresponding granularities. Finally, these are summed with the average of the regional feature maps from each layer to yield the hierarchically semantically enhanced features \(\tilde{\mathbf{F}}\). During training, this multi-granularity representation prevents support and query features from being locked into a single-granularity alignment, allowing them to find correspondences across multiple granularities, thereby mitigating over-alignment. During testing, the regional consistency up to different scales also helps suppress background noise. Orthogonal constraints (\(\mathcal{L}_{orth}\)) are applied to the slots of each layer to maintain mutual distinctiveness and prevent collapse into identical representations.

2. HCA: Hierarchical Channel Aggregation to Retain Domain-Sensitive Attributes

The root of attribute over-alignment lies in the fact that the same semantic region in different images may be activated by different channels, but channel-level alignment forces these channels to converge, leading to the gradual degradation of channels corresponding to attributes that are insensitive in the source domain but crucial in the target domain. The design of HCA is symmetric to HSA but operates along the channel dimension: by stacking the HSA-enhanced features \(\tilde{\mathbf{F}}\) with shallow detailed features \(\mathbf{F}^l\), flattening them along the spatial dimension, and transposing them, the channel dimension becomes the fundamental unit of aggregation. Multi-level learnable channel slots are set (\(L_c=2\), with 32 slots in the first layer and 64 in the second). Unlike HSA, HCA does not use RMAP to "fill back" attribute prototypes into their corresponding channels, as this would destroy the attribute diversity of the features. Instead, it directly concatenates all attribute prototypes from each layer to the original features along the channel dimension. This concatenation strategy retains all information of the original channels while incorporating multi-granularity attribute prototypes, allowing the model to self-select the matching granularity during inference based on the attribute sensitivity of the target domain. For example, if the target domain is sensitive to color, the model can utilize fine-grained attribute prototypes; if it is sensitive to texture, the model can select coarse-grained prototypes. HCA also imposes orthogonal constraints on the channel slots.

3. OPSB: Online Probabilistic Semantic Bank to Alleviate Support Scarcity

The testing phase contains only 1 or 5 support samples, making it difficult for the limited class information to cover cases with large intra-class variations in the target domain. Many existing methods opt to fine-tune the model online using each round's predictions, but this introduces the risk of catastrophic forgetting and is computationally expensive. OPSB takes a different path: it models the prototype of each class as a Gaussian distribution (parameterized by mean and variance), which is accumulated and updated episode-by-episode during inference. Specifically, each test episode first estimates the distribution parameters of that episode using foreground/background pixel features with high prediction confidence, and then updates them into the global semantic bank using a sample-size-weighted fusion formula (similar to the online Welford algorithm, taking a weighted average of both mean and variance). During inference, 20 pseudo-prototypes are sampled from the current distribution to serve as auxiliary support information, and their cosine similarity with query features is calculated. By taking both the maximum and mean values, two auxiliary confidence maps are obtained. The key advantage of probabilistic modeling over deterministic prototypes is that sampling multiple pseudo-prototypes can cover different intervals of intra-class variance, providing richer matching information than a single deterministic prototype. Experiments also confirm that the probabilistic prototype outperforms the deterministic prototype by nearly 1 mIoU percentage point.

Loss & Training¶

The total loss is \(\mathcal{L} = \mathcal{L}_{main} + \mathcal{L}_{ssp} + \lambda\mathcal{L}_{orth}\), where \(\mathcal{L}_{main}\) is the binary cross-entropy (BCE) segmentation loss of the main branch, \(\mathcal{L}_{ssp}\) is the self-support prototype auxiliary loss (identical to SSP), and \(\mathcal{L}_{orth}\) is the orthogonal constraint on slots (\(\lambda=0.1\)). Training uses an SGD optimizer with a momentum of 0.9, weight decay of 5e-4, a learning rate of 1e-3, and a batch size of 6, for a total of 5 epochs. In addition, one round of fine-tuning is performed on the target domain using the support set of the first episode (training for 1 epoch on 12 augmented episodes).

Key Experimental Results¶

Main Results¶

Comparison of 1-shot and 5-shot mIoU on four target domain datasets (Deepglobe for satellite remote sensing, ISIC for dermoscopy, Chest X-ray, and FSS-1000 for daily objects):

Setup	Method	Deepglobe	ISIC	Chest X-ray	FSS-1000	Average
1-shot	DHANet (Ours)	49.79	64.25	86.06	83.61	70.93
	HSL (AAAI-26 SOTA)	45.77	59.36	85.95	81.89	68.24
	Gain	+4.02	+4.89	+0.11	+1.72	+2.69
5-shot	DHANet (Ours)	56.13	67.38	86.87	86.67	74.26
	HSL (AAAI-26 SOTA)	54.56	64.62	86.25	83.84	72.32
	Gain	+1.57	+2.76	+0.62	+2.83	+1.94

Under the 1-shot setting, the average mIoU of DHANet outperforms the previous SOTA method HSL by 2.69 percentage points. Notably, the improvements are most significant on Deepglobe (satellite remote sensing) and ISIC (dermoscopy), which exhibit the largest semantic granularity differences from the source domain (+4.02 and +4.89 percentage points, respectively). Furthermore, the 1-shot average performance of DHANet even exceeds the 5-shot performance of all compared methods (excluding HSL).

Ablation Study¶

Configuration	Average mIoU (1-shot)	Description
Baseline (GSR + Fine-tuning + Adaptive Thresholding)	66.53	Only alleviates style discrepancies
+HSA	68.43	+1.90, mitigates semantic over-alignment
+HCA	67.90	+1.37, alleviates attribute over-alignment
+HSA + HCA	69.22	+2.69, mutual complementarity
+HSA + HCA + OPSB	70.93	+1.71, auxiliary support information

Ablation on HSA/HCA Internal Components	Average mIoU	Description
Full aggregation module	69.22	HSA + HCA enabled
w/o SlotAttn	67.39	Slots fail to aggregate distinctive features
w/o CrossAttn	67.81	Slots lose guidance from sample priors
w/o \(\mathcal{L}_{orth}\)	68.33	Slots tend to homogenize

Key Findings¶

The complementary gain of HSA and HCA (jointly +2.69%) is significantly greater than their individual gains (+1.90% and +1.37%, respectively), confirming that semantic over-alignment and attribute over-alignment are orthogonal issues that are correctly addressed through separate strategies.
The probabilistic prototype strategy of OPSB outperforms the deterministic prototype by 0.98 percentage points (70.93 vs. 69.95) and online fine-tuning by 3.05 percentage points (70.93 vs. 67.88). This indicates that probabilistic modeling is more effective than both deterministic prototypes and traditional fine-tuning when coping with intra-class variance.
An optimal value exists for the number of hierarchy levels: DHANet yields the best performance when \(L_s=2\) for HSA and \(L_c=2\) for HCA. Further increasing the number of levels disrupts original category hierarchical relationships due to excessively fine granularity divisions, resulting in performance degradation.
Regarding efficiency, DHANet has lower FLOPs than HSL (176.48G vs. 186.52G) and a higher FPS (30.77 vs. 27.25), demonstrating superior efficiency while delivering better performance.

Highlights & Insights¶

Systematic separation of two over-alignment problems: This is the first work in the CD-FSS domain to treat semantic over-alignment and attribute over-alignment as two orthogonal challenges. The experiments confirm their independence and complementarity, presenting a clean and compelling perspective.
Cross-dimensional reuse of slot attention: The same attention mechanism is utilized to aggregate features along the spatial and channel dimensions separately, varying only in the output strategies (RMAP back-projection for HSA, direct concatenation for HCA). This showcases the flexibility of slot attention as a general-purpose feature decoupling tool.
Concise and practical design of the probabilistic semantic bank: Upgrading deterministic prototypes to Gaussian distributions and using pseudo-prototype sampling to cover intra-class variance eliminates the need for auxiliary models or complex online training. This concept can be easily transferred to other few-shot learning, domain adaptation, or continual learning scenarios.

Limitations & Future Work¶

OPSB relies heavily on prediction confidence to filter high-quality features for library updates: if the overall prediction quality of the model on a certain target domain is extremely poor, the selected “high-quality” features may be unreliable, leading to accumulated errors. More robust confidence estimation or uncertainty-aware selection strategies are needed.
The hierarchical levels (\(L_s=2, L_c=2\)) are fixed hyperparameters and do not adaptively adjust to the target domain. Ideally, the model should automatically select different levels when facing coarse-grained domains (e.g., large-area ground object classification in satellite imagery) versus fine-grained domains (e.g., minute lesion segmentation in medical imagery).
The experiments are only verified on semantic segmentation tasks, without extension to instance segmentation or panoptic segmentation scenarios. Whether the probabilistic modeling mechanism of OPSB yields similar gains in instance-level segmentation remains to be explored.

vs HSL: HSL relies on an external superpixel segmentation model to generate multi-scale priors for learning hierarchical semantic features, introducing dependency on external models. In contrast, the spatial slot representation in this work is purely driven by learnable slot attention, which is more flexible and holds no extra segmentation prior dependencies.
vs SDRC: SDRC decouples features under a single granularity to mitigate semantic over-alignment, which limits its adaptability to varying granularities in different domains. The proposed multi-granularity hierarchical aggregation naturally handles diverse requirements from coarse to fine scales.
vs PATNet / APM / DRA: These methods focus on domain-agnostic feature extraction or style perturbation to reduce distribution discrepancies, completely overlooking semantic and attribute over-alignment. Consequently, their performance lags significantly in domains with large semantic granularity variances (e.g., Deepglobe, ISIC).

Rating¶

Novelty: ⭐⭐⭐⭐⭐ For the first time, both semantic and attribute over-alignments in CD-FSS are systematically modeled from spatial and channel dimensions. The problem perspective is highly novel, and the design of the probabilistic semantic bank is concise yet effective.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across four diverse target domain datasets, component ablations, hierarchical level analysis, query utilization strategy comparison, efficiency analysis, and visualization validations.
Writing Quality: ⭐⭐⭐⭐ Clear motivation and precise problem definitions, though the method section contains dense mathematical formulations that could be slightly streamlined to enhance readability.
Value: ⭐⭐⭐⭐⭐ Significantly outperforms the state-of-the-art in both 1-shot and 5-shot settings (+2.69 / +1.94 mIoU) with superior efficiency, representing a substantial advancement in the CD-FSS domain.