Capturing Spectral and Spatial Patterns for Federated Remote Sensing Segmentation

Conference: ECCV2026
Paper: ECCV official page
PDF: Full paper
Authors: Yixin Xue, Haonan Guo, Wenke Huang, Bo Du
Area: Remote Sensing
Keywords: federated learning, remote sensing semantic segmentation, domain-adaptive prototypes, spectral shift, spatial shift

TL;DR

Rather than requiring every remote-sensing client to use identical class centers, FedDap applies constrained local modulation to shared prototypes using spectral and spatial cues, reporting 70.42% mIoU on the compound-shift Potsdam(IRRG) versus Vaihingen(RGB) setting, 1.60 percentage points above FedSeg.

Background & Motivation

Remote sensing semantic segmentation assigns land-cover categories such as buildings, vegetation, and roads to individual pixels, but data from different institutions cannot necessarily be centralized. Federated learning lets clients train locally and exchange model information instead of raw images. The difficulty is that “building” does not have one common appearance distribution across locations. Sensors and imaging pipelines alter channel responses, while urban and rural layouts, scales, and textures alter spatial structure. Consequently, pixels from the same class may first separate into domain-specific clusters rather than organize primarily by semantic identity.

Prototype methods such as FedProto align clients through shared class centers, partly addressing inconsistent semantic coordinates across independently trained models. However, they may not accommodate the structured domain variation within remote-sensing images. Averaging every building into one center can erase useful domain-specific structure; allowing each client to define its building prototype independently can instead lose cross-domain semantic correspondence. This paper retains shared centers to define class identity and uses local modulation to explain how a class appears under the current sensor and geographic conditions.

The adaptation therefore operates in prototype space, rather than imposing one style transformation on all pixel features. Core Idea: preserve global class anchors, derive local prototypes separately from shallow spectral statistics and deep multi-scale spatial structure, fuse them by class, and supervise pixel representations against both global anchors and local prototypes.

Method

Overall Architecture

The inputs are independently held remote-sensing images with pixel annotations, and the outputs remain land-cover predictions from a segmentation network. FedDap's additional machinery primarily supports training: the server maintains shared semantic anchors, clients construct Domain-Adaptive Prototypes (DAPs) through spectral and spatial dual-branch modulation and shift-aware gating, and learning proceeds through dual supervision and cluster-wise synchronization. Shared anchors specify class identity, whereas DAPs specify where that class should lie in the current domain. Both constrain the encoder instead of treating every domain difference as noise.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Local images and pixel labels"] --> B["Shared semantic anchors"]
    B --> C["Spectral and spatial<br/>dual-branch modulation"]
    C --> D["Shift-aware gating"]
    D --> E["Dual supervision and<br/>cluster-wise synchronization"]
    B --> E
    E --> F["Update segmentation network<br/>and predict land-cover map"]
    E --> B

Key Designs

1. Shared semantic anchors: share class statistics rather than dense pixel representations

Each client accumulates embedding vectors and pixel counts separately for each labeled class. The server sums these uploaded statistics across participating clients, divides by the total pixel count for the class, and applies L2 normalization to obtain the global prototype bank. This is not an unweighted average of independently computed client centers: a client with more building pixels contributes correspondingly more to the building prototype, consistent with the definition of the sufficient statistics.

Global prototypes are broadcast with the shared model each round, providing class coordinates that all clients can reference. Even a client that only observes buildings under one sensor retains a cross-client building anchor. The prototypes do not require pixel positions, and individual pixel embeddings are not transmitted. However, aggregate statistics still contain data-dependent information, so keeping images local does not establish a formal privacy guarantee.

2. Spectral and spatial dual-branch modulation: move prototypes without confusing class identity

The spectral branch uses shallow features because channel brightness and radiometric differences are more readily preserved at this level. A lightweight 1×1 convolution first maps features into a common representation space. Channel-wise means and standard deviations are then collected and summarized across the mini-batch into a compact spectral descriptor. An MLP takes this descriptor together with a learnable class embedding and predicts channel-wise scale and shift parameters for each class. A sensor shift may affect several classes, but class conditioning allows buildings and vegetation to receive different corrections.

The spatial branch reads deep features and combines spatial pyramid pooling with global means and standard deviations. The paper's architecture figure shows 1×1, 2×2, and 4×4 pooling scales, intended to preserve texture and layout at different granularities rather than rely only on a global average. This branch likewise predicts modulation parameters from the spatial descriptor and class embedding. Its scale parameters additionally pass through twice a sigmoid, restricting them between zero and two to prevent unstable amplification. Both branches start from the same global anchor and normalize their outputs; neither applies its transformation serially to the other branch's output.

The affine modulation in both branches can be summarized with unified notation:

\[ p_c^{b}=\operatorname{Norm}\!\left(\gamma_c^{b}\odot p_c^{\mathrm{inv}}+\beta_c^{b}\right),\qquad b\in\{\mathrm{spec},\mathrm{spa}\}. \]

Here, \(p_c^{\mathrm{inv}}\) is the global anchor for class \(c\), \(\gamma_c^{b}\) and \(\beta_c^{b}\) are the scale and shift predicted by the corresponding branch, and \(\odot\) denotes channel-wise multiplication. This expression combines the paper's two structurally identical modulation equations. Local descriptors supply domain conditioning, class embeddings supply class conditioning, and normalization provides a consistent scale for subsequent cosine similarities.

3. Shift-aware gating: let classes select different spectral and spatial weights

Fixed averaging assumes that every client and class depends equally on spectral and spatial information, which is poorly matched to remote sensing. For example, vegetation may be sensitive to near-infrared information, whereas cross-city building differences under the same spectral modality may primarily involve structure. The gating network concatenates the two domain descriptors and applies an MLP followed by a sigmoid to produce one weight per class. Fusion is therefore class-wise, not a separate gate for every pixel.

\[ p_c^{\mathrm{dap}}=\operatorname{Norm}\!\left(\alpha_c^{(k)}p_c^{\mathrm{spec}}+(1-\alpha_c^{(k)})p_c^{\mathrm{spa}}\right). \]

A larger \(\alpha_c^{(k)}\) assigns more weight to the spectral branch on client \(k\) for class \(c\). The fused DAP is normalized again and becomes a class reference for local pixels. “Shift-aware” should be understood as task-trained conditional weighting, not as an externally calibrated physical measurement of true sensor-shift intensity; the paper does not provide evidence for that stronger interpretation.

4. Dual supervision and cluster-wise synchronization: accommodate local variation without losing shared semantics

Each client forms classification scores from cosine similarities between normalized pixel embeddings and the DAPs of all classes, then applies temperature scaling and cross-entropy. This pixel-to-prototype contrastive objective brings a pixel closer to its correct-class DAP relative to other classes. A parallel objective classifies the same pixel against the global prototype bank. With DAP supervision alone, local centers could increasingly follow local data; global-anchor supervision maintains the common cross-client relationships between classes such as buildings and vegetation.

Training also penalizes the cosine drift of spectral prototypes, spatial prototypes, and fused DAPs from their global anchors. Rather than prohibiting movement, this makes adaptation carry a cost: a shift can still be corrected when the segmentation and prototype-supervision benefits justify that correction. At the end of a federated round, the server aggregates segmentation models through FedAvg and updates global anchors from class statistics. For the modulators, clients upload parameters and compact shift descriptors smoothed by an exponential moving average. Every \(R\) rounds, the server clusters clients by these descriptors and aggregates spectral and spatial modulators within their corresponding clusters, while averaging the gating network globally. Clients with similar shifts can thus share modulation knowledge without merging every domain's corrections into one rule.

A Worked Example

Consider the main experiment with ten clients: five hold Potsdam(IRRG) data and five hold Vaihingen(RGB) data. At the start of a round, the server broadcasts the shared segmentation model and class anchors. An RGB client reads its local images, describing their appearance through shallow channel statistics and their layout through deep multi-scale features.

For the building class, both branches independently adjust the same global building anchor, and the gate fuses their outputs into a local building DAP. Every building-labeled pixel must match both this local center and the global building center. The modulator cannot reduce the local contrastive loss simply by moving its center arbitrarily far, because the drift penalty remains active. After two local training passes in the round, the client uploads its model and the specified compact statistics, but not images or DAPs. The server updates the model and global prototypes; when the synchronization interval is reached, it also shares modulators within clusters determined by shift descriptors. This illustrates the training procedure rather than an additional numerical gating experiment reported in the paper.

Loss & Training

The local objective combines segmentation loss, DAP supervision, global-anchor supervision, and drift regularization. Following the surrounding method description, it can be written as:

\[ \mathcal L_k=\mathcal L_{\mathrm{seg}}^{(k)}+\lambda_{\mathrm{dap}}\mathcal L_{\mathrm{con}}^{(k)}+\lambda_{\mathrm{inv}}\mathcal L_{\mathrm{inv}}^{(k)}+\lambda_{\mathrm{drift}}\mathcal L_{\mathrm{drift}}^{(k)}. \]

The paper gives pixel cross-entropy or cross-entropy plus Dice as examples of segmentation loss, without fixing one concrete implementation in the available full text. Exact settings for the temperature, loss weights, clustering algorithm, and synchronization interval \(R\) are also incompletely specified, so no defaults should be invented. Some equations are damaged in the text extraction; the expressions above only reconstruct modulation, fusion, and objective structures supported by the adjacent prose.

The three main settings use ResNet-50, ten clients with five per domain, 150 communication rounds, and two local epochs per round. Client dataset sizes are balanced, while domain distributions differ. Following the paper's protocol, Vaihingen images are resampled from 9 cm to 5 cm ground sampling distance to reduce the resolution difference from Potsdam. Resampling does not make the sensors' underlying information content identical.

Key Experimental Results

Main Results

mIoU averages intersection-over-union across classes, while Acc denotes pixel accuracy; higher is better for both. LoveDA uses seven classes, whereas Potsdam/Vaihingen uses five foreground classes and excludes clutter/background. Absolute scores should not be compared directly across these settings. The table retains the paper's reported mIoU columns, and all gains are percentage points rather than relative percentages.

Setting / Backbone	FedAvg mIoU ↑ (%)	FedSeg mIoU ↑ (%)	FedDap mIoU ↑ (%)	Gain over FedSeg
LoveDA urban / rural, ResNet-50	49.89	50.04	50.68	+0.64
Potsdam(IRRG) / Vaihingen(IRRG), ResNet-50	70.64	71.06	73.43	+2.37
Potsdam(IRRG) / Vaihingen(RGB), ResNet-50	67.17	68.82	70.42	+1.60
Potsdam RGB / IRRG, ResNet-50	73.71	74.70	75.80	+1.10
LoveDA-RGB + Potsdam-IRRG, SegFormer-B0, global metric	36.47	38.04	39.25	+1.21

FedSeg should not be called the strongest baseline in every row. Under same-modality cross-region shift, FPL reports 73.08, leaving a gain of only 0.35 percentage points for FedDap. In the compound-shift table, FedProto reports 69.71, above FedSeg's 68.82, so comparison of the reported columns gives FedDap a +0.71 gain over FedProto. However, the paper's prose calls FedSeg the best baseline in that setting, and the arithmetic mean of FedProto's five listed class IoUs is approximately 68.31, inconsistent with its reported 69.71. This source conflict remains unresolved; neither silently correcting the table nor claiming a fully verified strongest-baseline ranking would be justified.

FedDap does not lead every metric either. Its LoveDA Acc is 69.52, below FedSeg's 69.90; its global Acc in the cross-dataset SegFormer-B0 setting is 61.44, also below FedSeg's 61.92. Higher mIoU supports improved class-averaged performance, not simultaneous improvement in pixel accuracy and every individual class.

Ablation Study

The following excerpt from the paper's Table 4 uses the same backbone and federated schedule as the main experiments. The first two removals disable the corresponding prototype supervision; their names should not be read as removing every global-prototype data structure.

Config	LoveDA mIoU ↑ (%)	Same-modality cross-region mIoU ↑ (%)	Compound-shift mIoU ↑ (%)
Full FedDap	50.68	73.43	70.42
Without global-anchor supervision	50.38	72.10	68.35
Without DAP supervision	50.43	72.44	69.05
Spectral branch only	50.26	72.18	69.41
Spatial branch only	50.34	73.31	69.13
Fixed gate at 0.5	49.83	73.20	69.25

Key Findings

• Under compound shift, removing global-anchor supervision costs 2.07 percentage points, while removing DAP supervision costs 1.37 points. Shared semantics and local adaptation are not interchangeable.
• Under same-modality cross-region shift, the spatial-only branch reaches 73.31, close to the full model's 73.43, whereas the spectral-only branch reaches 72.18. This matches the setting's primarily spatial construction.
• Fixed gating reaches 69.25 under compound shift, 1.17 percentage points below the full model. However, the ablation does not separately isolate drift regularization or cluster-wise synchronization, so the total gain cannot be attributed entirely to either component.

Highlights & Insights

• Adaptation targets class references, not only images or features. Shared identity and domain-specific appearance can coexist, avoiding an interpretation of alignment that simply removes all within-class differences.
• The branches correspond to distinct sources of remote-sensing shifts, and ablations test their behavior under same-modality and compound shifts. The value lies in targeted validation of the architecture rather than merely adding two generic modules.

Limitations & Future Work

• The authors identify additional sensors and resolutions, reduced label dependence, more than two latent domains, and highly imbalanced client participation as future directions. The current balanced-size, two-domain main settings do not cover those conditions.
• The paper lacks multi-run variance, sufficient communication/runtime measurements, and some training and clustering hyperparameters. The stability of small gains and the extra cost of modulator synchronization remain to be checked.
• Exchanging statistics and parameters does not establish differential privacy or secure aggregation. The paper evaluates training without centralizing raw images, not formal protection against privacy attacks.
• The tables and prose have the baseline-ranking and numerical-consistency issues described above. The introduction also describes domain discovery through DAP similarity, whereas the method explicitly uploads smoothed descriptors and keeps DAPs local. This note follows the detailed method description; reproduction should check these discrepancies against the implementation.

Related Work & Insights

• vs FedProto / FPL: These methods address heterogeneity through prototype sharing or a prototype-based perspective. FedDap further separates global semantic references from local prototypes conditioned on spectral and spatial information; it adds constrained local adaptation rather than discarding shared prototypes.
• vs FedSeg: FedSeg is an important class-heterogeneous federated segmentation baseline. This work primarily targets structured domain shifts caused by sensors and regions. Their emphasis differs, so one setting cannot establish universal replacement.
• vs FiLM / AdaIN: The paper borrows conditional affine modulation and statistical descriptors but primarily modulates class prototypes, rather than directly applying an existing feature-level style transformation to all pixels. This is its key distinction from related conditioning methods.

Rating

• Novelty: 4/5. Spectral and spatial remote-sensing shifts are explicitly organized into class-prototype adaptation, although the underlying operations build on established modulation and prototype learning.
• Experimental Thoroughness: 3/5. Different shifts and an additional backbone are covered, but variance and cost analyses are missing, and the source tables contain numerical conflicts.
• Writing Quality: 3/5. The main method is understandable, but inconsistent domain-clustering descriptions, equation-extraction quality, and missing reproduction details complicate verification.
• Value: 4/5. Relevant to multi-sensor, multi-region federated remote-sensing segmentation, particularly for its division of roles between shared anchors and local prototypes.