Skip to content

title: >- [Paper Note] FeDepth: Federated Learning for Depth Estimation under Robot Heterogeneity description: >- [ECCV 2026][3D Vision][Federated Learning] A descriptor-based soft clustered federated learning framework that models overlapping client distributions via symmetric Jeffreys divergence and redundant cluster merging, achieving robust depth estimation under robot platform, environment, and range heterogeneity. tags: - ECCV 2026 - 3D Vision - Federated Learning - Robot Perception - Monocular Depth Estimation date: 2026-09-19 content_hash: 7ddb9c3b88be4ee0

FeDepth: Federated Learning for Depth Estimation under Robot Heterogeneity

Conference: ECCV 2026
Paper: ECCV 2026
Project Page: https://vision3d-lab.github.io/fedepth/
Area: 3D Vision
Keywords: Federated Learning, Robot Perception, Monocular Depth Estimation, Soft Clustering, Heterogeneous Systems

TL;DR

Addressing non-IID challenges caused by heterogeneous robot platforms, sensing environments, and depth ranges, FeDepth introduces a descriptor-based soft clustered federated learning framework that captures overlapping client distributions via symmetric Jeffreys divergence and redundant cluster merging, achieving superior generalization on unseen robot sequences.

Background & Motivation

Robot perception—including monocular depth estimation, object recognition, and semantic scene understanding—serves as the foundational cornerstone enabling autonomous mobile systems to interact safely with their operational environments. Modern visual perception architectures rely extensively on diverse, large-scale training data collected across varied environments to cultivate robust generalization capabilities. Nonetheless, most mainstream robotic perception paradigms remain anchored to the centralized learning (CL) paradigm, in which sensory streams collected in the field are transmitted to a central server for aggregated model training. Centralized learning incurs prohibitive wireless communication overhead when continuously streaming high-resolution visual feeds, exposes central servers to computational scalability bottlenecks, and presents severe data privacy and regulatory hazards when personal or industrial robots operate within private spaces.

Federated learning (FL) offers a compelling decentralized alternative by keeping raw perceptual data on local client devices and transmitting only model parameters or gradients to the coordination server. However, deploying conventional FL within real-world robotic fleets encounters severe performance degradation driven by robot-induced non-IID data distributions. These domain shifts stem from combinatorial heterogeneities: diverse robotic platforms (such as wheeled UGVs, agile UAVs, quadruped legged robots, and handheld scanners) induce sharp variations in camera intrinsics, mounting heights, viewing angles, and motion vibrations; heterogeneous operating environments (including dense forests, day and night urban corridors, and indoor rooms) present fundamentally distinct structural geometries and illumination profiles; and most critically, indoor environments (depths under 10 meters) versus outdoor roadways (depths up to 80 meters) generate a bi-modal depth distribution that destabilizes unified optimization. Under such severe divergence, standard federated aggregation schemes (e.g., FedAvg, FedProx, FedDyn, and SCAFFOLD) suffer from pronounced client drift and gradient conflict.

Clustered federated learning (CFL) attempts to resolve domain heterogeneity by grouping statistically similar clients to train specialized sub-domain models. However, standard CFL frameworks enforce mutually exclusive hard partitioning, assigning every client to exactly one discrete cluster. In multi-robot deployments, data distributions across platforms and environments are inherently continuous, combinatorial, and partially overlapping: for instance, a legged robot and a UAV operating across the same forest share ecological scene features despite distinct platform motion dynamics, while a single UGV operating across daytime and nighttime shifts shares mechanical dynamics despite varying photometric profiles. Forcing mutually exclusive cluster boundaries causes knowledge fragmentation and misassignment. Core idea: abandon rigid hard clustering assumptions in favor of a soft clustered federated learning framework based on Gaussian descriptor matching and symmetric Jeffreys divergence, permitting clients to join multiple overlapping clusters and adaptively merging redundant sub-clusters to capture continuous robotic domain shifts.

Method

Overall Architecture

FeDepth comprises three cooperative stages: Soft Clustering prior to federated training, Cluster Model Update across iterative communication rounds, and Sequence-level Inference for zero-shot deployment on unseen robot platforms. Before training begins, each robot client samples a subset of local frames, utilizes a frozen encoder coupled with Generalized Mean (GeM) pooling to extract channel-wise mean and variance vectors, and transmits this compact Gaussian distribution descriptor to the central server. The server constructs an inter-client pairwise distance matrix via symmetric Jeffreys divergence, initializes multi-membership clusters using a threshold \(\tau\), and iteratively merges subset clusters into dominant clusters to produce a refined cluster map \(\mathbf{M}^* \in \{0, 1\}^{K \times N}\). During training rounds, clients perform local supervised optimization on private depth data; the server aggregates client models cluster-wise, and each client downloads and averages the models of all clusters to which it belongs to initialize the next round. At test time, an unseen sequence extracts a descriptor on initialization, routes to candidate cluster models via descriptor matching, and performs depth prediction using averaged cluster weights.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Heterogeneous Robot Input Streams<br/>UGV / UAV / Legged / Indoor & Outdoor RGB-D"] --> B["Client Distribution Gaussian Descriptor Modeling<br/>GeM feature pooling with mean-variance extraction"]
    B --> C["Jeffreys Divergence Soft Clustering & Redundant Merging<br/>Symmetric divergence matrix + Soft cluster map with subset merging"]
    C --> D["Multi-Membership Cluster Model Aggregation & Routing<br/>Intra-cluster weighted averaging + Cross-cluster client model updates"]
    D --> E["Unseen Sequence Zero-Shot Inference<br/>Sequence-level descriptor matching and averaged model prediction"]

Key Designs

1. Client Distribution Gaussian Descriptor Modeling: Compact robotic domain characterization under data privacy

To characterize client-specific visual and geometric distributions without transmitting sensitive raw sensory frames to the server, FeDepth establishes a compact statistical descriptor extraction pipeline. Each client \(c\) randomly samples a representative subset \(X_c' \subseteq X_c\) containing \(n_c = \min(n, |X_c|)\) frames. Each sampled image \(x_c^i\) is processed by the frozen encoder of the backbone depth estimation network to obtain a multi-channel spatial feature map \(z_c^i = f_{\text{enc}}(x_c^i) \in \mathbb{R}^{H' \times W' \times D}\). To compress spatial dimensions while preserving salient structural activations, FeDepth applies Generalized Mean (GeM) pooling: $\(g_c^i = \left( \frac{1}{H' W'} \sum_{h=1}^{H'} \sum_{w=1}^{W'} \left( z_c^i(h, w) \right)^p \right)^{\frac{1}{p}}, \quad p > 0\)$ Pooling produces an aggregated feature collection \(G_c = \{g_c^i\}_{i=1}^{n_c}\). The client computes the empirical mean vector \(\mu_c \in \mathbb{R}^D\) and variance vector \(\sigma_c^2 \in \mathbb{R}^D\) across all sampled vectors. The mean captures the domain center of the robot's visual environment, while the variance encodes geometric and illumination variability encountered along its trajectory. The client transmits solely the descriptor tuple \(v_c = (\mu_c, \sigma_c^2)\) to the server, maintaining minimal communication overhead while completely safeguarding visual privacy.

2. Jeffreys Divergence Soft Clustering & Redundant Merging: Flexible multi-membership modeling with partition consolidation

To accommodate overlapping robotic domain attributes where hard partition boundaries fail, FeDepth operates in distribution space on the coordination server. The server models each client descriptor as an independent multivariate Gaussian distribution with diagonal covariance \(\mathcal{N}(\mu_c, \text{diag}(\sigma_c^2))\). Standard Kullback-Leibler (KL) divergence is asymmetric and directional; FeDepth therefore adopts the symmetric Jeffreys divergence to compute order-invariant pairwise domain discrepancies between client distributions: $\(A_{i, j} = D_{\text{KL}}\!\left(\mathcal{N}(\mu_i, \sigma_i^2) \,\|\, \mathcal{N}(\mu_j, \sigma_j^2)\right) + D_{\text{KL}}\!\left(\mathcal{N}(\mu_j, \sigma_j^2) \,\|\, \mathcal{N}(\mu_i, \sigma_i^2)\right)\)$ This yields a complete symmetric similarity distance matrix \(\mathbf{A} \in \mathbb{R}^{N \times N}\). Applying a clustering distance threshold \(\tau\), the server generates an initial binary cluster assignment matrix \(\mathbf{M} \in \{0, 1\}^{N \times N}\), naturally allowing a client to hold concurrent memberships in multiple clusters. However, raw thresholding can generate fragmented, redundant clusters with near-identical memberships. FeDepth introduces an iterative soft cluster merging protocol: for any two cluster rows \(\mathbf{M}_{i, :}\) and \(\mathbf{M}_{j, :}\), if the client set of one cluster forms a proper subset of the other, the subset cluster is absorbed into the dominant cluster. Merging repeats until topological convergence, producing a finalized compact cluster map \(\mathbf{M}^* \in \{0, 1\}^{K \times N}\), where \(K \le N\) is the number of surviving expert clusters.

3. Multi-Membership Cluster Model Aggregation & Routing: Cross-cluster knowledge federation and unseen client adaptation

The dual-membership structure requires a decoupled two-step model aggregation protocol. In communication round \(t\), the server first aggregates local model weights within each cluster \(k \in \{1, \dots, K\}\) across its constituent clients \(c \in \mathbf{M}^*_{k, :}\), weighted by local dataset size \(p_c^k = \frac{|X_c|}{\sum_{j \in \mathbf{M}^*_{k, :}} |X_j|}\): $\(w_k^t \leftarrow \sum_{c \in \mathbf{M}^*_{k, :}} p_c^k w_c^t\)$ Next, to supply client \(c\) with its updated model for round \(t+1\), the client downloads model weights from all clusters to which it is assigned and computes their arithmetic mean: $\(w_c^{t+1} \leftarrow \frac{1}{|\mathbf{M}^*_{:, c}|} \sum_{k \in \mathbf{M}^*_{:, c}} w_k^t\)$ Clients belonging to multiple clusters serve as continuous communication bridges, facilitating shared representation transfer across related visual domains while preserving specialized expert capabilities. During post-training inference on an unseen robotic platform, the test sequence extracts its Gaussian descriptor \(v_{\text{test}}\) via the identical frozen encoder. The server evaluates average Jeffreys divergence against each cluster \(k\): \(S_{\text{test}}^k = \frac{1}{|\mathbf{M}^*_{k, :}|} \sum_{c \in \mathbf{M}^*_{k, :}} A_{\text{test}, c}\), selects candidate clusters matching \(S_{\text{test}}^k \le \tau\), and averages their model parameters for depth prediction. If no cluster meets the threshold, the nearest cluster \(\arg\min_k S_{\text{test}}^k\) is assigned. This routing occurs once per sequence, preserving zero computational overhead during real-time frame inference.

Loss & Training

During local client training, robot clients optimize depth estimation parameters \(w_c\) using scale-invariant loss combined with pixel-wise depth regression objectives on local RGB-D pairs: $\(\mathcal{L}_{\text{SI}}(y, y^*) = \frac{1}{T} \sum_{i} d_i^2 - \frac{\lambda}{T^2} \left( \sum_{i} d_i \right)^2\)$ where \(d_i = \log y_i - \log y_i^*\) represents logarithmic depth error at pixel \(i\), \(T\) is the total count of valid depth pixels, and \(\lambda = 0.85\) balances global scale variance against fine-grained relative structure. In the HPE scenario, threshold \(\tau\) is calibrated to maintain an empirical cluster overlap ratio of 20%, whereas in the BMR scenario, \(\tau\) converges naturally to distinct binary clusters aligning with indoor and outdoor distributions.

Key Experimental Results

Main Results

Evaluation spans two challenging benchmark setups: 1. HPE Scenario (Heterogeneous Platform & Environment): Constructed from the multi-robot M3ED dataset, comprising 41 continuous training sequences across UGVs, UAVs, and legged robots traversing forest, urban day, urban night, and indoor environments, evaluated on 11 held-out unseen test sequences. 2. BMR Scenario (Bi-Modal Range): Combining indoor NYUv2 (15 clients, depth \(< 10\)m) and outdoor KITTI (17 clients, depth up to 80m) across 32 clients.

Baselines benchmark three representative monocular depth backbones: adaptive-bin regression AdaBins, graphical model NeWCRFs, and discrete cosine transform network DCDepth, across centralized learning (CL), standard FL (FedAvg, FedProx, FedDyn, SCAFFOLD), and clustered FL algorithms (FedClust, PACFL†).

Table 1: Quantitative evaluation on the HPE scenario across diverse depth estimation backbones

Backbone Method Abs Rel ↓ Sq Rel ↓ RMSE ↓ \(\text{RMSE}_{\text{log}}\) ↓ \(\delta < 1.25\) ↑ \(\delta < 1.25^2\) ↑ \(\delta < 1.25^3\) ↑
AdaBins CL (Upper Bound) 0.222 1.477 3.552 0.254 0.733 0.887 0.941
FedAvg 0.434 2.990 6.959 0.542 0.369 0.580 0.729
FedProx 0.439 3.029 6.933 0.533 0.369 0.585 0.737
FedDyn 0.440 3.063 7.027 0.551 0.367 0.573 0.721
SCAFFOLD 0.631 6.009 9.664 0.749 0.225 0.432 0.603
FedClust 0.444 3.263 7.521 0.566 0.277 0.550 0.733
PACFL† 0.602 6.447 7.348 0.509 0.372 0.605 0.768
Ours (FeDepth) 0.318 2.127 5.354 0.364 0.515 0.770 0.897
NeWCRFs CL (Upper Bound) 0.166 0.994 2.987 0.190 0.827 0.935 0.968
FedAvg 0.366 1.994 4.475 0.406 0.504 0.717 0.834
FedProx 0.364 1.983 4.479 0.405 0.504 0.718 0.834
FedDyn 0.363 1.957 4.473 0.404 0.505 0.719 0.834
SCAFFOLD 0.356 2.060 4.569 0.376 0.498 0.726 0.864
FedClust 0.487 4.325 7.760 0.615 0.241 0.507 0.701
PACFL† 0.318 1.934 4.265 0.327 0.540 0.791 0.928
Ours (FeDepth) 0.249 1.750 3.838 0.261 0.703 0.892 0.954
DCDepth CL (Upper Bound) 0.159 0.962 2.916 0.183 0.835 0.939 0.971
FedAvg 0.351 2.077 4.640 0.348 0.493 0.766 0.902
FedProx 0.354 2.152 4.641 0.348 0.492 0.769 0.902
FedDyn 0.352 2.109 4.699 0.349 0.491 0.767 0.903
SCAFFOLD 0.329 2.050 4.880 0.350 0.472 0.759 0.914
FedClust 0.349 3.132 6.423 0.434 0.498 0.712 0.815
PACFL† 0.321 1.976 4.338 0.325 0.537 0.796 0.924
Ours (FeDepth) 0.293 1.872 4.469 0.310 0.547 0.840 0.941

Table 2: Quantitative evaluation on the BMR scenario (NYUv2 indoor + KITTI outdoor)

Backbone Method Abs Rel ↓ Sq Rel ↓ RMSE ↓ \(\text{RMSE}_{\text{log}}\) ↓ \(\delta < 1.25\) ↑ \(\delta < 1.25^2\) ↑ \(\delta < 1.25^3\) ↑
AdaBins CL 0.166 0.369 2.073 0.191 0.768 0.945 0.984
FedAvg 0.357 1.153 3.329 0.409 0.211 0.673 0.913
PACFL† 0.124 0.263 2.022 0.164 0.831 0.970 0.994
Ours (FeDepth) 0.126 0.283 2.150 0.167 0.827 0.968 0.994
NeWCRFs CL 0.077 0.100 1.209 0.100 0.950 0.994 0.998
FedAvg 0.323 1.048 3.094 0.344 0.191 0.833 0.983
PACFL† 0.087 0.145 1.501 0.117 0.921 0.991 0.999
Ours (FeDepth) 0.089 0.145 1.499 0.118 0.921 0.991 0.999
DCDepth CL 0.074 0.101 1.246 0.099 0.952 0.994 0.998
FedAvg 0.158 0.265 1.789 0.183 0.801 0.969 0.996
PACFL† 0.082 0.130 1.428 0.109 0.933 0.993 0.999
Ours (FeDepth) 0.082 0.127 1.430 0.108 0.933 0.993 0.999

Ablation Study

All ablations use NeWCRFs on the HPE scenario to evaluate clustering mechanics, overlap sensitivity, and domain grouping strategies.

Table 3a: Hard vs. Soft clustering under the same clustering threshold \(\tau\)

Clustering Strategy Abs Rel ↓ Note
Hard Clustering 0.264 Standard agglomerative hard partition; enforces single cluster membership
Soft Clustering (Ours) 0.249 Allows overlapping cluster memberships; relative Abs Rel reduction of 5.68%

Table 3b: Sensitivity analysis across different client overlap ratios

Overlap Ratio (%) 0% (Hard) 10% 20% (Ours) 30% 40% 50% 100% (Single Cluster) FedAvg Baseline
Abs Rel ↓ 0.335 0.253 0.249 0.257 0.295 0.300 0.365 0.366

Table 3c: Evaluation of data-driven clustering vs. ground-truth platform/environment isolation (Abs Rel ↓)

Grouping Scheme Training Source UGV UAV Legged Forest Urban-Day Urban-Night Indoor All Test
Centralized (CL) All Data 0.122 0.158 0.261 0.091 0.112 0.280 0.284 0.167
FedAvg All Data 0.166 0.372 0.700 0.298 0.358 0.356 0.659 0.366
PACFL† All Data 0.174 0.305 0.580 0.236 0.338 0.414 0.309 0.318
Ours (FeDepth) All Data 0.148 0.262 0.411 0.203 0.185 0.420 0.275 0.249
Platform: UGV only UGV only 0.139 0.514 0.996 0.493 0.366 0.318 0.663 0.471
Platform: UAV only UAV only 0.295 0.640 1.358 0.790 0.549 0.471 0.610 0.662
Platform: Legged only Legged only 0.458 0.645 0.265 0.407 0.413 0.483 0.782 0.484
Environment: Forest Forest only 0.269 0.383 0.530 0.246 0.351 0.381 0.764 0.370
Environment: Urban-Day Urban-Day only 0.191 0.432 0.696 0.415 0.292 0.301 0.680 0.394
Environment: Urban-Night Urban-Night only 0.204 0.502 0.943 0.527 0.393 0.330 0.657 0.481
Environment: Indoor Indoor only 0.709 0.823 2.012 1.297 1.237 0.764 0.279 1.069

Key Findings

  • Soft overlap clustering bridges severe robotic domain shifts: In the complex HPE benchmark, FeDepth cuts NeWCRFs Abs Rel from FedAvg's 0.366 down to 0.249 (a 31.9% error reduction) and substantially outperforms PACFL†'s 0.318. This demonstrates that multi-attribute robotic domains benefit from multi-cluster bridge representations.
  • Convex trade-off in cluster overlap ratio: The ablation in Table 3b shows that 0% overlap (Abs Rel 0.335) restricts inter-domain feature transfer, while a 20% overlap achieves optimal balance (0.249). Expanding overlap to 50% and 100% collapses clusters into unified FedAvg behavior (0.366), causing performance degradation.
  • Unsupervised distribution matching surpasses ground-truth label partitioning: Table 3c demonstrates that isolating clients by static robot platform or environment labels yields suboptimal global performance (Abs Rel 0.471 to 0.662). FeDepth's data-driven Gaussian descriptor matching discovers intrinsic continuous geometric similarities, outperforming discrete manual categorization.

Highlights & Insights

  • Statistical Gaussian Descriptors via GeM Pooling: Transmitting low-dimensional \((\mu_c, \sigma_c^2)\) channel-wise statistics eliminates raw image transmission while capturing both domain centroids and motion dynamics. Coupled with symmetric Jeffreys divergence, it provides an order-invariant similarity metric.
  • Overlapping Memberships with Subset Cluster Merging: Allowing clients to belong to multiple clusters enables bridge nodes to share invariant features across domains, while iterative subset merging eliminates redundant clusters.
  • Universal Architecture-Agnostic Generalization: FeDepth demonstrates steady performance gains across bin-based (AdaBins), graph-based (NeWCRFs), and frequency-based (DCDepth) depth estimators. On sharply separated indoor/outdoor range shifts (BMR), it matches hard clustering and approaches centralized performance.

Limitations & Future Work

  • Reliance on Initial Frozen Feature Extractor: The clustering partition depends entirely on representations from a frozen pre-trained encoder. If the initial model fails to extract discriminative features in extreme degradation (such as adverse weather or total darkness), initial clusters may be suboptimally configured.
  • Multi-Model Transmission Bandwidth Overhead: Clients participating in multiple clusters must download multiple model parameter sets before local averaging, proportionally increasing downlink bandwidth consumption.
  • Future Directions: Exploring adaptive dynamic re-clustering during training, integrating communication-aware sparse model updates, and extending the framework to multi-modal embodied perception combining RGB, LiDAR, and event sensors.
  • vs FedAvg / FedProx / FedDyn / SCAFFOLD (Standard FL): Standard FL algorithms enforce a single global model across all clients, leading to catastrophic client drift under severe robotic non-IID conditions. FeDepth establishes multiple specialized cluster models that capture distinct geometric domains.
  • vs FedClust / PACFL (Clustered FL): Prior CFL methods enforce mutually exclusive hard partitions, failing to model overlapping platform and environment attributes. FeDepth enables multi-cluster membership and soft merging to maintain cross-domain knowledge flow.
  • vs Personalized FL (pFL): Most pFL methods require local adaptation on seen clients during inference (closed-client setting). FeDepth uses sequence-level descriptor routing to assign unseen robot platforms to relevant cluster models in a zero-shot manner.

Rating

  • Novelty: ⭐⭐⭐⭐ [Introduces soft clustering with Gaussian distribution matching and subset merging specifically formulated for multi-robot domain heterogeneity]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 3 depth architectures, 2 non-IID robotic scenarios, detailed ablations, and baseline comparisons]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous formal definitions, clear architecture figures, and sound theoretical grounding]
  • Value: ⭐⭐⭐⭐⭐ [Provides an actionable and scalable federated framework for real-world collaborative multi-robot fleets and autonomous driving systems]