Neural Collapse-Inspired Multi-Label Federated Learning under Label-Distribution Skew¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/CanPeng123/multi_label_fl_fednca
Area: Federated Learning
Keywords: multi-label classification, label-distribution skew, neural collapse, equiangular tight frame, class-wise attention

TL;DR¶

FedNCA-ML extracts class-wise image features and aligns their geometry across clients using a shared fixed equiangular tight frame and positive and negative feature regularization, raising macro-AUC from the strongest comparator's 83.63 to 87.55 in one skewed multi-label CIFAR-10 setting, without improving every micro metric.

Background & Motivation¶

Federated learning allows hospitals and other clients to retain raw images while exchanging model parameters, but local data retention does not make their learned representations consistent. One hospital may mostly see a few diseases that are almost absent at another; local optimization consequently favors different labels, and parameter averaging does not directly remove this bias. FedAvg provides the basic collaborative procedure, while methods such as FedProx constrain local drift, but strategies effective for single-label learning do not directly resolve entangled multi-label evidence. For example, a chest image can show both cardiomegaly and effusion, so a shared image vector must support several positive predictions and may exploit frequent co-occurrence as a shortcut for an individual disease.

The problem concerns not only different label frequencies across clients, but also different relationships between labels. If a hospital almost always observes two labels together, its model may fail to extract their evidence separately, while another hospital needs to distinguish them. Moreover, absence of positive examples for a class is different from unknown annotations for that class; this paper primarily studies label skew and missing classes, not pseudo-labeling unannotated diseases. The goal is a single global model covering every target class, rather than a separate personalized classification system for each client.

Neural collapse describes an ideal geometry late in balanced single-label training: same-class features cluster, class centers become symmetrically separated, and classifier directions align with them. The obstacle here is that a multi-label image cannot collapse as a whole toward several mutually exclusive directions. The authors therefore constrain disentangled class-wise features rather than forcing the shared backbone representation to discard semantic relationships. Core Idea: use class-wise attention to obtain alignable representations for multi-label prediction, then share a fixed ETF reference across clients so that local label preferences cannot readily redefine global class geometry.

Method¶

Overall Architecture¶

Each client receives images and multi-hot label vectors, and ResNet-18 extracts feature maps that retain spatial structure. Class vectors from the shared ETF query image tokens, and the Label-Aware Disentanglement Module, LADM, produces one feature per class. Each feature is compared with its corresponding ETF vector by an inner product, followed by a sigmoid yielding an independent class probability. During training, dual feature regularization and binary cross-entropy update the backbone and attention module; the server aggregates trainable parameters before the next round. The fixed ETF participates in both querying and classification, but is not updated according to an individual client's class frequencies or re-estimated from global class centers each round.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Shared ETF reference"] -->|Fixed class queries| C["Label-aware disentanglement<br/>LADM"]
    B["Local images<br/>Backbone spatial features"] --> C
    C --> D["Class-wise inner products<br/>Sigmoid probabilities"]
    A -->|Fixed classifier| D
    C --> E["Dual feature regularization"]
    A -->|Prototype reference| E
    Y["Multi-hot labels<br/>Training only"] --> E
    Y --> F["BCE and regularization<br/>Local parameter updates"]
    D --> F
    E --> F
    F --> G["Server averaging<br/>Next-round broadcast"]

The label and loss branches are used only during training; inference retains image features, LADM, the fixed classifier, and sigmoid. One forward pass returns probabilities for every class, rather than selecting one class first, and requires no ground-truth labels for the test image. There is no server-side teacher, text encoder, or additional prototype-fusion module.

Key Designs¶

1. Shared ETF reference: fix class directions for both querying and classification

An Equiangular Tight Frame, ETF, provides one vector per class with equal lengths and equal pairwise angles. For \(C\) classes, the paper uses a unit-norm simplex ETF; distinct vectors have inner product \(-1/(C-1)\). Its role is not to encode which diseases are more similar, but to assign symmetric positions to classes so that common labels cannot gain favorable directions through classifier updates. The reference is identical across clients and supplies both LADM queries and final classifier weights. Unlike class centers estimated from local examples, it remains available when a client lacks a class and does not drift with that client's sample composition.

Fixing the ETF does not freeze the model: the backbone and attention mappings still learn how to organize visual evidence around these directions. Real class relationships can remain in the shared spatial features, while stronger separation is imposed on the subsequent class-wise representations. This is therefore a neural-collapse-inspired structural prior, not a proof that heterogeneous multi-label training must satisfy classical neural collapse theory. The text extraction of Equation (1) loses some matrix-construction operators, so guessed repairs are not presented here as the authors' exact formula.

2. Label-aware disentanglement LADM: retrieve spatial evidence separately for each label

Pooling an entire image into one vector and imposing single-label ETF geometry still packs several coexisting semantics into the same representation. LADM instead flattens the backbone feature map into spatial tokens and incorporates fixed two-dimensional sine-cosine positional encoding. The positional encoding allows attention to use evidence locations without introducing independently learned positional references across clients. It then uses 4-head cross-attention: each ETF class vector supplies a query, and image tokens supply keys and values, producing a feature for each class. The query-based retrieval resembles DETR, but queries represent classes rather than object instances, and training uses image-level labels without bounding-box supervision.

One region can support several labels, so queries act as soft evidence selectors rather than mutually exclusive spatial partitions. For the feature \(\mathbf{h}_{ic}\) of class \(c\), only its inner product with the corresponding prototype \(\mathbf{m}_c\) determines that label's prediction. This also explains compatibility with multi-label output: every class has its own feature instead of making one vector compete for a unique class through softmax.

\[ \hat y_{ic}=\sigma(\mathbf{h}_{ic}^{\top}\mathbf{m}_c). \]

This is the class-wise prediction relationship in Equation (9), with no requirement that class probabilities sum to one. Shared queries reduce the risk that the same class acquires different query meanings across clients, while the attention parameters must still learn to read images through local training. Table 7 compares randomly initialized learnable queries, ETF-initialized learnable queries, and fixed ETF queries, supporting the practical value of a fixed reference without showing superiority on every metric.

3. Dual feature regularization: handle spurious evidence and positive clustering separately

Binary cross-entropy directly checks only whether each class-wise feature predicts correctly against its corresponding prototype. For an absent label, it can reduce the response along that class's direction without preventing the feature from pointing spuriously toward other class prototypes. Negative feature rejection therefore examines class-wise features whose ground-truth labels are zero, compares them with every non-self prototype, and penalizes excessively high similarity responses. Selection uses sigmoid-transformed inner products with threshold \(\tau=0.3\); pairs below the threshold do not contribute to this penalty. A negative feature is the feature produced for a class absent from the current sample, not a negative client or a contrastive example downloaded from the server.

Positive feature contrastive loss handles classes that are present: their class-wise features should prefer their own prototypes over all alternatives. The authors implement this comparison through a softmax over prototypes, encouraging both within-class compactness and between-class discrimination. This softmax is a training regularizer and should not be confused with the final multi-label sigmoid outputs. Its references are shared fixed prototypes rather than uploaded cross-client sample features, so it does not require additional data-dependent prototype transmission. The two terms address different gaps: negative features should not impersonate other classes, while positive features should cluster consistently toward their own directions. Their optimization effects can still interact; adding only negative regularization reduces macro-F1 in Table 6, so complementarity does not imply independent gains on every metric.

A Worked Example¶

Consider a chest image labeled with cardiomegaly and effusion but not fibrosis; this illustrates the mechanism and is not a new experimental result. The shared backbone extracts spatial evidence from the image, and different LADM class queries retrieve relevant regions. The two positive queries may attend to overlapping regions but produce distinct class-wise features, each compared with its own prototype. Positive contrastive regularization encourages these features toward their respective directions rather than requiring the entire image feature to represent only one disease. A fibrosis feature is still generated, but its label is zero; BCE lowers its own-class probability, and negative rejection additionally limits its use of other class directions. At another hospital without positive fibrosis examples, the fibrosis ETF reference remains the same, but that client does not thereby acquire missing positive visual evidence. The server can acquire that knowledge only through updates from other clients: fixed geometry coordinates learning rather than replacing missing data.

Loss & Training¶

The total objective follows Equation (14): BCE provides binary supervision per label, and the other terms implement negative rejection and positive contrastive regularization.

\[ \mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{BCE}}+\lambda_1\mathcal{L}_{\mathrm{Neg}}+\lambda_2\mathcal{L}_{\mathrm{Pos}}. \]

The extraction of Equations (12) and (13) contains damaged summation, logarithm, or denominator formatting, so their operations are explained without inventing unverified exact normalization expressions. Experiments use 10 clients with full participation, 100 communication rounds, and 1 local epoch per round. The optimizer is AdamW with batch size 32, initial learning rate \(10^{-4}\), and weight decay 0.01. CIFAR-10 is trained from scratch, while other datasets use an ImageNet-pretrained ResNet-18. Scratch training uses \(\lambda_1=1\), pretrained models use \(\lambda_1=0.01\), and all experiments use \(\lambda_2=1\). The authors select checkpoints by the best validation performance and report mean and standard deviation over 3 random seeds in the main experiments. Algorithm 1 explicitly shows equal averaging of client parameters; the prose calls this standard FedAvg but the pseudocode does not show sample-count weighting. Under quantity skew, this distinction matters for reproduction, so sample-count weighting should not be assumed without checking the implementation.

Key Experimental Results¶

Main Results¶

The following selection comes from Tables 1–5 on pages 11–12; values retain the percentage scale, and uncertainty is the standard deviation over 3 runs. \(\beta\) is the Dirichlet concentration and \(\gamma\) the client class-presence ratio; smaller \(\beta\) indicates stronger distribution skew. The strongest comparator is selected separately for each metric, not across all metrics; gains are percentage points.

Dataset and setting	Metric	Strongest comparator	FedNCA-ML	Gain	Source
CIFAR-10, \(\beta=0.5,\gamma=0.5\)	macro-AUC	SphereFed 83.63 ± 1.50	87.55 ± 0.31	+3.92	Table 1
CIFAR-10, \(\beta=0.5,\gamma=0.5\)	macro-F1	FedLGT 43.60 ± 1.68	48.17 ± 1.65	+4.57	Table 1
DermaMNIST, \(\beta=0.1,\gamma=0.71\)	macro-F1	FedLGT 45.61 ± 1.16	50.54 ± 1.37	+4.93	Table 2
VOC, \(\beta=0.01,\gamma=0.5\)	macro-F1	FedLGT 56.53 ± 2.42	61.08 ± 0.10	+4.55	Table 3
COCO, \(\beta=0.05,\gamma=0.75\)	macro-F1	FedLGT 55.68 ± 0.83	56.28 ± 0.32	+0.60	Table 4
ChestX-ray14, \(\beta=0.1,\gamma=0.5\)	macro-AUC	FedLGT 70.16 ± 0.37	71.28 ± 0.15	+1.12	Table 5

Multi-label CIFAR-10 and DermaMNIST are constructed by composing multiple images and taking the union of their labels; VOC, COCO, and ChestX-ray14 contain natural multi-label images. ChestX-ray14 has 14 disease classes; No Finding samples have all-zero disease labels, constitute 57% of training data, and are distributed evenly across clients. Macro metrics average across classes and give more weight to minority-class performance; the paper calls micro metrics instance-wise, but this does not imply equal importance for each class. Benefits have limits: in the severely skewed VOC setting, macro-AUC is 93.01 versus SCAFFOLD's 93.41; COCO micro-F1 is 61.71 versus FedLGT's 62.76.

Ablation Study¶

Table 6 on page 13 uses multi-label DermaMNIST with \(\beta=0.1,\gamma=0.71\); it provides no standard deviations and should be read separately from the repeated-run results in Table 2.

Config	macro-AUC	macro-F1	micro-AUC	micro-F1
Learnable classifier baseline	83.95	40.69	86.68	63.03
Fixed ETF classifier only	83.61	33.35	87.26	61.48
ETF + LADM	84.38	47.95	86.70	60.49
ETF + LADM + negative regularization	84.73	45.06	89.71	62.89
ETF + LADM + positive regularization	85.20	49.84	89.03	61.06
ETF + LADM + dual regularization	87.69	51.38	90.36	63.26

Fixing the classifier alone reduces macro-F1 from 40.69 to 33.35, showing that applying single-label geometry directly to image-level multi-label features can be harmful. ETF + LADM improves macro-F1 by 7.26 percentage points over the learnable classifier, and adding dual regularization produces another 3.43-point gain. The complete model is 87.69/51.38 in Table 6, versus 86.30/50.54 in the same-setting main results of Table 2; the paper does not sufficiently explain this difference, so they are not treated as one measurement.

Table 7 on page 13 further compares query designs under the same DermaMNIST skew setting.

Query type	Initialization	macro-AUC	macro-F1	micro-AUC	micro-F1
Learnable	Random	82.94	47.94	86.33	57.37
Learnable	ETF	85.39	46.92	84.94	53.37
Fixed	ETF	84.38	47.95	86.70	60.49

Key Findings¶

Fixed ETF queries lead on 3 of the 4 metrics in Table 7, but their macro-AUC is lower than that of ETF-initialized learnable queries, ruling out an across-the-board advantage. Table 8 on page 15 reports 12.25M total and 12.23M trainable parameters for FedNCA-ML, compared with 11.18M for FedAvg. One-way FP32 communication per client per round rises from 42.69 MB to 46.71 MB; relative training and inference times are 1.52 and 1.15 times the baseline. Thus, no additional prototype transmission does not mean unchanged communication volume: the added attention parameters still need uploading. Figures 3–5 provide qualitative clustering, similarity, and Grad-CAM explanations, but these visualizations alone cannot establish clinical diagnostic safety for rare diseases.

Highlights & Insights¶

The shared object is a class reference rather than a data statistic. A common coordinate system remains available when a client lacks a class, avoiding unreliable local centers as global definitions.
ETF constraints act only on class-wise representations, retaining semantic relationships in the shared backbone. This better fits the task than forcing a whole multi-label image into single-label classification geometry.
Queries and classifier weights use the same prototypes, connecting evidence retrieval with prediction directions. This offers a reusable design for cross-client consistency.
Treating positive and negative features separately reveals a blind spot of BCE. However, ablations show interactions between regularizers, so their weights and threshold still require validation when transferred.

Limitations & Future Work¶

Experiments fix 10 clients, full participation, and ResNet-18, without testing larger client populations, partial participation, asynchronous updates, or alternative backbones.
Two datasets use artificial compositions whose co-occurrence differs from natural clinical comorbidity; results on naturally multi-label data offer stronger evidence of applicability.
Improved macro-AUC does not directly establish fewer missed rare diseases; class-wise sensitivity, specificity, calibration, and clinical threshold evaluation remain necessary. These are reader-proposed validation requirements.
Some cached equations are damaged and no separate supplementary material is included; exact regularizer normalization, empty positive or negative set handling, and F1 thresholds cannot be fully reproduced from this source.
Page 12 claims a 1.21 macro-AUC gain for severely skewed ChestX-ray14, but Table 5 gives 71.28 minus 70.16, which is 1.12; this note retains the table values and explicitly flags the discrepancy.
Page 13 calls 30.27 the lowest class-wise F1 after adding LADM, but the same Table 6 row also contains 23.83 and 26.11; 30.27 is not its minimum, so that conclusion should not be repeated.
Fixed geometry avoids data-dependent prototype communication but provides no differential privacy or attack-resistance proof; keeping data local is not a complete privacy guarantee.

Compared with FedETF: FedETF aligns image-level features for single-label learning, whereas this paper adds class-wise extraction for coexisting labels; Table 6 illustrates the risk of fixing only the classifier.
Compared with MLC-NC: MLC-NC studies centralized long-tailed multi-label learning, while this paper uses a fixed shared reference to avoid biased local centers and additional global prototype transmission.
Compared with FedLGT: FedLGT uses label text and frozen CLIP text embeddings for semantic consistency, while this paper uses text-independent symmetric geometry; the micro metrics in Table 4 show that the latter is not always preferable.
Compared with FedMLP: FedMLP targets partial annotations and task heterogeneity with prototype-assisted pseudo-labeling; its problem is not interchangeable with the missing-class setting studied here.
Research direction: retaining a fixed global reference while adapting attention retrieval under partial participation is worth studying, but is an extension rather than a result established by this paper.

Rating¶

Novelty: 4/5. Shared queries, ETF classification, and class-wise multi-label representations form a targeted adaptation to federated heterogeneity.
Experimental Thoroughness: 4/5. The study covers 5 datasets, 9 settings, and component ablations, but lacks extensions in client scale and participation patterns.
Writing Quality: 3/5. The motivation is clear, while main-versus-ablation reporting, isolated arithmetic, and minimum-value descriptions need clarification.
Value: 4/5. The representation design is reusable for skewed multi-label federated learning, provided its overhead and metric trade-offs are acknowledged.