Fisher-Routed Mixture of Experts for Federated Class-Incremental Learning

Conference: ECCV2026
Paper: ECCV official page
PDF: Full paper
Area: Model Compression
Keywords: Continual learning, federated learning, mixture of experts, Fisher information, routing distillation

TL;DR

FedFMX selects sample-specific expert subsets during training using forgetting risk and learning gain, then distills a label-free inference gate; with ResNet-18, fine-grained increments, and Dirichlet label imbalance, it achieves 50.32% accuracy on CIFAR-100 versus 46.27% for FedCBDR.

Background & Motivation

Federated class-incremental learning requires multiple clients to acquire new classes continuously without exchanging raw data. The challenge extends beyond forgetting within an individual client: clients hold different classes, and new classes may arrive at different stages. Shared parameters must therefore accommodate new classes while absorbing inconsistent local gradients, without access to previous-task data under this paper's formulation. Global parameter regularization, replay, and personalized aggregation address parts of this problem, but do not directly determine which parameters should learn each incoming sample.

A mixture of experts creates room to separate updates, but always choosing one expert or a fixed number of experts is insufficient. Selection based only on current loss may repeatedly favor experts that already recognize certain classes well, yet are vulnerable to losing that knowledge under new updates. Activating more experts can instead introduce redundancy and interference. The paper therefore treats routing as a decision made before updating parameters: assess not only which expert can learn a sample, but also how strongly that update overlaps historically important parameter directions.

Core Idea: use Fisher information to characterize historically sensitive directions and current gradients to estimate plasticity, select a training subset through expert marginal contributions, and teach a lightweight gate to reproduce the routing rule at inference.

Method

Overall Architecture

The model comprises a shared backbone and a pool of experts, each with an independent expert body and classification head rather than merely an additional personalized head. Inputs are labeled images from a client's current task, and outputs are predictions over seen classes. Experts are not permanently assigned to clients or classes, allowing one expert to serve compatible updates from different clients.

Training proceeds through Fisher-Routed Expert Scoring (FRES), Adaptive Expert Selection (AES), and Routing-aware Regularization (RAR). The first two use labels and gradients to identify experts worth updating. RAR discourages concentrated routing while training a gate that observes only backbone features. At deployment, this gate selects experts without computing Fisher costs or Shapley values.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Current-task images and labels"] --> B["Shared backbone and candidate experts"]
    B --> C["Fisher-Routed Expert Scoring"]
    C --> D["Adaptive Expert Selection"]
    D --> E["Routing-aware Regularization"]
    E --> F["Local updates and federated aggregation"]
    E --> G["Deployment: gated experts<br/>Weighted prediction"]

Key Designs

1. Fisher-Routed Expert Scoring: assess whether an update threatens historically sensitive directions

For a labeled sample, each candidate expert produces a classification loss and a gradient with respect to its classification head. The paper maintains a diagonal empirical Fisher approximation only for these heads: an exponential moving average of historical squared gradients records parameter importance, with decay coefficient \(\rho\). This does not compute a full-model Hessian or directly insert Fisher information into a parameter-constraint loss; its first purpose is routing. Parameters with persistently large historical squared gradients are treated as directions requiring cautious updates. Scoring uses statistics accumulated before the current sample, avoiding the treatment of a newly encountered sample as historical knowledge that already needs protection.

FRES normalizes gradient directions and applies dimension-wise standardization and a mapping to the Fisher statistics to improve comparability across experts. Stability cost measures overlap between the current direction and historically sensitive directions, whereas plasticity gain divides unnormalized gradient strength by overall Fisher rigidity. This distinction matters: a large gradient alone does not establish that an update is desirable. The update might absorb new knowledge in a relatively flat region, or disturb important directions. Using the paper's notation, the scores are:

\[ \Delta_c^{(k)}=\frac12\left\|\operatorname{diag}(\widetilde F_k)^{1/2}\widetilde g_k\right\|_2^2, \qquad \Delta_g^{(k)}=\frac{\|g_k\|_2^2}{\operatorname{Tr}(\widetilde F_k)+\varepsilon}. \]

Here \(g_k\) is the expert-head gradient, \(\widetilde g_k\) its unit-norm direction, \(\widetilde F_k\) the processed diagonal Fisher statistics, and \(\varepsilon\) a numerical stabilizer. Lower stability cost and higher plasticity gain generally indicate greater suitability for the sample. These are proxies based on local gradients and historical statistics, not direct measurements of old-task accuracy. The main text also leaves the Fisher mapping unspecified, so a particular mapping function should not be assumed.

2. Adaptive Expert Selection: turn individual scores into subset marginal contributions

AES does not apply a fixed Top-K operation to an isolated score. Instead, it defines a utility for every candidate subset. Classification losses and normalized plasticity gains are averaged across members, while stability costs are summed. Adding an expert can therefore improve average quality but also increase the cost of disturbing historical knowledge. The paper defines:

\[ v(S)=-\ell_S(x,y)-\alpha\sum_{k\in S}\Delta_c^{(k)}+(1-\alpha)\Delta_g(S). \]

Here \(\ell_S\) averages the members' cross-entropy losses; it is not the cross-entropy obtained after mixing their predictions. Likewise, \(\Delta_g(S)\) averages normalized plasticity gains. This distinction specifies what the cooperative utility measures: it rewards a combination of loss, gain, and cost rather than directly measuring complementary prediction errors. The coefficient \(\alpha\) controls the relative weighting of stability cost and plasticity gain.

AES then computes each expert's Shapley value \(\phi_k\): the weighted average increase in utility when that expert joins different subsets of the other experts. Its usefulness therefore depends not only on standalone performance but also on its effect when added to other candidate combinations. The paper computes these contributions exactly for small expert pools, typically discussing \(K\leq8\). For larger pools, it suggests permutation sampling but provides no large-scale results in the main text. Caching subset losses, costs, gains, and cardinalities accelerates individual utility updates without removing the combinatorial enumeration cost of exact Shapley values.

Experts are added in descending order of \(\phi_k\). AES compares the latest utility increment with the preceding mean increment multiplied by a threshold \(\mu\), stopping when the diminishing-gain condition is met. The stated rule starts checking at the second addition and retains the expert that triggers stopping. However, the claim that a smaller threshold causes earlier stopping is not evidently consistent with the given inequality, and empty-set utility is not explicitly defined. This note therefore explains the stopping mechanism without asserting a monotonic threshold effect.

3. Routing-aware Regularization: convert expensive training decisions into a deployable gate

The preceding routing procedure depends on ground-truth labels and gradients, so it cannot be used unchanged at test time. RAR converts Shapley values into a teacher distribution \(q\) using a temperature-scaled softmax, while a lightweight gate maps shared-backbone features to a student distribution \(p\). Minimizing \(\operatorname{KL}(q\|p)\) teaches the gate which features should invoke which experts. The distilled target is a routing distribution, not an old model's class predictions. Shapley values are used for selection and weighting as non-differentiable supervision signals; gradients do not flow through the combinatorial selection procedure.

RAR also minimizes the negative entropy of the student routing distribution—the sum of each probability times its logarithm—to discourage overly sharp per-sample soft routing. This encourages dispersed soft probabilities but does not strictly enforce equal hard activation counts across the batch. The training task loss includes only active experts, weighted by their normalized positive Shapley values to reduce the influence of negative-contribution or redundant experts. The shared backbone continues to receive updates through sample task losses. The paper does not explicitly handle a denominator of zero when all active experts have zero positive contribution, an implementation edge case that requires verification.

A Worked Example

Consider the default pool of \(K=4\) experts. After entering a new stage, a client can access only images from its current classes. During training, an image passes through the shared backbone; candidate experts then compute classification losses and head gradients, which FRES compares with their historical Fisher statistics. An expert with a low current loss need not receive the highest routing priority if its gradient points mainly along historically sensitive directions.

AES next computes marginal contributions from subset utility, ranks experts, and selects an active subset without requiring every training image to use the same number of experts. RAR teaches the gate these selection signals while updating active experts and the shared backbone before federated aggregation. This is a mechanism walkthrough, not additional experimental data. At test time, labels are unavailable, and the gate selects Top-2 experts by default, combining their outputs with normalized gating weights. Adaptive training subsets must be distinguished from fixed Top-2 inference.

Loss & Training

The local objective combines task cross-entropy, negative-entropy load regularization, and routing distillation:

\[ \mathcal L=\mathcal L_{\mathrm{task}}+\beta\mathcal L_{\mathrm{bal}}+\gamma\mathcal L_r. \]

The main text sets \(\rho=0.95\), \(\alpha=0.6\), \(\beta=0.05\), and \(\gamma=0.6\). Training uses SGD with learning rate 0.01, momentum 0.9, weight decay \(10^{-5}\), batch size 64, and 5 local epochs per round. There are 100 clients, with 20% sampled randomly each round. The theory interprets expert selection as a projected update onto expert-parameter subspaces. Its global bound contains a decaying \(O(1/T)\) term plus an error floor from gradient noise and residual gradients outside the active subspace, rather than unconditional convergence to zero under general stochastic training.

Key Experimental Results

Main Results

The following entries come from main-text Table 1(b), all using ResNet-18, fine-grained class increments, and Dirichlet distribution-based label imbalance with \(\alpha_D=0.5\). Accuracy is reported in %, with higher values better. CIFAR-10, CIFAR-100, Tiny-ImageNet, and DomainNet use 5, 10, 20, and 10 tasks, respectively. Gains are differences between mean accuracies in percentage points, not relative percentages. The original “±” values are retained, although their statistical interpretation is not clearly defined in the main text.

Dataset	FedCBDR accuracy ↑	pFedMxF accuracy ↑	FedFMX accuracy ↑	Gain over FedCBDR
CIFAR-10	60.58 ± 1.6	60.43 ± 1.5	64.68 ± 1.2	+4.10
CIFAR-100	46.27 ± 1.4	46.09 ± 1.6	50.32 ± 1.5	+4.05
Tiny-ImageNet	14.38 ± 0.6	14.25 ± 0.7	19.14 ± 0.7	+4.76
DomainNet	42.72 ± 1.5	40.56 ± 1.0	44.29 ± 1.1	+1.57

The paper also evaluates quantity-based label imbalance (QLI) and ViT-B/16 initialized with DINO self-supervised pretrained weights. The latter should not be mixed with the ResNet-18 results above. This note selects a common backbone and imbalance parameter to avoid presenting protocol differences as method gains.

Ablation Study

The following training-time expert-selection comparison comes from main-text Table 2, using fine-grained increments and DLI, with the default \(\alpha_D=0.5\). It changes the training selection strategy and should not be interpreted as a comparison between Top-1 and Top-2 inference. The table does not explicitly specify K for its fixed Top-K training comparison.

Dataset	Training selection strategy	Accuracy ↑ (%)	Note
CIFAR-100	Top-1	45.96	Fixed single expert
CIFAR-100	Top-K	48.54	Fixed expert count
CIFAR-100	AES	50.32	Adaptive expert subset
DomainNet	Top-1	41.59	Fixed single expert
DomainNet	Top-K	44.86	Fixed expert count
DomainNet	AES	44.29	Adaptive expert subset

Key Findings

• AES exceeds training-time Top-K by 1.78 percentage points on CIFAR-100 but trails it by 0.57 percentage points on DomainNet. Table 2 therefore does not support the text's claim of best performance in every setting.
• Enlarging the expert pool does not always help. The main text reports declining performance for excessively large pools, showing that greater capacity and better expert specialization are not equivalent. The default pool contains 4 experts.
• Computation is not free: in Table 4's ResNet-18 comparison on CIFAR-100, pFedMoE and FedFMX use 11.7M and 14.6M parameters, with reported times of 75 s and 82 s. The main text does not clearly specify the timing unit of work or provide communication-byte measurements sufficient to establish communication savings.

Highlights & Insights

• Fisher information is moved upstream to the decision about which parameters should learn, rather than only constraining an already chosen update. This connects continual-learning protection signals to MoE routing.
• Separating the training teacher from the inference gate keeps label-dependent contribution estimates out of deployment. This distinction explains how the method supports label-free inference.
• Summing costs while averaging gains in subset utility explicitly charges for additional expert activation. This is more selective than simply increasing Top-K, but remains a proxy objective rather than direct optimization of measured forgetting.

Limitations & Future Work

• The available cache contains the complete main text and references but not the cited appendices. The Fisher mapping, empty-set utility, handling of zero positive contributions, some experimental details, and proof details cannot be fully reproduced from it; implementation requires checking the original supplementary material.
• Reporting is inconsistent: Table 3(a) gives 19.74 for the full model on fine-grained Tiny-ImageNet, whereas Tables 1(a), 2, and 4 give 19.47. The comparisons above avoid mixing these values; neither should be silently adopted as a corrected ground truth.
• Exact Shapley routing with small expert pools and head-only Fisher approximations do not automatically ensure efficient training with large pools or capture all forgetting in expert bodies and the shared backbone. Removing contribution computation at deployment does not eliminate training cost.
• The theoretical result includes a noise and projection-residual error floor. The experiments also leave the distinction between final accuracy and across-stage average accuracy, and the interpretation of error bars, insufficiently clear. A complete protocol and implementation specification matter more than broad claims of superiority.

Related Work & Insights

• vs FedEWC: Classical Fisher regularization constrains changes to important parameters, whereas FedFMX first uses Fisher information to choose update channels. They use related importance signals at different points in optimization.
• vs pFedMoE: Conventional personalized MoE primarily addresses client differences. FedFMX explicitly adds historical sensitivity and incremental-learning gain to routing utility, with additional training and parameter overhead.
• vs FedCBDR / pFedMxF: The former emphasizes class-balanced replay, while the latter emphasizes personalized frequency aggregation. FedFMX primarily changes sample-level expert selection. Higher tabulated accuracy does not establish that all capacity, storage, and pretraining differences have been controlled.

Rating

• Novelty: 4/5. Fisher routing, Shapley subset selection, and gate distillation form a coherent approach to federated class-incremental learning.
• Experimental Thoroughness: 3/5. Multiple datasets, backbones, and routing comparisons are included, but unclear statistics and table–text inconsistencies weaken the conclusions.
• Writing Quality: 3/5. The training–inference separation is clear, while threshold descriptions, edge cases, and appendix dependencies impede reproduction.
• Value: 4/5. The paper provides a concrete way to use knowledge-retention signals for expert routing; practical value still depends on training overhead and protocol verification.