Identifiable Gated Residual Personalization for Federated Parameter-Efficient Fine-Tuning¶

Conference: ECCV 2026
Paper: ECCV 2026 Oral/Poster
Code: https://github.com/ahuang0324/fedsdg
Area: Model Compression
Keywords: Personalized Federated Learning, Parameter-Efficient Fine-Tuning, Scale Identifiability, Dynamic Alignment, Dual-Branch Residual Adaptation

TL;DR¶

Addressing the intrinsic scale non-identifiability in gated residual mixing for federated PEFT, FedSDG introduces structure-decoupled dual-path LoRA, projection-level scalar gating, and backbone-anchored Dynamic Alignment to enable depth-aware, scale-identifiable personalization without additional communication.

Background & Motivation¶

In contemporary transfer learning, adapting large frozen pre-trained backbones (such as Vision Transformers) via parameter-efficient fine-tuning (PEFT) has emerged as the standard paradigm for edge deployments. In federated learning (FL), however, clients routinely exhibit substantial non-IID statistical heterogeneity stemming from differences in acquisition conditions, label skews, and domain shifts. Aggregating a uniform global LoRA adapter across heterogeneous clients via Federated Averaging (FedAvg) triggers severe client drift, biased aggregation, and negative transfer; conversely, relying strictly on isolated local training leads to severe overfitting and sacrifices the transferable representations learned across the population. Personalized federated learning (PFL) thus seeks to retain a robust transferable scaffold while supporting controlled, client-specific adaptation.

To combine shared knowledge with client personalization, recent federated PEFT methods commonly deploy dual-branch architectures (such as FedDPA) featuring a globally synchronized shared branch and an on-device private branch combined via weighted residual mixing. However, this parameterization suffers from a fundamental, previously overlooked structural defect: scale non-identifiability. Specifically, the functional contribution of the private residual depends solely on the product of the residual vector magnitude and its scalar mixing coefficient. For any positive scalar factor $c$, scaling the private residual by $1/c$ while multiplying the mixing gate by $c$ yields an identical functional output and task loss. Because private residual modules are optimized locally and never aggregated, their numerical scales drift uncontrollably across clients and communication rounds. Consequently, mixing coefficients completely lose their physical interpretation as measures of personalization strength, $\ell_1$ sparsity regularization over gates fails to enforce a consistent budget, and gate optimization becomes highly unstable due to noisy gradient scales.

The key insight is that private residual energy must be pinned to an invariant, client-internal reference scale before gated mixing occurs, requiring zero modifications to the standard communication protocol. Core idea: introduce FedSDG, a structure-decoupled gating framework for federated PEFT that combines globally shared and locally private dual-path LoRA with projection-level scalar gates, and eliminates the scale degree of freedom via backbone-anchored Dynamic Alignment to make gating coefficients physically identifiable personalization injection ratios.

Method¶

Overall Architecture¶

FedSDG operates on the principle of "shared scaffold first, decoupled scale calibration, and depth-aware gated injection." Building upon a frozen Vision Transformer backbone, the architecture inserts symmetric dual-path LoRA adapters at the attention output projection ($W_o$) and the feed-forward network output projection ($W_2$) of each Transformer block. The global LoRA path participates in federated aggregation across clients, whereas the private LoRA path and projection-level scalar gates remain strictly on-device. To eliminate scale non-identifiability, FedSDG normalizes the private residual and rescales it using the per-sample root-mean-square (RMS) activation of the corresponding frozen backbone projection before applying the scalar sigmoid gate.

The overall pipeline and forward dataflow within an injected Transformer block are illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input feature $h_{k,l}$"] --> B["Frozen backbone forward<br/>Compute $b_{k,l}$ and extract RMS reference"]
    A --> C["Dual-path LoRA adapters<br/>Compute shared $g_{k,l}$ and private $p_{k,l}$"]
    B --> D["Dynamic Alignment (DA)<br/>Anchor private residual scale to backbone RMS"]
    C --> D
    D --> E["Projection-level scalar gating<br/>Scalar gate $m_{k,l}$ controls injection ratio"]
    B --> F["Identifiable residual mixing<br/>$out_{k,l} = b_{k,l} + g_{k,l} + m_{k,l} \cdot p^{\text{align}}_{k,l}$"]
    C --> F
    E --> F

Key Designs¶

1. Structure-decoupled dual-path LoRA parameterization: balancing shared transfer and client adaptation

Rather than relying on coarse head privatization or uniform adapter sharing, FedSDG introduces symmetric dual-path LoRA adapters at $L_{\text{inj}} = 2L$ linear projections ($W_o$ and $W_2$ across all $L$ Transformer blocks). Both the shared adapter $\boldsymbol{\theta}_g$ and the private adapter $\boldsymbol{\theta}_{p,k}$ share identical rank $r$ and scaling factor $s_{\text{lora}} = \alpha / r$. The global branch is broadcast, updated locally, and aggregated across clients to capture domain-invariant representations. The private branch remains entirely on-device, adapting to localized feature shifts. Because the two branches share an identical architectural capacity, behavioral differences stem purely from optimization dynamics and statistical heterogeneity rather than structural asymmetry.

2. Projection-level scalar gating: depth-aware and interpretable modulation

At each injection site $l$, client $k$ maintains an independent learnable logit $a_{k,l}$, converted via sigmoid to a scalar gate $m_{k,l} = \sigma(a_{k,l}) \in (0, 1)$. Broadcast across token and channel dimensions, this single scalar governs the injection intensity of the private residual without adding significant parameter overhead (only $2L$ extra scalar parameters per model). To prevent premature private deviations from disrupting early shared representation learning, FedSDG initializes logits with a negative constant $a_{\text{init}} = -3$ ($m_{k,l} \approx 0.047$). This "shared-scaffold-first" initialization guarantees that early rounds focus on establishing a shared feature baseline before private branches begin contributing substantial corrections.

3. Backbone-anchored Dynamic Alignment: eliminating scale non-identifiability

Under naive gated mixing $b_{k,l} + g_{k,l} + m_{k,l} p_{k,l}$, scaling $(c m_{k,l}, p_{k,l}/c)$ produces identical functional output, decoupling gate magnitudes from actual energy contributions and distorting regularization. While one might consider anchoring to the global residual $g_{k,l}$, its scale fluctuates across communication rounds due to aggregation. FedSDG instead anchors private residual energy to the output of the frozen backbone $b_{k,l} = b_l(h_{k,l})$, which provides an invariant reference scale for any given input. Dynamic Alignment normalizes $p_{k,l}$ by its per-sample RMS across token and channel dimensions and rescales it by the backbone RMS under stop-gradient: $$p^{\text{align}}_{k,l} = \frac{p_{k,l}}{\text{sg}(\text{RMS}(p_{k,l}))} \cdot \text{sg}(\text{RMS}(b_{k,l}))$$ Because $\text{RMS}(p^{\text{align}}_{k,l}) = \text{RMS}(b_{k,l})$, the gate $m_{k,l}$ directly represents the injected private residual energy relative to the backbone activation. This restores full physical identifiability to the gate, gives $\|\mathbf{m}_k\|_1$ a well-grounded interpretation as a layer-wise personalization budget, and stabilizes gate gradient magnitudes against arbitrary scale drift.

Loss & Training¶

In communication round $t$, the central server samples a client cohort and broadcasts the global parameters $\boldsymbol{\theta}_g^t$. Each selected client initializes $\boldsymbol{\theta}_g^{(k)} \leftarrow \boldsymbol{\theta}_g^t$ and optimizes the joint parameter set $(\boldsymbol{\theta}_g^{(k)}, \boldsymbol{\theta}_{p,k}, \mathbf{a}_k)$ over local dataset $\mathcal{D}_k$ by minimizing the composite objective: $$\mathcal{L}_k = \mathcal{L}_{\text{task}}(f_k; \mathcal{D}_k) + \lambda_1 \|\mathbf{m}_k\|_1 + \lambda_2 \|\boldsymbol{\theta}_{p,k}\|_2^2$$ where $\mathcal{L}_{\text{task}}$ is the cross-entropy classification loss, $\lambda_1$ enforces projection-level sparsity on private residual activation, and $\lambda_2$ constrains private adapter capacity against local overfitting. After $E$ local epochs, the client transmits only the global update $\Delta \boldsymbol{\theta}_g^{(k)} = \boldsymbol{\theta}_g^{(k)} - \boldsymbol{\theta}_g^t$ back to the server for standard aggregation. All private adapters, gate logits, and backbone RMS statistics remain strictly local, preserving identical communication overhead to standard single-branch federated LoRA.

Key Experimental Results¶

Main Results¶

Experiments are conducted on an ImageNet-21k pre-trained ViT-Tiny (24 injection sites, rank $r=8$, $\alpha=16$) across four standard benchmarks: CIFAR-100 and Tiny-ImageNet for label skew (Dirichlet $\alpha \in \{0.1, 0.3, 0.5, 1.0\}$), and Office-Home and Mini-DomainNet for domain/feature shifts. The simulated federation comprises 100 clients, participation rate $C=0.1$, $E=5$ local epochs, and $R=80$ total communication rounds.

The table below summarizes average local top-1 test accuracy (mean±std over three random seeds) evaluated across the final 10 communication rounds:

Setting / Benchmark	Local-only	FedAvg	FedRep	Ditto	FA+LoRA	FedDPA	FedSDG (Ours)
CIFAR-100 ($\alpha=0.1$)	65.16 (1.50)	79.22 (1.10)	86.50 (0.03)	83.75 (0.70)	71.67 (0.83)	82.88 (0.50)	88.20 (0.09)
CIFAR-100 ($\alpha=0.3$)	47.46 (2.15)	82.20 (0.49)	77.11 (0.78)	75.70 (1.53)	76.34 (0.95)	74.33 (0.94)	82.57 (0.38)
CIFAR-100 ($\alpha=0.5$)	40.17 (2.53)	83.02 (0.35)	72.48 (0.56)	72.58 (1.11)	78.16 (0.25)	71.42 (0.25)	80.81 (0.11)
CIFAR-100 ($\alpha=1.0$)	32.82 (2.32)	82.95 (0.83)	67.31 (1.08)	69.50 (1.67)	79.16 (0.27)	68.32 (0.65)	79.24 (0.31)
Tiny-ImageNet ($\alpha=0.1$)	53.75 (0.83)	71.28 (0.08)	79.23 (0.55)	77.43 (0.57)	61.46 (0.46)	73.28 (0.24)	80.63 (0.57)
Tiny-ImageNet ($\alpha=0.3$)	36.65 (0.24)	74.35 (0.19)	68.10 (0.27)	69.30 (0.63)	68.22 (0.32)	64.40 (0.12)	74.23 (0.05)
Office-Home (dom-0.1)	55.11 (0.99)	72.34 (0.08)	69.56 (0.64)	70.53 (0.89)	55.00 (0.38)	73.36 (0.25)	75.40 (0.46)
Office-Home (global-0.1)	50.26 (0.42)	72.81 (0.58)	66.94 (0.70)	67.48 (1.40)	55.72 (0.46)	69.11 (0.88)	75.76 (1.11)
Mini-DomainNet (dom-0.1)	48.48 (0.07)	38.36 (0.96)	58.26 (0.13)	62.02 (0.15)	23.14 (0.94)	56.74 (0.10)	62.80 (0.11)
Mini-DomainNet (global-0.1)	49.39 (0.21)	41.99 (0.18)	59.41 (0.31)	62.95 (0.07)	26.72 (0.14)	57.02 (0.27)	63.58 (0.25)
Overall Average	42.13	71.67	64.56	67.48	61.95	66.56	74.27

Ablation Study¶

The table below details ablations on gating granularity, initialization parameters, and Dynamic Alignment anchor targets:

Ablation Category	Variant Description	Tiny-ImageNet ($\alpha=0.1$)	Office-Home (dom-0.1)	Empirical Finding
Gating Mechanism	Fixed gate $m=0$ (no personalization)	64.77 (0.24)	56.16 (0.57)	Degrades under severe client heterogeneity
	Fixed gate $m=0.047$	76.63 (0.64)	70.55 (0.39)	Conservative static gate outperforms open gate
	Fixed gate $m=0.5$	44.78 (2.21)	45.98 (0.92)	High fixed gate causes severe client overfitting
	Fixed gate $m=1.0$ (full personalization)	31.05 (0.45)	36.88 (0.89)	Complete decoupling causes catastrophic failure
	Coarse gate (shared per block)	76.57 (0.64)	70.59 (0.21)	Cannot differentiate Attention vs FFN demands
	Initialization $a_{\text{init}} = 0$ ($m=0.5$)	52.89 (3.30)	49.17 (1.03)	Early private noise disrupts shared convergence
	FedSDG Default (learned projection-level)	80.63 (0.57)	75.40 (0.46)	Layer-wise adaptive gating achieves optimal trade-off
Dynamic Alignment	FedSDG w/o DA (uncalibrated)	79.32 (0.30)	68.24 (0.64)*	Scale drift degrades interpretability and accuracy
	Anchored to global residual $\text{DA}_{\text{global}}$	80.09 (0.43)	64.64 (1.20)	Aggregation updates shift anchor dynamically
	Anchored to backbone $\text{DA}_{\text{base}}$ (Ours)	80.63 (0.57)	75.40 (0.46)	Frozen backbone provides perfectly stable reference

(Note: Without DA, inter-layer variance and cross-round temporal fluctuations in gate values are significantly amplified.)

Key Findings¶

Depth concentration and FFN preference: Visualizing learned gate activations across Transformer layers reveals that personalization heavily concentrates in deeper blocks, while shallow layers remain almost purely shared to preserve generic low-level features. Furthermore, MLP/FFN gates ($W_2$) activate with substantially larger amplitudes than attention gates ($W_o$) in deeper blocks, corroborating mechanistic interpretability findings that feed-forward layers act as key-value semantic memories.
Superiority in extreme non-IID regimes: The personalization advantage of FedSDG scales with statistical heterogeneity. On CIFAR-100 with $\alpha=0.1$, FedSDG outperforms FedRep by +1.70% and dual-path FedDPA by +5.32%. Fairness quantile evaluation (P10/P50/P90) demonstrates that FedSDG elevates the worst-performing 10% of clients (P10) to 56.64% on Office-Home (dom-0.1), compared to 48.41% for FedRep and 48.98% for Ditto.
Frozen backbone reference outclasses moving global targets: Aligning against the moving global residual branch ($\text{DA}_{\text{global}}$) severely degrades performance on domain-shift tasks (Office-Home dropping from 75.40% to 64.64%), confirming that an anchor reference must itself remain invariant to federated aggregation dynamics.

Highlights & Insights¶

First formalization of scale non-identifiability in federated PEFT: The paper rigorously identifies that multiplicative coupling between private residual magnitudes and scalar mixing gates invalidates $\ell_1$ sparsity budgets and creates optimization instability across non-IID rounds.
Zero-communication Dynamic Alignment: By leveraging the on-device frozen backbone activations as an invariant energy baseline, FedSDG resolves the scale ambiguity entirely during local forward passes, requiring zero server-client communication overhead.
Shared-scaffold-first initialization: Initializing gate logits to $a_{\text{init}} = -3$ ($m \approx 0.047$) effectively protects the initialization phase of global representation learning, allowing stable collaborative pre-alignment before releasing localized adaptation.

Limitations & Future Work¶

Scope limited to Vision Transformers with LoRA: Experimental validation is conducted primarily on ViT-Tiny/ViT-Base with low-rank adaptation, leaving prompt tuning, prefix tuning, and CNN/Mamba architectures unexplored.
Privacy and inversion vulnerability: While private adapters and gating statistics remain local, communicated global adapter updates could still be susceptible to white-box gradient inversion or membership inference attacks; pairing with Differential Privacy (DP) or Secure Aggregation requires investigation.
Extension to LLMs and multimodal models: Future work should scale FedSDG to distributed instruction tuning of massive language models and cross-modal vision-language models (e.g., LLaVA/CLIP).

vs FedRep / FedPer: FedRep and FedPer adopt rigid parameter splits between a shared feature extractor and private classification heads. This cannot capture deep semantic domain shifts. FedSDG pushes adaptive personalization into intermediate residual layers while maintaining a unified backbone.
vs FedDPA: FedDPA employs dual-branch LoRA but relies on uncalibrated residual addition, suffering from scale non-identifiability and cross-client gate drift. FedSDG resolves this issue via Dynamic Alignment, yielding a +7.71% average accuracy gain across benchmarks.
vs Ditto / pFedMe: Optimization-based PFL methods enforce proximity between local and global models using proximal loss terms, which require delicate hyperparameter tuning and struggle in parameter-efficient regimes. FedSDG achieves smooth, interpretable personalization directly in the low-rank subspace.

Rating¶

Novelty: ⭐⭐⭐⭐ [Identifies scale non-identifiability in federated PEFT and proposes backbone-anchored Dynamic Alignment]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluation across 4 benchmarks, diverse Dirichlet skews, domain shifts, ablations, and quantile fairness metrics]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulation, thorough motivation, and intuitive visualizations]
Value: ⭐⭐⭐⭐ [Highly practical, communication-free enhancement for personalized parameter-efficient federated learning]

Setting / Benchmark	Local-only	FedAvg	FedRep	Ditto	FA+LoRA	FedDPA	FedSDG (Ours)
CIFAR-100 (\(\alpha=0.1\))	65.16 (1.50)	79.22 (1.10)	86.50 (0.03)	83.75 (0.70)	71.67 (0.83)	82.88 (0.50)	88.20 (0.09)
CIFAR-100 (\(\alpha=0.3\))	47.46 (2.15)	82.20 (0.49)	77.11 (0.78)	75.70 (1.53)	76.34 (0.95)	74.33 (0.94)	82.57 (0.38)
CIFAR-100 (\(\alpha=0.5\))	40.17 (2.53)	83.02 (0.35)	72.48 (0.56)	72.58 (1.11)	78.16 (0.25)	71.42 (0.25)	80.81 (0.11)
CIFAR-100 (\(\alpha=1.0\))	32.82 (2.32)	82.95 (0.83)	67.31 (1.08)	69.50 (1.67)	79.16 (0.27)	68.32 (0.65)	79.24 (0.31)
Tiny-ImageNet (\(\alpha=0.1\))	53.75 (0.83)	71.28 (0.08)	79.23 (0.55)	77.43 (0.57)	61.46 (0.46)	73.28 (0.24)	80.63 (0.57)
Tiny-ImageNet (\(\alpha=0.3\))	36.65 (0.24)	74.35 (0.19)	68.10 (0.27)	69.30 (0.63)	68.22 (0.32)	64.40 (0.12)	74.23 (0.05)
Office-Home (dom-0.1)	55.11 (0.99)	72.34 (0.08)	69.56 (0.64)	70.53 (0.89)	55.00 (0.38)	73.36 (0.25)	75.40 (0.46)
Office-Home (global-0.1)	50.26 (0.42)	72.81 (0.58)	66.94 (0.70)	67.48 (1.40)	55.72 (0.46)	69.11 (0.88)	75.76 (1.11)
Mini-DomainNet (dom-0.1)	48.48 (0.07)	38.36 (0.96)	58.26 (0.13)	62.02 (0.15)	23.14 (0.94)	56.74 (0.10)	62.80 (0.11)
Mini-DomainNet (global-0.1)	49.39 (0.21)	41.99 (0.18)	59.41 (0.31)	62.95 (0.07)	26.72 (0.14)	57.02 (0.27)	63.58 (0.25)
Overall Average	42.13	71.67	64.56	67.48	61.95	66.56	74.27

Ablation Category	Variant Description	Tiny-ImageNet (\(\alpha=0.1\))	Office-Home (dom-0.1)	Empirical Finding
Gating Mechanism	Fixed gate \(m=0\) (no personalization)	64.77 (0.24)	56.16 (0.57)	Degrades under severe client heterogeneity
	Fixed gate \(m=0.047\)	76.63 (0.64)	70.55 (0.39)	Conservative static gate outperforms open gate
	Fixed gate \(m=0.5\)	44.78 (2.21)	45.98 (0.92)	High fixed gate causes severe client overfitting
	Fixed gate \(m=1.0\) (full personalization)	31.05 (0.45)	36.88 (0.89)	Complete decoupling causes catastrophic failure
	Coarse gate (shared per block)	76.57 (0.64)	70.59 (0.21)	Cannot differentiate Attention vs FFN demands
	Initialization \(a_{\text{init}} = 0\) (\(m=0.5\))	52.89 (3.30)	49.17 (1.03)	Early private noise disrupts shared convergence
	FedSDG Default (learned projection-level)	80.63 (0.57)	75.40 (0.46)	Layer-wise adaptive gating achieves optimal trade-off
Dynamic Alignment	FedSDG w/o DA (uncalibrated)	79.32 (0.30)	68.24 (0.64)*	Scale drift degrades interpretability and accuracy
	Anchored to global residual \(\text{DA}_{\text{global}}\)	80.09 (0.43)	64.64 (1.20)	Aggregation updates shift anchor dynamically
	Anchored to backbone \(\text{DA}_{\text{base}}\) (Ours)	80.63 (0.57)	75.40 (0.46)	Frozen backbone provides perfectly stable reference