FedNASP: Federated Vision-Language Navigation with Adaptive Step-Wise Personalization

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://github.com/IntelliSys-Lab/FedNASP.git
Area: Robotics
Keywords: Vision-Language Navigation, Personalized Federated Learning, Embodied AI, Adaptive Step-Wise Personalization, Prefix Adapter

TL;DR

To resolve the vulnerability of conventional federated averaging and static personalization under severe cross-environment non-IID shifts and time-varying decision uncertainty in Vision-Language Navigation (VLN), FedNASP introduces a lightweight Step-wise Personalization Modulator (SPM) that dynamically predicts step-wise modulation scalars from navigation state and history, combined with structure-aware personalized prefix injection (SPI) on critical attention blocks, achieving robust decentralized navigation without communicating local personalized weights.

Background & Motivation

Vision-Language Navigation (VLN) requires an embodied agent to understand natural-language instructions, continuously process panoramic visual observations in complex 3D environments, and plan long-horizon trajectories to reach target locations. In real-world robotic applications, indoor navigation data typically originate from private households or confidential offices: panoramic captures, trajectories, and instructional transcripts inevitably disclose sensitive spatial layouts, furniture distributions, and inhabitants' daily habits. These critical privacy barriers prohibit centralizing raw embodied data onto a centralized cloud server. Federated learning (FL) presents an appealing decentralized paradigm by training agents locally and communicating only parameter updates or gradients. However, because each client corresponds to a distinct physical residence or building scan, the data across clients exhibit extreme non-IID characteristics: floorplans, room counts, visual appearances, and human instruction styles vary drastically across clients. Standard federated aggregation algorithms such as FedAvg force a unified global model that inevitably ends up as a compromised policy, failing to perform optimal action grounding on any specific local environment.

Personalized federated learning (pFL) is therefore essential for federated VLN, yet existing pFL techniques—predominantly tailored for unimodal image classification—struggle when applied to embodied navigation. The core tension stems from the temporal and structural complexity of VLN: it is a long-horizon sequential decision process where decision difficulty and uncertainty fluctuate non-uniformly across navigation steps. Conventional pFL methods assign a static, episode-level personalization weight to each client. In a long-horizon episode, this static strategy becomes brittle: during straightforward segments (such as straight hallways) where shared navigation commonalities should dominate, overly aggressive personalization degrades generalization and causes overfitting; conversely, at complex intersections or ambiguous waypoints where fine-grained local disambiguation is vital, insufficient adaptation leads to cascading errors and failed trajectories. Moreover, multimodal backbones exhibit disparate sensitivity across components: language encoders, visual representations, and cross-modal fusion layers react very differently to environmental shifts, meaning that naive full-parameter personalization easily destabilizes multimodal alignment while incurring prohibitive client memory overhead.

To overcome the combined challenge of time-varying decision contexts and multimodal structural sensitivity, an embodied agent must be able to gauge its immediate state and trajectory progress, dynamically calibrating the balance between global navigation knowledge and local personalization. Core idea: propose a step-wise personalized federated learning framework that employs a lightweight dual-branch modulator to predict dynamic scaling factors at each step, cooperating with structure-aware prefix adapters on sensitive attention blocks to adaptively balance shared general priors and local cross-modal grounding.

Method

Overall Architecture

FedNASP treats each autonomous agent deployed in a distinct physical environment (Scan) as an FL client. Across communication rounds, the central federated server maintains and broadcasts a shared global backbone \(\Theta_g\), while each client locally retains a lightweight Step-wise Personalization Modulator (SPM), structure-aware prefix adapters (SPI), and a local reinforcement learning critic. During local trajectory rollouts, the client agent extracts the recurrent hidden state from the previous step and the immediate candidate visual observation features, feeding them into SPM to predict block-wise modulation factors \(\boldsymbol{\alpha}_{i,j}\). Subsequently, the SPI adapters generate additive projection offsets \((\Delta Q, \Delta K, \Delta V)\) scaled by \(\boldsymbol{\alpha}_{i,j}\) and inject them into designated cross-modal and visual attention blocks. The agent executes action selection, computes task losses (imitation and reinforcement learning), and updates parameters. Finally, each client uploads only the backbone update \(\Delta \Theta\) to the server for standard FedAvg aggregation, while all personalized parameters remain strictly on the client.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Recurrent History h and Observation Feature O"] --> B["Step-wise Personalization Modulator SPM<br/>Dual-branch MLP predicts dynamic scalar alpha"]
    B --> C["Structure-aware Personalized Prefix Injection SPI<br/>Local adapter generates Delta QKV scaled by alpha"]
    C --> D["Action Prediction & Local State Update<br/>Multimodal fusion decides next action and updates Critic"]
    D --> E["Dual-track Communication & Aggregation<br/>Upload shared backbone Delta Theta for FedAvg, keep pFL parameters local"]

Key Designs

1. Step-wise Personalization Modulator (SPM): dynamically calibrating time-varying decision uncertainty

Decision difficulty varies non-uniformly along an embodied navigation episode. In certain steps, an agent can confidently navigate via shared spatial instincts, whereas in ambiguous states it relies heavily on environment-specific visual-language grounding. To handle this, FedNASP designs a lightweight two-branch MLP modulator (SPM). SPM decouples and ingests two complementary signals: the decoder recurrent hidden state \(h_{i,j-1}\) (encoding historical decision momentum, instruction execution progress, and path memory) and the current panoramic candidate visual feature \(O_{i,j}\) (capturing immediate local visual context and candidate ambiguity). Conditioned on the concatenated state \(z_{i,j} = [h_{i,j-1}; O_{i,j}]\), SPM maps the state to bounded continuous scaling factors for the selected attention blocks:

\[\boldsymbol{\alpha}_{i,j} = \sigma\left(g_{\Phi_i}(z_{i,j})\right) \in (0,1)^{|\mathcal{B}|}\]

where \(\Phi_i\) represents client \(i\)'s local modulator parameters, \(\mathcal{B}\) denotes the target attention blocks, and \(\sigma\) is the Sigmoid activation. This soft, continuous scaling avoids unstable hard switching and guarantees smooth, stable optimization throughout long-horizon rollouts.

2. Structure-aware Personalized Prefix Injection (SPI): targeting sensitive modules while preserving multimodal foundations

Different functional components of a multimodal VLN architecture exhibit distinct vulnerability to cross-environment shifts. Empirical sensitivity studies in the paper reveal that the language encoder is largely client-invariant; localizing it degrades performance by distorting universal linguistic representations. Conversely, cross-modal attention fusion and visual encoders are the primary loci of client heterogeneity, and a localized critic offers more accurate value estimation for specific floorplans. Accordingly, SPI introduces lightweight adapters \(\Psi_i^{(b)}\) strictly on sensitive cross-modal and visual attention blocks. For each block \(b \in \mathcal{B}\) with input \(x_{i,j}^{(b)}\), the adapter generates additive query, key, and value offsets:

\[(\Delta Q_{i,j}^{(b)}, \Delta K_{i,j}^{(b)}, \Delta V_{i,j}^{(b)}) = \Psi_i^{(b)}(x_{i,j}^{(b)})\]

These offsets are modulated by the step-wise factor \(\alpha_{i,j}^{(b)}\) and injected into the linear projections:

\[\tilde{Q} = Q + \alpha_{i,j}^{(b)}\Delta Q, \quad \tilde{K} = K + \alpha_{i,j}^{(b)}\Delta K, \quad \tilde{V} = V + \alpha_{i,j}^{(b)}\Delta V\]

This parameter-efficient prefix injection enables fine-grained local environmental adaptation with less than 10% additional client-side storage, leaving generic representations in the global backbone intact.

3. Privacy-preserving and communication-efficient dual-track federated optimization

Exchanging personalized parameters across federated rounds incurs severe communication overhead and risks reconstructing private floorplans. FedNASP establishes a dual-track training protocol: during local client optimization, the agent joint-trains the backbone \(\Theta_i\), modulator \(\Phi_i\), and adapter \(\Psi_i\) end-to-end using imitation learning \(\mathcal{L}_{\mathrm{IL}}\) and policy reinforcement learning \(\mathcal{L}_{\mathrm{RL}}\). Upon round completion, the personalized modules \((\Phi_i, \Psi_i)\) and critic network remain strictly private on the client. Only the parameter difference of the shared backbone \(\Delta \Theta_i^t = \Theta_i - \Theta_g^t\) is communicated to the server for standard FedAvg aggregation. Consequently, per-round wall-clock communication time remains virtually unchanged (14.37 minutes for FedNASP vs. 14.12 minutes for standard FedVLN), while safeguarding spatial topologies from leakage.

Loss & Training

Each selected client \(i \in \mathcal{S}_t\) optimizes the combined navigation objective:

\[\min_{\Theta_i, \Phi_i, \Psi_i} \mathbb{E}_{\tau \sim \mathcal{D}_i} \sum_{j=1}^L \left[ \mathcal{L}_{\text{IL}}\big(\pi_{\Theta_i,\Phi_i,\Psi_i}(s_{i,j}), a^\star_{i,j}\big) + \mathcal{L}_{\text{RL}}\big(\pi_{\Theta_i,\Phi_i,\Psi_i}(s_{i,j})\big) \right]\]

where \(\mathcal{L}_{\text{IL}}\) denotes cross-entropy imitation learning against teacher actions \(a^\star_{i,j}\), and \(\mathcal{L}_{\text{RL}}\) optimizes policy exploration via actor-critic reinforcement learning guided by the local critic. The server aggregates updates via sample-size weighting: \(\Theta_g^{t+1} \leftarrow \Theta_g^t + \sum_{i \in \mathcal{S}_t} \frac{|D_i|}{\sum_{k \in \mathcal{S}_t} |D_k|} \Delta \Theta_i^t\).

Key Experimental Results

Main Results

Evaluations were run on an NVIDIA RTX 3090 GPU with client fraction \(C = 0.2\) and \(E = 3\) local epochs. REVERIE trained for 400 rounds and CVDN for 2000 rounds. All reported metrics are averaged over five independent runs.

Results on the REVERIE benchmark (fine-grained object-grounding navigation in unseen test environments, all metrics in %):

Method	SR ↑	SPL ↑	OSR ↑	RGS ↑	RGSPL ↑	Client Storage (Params)
Centralized (Upper bound)	41.78	36.33	46.65	32.87	29.23	–
FedVLN (Standard FL baseline)	29.83	25.08	34.17	22.95	19.81	235.18M (100%)
pFedNavi (Prior pFL)	–	–	–	–	–	470.10M (199.9%)
Per-FedAvg (Classic pFL)	12.51	11.33	13.76	5.10	4.71	235.18M (100%)
FedPerfix (Adapter pFL)	37.22	31.40	40.67	31.18	26.24	253.92M (107.9%)
FedNASP (Ours)	43.23	36.62	45.88	35.22	30.70	256.86M (109.2%)

Results on the multi-turn dialog-driven CVDN navigation benchmark (SR, SPL, OSR, and OPSR in %):

Method	SR ↑	SPL ↑	NE ↓	OSR ↑	OPSR ↑	DER ↑	Len
Centralized (Upper bound)	43.56	33.13	6.11	62.85	76.20	5.63	16.27
FedVLN (Standard FL baseline)	21.19	8.24	7.61	42.80	53.76	3.85	22.84
pFedNavi (Prior pFL)	19.61	13.62	9.71	45.54	55.87	2.58	14.02
Per-FedAvg (Classic pFL)	22.16	10.56	7.58	43.42	53.19	4.27	28.67
FedCP (Conditional policy pFL)	31.82	14.49	6.42	49.56	62.10	5.32	21.32
FedNASP (Ours)	39.02	23.14	5.98	57.09	71.40	5.62	21.06

Ablation Study

The table below isolates the individual contributions of the step-wise modulator (SPM) and structure-aware prefix injection (SPI):

Variant	REVERIE SR ↑	REVERIE OSR ↑	REVERIE RGS ↑	CVDN SR ↑	CVDN SPL ↑	CVDN DER ↑	Note
w/o SPI	29.83	34.17	22.95	21.19	8.24	3.85	Degrades to global FedVLN without local adaptation
w/o SPM	39.96	41.20	30.91	32.40	16.80	4.23	Constant prefix injection without dynamic step modulation
FedNASP (Full model)	43.23	45.88	35.22	39.02	23.14	5.62	Adaptive step-wise modulation + structure-aware injection

Furthermore, the architectural sensitivity investigation in Tab. 1 on REVERIE reveals: - Personalizing solely the language encoder degrades SR drastically from 27.83% down to 13.23%, verifying that localizing text representations corrupts cross-client general linguistic understanding. - Personalizing cross-modal attention increases SR to 33.25%, and the joint configuration of "Cross-modal + Visual + Local Critic" achieves the top performance of 37.35% SR and 31.21% SPL.

Key Findings

• Crucial role of step-wise dynamic adaptation: Removing SPM causes SR to drop by 3.27% on REVERIE and by 6.62% on dialog-heavy CVDN (from 39.02% to 32.40%), confirming that dynamic modulation across long-horizon decision states is critical.
• Substantial gains in high-ambiguity environments: On challenging scan subsets with high branching ratios, FedNASP improves SR by +17.89% over FedAvg (+13.76% on dense topological graphs), showing superior disambiguation at complex intersections.
• Communication efficiency matching standard FL: Client-side storage expands by only 9.2%, communication payload per round is unchanged, and per-round execution time increases by just 1.8% (14.37 min vs. 14.12 min), while surpassing centralized training in REVERIE SR (43.23% vs. 41.78%).

Highlights & Insights

• Recognizes the fundamental physical reality that decision difficulty fluctuates along embodied navigation trajectories, addressing it with a lightweight dual-branch modulator that produces continuous scaling scalars.
• Systematically decodes the heterogeneity sensitivity across multimodal VLN sub-modules, showing that freezing universal language representations while adapting cross-modal attention is far superior to blind parameter-level personalization.
• Closes the substantial performance gap between privacy-preserving decentralized learning and centralized training in embodied AI without introducing communication bandwidth overhead.

Limitations & Future Work

• Limitations acknowledged by authors: Experiments are centered on discrete topological navigation graphs (Matterport3D environments) and specific backbones (ViLBERT, Seq2Seq+LSTM), without yet extending to continuous physics-based control engines (e.g., Habitat, Robo-VLN).
• Potential research limitations: SPM relies on recurrent hidden state \(h_{t-1}\); modern transformer-based long-context VLMs without explicit recurrent states will require alternative memory summarization mechanisms. Furthermore, clients with extremely sparse trajectories face underfitting risks in training local adapters.
• Future directions: Integrating test-time adaptation with shared retrieval-augmented spatial memory, and deploying the decentralized framework onto physical ground robots for collaborative indoor mapping and navigation.

Related Work & Insights

• vs FedVLN [27]: FedVLN first applied FedAvg to VLN but utilized a single global model that suffers catastrophic compromise in non-IID homes; FedNASP improves REVERIE remote grounding success (RGS) by an absolute 12.27% (a 53% relative gain) via step-wise adaptation.
• vs pFedNavi [23]: pFedNavi uses coarse, static parameter-selection personalization that doubles client-side storage (+199.9%); FedNASP achieves higher performance across benchmarks with only a 9.2% parameter footprint and dynamic step-wise modulation.
• vs FedPerfix [18]: FedPerfix is a static prefix method designed for vision classification; FedNASP tailors prefix injection to multimodal cross-attention and dynamically gates injection strength using sequential trajectory states.

Rating

• Novelty: ⭐⭐⭐⭐ [Step-wise dynamic personalization and structure-aware prefix injection precisely tackle temporal and multimodal heterogeneity in embodied navigation]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across REVERIE, CVDN, and R2R, complete with module sensitivity analyses, subset breakdowns, and ablations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, well-formulated methodology, and cohesive diagram-to-design alignment]
• Value: ⭐⭐⭐⭐ [Provides a highly practical, low-overhead blueprint for deploying privacy-preserving collaborative robotic systems in domestic environments]

Related Papers

• [ECCV 2026] Let's Reward Step-by-Step: Step-Aware Contrastive Alignment for Vision-Language Navigation in Continuous Environments
• [ACL 2026] When Does Language Matter? Multilingual Instructions Reveal Step-wise Language Sensitivity in Vision-Language-Action Models
• [ECCV 2026] Path-level Hindsight Instructions for Semantic Exploration in Vision-Language Navigation
• [ICLR 2026] JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation
• [ICLR 2026] Block-wise Adaptive Caching for Accelerating Diffusion Policy