Skip to content

Divide-and-Adapt: Fine-Tuning VLMs via Model Division and Routing

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/haiiling/DivideAndAdapt
Area: Multimodal VLM
Keywords: Vision-Language Models, Model Division, Mixture-of-Experts, Diversity Pruning, Dynamic Routing

TL;DR

Challenging the long-standing assumption that pre-trained dense architectures must remain intact during adaptation, this paper proposes Divide-and-Adapt (DA-MoE), which decomposes pre-trained FFNs into complementary sparse subnetwork experts via orthogonal pruning criteria (Wanda and SparseGPT) on multi-split calibration sets and coordinates them through alternating top-2 MoE routing, surpassing full fine-tuning while reducing active inference parameters.

Background & Motivation

Driven by web-scale multimodal pre-training, vision-language models have achieved remarkable generalization across diverse visual-textual tasks such as image captioning, visual question answering, and multimodal reasoning. However, when deploying pre-trained VLMs to real-world vertical domains (such as egocentric smart-home surveillance or long-form video causal reasoning), substantial distribution shifts invariably emerge, necessitating downstream adaptation. While full fine-tuning adapts all parameters to downstream distributions, its surging computational and GPU memory footprint quickly becomes prohibitive as foundation backbones scale. Conversely, existing parameter-efficient fine-tuning (PEFT) techniquesโ€”including LoRA, DoRA, VeRA, and modular Adaptersโ€”drastically cut trainable budgets, yet they frequently lag behind full fine-tuning when confronted with intricate reasoning and fine-grained visual counting tasks.

A deeper examination reveals that virtually all existing adaptation paradigms, whether full fine-tuning or PEFT, adhere strictly to a shared premise: the dense structural integrity of the pre-trained architecture must be preserved unconditionally throughout downstream tuning. However, the lottery ticket hypothesis and modern pruning literature (e.g., Wanda, SparseGPT) have established that overparameterized foundation networks implicitly harbor performant, highly specialized sparse subnetworks. Treating the pre-trained model as an indivisible monolithic whole not only inherits redundant noise pathways during gradient updates but also deprives the model of input-adaptive pathway specialization. At the same time, standard pruning methods focus exclusively on extracting a single winning subnetwork, thereby suffering from severe capacity bottlenecks when addressing diverse downstream multimodal distributions.

To resolve this fundamental tension, the authors rethink downstream fine-tuning from a decompositional perspective, positing that a pre-trained model can be split into multiple complementary subnetworks and collaboratively adapted via dynamic sparse routing. Core idea: decompose the dense pre-trained FFN into structurally diverse sparse subnetwork experts through multi-seed calibration and orthogonal pruning strategies, then fine-tune them collaboratively under an alternating DA-MoE architecture with top-2 dynamic routing, achieving superior adaptation accuracy over full fine-tuning with only 1% trainable parameters and reduced active inference footprint.

Method

Overall Architecture

The Divide-and-Adapt (DA-MoE) framework transforms a pre-trained dense decoder into a dynamically routed sparse network. Instead of replacing every Transformer layer, DA-MoE adopts an interval-based alternating layout: even decoder layers retain a single static, 50% sparse FFN, while odd decoder layers are converted into DA-MoE blocks housing four pruned experts. Within each DA-MoE block, hidden states emerging from self-attention and layer normalization are evaluated by a parameterized MoE router, which adaptively dispatches tokens to the top-2 scoring pruned experts. Each expert originates from the pre-trained dense FFN via structured-diversity pruning, inherently inheriting robust foundation semantic priors while embodying distinct structural inductive biases.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Video & Text Input<br/>Frame sequence + Query prompt"] --> B["Pre-trained Dense Feature Extraction<br/>Vision Encoder + Tokenizer"]
    B --> C["Diversity Pruning Expert Construction<br/>Wanda/SparseGPT ร— Dual Calibration Splits"]
    C --> D["Alternating MoE Layer Architecture<br/>DA-MoE blocks interleaved with static sparse FFN"]
    D --> E["Dynamic Routing & Load-Balanced Training<br/>Top-2 Router + Auxiliary Balancing Loss"]
    E --> F["Downstream Multi-Task Inference Output<br/>High-capacity responses with reduced active params"]

Key Designs

1. Diversity Pruning Expert Construction: Overcoming Single-Subnetwork Bottlenecks and Cold-Start Homogeneity
In conventional pruning, networks are trimmed according to a single importance metric to isolate one winning subnetwork, which significantly constrains representation capacity. Conversely, randomly initialized MoE experts discard valuable pre-trained representations and suffer from unstable convergence. To cultivate diverse yet capable experts, DA-MoE extracts four distinct 50% sparse subnetworks from the dense FFN by combining two orthogonal pruning criteria across two distinct calibration subsets sampled from the downstream training data using fixed random seeds. The first criterion, Wanda, scores parameter importance by coupling weight magnitudes with activation norms: \(S_{ij} = |W_{ij}| \cdot \|X_j\|_2\). The second criterion, SparseGPT, formulates pruning as a layer-wise local reconstruction problem, estimating the error perturbation induced by zeroing weight \(W_{ij}\) via the inverted damped Hessian:

\[S_{ij} = \frac{|W_{ij}|^2}{\left[\text{diag}(X^T X + \lambda I)^{-1}\right]_{ij}}\]

By retaining the top 50% weights under these distinct perspectives, each expert receives a unique binary structural mask \(M_i \in \{0, 1\}^{d \times d}\). This setup ensures that individual experts specialize along distinct signal-flow topologies while sharing foundational domain priors, effectively eliminating cold-start expert competition.

2. Alternating MoE Layer Architecture: Harmonizing Global Flow Stability with Dynamic Sparsity
Converting every decoder layer into an MoE structure inflates total model parameters beyond edge deployment thresholds and introduces severe routing instability across stacked gating modules. To strike an optimal balance, DA-MoE introduces an interval-based alternating architecture: across all decoder blocks, DA-MoE modules are deployed only at odd-indexed layers, while even-indexed layers are configured as static 50% sparse FFNs without routing overhead. For an incoming token hidden state \(h\), the DA-MoE block performs self-attention followed by dynamic top-\(k\) expert routing (\(k=2\)):

\[h' = \text{LayerNorm}\left(h + \text{Attention}(h) + \sum_{i=1}^{k} R(h)_i M_i E_i(h)\right)\]

where \(R(h)_i\) is the dynamic dispatch weight and \(E_i(h)\) is the forward mapping of expert \(i\). This alternating design guarantees uninterrupted gradient backpropagation through deterministic intermediate layers while affording critical layers the flexibility of conditional computation, successfully trimming inference active parameters from 8.0B to 6.6B on InternVL3-8B.

3. Dynamic Routing & Load-Balanced Training: Mitigating Expert Collapse under Parameter-Efficient Fine-Tuning
Without explicit regularization, learned routers inevitably degenerate into trivial local minima where a small subset of dominant experts absorbs all tokens, leaving remaining subnetworks completely inactive. To enforce balanced expert utilization across complex multimodal sequences, DA-MoE integrates a differentiable auxiliary load-balancing loss:

\[\mathcal{L}_{\text{aux}} = E \cdot \sum_{i=1}^{E} F_i \cdot G_i\]

where \(F_i = \frac{1}{K}\sum_{j=1}^K \mathbf{1}[\arg\max P(x_j) = i]\) represents the fraction of tokens routed to expert \(i\), and \(G_i = \frac{1}{K}\sum_{j=1}^K P(x_j)_i\) denotes its mean routing probability across all \(K\) tokens in the batch. During downstream adaptation, the base backbone weights and structural masks remain strictly frozen; parameter updates are confined to rank-32 LoRA adapters inserted into attention and FFN modules alongside the lightweight gating network. This configuration restricts trainable parameters to just 1.05%, enabling efficient collaborative specialization.

A Worked Example

Consider an egocentric smart-home video query: "How many items did the boy put on the shelf in this video?" (The boy holds one toy with both hands across distinct grasps, then places his toy and a man's toy onto the shelf sequentially, totaling two placement actions). 1. Multimodal Encoding: A 16-frame video sequence and the question text are projected into unified sequence tokens. 2. Layer-wise Routing: When tokens corresponding to the visual boundary between the boy's hands and the toy reach Layer 0 (DA-MoE), the router assigns high dispatch weights to the SparseGPT expert (attuned to fine-grained geometric boundaries) and the Wanda expert (sensitive to high-energy motion activations), while deactivating background-oriented experts. 3. Alternating Interleaved Processing: The top-2 activated outputs are dynamically merged and passed into Layer 1 (a deterministic 50% sparse FFN), smoothing representations without routing jitter. 4. Disambiguated Counting: Dense baselines (Zero-Shot, Full FT, and LoRA FT) erroneously count the two initial hand grasps as separate objects and predict "Three". In contrast, DA-MoE leverages specialized expert routing to track object persistence correctly, resolving the compound visual action and predicting the exact answer: "Two".

Loss & Training

The overall training objective combines autoregressive sequence modeling with the load-balancing penalty weighted by \(\alpha = 0.01\):

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{regressive}} + \alpha \cdot \mathcal{L}_{\text{aux}} = -\sum_{i=1}^K \log p_\theta(y_i \mid X, y_{<i}) + \alpha \cdot E \sum_{j=1}^E F_j G_j\]

Models are trained using the AdamW optimizer (\(\beta_1=0.9, \beta_2=0.999, \epsilon=10^{-8}\), weight decay 0.02, warm-up ratio 0.02) across four NVIDIA A100 (80GB) GPUs using DeepSpeed-MoE. The maximum sequence length is fixed at 8192 tokens to support long-context multimodal video inputs.

Key Experimental Results

Main Results

DA-MoE is evaluated against zero-shot inference, full fine-tuning, and LoRA fine-tuning on NExT-QA (5,440 videos, 52K QA pairs spanning causal AccC, temporal AccT, and descriptive AccD reasoning) and a proprietary real-world smart-home benchmark (1,143 videos across 7 functional categories) using InternVL3-8B and SmolVLM2-2.2B backbones.

Model Method Causal Reasoning (AccC) Temporal Reasoning (AccT) Descriptive Reasoning (AccD) Smart-Home (AVG) Overall Accuracy (Acc)
InternVL3-8B Zero Shot 79.1% 71.9% 86.3% 71.07% 79.1%
InternVL3-8B Full FT 80.4% 78.8% 86.5% 71.73% 80.9%
InternVL3-8B LoRA FT 80.0% 78.0% 85.3% 71.87% 80.2%
InternVL3-8B DA-MoE (Ours) 81.0% 79.0% 87.5% 73.35% 81.4%
SmolVLM2-2.2B Zero Shot - - - 48.35% -
SmolVLM2-2.2B Full FT - - - 49.56% -
SmolVLM2-2.2B LoRA FT - - - 46.61% -
SmolVLM2-2.2B DA-MoE (Ours) - - - 51.42% -

On granular NExT-QA question categories, DA-MoE excels in Count questions with 70.2% accuracy (beating Full FT's 68.3% and LoRA's 67.7%) and Location questions with 96.1% accuracy (beating Full FT's 94.6% and LoRA's 93.2%).

Ablation Study

A step-by-step ablation on NExT-QA demonstrates the progressive gains delivered by subnetwork pruning, adaptation, MoE routing, and expert diversification on InternVL3-8B:

Configuration Architectural Specification Overall Accuracy (Acc) Relative Change vs. Zero-Shot
Zero Shot Untuned dense pre-trained model 79.1% Baseline
SP Init Single pruned subnetwork (untuned) 77.4% -1.7% (capacity loss from pruning)
SP FT Single pruned subnetwork with downstream FT 79.4% +0.3% (task-specific adaptation)
SP MoE MoE constructed with single pruning criterion 80.1% +1.0% (gains from dynamic routing)
DP Init Diversity-pruned subnetworks as MoE (untuned) 79.5% +0.4% (surpasses dense without training)
DA-MoE (Full) Diverse pruned experts + alternating MoE + FT 81.4% +2.3% (optimal cooperative specialization)

Layer placement ablations further corroborate the alternating design: First-Half MoE yields 80.6%, Second-Half MoE achieves 80.0%, All-Layer MoE attains 81.2%, while the proposed alternating interval architecture reaches 81.4%. In computational efficiency, DA-MoE reduces inference computation from 13.05 GFLOPs to 10.20 GFLOPs per token (a 21.8% saving).

Key Findings

  • Zero-Shot Superiority via Subnetwork Division: Decomposing the dense network into diverse subnetworks with initial uniform routing (DP Init) yields 79.5% accuracy without any fine-tuning, directly outperforming the dense zero-shot model (79.1%). This indicates that dense overparameterization contains conflicting representation pathways that benefit from structural disentanglement.
  • Diversity Drives Specialization: As shown in Table 8, homogeneous experts trimmed under identical criteria yield 79.1% zero-shot accuracy, whereas PCA-guided (79.5%) and strategy-guided (79.5%) diverse experts exhibit significant performance boosts, verifying that structural heterogeneity is critical for routing efficacy.
  • Concentrated Gains on Fine-Grained Perception: The most prominent improvements occur on Count and Location questions, where conditional subnetwork routing isolates localized target cues from sprawling background context.

Highlights & Insights

  • Challenging the Architectural Invariance Dogma: Proves that downstream adaptation does not require maintaining dense backbone structures; structured decomposition provides a superior starting point for specialization.
  • Synergistic Pruning-MoE Fusion: Repurposes one-shot pruning techniques (Wanda and SparseGPT) from downstream compression tools into generative subnetwork initializers for MoE architectures, bypassing costly pre-training of sparse experts.
  • Triple Pareto Efficiency: Simultaneously achieves ~1% trainable parameters during fine-tuning, a 21.8% cut in per-token inference FLOPs, and higher accuracy than full fine-tuning.

Limitations & Future Work

  • Total Physical Footprint Expansion: Housing four sparse experts per MoE block increases physical disk/RAM footprint to 10.8B parameters on InternVL3-8B (despite active parameters dropping to 6.6B), posing memory capacity challenges on resource-constrained devices.
  • Unstructured Sparsity Constraints: Masks generated by Wanda and SparseGPT remain largely unstructured, preventing direct speedups on standard dense tensor cores without specialized sparse execution kernels. Future work should investigate block-sparse or channel-level division schemes.
  • vs LoRA / DoRA: Standard PEFT treats the dense backbone as an immutable black box and appends peripheral low-rank adapters; DA-MoE reconstructs internal FFN pathways into specialized sparse routes, unlocking superior capacity at identical 1% training cost.
  • vs Wanda / SparseGPT: Prior pruning literature targets single-model post-training compression and discards discarded weights permanently; DA-MoE re-conceptualizes pruning as a diversified model division operator to spawn multi-expert architectures.

Rating

  • Novelty: โญโญโญโญโญ Pioneering paradigm that departs from dense architectural preservation, marrying model pruning with MoE routing for VLM fine-tuning.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorously validated on both public benchmarks and real-world smart-home video datasets with detailed FLOPs, parameter, and ablation metrics.
  • Writing Quality: โญโญโญโญโญ Cohesive narrative progression with clear mathematical grounding and illustrative qualitative examples.
  • Value: โญโญโญโญโญ Provides profound insights and actionable recipes for next-generation efficient multimodal model tuning and deployment.