Layer-Specific Prompt Fusion Discovery via Differentiable Search in Vision Foundation Models¶

Conference: ECCV 2026
arXiv: 2606.26379
Project Page: https://xixiaouab.github.io/Prompt-Fusion-Discovery/
Code: See project page (GitHub link not directly provided in the paper)
Area: Multimodal VLM / Vision Foundation Models / Parameter-Efficient Fine-Tuning
Keywords: visual prompt tuning, differentiable architecture search, layer-wise fusion, DARTS, information bottleneck

TL;DR¶

This paper reformulates the prompt-image token fusion strategy (traditionally hardcoded as concatenation/addition in visual prompt tuning) as a bi-level optimization problem. Utilizing differentiable architecture search (DARTS), it discovers a layer-specific fusion operator (concat / add / affine / cross-attention) for each layer of a frozen ViT. At a cost of tuning only 0.75% of parameters, it pushes the VTAB-1k average score to 77.01%, revealing that "fusion mechanism itself" is a previously overlooked first-order variable in prompt tuning.

Background & Motivation¶

Visual Prompt Tuning (VPT) has become a mainstream parameter-efficient method for transferring large-scale ViT backbones to downstream tasks: the backbone is frozen throughout, and a small set of learnable prompt tokens are prepended to the input/feature space, updating only these tokens during training. The core operation is simple: fusing prompt tokens and image tokens before passing the combination into the transformer layers. However, from VPT to subsequent works like E2VPT, SA2VP, VFPT, and LoR-VP, the vast majority of research focuses on "where prompts should be inserted, their optimal length, whether they should be represented in the frequency domain, or if they need spatial structuring." The fusion mechanism itself remains largely unchallenged: it is almost always concatenation or addition, applied uniformly across all layers from the 1st to the 12th.

The issue is that different depths of a ViT capture fundamentally different semantics—shallow layers process low-level structural cues, while deep layers handle high-level semantics, which is a well-established hierarchical property. Thus, using a "fixed, one-size-fits-all fusion ruler" is highly questionable. Is concatenation or addition always superior? Could a hybrid strategy leverage the strengths of different operators across layers? If a hybrid strategy is indeed superior, are simple concatenation and addition sufficient to construct it? The authors make a preliminary observation (Sec. 3.3 Ablation) showing that simply mixing concat and add does not yield optimal results, suggesting that the search space lacks "raw material." The main contradiction lies in the fact that the choice of fusion methods is discrete, layer-wise, and coupled with the prompt parameters (the optimal prompt changes when the fusion method varies). Manually searching through a space of \(4^{12}\) combinations is impractical. This work aims to shift "fusion selection" from manual heuristics to a learnable, searchable process that automatically adapts to tasks and layer depths.

The core idea of this work is: to treat prompt-token fusion as a searchable, differentiable component. Using a DARTS-like bi-level optimization, each layer of a frozen ViT automatically selects the most suitable fusion operator from {concat, add, affine, cross-attention} for its respective semantics—with the inner loop optimizing the prompts and the outer loop optimizing the architecture weights for layer-wise fusion selection. In addition, two lightweight operators, affine and cross-attention, are introduced to complete the representation capacity of the search space.

Method¶

Overall Architecture¶

The proposed method is named Auto-Prompting. Standard VPT uses a single fixed fusion operator at each layer to combine prompt and patch tokens before feeding them to the frozen transformer. This work replaces the fixed operator with a searchable fusion module, operating in two stages. Search Stage: Each layer maintains a set of "architecture logits" \(\alpha^{(l)}\), which are softmax-transformed to obtain layer-wise preference weights \(\pi^{(l)}=(\pi_1,\pi_2,\pi_3,\pi_4)\). Meanwhile, the four operators process the same input in parallel to yield intermediate outputs \(\Delta_1,\dots,\Delta_4\), which are weighted and summed to form a soft fusion input for the frozen block. Discretization Stage: Once training converges, the operator with the highest preference weight is kept for each layer, others are discarded, and the prompts are briefly fine-tuned to restore performance.

Key injection point: For a frozen ViT with a Pre-LayerNorm structure, the fusion module \(\Delta^{(l)}\) is inserted after the LayerNorm of the current layer and before the frozen MSA, without adding post-normalization. That is, \(\widehat{x}_{\mathrm{msa}}^{(l-1)}=\Delta^{(l)}(p^{(l)},\tilde{x}^{(l-1)})\), where \(\tilde{x}^{(l-1)}=\mathrm{LN}(x^{(l-1)})\). The module is inserted after the LN because LayerNorm removes any constant bias added before it (\(\mathrm{LN}(z+b)=\mathrm{LN}(z)\)). Positioning it after the LN ensures that additive and affine modulations are preserved. All operators are strictly constrained to output dimensions of \(k\times d\) (where \(k\) is the number of tokens including the CLS token), keeping the frozen backbone's interface, positional embeddings, attention map size, and within-block FLOPs unchanged.

There are two groups of trainable parameters: model parameters \(\phi\) (including prompt tokens and internal operator weights, such as the down-projection matrix for concat, FiLM MLPs for affine, and QKV projections for cross-attention) and architecture parameters \(\alpha\) (a 4-dimensional logit vector per layer mapping to the choice of fusion). They are optimized on the training and validation sets, respectively, forming a bi-level optimization problem.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Frozen ViT Layer l<br/>Tokens passing through Pre-LN"] --> B
    P["prompt token p^(l)"] --> B
    subgraph B["Four-Operator Fusion Base<br/>concat / add / affine / cross-attn"]
        direction TB
        B1["Δ_concat: token-preserving down-projection"]
        B2["Δ_add: mean-broadcast bias"]
        B3["Δ_affine: FiLM scaling and shifting"]
        B4["Δ_cross: prompt as memory"]
    end
    B --> C["Differentiable Layer-wise Operator Search<br/>π=softmax(α/τ) soft mixture"]
    C --> D["Bi-level Optimization + Stability Regularization<br/>entropy term prevents collapse + cost term favors lightweight operators"]
    D -->|After annealing τ→0| E["Discretization<br/>keep argmax operator per layer"]
    E --> F["Short fine-tuning of prompts<br/>inference path matches fixed-fusion length"]

Key Designs¶

1. Four-Operator Fusion Base: Organizing "potentially useful fusion methods" into a complete and lightweight search space

The search space \(\mathcal{S}=\{\Delta_{\texttt{concat}},\Delta_{\texttt{add}},\Delta_{\texttt{affine}},\Delta_{\texttt{cross}}\}\) serves as the raw material. Preliminary ablations indicate that combining only concat and add does not yield optimal solutions. Therefore, the authors introduce affine and cross-attention, arguing that these four are complementary, lightweight, and that more heavy fusion designs (gated residuals, dynamic projections) can be accurately combined or approximated by them, forming a "complete basis for feature modulation." The specific mechanisms are:

concat (token-preserving concatenation): Unlike standard VPT which changes the interface by feeding \((m{+}k)\) tokens into the block, this method prepends prompt tokens and uses a column-stochastic matrix \(R^{(l)}=\mathrm{softmax}_{\mathrm{col}}(U^{(l)})\) to project the sequence back to \(k\) tokens: \(\Delta_{\texttt{concat}}=(R^{(l)})^{\top}[p^{(l)};\tilde{x}^{(l-1)}]\). Each output token is a convex combination of \((m{+}k)\) inputs, which avoids QKV projections and is cleaner and more stable than cross-attention.
add (mean bias): Summarizes prompts into a single vector \(s=\mathrm{LN}(\mathrm{mean}(p^{(l)}))\), which is broadcast-added to each token: \(\Delta_{\texttt{add}}=\tilde{x}^{(l-1)}+\mathbf{1}_k s^{\top}\). The overhead is negligible.
affine (FiLM scaling and transfer): Reuses the same prompt summary \(s\), using two small MLPs to generate channel-wise scaling \(\gamma=\sigma(\phi_\gamma(s))\) and shift \(\beta=\phi_\beta(s)\): \(\Delta_{\texttt{affine}}=\gamma\odot\tilde{x}^{(l-1)}+\mathbf{1}_k\beta^{\top}\). The add operator is a special case of this with \(\gamma=\mathbf{1},\beta=s\).
cross-attention (prompt as memory): Uses image tokens as queries to attend to prompt memories: \(A^{(h)}=\mathrm{softmax}(Q_x^{(h)}(K_p^{(h)})^{\top}/\sqrt{d_h})\in\mathbb{R}^{k\times m}\), followed by head concatenation and residual addition. It provides content-adaptive routing and is the heaviest of the four.

The paper explicitly discusses why the "redundancy" between add and affine is retained (Sec. 2.5): although add is a special case of affine, training an affine MLP from scratch introduces a much more complex optimization landscape. The parameter-free add operator acts like a residual connection, providing stable, near-identity gradient paths during early search stages. Additionally, the outer cost-regularization leverages this redundancy for implicit pruning—activating affine only when its extra expressiveness reduces validation loss, otherwise defaulting to the zero-cost add operator. The authors report that without this stabilization strategy utilizing operator redundancy, there is a 65% probability of the search collapsing into sub-optimal solutions.

2. Differentiable Layer-wise Operator Search (DARTS Bi-level Optimization + Stability Regularization): Formulating "fusion selection" as a gradient-tractable problem

This is the core of the "differentiable search." The discrete choice of "which operator to use at layer \(l\)" is relaxed into a continuous soft mixture: \(\pi^{(l)}(\tau)=\mathrm{softmax}(\alpha^{(l)}/\tau)\), with soft fusion \(\Delta_{\text{soft}}^{(l)}=\sum_{i\in\mathcal{S}}\pi_i^{(l)}\Delta_i\). The temperature \(\tau\) is annealed from high (exploration) to low (exploitation). As \(\tau\to0\), the soft fusion converges to the argmax hard choice. The optimization objective is a standard DARTS bi-level formulation—updating architecture parameters \(\alpha\) on the validation set, while updating model parameters \(\phi\) on the training set:

\[\min_{\alpha}\ \mathcal{L}_{\mathrm{val}}\big(\phi^{\ast}(\alpha),\alpha\big),\quad\text{s.t.}\ \ \phi^{\ast}(\alpha)=\arg\min_{\phi}\mathcal{L}_{\mathrm{train}}(\phi,\alpha).\]

\(\nabla_\alpha\mathcal{L}_{\mathrm{val}}\) is approximated using a single-step unrolled gradient of DARTS (taking a step \(\phi'=\phi-\eta_\phi\nabla_\phi\mathcal{L}_{\text{train}}\), then computing the correction term containing Hessian-vector products). The HVP is calculated via two backward passes without explicitly constructing the Hessian. To prevent premature collapse and prioritize cheaper operators when performance is comparable, two regularization terms are added to the outer loop: a negative entropy term \(\widetilde{\mathcal{R}}_{\mathrm{ent}}=-\sum_l H(\pi^{(l)})\) to encourage exploration, and a cost term \(\widetilde{\mathcal{R}}_{\mathrm{cost}}=\sum_l\sum_i c_i\pi_i^{(l)}\) to penalize heavy operators. The normalized cost prior is micro-benchmarked as \(c=[0,0.06,0.30,1.00]\) (for add, affine, concat, and cross, respectively). The total outer objective is:

\[\mathcal{L}_{\text{outer}}=\mathcal{L}_{\mathrm{val}}+\lambda_{\mathrm{ent}}\widetilde{\mathcal{R}}_{\mathrm{ent}}+\lambda_{\mathrm{cost}}\widetilde{\mathcal{R}}_{\mathrm{cost}}.\]

Discretization is performed after a certain number of search epochs: each layer selects \(\widehat{i}^{(l)}=\arg\max_i\pi_i^{(l)}\), \(\alpha\) is frozen, only the selected operator is retained, and \(\phi\) is briefly fine-tuned for a few epochs. Thus, the final deployed inference path is of identical length to fixed fusion methods—search overhead is confined to the training stage, resulting in zero additional branches during inference. This is the practical definition of "layer-specific fusion": rather than sharing a single fusion scheme, a blueprint of "which operator to use in which layer" is discovered.

3. Information Bottleneck Perspective: Why "layer-specific self-selected fusion" is theoretically superior

The authors provide a theoretical rather than merely empirical justification for their method using the Information Bottleneck (IB) framework. Let the representation entering layer \(l\) be \(T^{(l)}\). The layer-wise IB objective is defined as \(\mathcal{L}_{\mathrm{IB}}^{(l)}=I(T^{(l)};X)-\beta I(T^{(l)};Y)\) (where \(X\) is the input image and \(Y\) is the label). The former term compresses input-related redundancy, while the latter retains label-related information. Because mutual information cannot be calculated precisely, variational estimators (MINE, InfoNCE, and CLUB) are used, yielding a consistent trend. Observation (CUB-200, ViT-B/16): compared to fixed-fusion VPT, the proposed method achieves stronger compression of \(\widehat{I}(T^{(l)};X)\) and higher preservation of \(\widehat{I}(T^{(l)};Y)\) in deeper layers, consistently yielding lower empirical IB proxy values—achieving a superior compression-relevance trade-off. The authors emphasize that this provides trend-based evidence rather than a deconfounded causal proof. This perspective also explains the patterns learned during search: shallow layers prefer lightweight operators (concat/add) to preserve structural cues, while deep layers shift toward semantic operators (affine/cross-attn) to emphasize label-related patterns, aligning with the attention visualizations in Sec. 3.4.

Loss & Training¶

Two-stage training (Algorithm 1): Phase I: Differentiable Search—within each epoch, model parameters \(\phi\) are updated using soft mixtures in the inner loop, architecture parameters \(\alpha\) are updated using single-step unrolled gradients in the outer loop, and \(\tau\) is cosine-annealed; Phase II: Post-Discretization Short Fine-Tuning—each layer uses only the selected operator, and \(\phi\) is briefly fine-tuned to restore performance. Training details: AdamW, learning rate \(1\times10^{-3}\), weight decay 0.01, cosine learning rate scheduler, 100 epochs, batch size 64, A100 GPU, reporting top-1 accuracy averaged over three seeds. By default, 10 prompt tokens (\(d=768\)) are used per layer.

Key Experimental Results¶

Main Results¶

Evaluated on 34 datasets across 3 benchmarks (19 in VTAB-1k, 5 in FGVC, 10 in HTA) using a frozen ViT-Base/16 backbone. Core comparisons (Table 1):

Method	Tuned Params (%)	FGVC Avg	HTA Avg	VTAB-1k Avg	Structured
VPT-Deep (ECCV22)	0.73	89.11	85.5	69.43	54.98
VFPT (NeurIPS24)	0.66	89.24	—	75.49	60.19
SA2VP (AAAI24)	0.81	90.08	91.5	75.83	60.80
LoR-VP (ICLR25)	—	91.22	—	74.69	58.71
Ours (Auto-Prompting)	0.75	91.60	92.5	77.01	62.55

Compared to fixed-fusion VPT-Deep, the VTAB-1k average score increases by +7.58, FGVC by +2.49, and HTA by +7.0; compared to VFPT under a similar budget, VTAB increases by +1.52; compared to SA2VP, the proposed method achieves state-of-the-art results on both Natural (82.88) and Structured (62.55). The improvement is most pronounced in the Structured group (+7.57 over VPT-D), especially on geometric reasoning tasks: +6.6% on dSprites/Orientation and +14.7% on CLEVR/Count (both over VPT-Deep). Fixed-fusion fails to capture spatial layouts, whereas search automatically assigns cross-attention to deep layers for content-adaptive routing. Across pre-training paradigms (Table 2): under MoCo v3, tuning only 0.25% parameters yields overall state-of-the-art results (Natural 79.60 / Specialized 86.86 / Structured 61.01); under MAE, it outperforms VFPT across the board with a smaller budget. When using Swin-Base (Table 3), the proposed method achieves the best overall performance, with a +10.51% increase in the Structured group compared to VPT-D.

Ablation Study¶

Modifying only the fusion rules while keeping the prompt length and backbone fixed (Table 4, VTAB-1k three-category averages):

Fusion Method	Search?	Natural	Specialized	Structured	Gain over Best Fixed
Concat	✗	78.42	82.10	56.33	-1.73
Add	✗	77.63	81.47	55.98	-2.32
Affine	✗	78.86	82.62	57.40	-1.05
Cross-Attn	✗	79.91	83.44	58.67	0.00 (Best Fixed)
Ours (Search)	✓	82.88	85.61	62.55	+3.00

Key Findings¶

No single operator is globally optimal: Among individual operators, cross-attention is the strongest. However, layer-wise search still outperforms it by +3.00 on average (with the largest gain of +3.88 on Structured)—proving that "fusion mechanisms" deserve layer-wise consideration, and merely tuning prompt contents is insufficient.
The searched operator distribution exhibits highly regular patterns across depths: On VTAB-1k, layers 1–4 assign 0.69 of the probability mass to lightweight operators (concat/add), layers 9–12 assign 0.73 to affine/cross-attention, while intermediate layers (5–8) exhibit the highest entropy and ambiguity. Across three seeds, the final operators chosen for 9–10 out of the 12 layers are consistent. Furthermore, the "layers that eventually choose cross-attention" are precisely the ones that suffer the largest performance drops if replaced with fixed rules—where to switch operators is just as important as which operator to choose.
Task adaptability: FGVC shifts cross-attention deeper (to associate fine-grained instance attributes), while Structured selects cross/affine more frequently than Natural (emphasizing geometric alignment). Any static recipe fails to capture these task-specific nuances.
Acceptable efficiency: With 4 candidates, the search overhead is 1.38× that of VPT-Deep (10.8 vs 7.8 GPU-hours) but yields a +3.5 average gain on VTAB. After discretization, the per-image inference speed is 15.9ms vs. 14.8ms for VPT-Deep (only +7.4% slower)—because only one operator is kept per layer during inference, and heterogeneous operator scheduling assigns cheaper operations to shallow layers and expensive operations to deep layers.

Highlights & Insights¶

Elevating "fusion schemes" to first-class searchable variables: This is the most insightful contribution of this work. While the domain has focused on prompt locations, lengths, or transformation in the frequency domain for years, no prior work questioned whether the fusion operator itself could vary across layers or be learned. The authors highlighted this overlooked dimension with simple preliminary ablations (showing that merely combining concat and add cannot yield optimal performance).
Solid justification for the "complete basis" of the search space: Instead of randomly selecting four operators, the authors justify that add/affine/concat/cross correspond to bias injection, feature calibration, convex combination, and content-adaptive routing, which collectively approximate heavier fusion designs. They also defend the retention of apparent redundancy between add and affine as essential for search stability and implicit pruning (removing it leads to a 65% collapse rate).
Cost-prior-driven implicit pruning: Normalizing micro-benchmarked operator latencies into \(c=[0,0.06,0.30,1.00]\) and embedding them in the outer loop regularization enables the search to fall back to cheaper operators when accuracy is comparable. This "accuracy-cost frontier tunable" mechanism can be directly extended to any DARTS-based PEFT search.
Symmetric relationship between the IB perspective and operator scheduling: Connecting the empirical observation of "preserving structure in shallow layers and capturing semantics in deep layers" to the compression-relevance trade-off of the Information Bottleneck provides a principled explanation for why layer-specific self-selected fusion works, going beyond simple benchmark scaling.

Limitations & Future Work¶

Extra training overhead during search: While the 1.38× search overhead is tolerable on small datasets, the bi-level optimization and HVP computations can scale up significantly on massive datasets, with more layers, or with more candidate operators (this is attributed to evaluating cross-attention in deep layers).
The IB analysis indicates correlation rather than causation: This is repeatedly stressed by the authors; mutual information trend observations represent "trend-conducive evidence." Attention focus does not equal causal deconfounding, and these results should not be over-interpreted.
The search space is still restricted to four manually designed operators: Although scalable, the current claims of completeness rely on the assumption that heavier fusion schemes can be approximated by these four. If a task inherently requires a fusion scheme outside this basis, the performance ceiling of this method is constrained.
Evaluation is mainly limited to classification: All 34 datasets represent recognition or classification tasks. The appendix only briefly reports "preliminary dense prediction extensions"; whether the method is equally effective for structured output tasks like detection and segmentation remains to be fully verified.
⚠️ The arXiv ID 2606.26379 (2026-06) is a future placeholder date, and some citations (such as liu2026, xiao2026) point to future years as well. Specific values and citations should refer to the official published version.

vs VPT / VPT-Deep: Standard VPT uses a single fixed fusion scheme (concatenation or addition) across all layers, whereas this work treats fusion as a layer-specific searchable component. The difference lies in standard VPT learning only the prompt "content," whereas this work learns both the content and the "fusion scheme." The advantage is a +7.58 average gain on VTAB-1k and substantial improvement on geometric reasoning tasks, at the expense of 38% more training overhead during the search phase.
vs E2VPT / SA2VP / VFPT / LoR-VP: These subsequent works innovate on prompt decoupling, residual interactions, spatial structures, frequency domains, or low-rank representations, but still use fixed fusions. This work orthogonally addresses the "fusion selection" gap, outperforming all of them under comparable parameter budgets.
vs Traditional NAS in ViTs: Previous NAS approaches searched for transformer block layouts, widths, head counts, or dynamic token routing. This work introduces prompt tuning into the DARTS framework, searching not for the architecture of the backbone itself but for the "prompt fusion operators on a frozen backbone," preserving the backbone interface. This represents a novel marriage of NAS and parameter-efficient fine-tuning.
vs FiLM / Cross-attention: The affine operator is borrowed from FiLM's channel-wise scaling/shifting, while cross-attention is based on standard attention. However, the contribution does not lie in the operators themselves, but in "letting each layer automatically decide which template to use," giving a data-driven answer on when to use heavy structures (deeper layers, geometric tasks).

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formulating "prompt fusion" from a hardcoded design to a layer-wise navigable selection introduces a genuinely overlooked first-order variable in prompt tuning. The entry point is novel and persuasive.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covering 34 datasets across 3 benchmarks, cross-pre-training paradigms (MAE/MoCo v3), different backbones (ViT/Swin), layer-wise operator distribution analysis, attention visualization, and efficiency-latency evaluations.
Writing Quality: ⭐⭐⭐⭐ Method and observations are clearly structured, and limitations such as the correlative nature of the IB analysis are transparently detailed. However, equations and the appendix are dense, making full reproduction of the search stabilization details difficult without referring to the supplementary material.
Value: ⭐⭐⭐⭐ With controllable costs, zero extra inference overhead, and scalability to new operators, the insight of "layer-wise fusion scheduling" can directly inspire other PEFT search frameworks, rendering it highly practical and inspiring.