Learning Probabilistic Prompt for Continual Learning

Conference: ECCV 2026
Paper: ECCV Official
Project: http://cvlab.yonsei.ac.kr/projects/ProbPrompt
Area: Continual Learning / Prompt Tuning
Keywords: Continual Learning, Class-Incremental Learning, Prompt Tuning, Probabilistic Prompt, Prompt Collapse

TL;DR

To tackle the ubiquitous "prompt collapse" issue where deterministic prompt vectors become highly correlated and fail to represent diverse distributions across incremental tasks, this paper models prompts as Gaussian distributions, samples diverse prompt tokens from a query-conditioned mixture distribution, and employs a distribution regularization loss to stabilize continuous learning.

Background & Motivation

Continual learning aims to sequentially acquire new knowledge across a sequence of tasks while preventing catastrophic forgetting of previously learned classes. While conventional regularization and rehearsal-based strategies incur prohibitive compute and memory burdens when paired with large Vision Transformers (ViTs), prompt-based methods (e.g., L2P, DualPrompt, CODA-P, and VQ-Prompt) have emerged as the dominant paradigm. These methods freeze the pre-trained transformer backbone to retain upstream representations and optimize only a small set of prompt tokens conditioned on query features from input images via prefix tuning.

However, existing approaches fundamentally rely on deterministic, static prompt embeddings. In class-incremental learning (CIL), individual tasks exhibit substantial intra-task variation across diverse object classes (e.g., sketches, cartoons, and graffiti in ImageNet-R), and cross-task feature diversity escalates as new categories arrive. Empirical analysis reveals that deterministic prompts inevitably suffer from a severe "prompt collapse" problem: the pairwise cosine similarity across prompt tokens climbs to extreme levels (frequently reaching 0.88–0.96), collapsing their expressive capacity and leaving them incapable of capturing heterogeneous data patterns. Merely increasing the number of prompt components fails to break this correlation trap.

The essential breakthrough requires expanding the representation space from isolated deterministic vectors to continuous probabilistic distributions. The core idea is to parameterize each prompt component as a Gaussian distribution, dynamically construct a query-conditioned Gaussian mixture distribution, draw diverse candidates via stochastic reparameterization, and aggregate them by query cosine similarity into a single prefix token alongside a temporal KL distribution regularizer.

Method

Overall Architecture

The ProbPrompt framework operates on a frozen pre-trained ViT encoder. For an input image, a query feature vector \(q\) is extracted by the backbone. To generate \(M\) prompt tokens, each token is associated with a pool containing \(N\) independent Gaussian prompt distributions. The framework first measures the Mahalanobis distance between \(q\) and each distribution to compute Softmax relevance weights, synthesizing an input-adaptive Gaussian mixture distribution. Next, \(N_s\) candidate prompts are stochastically sampled via the reparameterization trick and aggregated through query-relevance cosine similarity weights. Finally, the \(M\) aggregated prompt tokens are prepended to the input sequence via prefix tuning for a single forward pass, optimized end-to-end with cross-entropy and a distribution regularization loss.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Query Extraction<br/>Frozen ViT extracts query q"] --> B["Adaptive Gaussian Mixture Construction<br/>Mahalanobis distance weights components"]
    B --> C["Stochastic Sampling & Query Reweighted Aggregation<br/>Reparameterization and cosine fusion"]
    C --> D["Prefix Concatenation & Single Forward Pass<br/>Prefix-Tuning into frozen ViT"]
    D --> E["Dual Objective Joint Optimization<br/>Cross-entropy loss + Temporal KL regularization"]

Key Designs

1. Probabilistic Prompt Modeling and Adaptive Gaussian Mixture Distribution: Resolving Prompt Collapse at the Root

To overcome the expressiveness bottleneck of deterministic vectors, each \(n\)-th prompt component in the \(m\)-th pool is modeled as a \(D\)-dimensional multivariate Gaussian distribution \(\mathcal{N}(\mu(m,n), \Sigma(m,n))\), where \(\Sigma(m,n) = \text{diag}(\sigma^2(m,n))\) is a diagonal covariance matrix parameterized by log standard deviations for numerical stability. Given the query feature \(q \in \mathbb{R}^D\), the model calculates the squared Mahalanobis distance to each distribution component:

\[S^2(q, \mu(n), \Sigma(n)) = (q - \mu(n))^\top \Sigma(n)^{-1} (q - \mu(n))\]

Since \(\Sigma(n)\) is diagonal, this is computed element-wise across feature dimensions. Normalized relevance scores \(s(n)\) are derived through Softmax:

\[s(n) = \frac{\exp(-S^2(q, \mu(n), \Sigma(n)))}{\sum_{n'=1}^N \exp(-S^2(q, \mu(n'), \Sigma(n')))}\]

Using these scores, the \(N\) component distributions are synthesized into a closed-form Gaussian mixture \(\mathcal{N}_{\text{GM}}(\mu_{\text{GM}}, \Sigma_{\text{GM}})\), whose mean and diagonal covariance are given by:

\[\mu_{\text{GM}} = \sum_{n=1}^N s(n) \mu(n), \quad \sigma_{\text{GM}}^2 = \sum_{n=1}^N s(n) \left(\sigma^2(n) + \mu(n)^2\right) - \mu_{\text{GM}}^2\]

This formulation allows the prompt space to represent a continuous density over features rather than fixed point vectors, directly mitigating prompt collapse.

2. Stochastic Sampling and Query-Adaptive Reweighted Aggregation: Preserving Diversity with Single-Pass Efficiency

Drawing only a single prompt sample results in excessive gradient variance, making optimization unstable; conversely, passing multiple samples through the full transformer backbone would multiply computational and memory requirements. ProbPrompt resolves this trade-off by sampling \(N_s\) candidates before backbone insertion and compressing them adaptively. Specifically, candidates are drawn via the reparameterization trick:

\[\tilde{P}_k = \mu_{\text{GM}} + \sigma_{\text{GM}} \odot \epsilon_k, \quad \epsilon_k \sim \mathcal{N}(0, \mathbf{I})\]

Instead of unweighted averaging (\(w_k = 1/N_s\)), which dilutes sample-specific alignment with the query, ProbPrompt computes Softmax weights \(w_k\) based on the cosine similarity between each sample \(\tilde{P}_k\) and \(q\):

\[w_k = \frac{\exp(\text{sim}(\tilde{P}_k, q))}{\sum_{j=1}^{N_s} \exp(\text{sim}(\tilde{P}_j, q))}, \quad \hat{P} = \sum_{k=1}^{N_s} w_k \tilde{P}_k\]

The resulting \(M\) prompt tokens \(\hat{\mathbf{P}} = [\hat{P}(1), \dots, \hat{P}(M)] \in \mathbb{R}^{M \times D}\) are prepended to the input sequence via Prefix Tuning. Because aggregation happens prior to backbone execution, the entire inference process requires only one standard forward pass, adding virtually zero runtime or memory overhead.

3. Temporal Distribution Regularization: Anchoring Prior Knowledge against Distributional Drift

In class-incremental learning, unbounded gradient updates from new tasks can abruptly shift the learned mean and variance of prompt distributions, causing catastrophic forgetting of earlier classes. ProbPrompt addresses this stability-plasticity dilemma through a distribution regularization loss \(\mathcal{L}_{\text{DR}}\). By caching prompt distribution parameters \(\mathcal{N}(\hat{\mu}(m,n), \hat{\Sigma}(m,n))\) from previous training iterations, it enforces smooth distributional transitions via Kullback-Leibler (KL) divergence minimization:

\[\mathcal{L}_{\text{DR}} = \frac{1}{MN} \sum_{m=1}^M \sum_{n=1}^N \text{KL}\left(\mathcal{N}(\mu(m,n), \Sigma(m,n)) \,\parallel\, \mathcal{N}(\hat{\mu}(m,n), \hat{\Sigma}(m,n))\right)\]

The overall training objective combines cross-entropy with the regularizer: \(\mathcal{L} = \mathcal{L}_{\text{CE}} + \lambda \mathcal{L}_{\text{DR}}\). This acts as a probabilistic anchor, preventing abrupt distortions of previously formed prompt clusters while accommodating new task representations.

Loss & Training

The backbone is a frozen ViT-B/16. Only prompt distribution parameters (\(\mu\) and \(\log \sigma\)) and linear classification heads are updated. Optimization uses AdamW (\(\beta_1 = 0.9, \beta_2 = 0.999\)) with an initial learning rate of \(2.5 \times 10^{-3}\) and cosine decay over 20 epochs across all benchmarks. The batch size is set to 64 for ImageNet-R and 128 for CIFAR-100 / CUB-200. Hyperparameters include sampling size \(N_s = 30\), balancing weight \(\lambda = 10^{-6}\), token count \(M = 8\), and pool distribution count \(N = 10\).

Key Experimental Results

Main Results

Evaluated on ImageNet-R, CIFAR-100, and CUB-200 across standard class-incremental splits, reporting Final Average Accuracy (FAA) and Cumulative Average Accuracy (CAA) averaged over 5 runs with standard deviations.

Table 1: Quantitative results across 5, 10, and 20 tasks on ImageNet-R (ViT-B/16):

Method	5-Task FAA (%)	5-Task CAA (%)	10-Task FAA (%)	10-Task CAA (%)	20-Task FAA (%)	20-Task CAA (%)
Fine-Tuning (FT)	18.74 ± 0.44	48.39 ± 0.58	10.12 ± 0.51	35.23 ± 0.92	4.75 ± 0.40	22.80 ± 0.37
FT++	60.42 ± 0.87	71.59 ± 0.50	48.93 ± 1.15	66.79 ± 0.92	35.98 ± 1.38	59.68 ± 0.95
L2P (CVPR'22)	70.83 ± 0.58	78.34 ± 0.47	69.29 ± 0.73	78.30 ± 0.69	65.89 ± 1.30	77.15 ± 0.65
DualPrompt (ECCV'22)	73.05 ± 0.50	79.47 ± 0.40	71.32 ± 0.62	78.94 ± 0.72	67.87 ± 1.39	77.42 ± 0.80
CODA-P (CVPR'23)	76.51 ± 0.38	82.04 ± 0.54	75.45 ± 0.56	81.59 ± 0.82	72.37 ± 1.19	79.88 ± 1.06
HiDePrompt (NeurIPS'23)	76.29 ± 0.10	78.77 ± 0.11	76.74 ± 0.18	78.76 ± 0.11	76.46 ± 0.06	78.76 ± 0.11
EvoPrompt (AAAI'24)	77.16 ± 0.18	82.22 ± 0.54	76.83 ± 0.08	82.09 ± 0.68	74.41 ± 0.23	80.96 ± 1.42
VQ-Prompt (NeurIPS'24)	79.23 ± 0.29	82.96 ± 0.50	78.71 ± 0.22	83.24 ± 0.68	78.10 ± 0.22	82.70 ± 1.16
APT (ICCV'25)	79.20 ± 0.38	83.07 ± 0.45	79.05 ± 0.41	83.41 ± 0.54	75.94 ± 0.04	79.46 ± 0.46
Ours (ProbPrompt)	80.53 ± 0.37	83.90 ± 0.23	80.23 ± 0.31	84.21 ± 0.26	79.01 ± 0.44	83.58 ± 0.59

Table 2: Quantitative comparison on CIFAR-100 and CUB-200 (10-task split):

Method	CIFAR-100 FAA (%)	CIFAR-100 CAA (%)	CUB-200 FAA (%)	CUB-200 CAA (%)
Joint-Training (Upper Bound)	91.38	-	88.41	-
DualPrompt	79.81 ± 1.19	88.48 ± 1.32	65.01 ± 1.08	77.56 ± 0.84
CODA-P	81.03 ± 0.78	84.26 ± 0.84	73.44 ± 0.62	81.55 ± 0.70
VQ-Prompt	88.73 ± 0.27	92.84 ± 0.73	86.72 ± 0.94	90.33 ± 1.03
APT	88.85 ± 0.63	92.84 ± 0.59	78.50 ± 0.94	-
Ours (ProbPrompt)	89.38 ± 0.22	93.34 ± 0.51	87.52 ± 0.41	90.93 ± 0.70

Ablation Study

Tables 4, 6, and 8: Component and parameter ablations on ImageNet-R (10-task setting):

Configuration / Sample Count \(N_s\)	Mechanism Description	FAA (%)	CAA (%)
Deterministic Baseline (\(\Sigma = \mathbf{I}\), Mean only)	Strips stochasticity; collapses to deterministic vectors	77.49 ± 0.27	80.88 ± 0.43
Independent Prompt Sampling (No \(\mathcal{N}_{\text{GM}}\))	Samples from individual distributions before weighting	78.73 ± 0.51	82.46 ± 0.47
Mixture Sampling (w/o \(\mathcal{L}_{\text{DR}}\))	Mixture distribution + reweighted aggregation	79.98 ± 0.49	83.63 ± 0.39
Uniform Aggregation (\(w_k = 1/N_s\), w/o \(\mathcal{L}_{\text{DR}}\))	Replaces cosine reweighting with simple mean	79.06 ± 0.58	83.21 ± 0.51
No Sampling Inference (Direct \(\mu_{\text{GM}}\))	Bypasses stochastic sampling during optimization	79.18 ± 0.18	82.54 ± 0.20
Sample size \(N_s = 1\)	Single candidate vector drawn per distribution	Fail to converge	-
Sample size \(N_s = 5\)	High sampling variance destabilizes training	33.55 ± 13.61	36.57 ± 11.24
Sample size \(N_s = 10\)	Insufficient sample support	61.04 ± 7.19	62.16 ± 8.05
Full Model (\(N_s = 30\), with \(\mathcal{L}_{\text{DR}}\))	Mixture + Reweighting + Temporal KL Regularization	80.23 ± 0.31	84.21 ± 0.26

Table 7: Computational cost and efficiency comparison on ImageNet-R (10-task split, ViT-B/16): - L2P: Memory 342.76 MB, Training 5.64 min/epoch, Inference 74.54 FPS, FAA 69.29% - CODA-P: Memory 352.59 MB, Training 6.09 min/epoch, Inference 74.38 FPS, FAA 75.45% - VQ-Prompt: Memory 343.87 MB, Training 6.30 min/epoch, Inference 74.15 FPS, FAA 78.71% - Ours: Memory 347.85 MB, Training 6.33 min/epoch, Inference 74.03 FPS, FAA 80.23%

Key Findings

• Defeating Prompt Collapse: Transitioning from a deterministic baseline (77.49%) to probabilistic distributions (78.73%) and mixture modeling (79.98%) yields consistent performance gains. Figure 1(b) verifies that average pairwise prompt similarity drops from ~0.90 in prior works to ~0.80 in ProbPrompt, empirically confirming that prompt collapse is alleviated.
• Phase Transition in Sample Size: \(N_s\) exhibits a sharp convergence threshold: at \(N_s=1\) training fails to converge, at \(N_s=5\) accuracy is only 33.55%, while reaching \(N_s=20\)–30 stabilizes accuracy at 80.23%, demonstrating that Monte Carlo averaging effectively dampens stochastic variance.
• Computational Overhead: By aggregating candidate samples into a single token before prefix injection, ProbPrompt achieves state-of-the-art accuracy with under 0.1 FPS difference in inference throughput and negligible memory overhead compared to deterministic baselines.

Highlights & Insights

• Parameter-Level Probabilistic Elevation: Instead of introducing ad-hoc Gaussian noise into output feature embeddings, ProbPrompt directly parameterizes prompt parameters as continuous Gaussian densities with Mahalanobis matching, providing an elegant general solution to representation collapse in parameter-efficient tuning.
• Pre-Injection Aggregation for Single-Pass Inference: Aggregating sampled prompts prior to transformer injection merges stochastic exploration with deterministic execution, unlocking the expressiveness of sampling without increasing forward passes.
• Distribution Regularization as a Probabilistic Memory Anchor: Enforcing step-wise KL constraints between current and cached prompt distributions provides an effective balance between plasticity and rigidity in continual learning.

Limitations & Future Work

• Architectural & Setting Scope: Evaluated primarily on standard class-incremental classification using ViT backbones, leaving open-world continual learning and multimodal LLM continual adaptation to future exploration.
• Diagonal Covariance Assumption: Assuming independent diagonal variance neglects feature covariance across prompt dimensions, which could be addressed by structured low-rank covariance approximations.
• Fixed Sampling Hyperparameters: Sample count \(N_s\) requires tuning via validation; adaptive dynamic sampling schedules could benefit resource-constrained edge deployments.

Related Work & Insights

• vs L2P / DualPrompt: L2P uses non-differentiable nearest-neighbor selection while DualPrompt splits prompts into complementary G/E pools; both rely on deterministic vectors susceptible to collapse. ProbPrompt offers fully differentiable, continuous probabilistic distributions with richer representation capacity.
• vs CODA-P: CODA-P dynamically weights deterministic components via learned attention matrices; ProbPrompt models full distribution densities (means and variances) with stochastic sampling, yielding stronger resilience against representation collapse.
• vs VQ-Prompt: VQ-Prompt relies on discrete vector quantization codebooks requiring straight-through estimators; ProbPrompt models continuous Gaussian mixtures via standard reparameterization, resulting in a cleaner and more stable training formulation.

Rating

• Novelty: ⭐⭐⭐⭐☆ [Pioneering study identifying prompt collapse in CIL and addressing it via parameter-level probabilistic prompt mixtures]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive benchmarking across ImageNet-R, CIFAR-100, and CUB-200 with extensive ablations, efficiency metrics, and SSL backbone tests]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical formulation, and well-structured empirical validation]
• Value: ⭐⭐⭐⭐☆ [Plug-and-play, computationally lightweight design with strong potential across vision and multimodal prompt-tuning workflows]

Related Papers

• [NeurIPS 2025] C²Prompt: Class-aware Client Knowledge Interaction for Federated Continual Learning
• [ACL 2025] Recurrent Knowledge Identification and Fusion for Language Model Continual Learning
• [ICLR 2026] Prompt-MII: Meta-Learning Instruction Induction for LLMs
• [NeurIPS 2025] System Prompt Optimization with Meta-Learning
• [ICCV 2025] Any-SSR: How Recursive Least Squares Works in Continual Learning of Large Language Models