MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/sajjad-ucsb/MMLoP
Area: VLM Efficiency
Keywords: multimodal prompt learning, low-rank factorization, cross-modal sharing, consistency regularization, few-shot generalization

TL;DR¶

MMLoP factorizes deep prompts in both CLIP encoders into a shared low-rank factor and modality-specific factors, then protects generalization with text drift correction and zero-shot consistency, achieving 79.70% base-to-novel harmonic mean accuracy with about 11.5K trainable parameters without claiming the highest absolute accuracy.

Background & Motivation¶

CLIP classifies images by comparing image features with class text features, so downstream adaptation need not modify the entire model. Methods such as CoOp learn only continuous context vectors in the text input, exploiting limited labeled data with a small parameter budget. However, changing only class descriptions makes the text branch adapt to fixed image representations and limits coordinated changes to cross-modal matching. Subsequent methods insert prompts into multiple Transformer layers of both visual and text encoders, influencing representation formation more deeply. This increases adaptation capacity but can introduce large projection modules; the MaPLe configuration compared here has approximately 3.55M trainable parameters.

Low-rank factorization appears to offer a direct compression strategy: instead of allowing every prompt vector to vary freely, construct them from a small set of shared directions. Yet fewer parameters do not automatically imply better generalization, and independently learned low-rank prompts can still produce uncoordinated changes across modalities. The paper's ablation demonstrates this directly: replacing independent vision-language prompts with low-rank prompts reduces harmonic mean accuracy from 77.51% to 77.06%. Meanwhile, supervision on base classes can pull the model toward training categories and weaken CLIP's recognition of unseen classes. The problem is therefore not merely parameter reduction, but retaining pretrained knowledge and coordinating both modalities within a restricted adaptation space.

The authors further distinguish class-specific residuals that support discrimination from a global shift shared by all class text features. The former should retain some freedom to adapt, whereas the latter is treated as redundant drift that may encode base-class bias. This motivates combining structural sharing, explicit drift removal, and zero-shot constraints instead of compensating for compression with a larger prompt network. Core Idea: constrain multimodal prompts with shared low-rank factors, remove common text-feature drift, and then regularize the remaining adaptation against frozen CLIP features and predictive distributions.

Method¶

Overall Architecture¶

The inputs are an image and candidate class names; the output remains a classification based on cosine similarity between image and class text representations. MMLoP freezes the CLIP backbone, inserts trainable prompts into selected early layers of both encoders, and generates modality-specific prompt matrices at each layer through shared low-rank prompting. After text encoding, uniform drift correction adjusts the class features, which are then matched with the prompted image features. Training additionally runs a frozen zero-shot path, using self-regulating consistency to provide feature-level and predictive-distribution references; these training losses are not computed at test time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Image + class names"] --> Prompts["Shared low-rank prompts"]
    Prompts --> Vision["Frozen visual encoder<br/>Prompted image features"]
    Prompts --> Text["Frozen text encoder<br/>Prompted class features"]
    Input --> Anchor["Frozen zero-shot path"]
    Text --> UDC["Uniform drift correction"]
    Anchor -->|Text anchors| UDC
    Vision --> Scores["Cosine classification scores"]
    UDC --> Scores
    Scores --> Prediction["Training classification loss / test prediction"]
    Vision -.-> SCL["Self-regulating consistency"]
    UDC -.-> SCL
    Scores -.-> SCL
    Anchor -.-> SCL
    SCL -.->|Training only: update prompt factors| Prompts

The frozen zero-shot path is the original CLIP without prompts, not a newly trained teacher model. Dashed edges denote training constraints rather than an additional sequential prediction module required at test time. Uniform drift correction remains active during both training and evaluation and requires the corresponding zero-shot text anchors, unlike the image consistency branch used only for training.

Key Designs¶

1. Shared low-rank prompts: couple modalities across prompt positions

Conventional deep prompting learns separate visual and text prompt matrices at each selected layer, with rows representing prompt tokens and columns representing encoder hidden dimensions. MMLoP optimizes two small factors instead of each full matrix and multiplies them to produce the prompts supplied to the Transformer. The default visual and text prompt lengths are equal, denoted by \(T\); layer \(l\) shares a \(T\times r\) factor while retaining separate feature-direction factors for the two branches. The readable dimensions in Sections 4.2 and 4.5 and Algorithm 1 establish the following construction:

\[ P_v^{(l)}=U^{(l)}V_v^{(l)},\qquad P_t^{(l)}=U^{(l)}V_t^{(l)},\qquad U^{(l)}\in\mathbb{R}^{T\times r},\quad V_v^{(l)}\in\mathbb{R}^{r\times d_v},\quad V_t^{(l)}\in\mathbb{R}^{r\times d_t}. \]

The shared factor controls how prompt positions combine latent directions, while modality-specific factors map that combination into visual or text hidden spaces. The encoders can therefore have different hidden dimensions without forcing visual prompts into the text feature space. When \(r=1\), prompt vectors within one modality and layer are scaled versions of a single direction; the modalities share the positional coefficients but not the direction itself. This constraint applies to the added prompts, not to the entire CLIP representation, and does not replace Transformer weights with rank-1 weights.

These dimensions imply that the per-layer parameter count decreases from \(T(d_v+d_t)\) to \(r(T+d_v+d_t)\); this is a derivation in this note, not an additional experiment. Each layer has its own factors: cross-modal sharing does not mean that all layers use one common matrix. The shared factor receives gradients through both visual and text paths, preventing fully independent prompt updates. Although backbone parameters are frozen, training still back-propagates through the backbone to the prompts, so parameter savings do not imply proportional compute savings. The paper describes the sharing as an identical row space; because \(d_v\) and \(d_t\) can differ, the more precise interpretation is a shared factor and column-space constraint along the token dimension, not identical feature spaces.

2. Uniform drift correction: retain class differences and remove common text shifts

For each class, the method first compares its prompted text feature with its frozen zero-shot text feature to obtain an adaptation residual. Averaging residuals over classes estimates the common drift, which is then subtracted from each class residual. This retains changes that distinguish a class from other classes instead of simply resetting all prompted features to their zero-shot representations. The operation described in Section 4.4 and Algorithm 1 is:

\[ r_k=\tilde g_{pk}-\tilde g_k,\qquad \bar r=\frac{1}{C}\sum_{k=1}^{C}r_k,\qquad \hat g_k=\frac{\tilde g_k+r_k-\bar r}{\|\tilde g_k+r_k-\bar r\|_2}. \]

Here \(C\) counts the classes participating in the mean, not the images in a training mini-batch. Before normalization, subtracting a common vector preserves differences between class vectors; after normalization, cosine geometry relative to an image can still change, allowing predictions to improve. Thus, the absence of class-specific information in a common shift is the authors' design motivation, not a mathematical guarantee that every common shift is harmless to cosine classification. UDC introduces no trainable parameters and adds only \(O(Cd)\) mean and subtraction operations to existing class features, where \(d\) is the final embedding dimension.

Correction is performed repeatedly during optimization rather than once after training, so cross-entropy and text consistency operate on the same corrected features. This avoids an objective mismatch in which training accommodates a common offset that is suddenly removed during evaluation. Text consistency also need not spend capacity suppressing the component already removed explicitly. The main text defines the mean using training classes, while the inference line in the algorithm only says to apply UDC again; the cache does not specify whether the candidate-set mean is recomputed for novel-class evaluation, so that implementation detail remains unresolved.

3. Self-regulating consistency: preserve representations and class distributions

Even limited prompt capacity can over-specialize to base-class labels, so the authors additionally use frozen CLIP as a zero-shot reference. An \(L_1\) penalty compares prompted and original image features, while its text counterpart compares UDC-corrected class features with original class features. These terms constrain representation movement, but feature matching alone does not directly regulate predictive probabilities across multiple classes. The method therefore adds symmetric KL divergence between prompted and zero-shot output distributions rather than distilling in only one direction.

The following objective summarizes the textual description in Section 4.3; \(p\) and \(q\) denote prompted and zero-shot class probability distributions, respectively, with batch and class aggregation notation omitted:

\[ \mathcal L=\mathcal L_{\mathrm{CE}} +\lambda_1\|\hat g-\tilde g\|_1 +\lambda_2\|\tilde f_p-\tilde f\|_1 +\frac12D_{\mathrm{KL}}(p\|q) +\frac12D_{\mathrm{KL}}(q\|p). \]

The source places similarity notation directly inside KL; this note explicitly uses probability distributions to avoid treating unnormalized cosine scores as probabilities. The cache does not clearly expose distribution temperatures and all reduction details, so this expression captures the confirmed loss components rather than serving as complete reproduction code. Symmetric KL constrains predictions in both directions; it does not prove equal gradient magnitudes or universal superiority over one-way KL. The authors place that comparison in Appendix Table A3, which is absent from the supplied cache, so no numerical advantage from that table is reported here.

The three designs act at different locations: shared low-rank prompts constrain parameter structure, UDC directly modifies text representations, and consistency controls the learning objective. They are not interchangeable post-processing stages, and text consistency specifically follows correction. This interpretation matches the ablation: rank restriction alone loses accuracy, while explicit zero-shot constraints substantially restore novel-class recognition.

Loss & Training¶

The default backbone is CLIP ViT-B/16, with 4 prompt tokens in each modality and factorization rank 1. Base-to-novel and all-to-all few-shot experiments prompt the first 9 layers; domain generalization uses the first 3 layers, and these configurations should not be conflated. Low-rank factors use normal initialization with standard deviation 0.05, while all other CLIP parameters remain unchanged. Training uses SGD with learning rate 0.0025 and text and image consistency weights \(\lambda_1=25\) and \(\lambda_2=10\). Base-to-novel training runs for 30 epochs, while all-to-all few-shot and domain generalization training run for 50 epochs; results are averaged over 3 random seeds. Zero-shot text anchors use an ensemble of 60 standard text templates rather than a single manually written sentence. At test time, prompted image features are matched against UDC-corrected candidate class features by maximum cosine similarity, without test-image labels or further prompt updates.

Key Experimental Results¶

Main Results¶

The base-to-novel protocol in Table 1, page 11, splits each dataset's classes equally and trains only on base classes with 16 images per class. The following reproduces the source's Average columns across 11 datasets; Base, Novel, and HM are percentages, with HM defined as \(2BN/(B+N)\) for the corresponding base and novel accuracies \(B\) and \(N\).

Method	Base	Novel	HM
CLIP	69.34	74.22	71.70
CoOp	82.69	63.22	71.66
MaPLe	82.28	75.14	78.55
PromptSRC	84.23	75.48	79.62
CLIP-LoRA	85.32	70.63	77.28
MMA	83.20	76.80	79.87
CoPrompt	84.00	77.23	80.48
MMLoP	83.79	75.98	79.70

MMLoP exceeds PromptSRC by 0.08 percentage points in HM but remains below CoPrompt's 80.48%, supporting a favorable accuracy-budget tradeoff rather than absolute accuracy leadership. For all-to-all few-shot classification in Table 3, page 13, ViT-B/16 achieves mean accuracies of 77.5%, 79.6%, and 81.5% at 4, 8, and 16 shots. The 4-shot result slightly exceeds CLIP-LoRA's 77.4%, whereas the 16-shot result trails its 83.0%, showing that the advantage depends on data availability and evaluation protocol. In Table 2, page 12, average accuracy across four target domains is 60.46%, close to MMA's 60.48%; ImageNet-R accuracy is 77.63%.

Ablation Study¶

Table 4, page 14, adds components sequentially under the base-to-novel setting averaged over 11 datasets; all entries below are accuracy or HM percentages.

Config	Base	Novel	HM
Independent vision-language prompting, IVLP	84.21	71.79	77.51
+ Low-rank prompting	83.70	71.39	77.06
+ Consistency loss, SCL	83.60	74.48	78.78
+ Uniform drift correction, UDC	83.78	75.12	79.20
+ Shared up-projection, full MMLoP	83.79	75.98	79.70

Consistency provides the largest single-step novel-class gain, increasing accuracy from 71.39% to 74.48%, or 3.09 percentage points. UDC adds another 0.64 percentage points, and shared up-projection adds 0.86 percentage points; the latter also increases HM by 0.50 percentage points. These are conditional gains under a particular addition order, not a complete causal decomposition of independent component contributions.

Key Findings¶

Table 6, page 15, measures system costs on a single A6000 with ViT-B/16, averaged over 11 datasets; the following selects columns relevant to parameter efficiency. Parameters are in M, VRAM in GB, and training and inference times in ms/img; 0.012M is the rounded table entry, whereas the main text reports approximately 11.5K.

Method	Trainable parameters	Peak VRAM	Training time	Inference time
CoOp	0.008	1.96	19.39	1.94
PromptSRC	0.046	2.99	30.16	2.29
MaPLe	3.555	2.27	19.75	2.03
CoPrompt	3.818	2.62	20.49	1.39
CLIP-LoRA	0.184	4.39	34.32	2.27
MMLoP	0.012	2.33	20.28	1.79

MMLoP uses far fewer trainable parameters than MaPLe, but its 2.33 GB VRAM exceeds 2.27 GB, and its per-image training time is not lower. The discussion of Table 6 calls its parameter count the lowest, yet CoOp has only 0.008M in that table; the claim should be restricted to the compared multimodal approaches. The hyperparameter analysis in Table 5, page 14, averages only 9 datasets and should not be directly equated with the 11-dataset results in Table 1. Its HM values for ranks 1, 2, and 4 are 79.91%, 79.59%, and 79.71%, respectively, providing no evidence that increasing rank necessarily improves generalization.

Highlights & Insights¶

Sharing the low-rank factor over prompt positions avoids requiring identical hidden dimensions. It coordinates structural updates while retaining modality-specific feature directions.
Separating common and class-specific adaptation residuals makes regularization more targeted. This is more selective than simply forcing every prompted feature back to its original position.
The paper exposes both the failure of compression alone and its subsequent recovery. Table 4 directly tests why low rank is insufficient by itself to improve accuracy.

Limitations & Future Work¶

Fine-grained recognition still shows a capacity bottleneck: Table 1 reports FGVCAircraft HM of 38.01%, below PromptSRC's 39.69% and CoPrompt's 39.76%.
Fewer trainable parameters do not guarantee less total compute, minimum VRAM, or fastest inference. Frozen backbone encoding remains substantial, and consistency training requires zero-shot references.
The cache contains damaged equation extraction and no appendix; the symmetric-KL ablation, detailed ViT-B/32 results, and inference class-mean implementation could not be independently verified.
A follow-up suggested by this note is to test UDC with changing candidate class sets. Its mean depends on the participating classes, so stability under open-vocabulary expansion deserves separate evaluation.

Compared with CoOp / IVLP: CoOp primarily adapts text, while IVLP learns prompts independently in both branches; MMLoP preserves deep multimodal adaptation but constrains how its prompts are generated.
Compared with PromptSRC: MMLoP builds on zero-shot self-regularization and adds symmetric KL and drift correction; using frozen CLIP as a consistency reference is not itself a new contribution.
Compared with CLIP-LoRA: CLIP-LoRA adapts backbone weights in a low-rank space, whereas MMLoP operates in prompt space. The former is stronger at 16-shot all-to-all classification, while the latter has higher base-to-novel HM.
Research direction: Allocate prompt capacity by class or layer while retaining inexpensive sharing. This is a suggestion motivated by the fine-grained bottleneck, not a method already validated in the paper.

Rating¶

Novelty: 4/5. Low-rank prompts are not new, but cross-modal factor sharing and drift correction form a concrete, interpretable combination.
Experimental Thoroughness: 4/5. Multiple classification protocols, component ablations, and system costs are covered, but the supplied cache omits the appendix and table-level variance does not support the smallest gains.
Writing Quality: 4/5. The method maps clearly to the incremental ablation, although subspace terminology and some efficiency claims need tighter qualification.
Value: 4/5. Useful for CLIP adaptation with constrained per-task trainable-parameter and storage budgets, rather than a universal replacement for methods targeting maximum accuracy.