AdaBoosting Text Prompts for Vision-Language Models

Conference: ECCV 2026
arXiv: 2607.00684
Code: https://sung0503.github.io/TPB
Area: Multimodal VLMs
Keywords: Text Prompt Boosting, AdaBoost, Few-shot Classification, Cross-model Transfer, Prompt Ensembling

TL;DR

This paper proposes Text Prompt Boosting (TPB), which introduces the AdaBoost framework into the construction of text prompts for VLMs. By treating the collection of prompts for each category as a weak classifier, it iteratively reweights hard samples, constructs new weak prompt classifiers round by round, and ensembles them into a strong classifier. This achieves sustained, sample-driven performance improvements (shot scalability) in few-shot scenarios. Moreover, this natural language prompt ensemble can be directly re-embedded and transferred across heterogeneous VLMs, preserving the performance gains after cross-model transfer.

Background & Motivation

Background: When pre-trained VLMs (such as CLIP) perform zero-shot image classification, the classification accuracy highly depends on the quality of text prompts. The current mainstream approaches are split into two paths: First, soft prompting (e.g., CoOp), which learns differentiable context vectors in a continuous embedding space. While achieving good performance, the learned prompts are tied to the representation space of a specific model, making them non-transferable across models and unreadable. Second, hard prompting, which constructs prompts using natural language text and optimizes few-shot performance through LLM-generated candidate descriptions (e.g., CuPL, DCLIP) or evolutionary search (e.g., ProAPO, LLMbo), preserving human readability and cross-model transferability.

Limitations of Prior Work: Existing few-shot hard prompting methods (e.g., ProAPO, LLMbo) optimize only a single global aggregate metric (such as overall validation accuracy) when constructing prompts. This metric is easily dominated by the majority of "easy samples." Once a set of candidate prompts correctly classifies these easy samples, the validation accuracy saturates prematurely, causing the method to stop searching for new semantic clues and failing to cover the broader visual diversity of the target domain. Consequently, even when the number of labeled samples increases (from 1-shot to 16-shot), the performance improvement of these methods is highly limited—ProAPO only improves by 1.7 percentage points from 1-shot to 16-shot on RN50.

Key Challenge: Hard prompting methods must preserve the cross-model transferability of natural language while also fully extracting information from limited annotations to cover long-tail and hard samples. A single global optimization objective cannot distinguish between "pseudo-saturation dominated by easy samples" and "true coverage of visual diversity," resulting in wasted annotated samples.

Goal: Design a hard prompting framework such that: (1) performance continuously scales with the number of labeled samples (shots); (2) the constructed prompt ensemble remains in natural language format, enabling direct transfer across heterogeneous VLMs; (3) the shot-driven performance gains are preserved even after transfer.

Key Insight: The authors observe that the core mechanism of AdaBoost—iteratively reweighting misclassified samples to force subsequent weak classifiers to focus on hard samples—perfectly addresses the limitation of existing hard prompting methods being "dominated by easy samples and failing to utilize hard sample signals." By treating the text prompt collection for each class as a weak classifier, the AdaBoost framework can naturally ensemble multiple sets of prompts into a strong classifier, with each round explicitly focusing on the samples misclassified in the previous round.

Core Idea: Replace single-pass global optimization with AdaBoost-style iterative reweighting combined with Greedy Prompt Composition (GPC) to ensemble text prompt weak classifiers into a strong classifier round by round, enabling hard samples to continuously drive the discovery of new prompts.

Method

Overall Architecture

TPB aims to solve the following problem: given a \(K\)-class few-shot labeled image set \(D\) and a pre-trained VLM, how to construct a set of natural language prompts such that the classifier's performance scales continuously with the number of shots on the source model, and this prompt set can seamlessly transfer to other VLMs. The overall framework is an \(M\)-round boosting loop: in each round, the GPC (Greedy Prompt Composition) algorithm is run under the current sample weights to select an optimal set of prompts for each class from a large-scale prompt pool, forming a weak classifier; then, this weak classifier is evaluated on the original training set to update sample weights according to the SAMME.R rule—increasing the weights of misclassified samples and decreasing those of correctly classified ones; the next round of GPC then re-selects prompts under these new weights, naturally focusing on the hard samples from the previous round. After \(M\) rounds, the outputs of all weak classifiers are summed according to the score aggregation formula of SAMME.R to yield the final prediction of the strong classifier.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Few-shot Labeled Set D<br/>+ Large-scale Prompt Pool P"] --> B["Initialize Sample Weights<br/>Uniform Distribution"]
    B --> C["Image Augmentation<br/>Randomly transform each image into 'a' copies"]
    C --> D["GPC: Single Template Initialization<br/>Select template with lowest weighted error rate from 80 options"]
    D --> E["GPC: Class-wise Greedy Insertion<br/>Select best prompt + optimal repetition count from prompt pool"]
    E --> F["Obtain Weak Classifier B*(m) for this round"]
    F --> G["Evaluate on original D<br/>Update sample weights via SAMME.R"]
    G -->|"Increase weights of misclassified samples<br/>Decrease weights of correct samples"| H{"m < M ?"}
    H -->|Yes| C
    H -->|No| I["SAMME.R Aggregation<br/>M Weak Classifiers → Strong Classifier"]

Diagram Description: Steps D and E combined represent the two stages of the GPC weak learner, while step C (image augmentation) performs data augmentation on the training set before each boosting round (to prevent fast overfitting under few-shot settings). The weight update in step G is the core of AdaBoost: weights of misclassified samples increase, which naturally biases the next round of GPC toward prompts that correct these misclassified samples. The entire loop runs for \(M=50\) rounds (\(M=30\) for ImageNet). All weak classifier text prompts are retained in natural language format, so transferring them only requires re-encoding the text embeddings on the new target VLM.

Key Designs

1. GPC Weak Learner: Two-Stage Greedy Prompt Composition

The problem GPC aims to solve is: given current sample weights \(w\) (where misclassified samples have higher weights and correct ones have lower weights), how to select a prompt set \(B_k\) for each category from a large-scale prompt pool \(P\) to minimize the weighted classification error rate. Exhaustively searching all prompt combinations across categories is intractable; hence, GPC adopts a two-stage greedy strategy.

The first stage is single-template initialization. From 80 standard templates provided by CLIP (e.g., "a photo of a {class}."), the template \(\phi^*\) that yields the lowest weighted error rate is selected as the shared template. The prompt bank \(B_k\) for each category is then initialized as \([\phi^*(c_k)]\). This has two advantages: first, any classifier needs at least one prompt per category, and the shared template provides this with minimal overhead; second, the single template itself acts as a reasonable baseline, providing a stable starting point for subsequent greedy refinement.

The second stage is class-wise cyclic greedy insertion. For each class \(k\), each candidate prompt \(t\) in its prompt pool \(P_k\) is evaluated: \(t\) is temporarily added to \(B_k\) to compute the new weighted error rate. The core innovation lies in the "repetition insertion" mechanism: since class scores are unweighted averages of the cosine similarities of all prompts in the bank, adding a single prompt to a bank that already has multiple prompts might have negligible impact. GPC allows selecting the same prompt repeatedly \(d\) times, which effectively approximates continuous weighting in a gradient-free discrete space. For each candidate \(t\), the optimal repetition count \(d(t)\) is computed (via closed-form analysis of upside/downside flips), and the candidate that minimizes \(\Delta\epsilon(k, t, d)/d\) is selected. If this value is negative (indicating a reduction in the weighted error rate), \(t\) is added \(d\) times to \(B_k\). The loop iterates over all classes until a full round of passes yields no further updates.

Why this works: Greedy insertion directly optimizes the reduction of the weighted error rate, naturally aligning with AdaBoost's sample weighting mechanism—hard samples with higher weights contribute more to the error rate, so GPC prioritizes prompts that can correct these samples. Repetition insertion, as a parameter-free continuous approximation, avoids the overfitting risk of introducing learnable weights (especially when the number of parameters is excessively large compared to the training signals in few-shot scenarios).

2. Large-scale Prompt Pool Construction: Templates + LLM Descriptions + Concatenation

TPB requires the prompt pool to have sufficient expressiveness to cover diverse visual features. The pool is constructed from three sources: (1) CLIP's original 80 hand-crafted templates, such as "a photo of a {class}." or "a blurry photo of a {class}.", which provide basic class name embedding formats; (2) LLM-generated descriptions from prior works, including DCLIP, CuPL, GPT4Vis, etc. (e.g., "beagles have large, floppy ears that hang down to the sides of their face."), which provide rich visual attribute descriptions; (3) concatenations of templates and descriptions (e.g., "a photo of a beagle, which has large, floppy ears."), which further expand the discrete search space and semantic diversity.

Ablation studies show that using CuPL descriptions alone is less effective than using CuPL and DCLIP combined, even though DCLIP alone has lower zero-shot accuracy as a standalone method. This indicates that the diversity of the prompt pool is more critical than the quality of any single database. A diverse candidate space allows GPC to have more choices in different boosting rounds to address different types of hard samples.

3. Image Augmentation in the Boosting Loop: Preventing Few-shot Overfitting

Before each GPC round, TPB performs image augmentation (random cropping + horizontal flipping) on the few-shot training set with an augmentation factor of \(a=4\) (i.e., each original image is augmented into 4 variants). The sample weight \(w_i\) is copied to each variant and re-normalized. Without augmentation, AdaBoost in few-shot settings would quickly fit the training set within a few rounds, reaching nearly 100% accuracy. Consequently, subsequent weight updates would generate no informative error signals, weak classifiers would stop improving, and test accuracy would stagnate. Augmentation forces the framework to see different transformed versions of the original images in each round, preventing training accuracy from saturating prematurely. This ensures that meaningful misclassified samples are generated in each round for reweighting, allowing boosting to steadily and continuously improve test accuracy over dozens of rounds.

A key ablation study fixes the total exposure budget to \(M \times a = 200\) and varies the assignment of \((M, a)\). Large \(a\) and small \(M\) (e.g., \(a=200, M=1\)) yields the highest accuracy on the source model (74.42%), but its accuracy drops when transferred to ViT-L/H target models; conversely, small \(a\) and large \(M\) (e.g., \(a=4, M=50\)) is slightly lower on the source model (73.36%) but yields the highest transfer accuracy (82.24% / 84.49%). This demonstrates that the core driver of transfer robustness is the iterative boosting reweighting process rather than mere data augmentation exposure.

4. SAMME.R Aggregation: Multi-class Confidence-weighted AdaBoost Variant

TPB adopts SAMME.R (Zhu et al., 2005) as the concrete implementation of boosting, instead of the weighted majority vote of the original binary AdaBoost. The core advantage of SAMME.R is that weak classifiers output probability estimates for the \(K\) classes rather than hard predicted labels. Both reweighting and aggregation utilize these real-valued confidences, which carry more information.

Specifically, the weak classifier \(F^{(m)}(x)\) produced in each round of GPC outputs a \(K\)-dimensional probability vector (computed via the softmax in Eq. (3)). The sample weight update formula is \(w'_i = w_i \times \exp\left(-\frac{K-1}{K} y_i^T \log F^{(m)}(x_i)\right)\), where \(y_i\) is a \(K\)-dimensional symmetrically encoded vector (1 for the correct class and \(-1/(K-1)\) for other classes). The final prediction of the strong classifier is not a vote but is obtained by taking the \(\operatorname{argmax}\) after summing the class-wise scores of the weak classifiers from all rounds: the score for class \(k\) in round \(m\) is \(s_k^{(m)}(x) = (K-1) \left(\log F_k^{(m)}(x) - \frac{1}{K} \sum_j \log F_j^{(m)}(x)\right)\), which is mean-centered within the log probabilities of the classes.

Loss & Training

TPB requires no traditional backpropagation training. The objective function of the weak learner GPC is the minimization of the weighted 0-1 error rate (Eq. (5)), achieved through greedy search instead of gradient optimization. SAMME.R's weight updates implicitly optimize the multi-class exponential loss. Key hyperparameters: boosting rounds \(M=50\) (\(M=30\) for ImageNet), augmentation factor \(a=4\), temperature parameter \(\tau=1\). During inference, the text embeddings of all prompts can be pre-computed. Consequently, TPB only requires a single VLM forward pass, which keeps the inference overhead comparable to single-prompt methods.

Key Experimental Results

Main Results

Few-shot classification results across 11 datasets on OpenAI's CLIP RN50 (Table 1):

Method	Type	1-shot Avg Accuracy	16-shot Avg Accuracy	1-shot to 16-shot Gain
CLIP (ZS)	Zero-Shot	56.0	—	—
CuPL	Zero-Shot	60.4	—	—
CoOp	Soft Prompting	59.3	73.2	+13.9 pp
PEZ	Gradient Hard Prompting	42.4	61.1	+18.7 pp
LLMbo	Hard Prompting	61.1	61.4	+0.3 pp
ProAPO	Hard Prompting	62.7	64.4	+1.7 pp
TPB	Hard Prompt Ensemble	63.8	70.3	+6.5 pp

TPB is already slightly superior to ProAPO at 1-shot (+1.1 pp) and widens the gap significantly at 16-shot (+5.9 pp). LLMbo and ProAPO show virtually no improvement from 1-shot to 16-shot (+0.3 pp and +1.7 pp), validating their core limitation of "being dominated by easy samples and failing to benefit from extra annotations." Conversely, TPB's upgrade of +6.5 pp demonstrates that the AdaBoost reweighting mechanism effectively drives the utilization of hard samples. While CoOp reaches the highest accuracy of 73.2% at 16-shot, it is a model-bound soft prompting method that cannot transfer across models.

Cross-model transfer experiments on ViT-B/32 (Table 2, source model: ViT-B/32, target model groups: ViT-L and ViT-H):

Target Model	Method	1-shot	16-shot	1-shot to 16-shot Gain
ViT-L Group Avg	ZS-CLIP	76.76	—	—
	CoOp+EFT	63.81	74.05	+10.24 pp
	ProAPO	79.59	80.01	+0.42 pp
	TPB	79.66	82.07	+2.41 pp
ViT-H Group Avg	ZS-CLIP	78.75	—	—
	CoOp+EFT	62.69	72.73	+10.04 pp
	ProAPO	82.07	82.39	+0.32 pp
	TPB	81.73	84.24	+2.51 pp

After transfer, ProAPO's shot-driven gains are nearly zero (+0.42 pp on ViT-L and +0.32 pp on ViT-H), whereas TPB maintains clear shot-driven gains (+2.41 pp and +2.51 pp). Additionally, although CoOp+EFT achieves transfer via simulated fine-tuning, it requires three model forward passes and suffers severe degradation (dropping to 22-40%) on certain target models (such as DFN).

Ablation Study

Ablation Configuration	Source ViT-B/32	Target ViT-L	Target ViT-H	Description
TPB (\(M=50\), \(a=4\), full pool)	72.87	82.07	84.24	Full model
w/o Repetition Insertion	72.77	81.64	83.99	Without repetition insertion, source model is almost unchanged, and transfer slightly drops
CuPL+DCLIP pool only	—	81.12	82.88	Shrinking prompt pool degrades performance, but still exceeds ProAPO
\(M=1\) (single weak classifier, 1-shot)	61.80	—	—	A single round is inferior to ProAPO (66.27)
\(M=10\) (1-shot)	66.77	—	—	10 rounds already match ProAPO
\(M=50\) (1-shot)	67.39	—	—	50 rounds is only slightly better than 30 rounds, tending to saturate
\(M=1, a=200\) (fixed exposure budget)	74.42	80.77	83.12	Large 'a', small 'M' is best on the source model but poor in transfer
\(M=50, a=4\) (fixed exposure budget)	73.36	82.24	84.49	Small 'a', large 'M' yields the best transfer

Key Findings

• Trade-off between Boosting Rounds and Augmentation: Under a fixed total exposure \(M \times a = 200\), large \(M\) and small \(a\) (more rounds, less augmentation) is slightly lower on the source model but significantly better during transfer. This indicates that cross-model transfer robustness stems from iterative reweighting rather than more augmented data. This is because multi-round boosting forces each round to discover complementary semantic prompts from different angles (different distributions of misclassified samples), and these complementary semantics are insensitive to model architecture.
• Importance of Prompt Pool Diversity: Even though DCLIP alone yields lower zero-shot accuracy, adding it to the prompt pool still improves TPB's performance, especially as the number of shots increases. A diverse candidate space offers more options for later boosting rounds to cover long-tail hard samples.
• Root Cause of PEZ's Failure in Gradient Hard Prompt Transfer: The text prompts learned by PEZ appear to be natural language characters but are actually model-specific codes. For example, the context generated for the class "pug" is "hey darby pls violets kissestc accept liza any,, adorable gorgeous heart heart :-) life behaved pug". These nonsensical word strings act as discriminative features in the representation space of RN50, but their accuracy plummets from 71.07% to 53.02% (below the zero-shot baseline) on ViT-L/14. This proves that prompt transferability does not stem from "being text", but rather from "being close to fluent natural language."
• Role of Repetition Insertion: About 53% of the selected prompts are repeatedly chosen more than once. Even if repetition insertion is completely removed, TPB still far outperforms ProAPO (72.77% vs 67.35% for 16-shot). This indicates that the core benefit originates from the boosting framework itself, and repetition insertion serves as a nice-to-have refining mechanism.
• TPB Supports Bidirectional Transfer: Reverse transfer experiments from ViT-L/14 to EVA-02 B/16 show that TPB achieves a +10.17 pp improvement on the source model and retains a +6.98 pp gain after transfer, preliminarily validating its bidirectional transfer capability.

Highlights & Insights

• Introducing AdaBoost to text prompt construction is an elegant cross-paradigm marriage: AdaBoost's "focusing on hard samples" mechanism directly addresses the core pain point of hard prompting methods being "dominated by easy samples, leading to early saturation." The integration of the two is natural rather than contrived. Each weak classifier consists of natural language prompts, preserving interpretability—allowing users to observe which semantic cues are introduced for what hard samples in each round.
• The repetition insertion mechanism in GPC is an ingenious discrete approximation of continuous weighting: In few-shot settings, introducing learnable weights for each prompt easily leads to overfitting due to excessive parameters. Allowing the same prompt to be inserted repeatedly is equivalent to regulating its contribution weight in a discrete space. This approach achieves a similar effect using zero parameters, zero gradients, and pure searching.
• Image augmentation acts as a "perpetual motion machine" for boosting: Under low-shot limits, AdaBoost is prone to rapid overfitting. Here, image augmentation is not merely a regularization technique, but a way to generate new "error signals" for each boosting round. For instance, different crops of the same dog might be described by different prompts, providing continuous informational increments to the boosting process.
• Exquisite design of the exposure budget experiment: Fixing \(M \times a = 200\) and sweeping the allocation ratio cleanly decouples the confounding factors of "data volume scaling from augmentation" and "focusing effects from boosting reweighting," representing one of the most valuable ablations.
• The failure case of PEZ is a hidden highlight of the paper: Text prompts optimized via gradients degrade into strings of meaningless characters. This provides a clear cautionary tale for the community: transferability depends on semantic fidelity rather than the medium type itself. This insight can be transferred to other prompt optimization tasks requiring cross-model generalization.

Limitations & Future Work

• The construction cost is non-negligible on large-scale datasets: In each round, GPC must evaluate candidate prompts in the prompt pool for each class individually. Although accelerated by two-stage greedy search and closed-form flip analysis, the prompt pool and candidate space remain massive on datasets like ImageNet-1K (1,000 classes). The authors lowering \(M\) from 50 to 30 on ImageNet implicitly reflects this limitation.
• AdaBoost reweighting can be highly sensitive to noisy samples: The authors acknowledge in the limitations that if the few-shot set contains labeling errors or highly atypical samples, AdaBoost might over-focus on these samples, leading to performance degradation. Robustness experiments against annotation noise are lacking.
• Prompt pool quality is a critical but under-explored upper bound: Since all prompts are selected from a fixed, pre-constructed pool, it might lack sufficiently precise semantic descriptions for fine-grained or ambiguous classes. Introducing an "on-demand generation" mechanism (such as querying an LLM dynamically for new descriptions based on current hard samples) is a natural direction for improvement.
• The comparison with CoOp is somewhat unfair: CoOp is a model-bound soft prompting method, and TPB's accuracy on the source model trails behind CoOp (16-shot: 70.3% vs 73.2%). The authors attribute this to "the cost of transferability" but do not explore whether this gap can be narrowed under the constraint of retaining text formats via some lightweight fine-tuning.
• Generalization beyond multimodal backbones is unverified: All experiments are conducted on CLIP-style dual-tower VLMs, without validation on single-tower architectures (e.g., ViLT), generative VLMs, or other multimodal paradigms.

Related Work & Insights

• vs ProAPO / LLMbo: Both guide LLMs to generate or search for prompts using a small amount of labeled images, but optimize only a single global accuracy metric, leading to performance saturation as shots increase. TPB replaces single-pass global optimization with AdaBoost reweighting. The fundamental difference lies in the granularity of the objective function: ProAPO asks "Is this set of prompts generally good?" whereas TPB asks "Is this set of prompts good for the samples that are currently misclassified?"
• vs CoOp / PromptSRC (Soft Prompting): Soft prompting methods can achieve higher accuracy on the source model, but the learned continuous vectors are tied to the representation space of a specific VLM and cannot be reused across models. TPB chooses the route of "sacrificing a small amount of source model performance in exchange for complete transferability." This is highly valuable in practical deployments, as one can construct prompts using a small model and directly deploy them on a larger model.
• vs PEZ (Gradient Hard Prompting): PEZ demonstrates that the "discrete text + gradient optimization" path fails in terms of transferability, as gradients drive the text toward model-specific non-semantic codes. The success of TPB conversely suggests that preserving natural language fluency is vital for cross-model generalization, a constraint that future hard prompting methods should respect.
• vs Classic Applications of AdaBoost (e.g., Viola-Jones): Traditional applications of AdaBoost in computer vision are mostly cascade detectors where weak classifiers are simple Haar-like feature decision stumps. TPB replaces weak classifiers with "VLMs + natural language prompts," showcasing a fresh utility for AdaBoost in the deep learning era—not for creating faster or stronger features, but for building transferable semantic ensembles.

Rating

• Novelty: ⭐⭐⭐⭐☆ Introducing the classic AdaBoost framework into VLM prompt construction is an elegant cross-paradigm transplantation. The idea is clean and natural, yet unexplored before. The repetition insertion mechanism in GPC also shows some ingenuity.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ 11 datasets, 6 target VLMs, 1-16 shots, bidirectional cross-model transfer, exposure budget isolation analysis, and qualitative visualization. The ablation studies are comprehensive, with each answering a precise question.
• Writing Quality: ⭐⭐⭐⭐☆ The method is clearly motivated (deriving the AdaBoost solution directly from ProAPO's saturation issue), with rigorous experimental logic. The analysis of PEZ's failure case is particularly outstanding. The overview diagram in Figure 1 effectively conveys the core concepts.
• Value: ⭐⭐⭐⭐☆ It provides a clean and effective solution to the shot scalability problem of hard prompting methods. The cross-model transfer characteristics of prompt ensembles have practical deployment value. The limitation of a static prompt pool also leaves clear room for future work.

Related Papers

• [CVPR 2026] Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models
• [ICML 2026] TGV-KV: Text-Grounded KV Eviction for Vision-Language Models
• [CVPR 2025] Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
• [CVPR 2026] TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment
• [CVPR 2026] TANGO: Text-Anchored Guided Optimization for Robust Fine-tuning Vision-Language Models under Label Noise