An Inverse-Adversarial and Difficulty-Adaptive Robust Vision-Language Model

Conference: ECCV2026
Paper: Official paper page / PDF
Area: Multimodal VLM
Keywords: inverse-adversarial learning, few-shot robust classification, visual prompts, text adapter, difficulty-adaptive fusion

TL;DR

IADA-RVLM constrains adversarial training with inverse-adversarial examples moved toward the correct class, learns specialized knowledge through visual prompts and a text adapter, and adjusts text knowledge fusion using class semantic distances, achieving the best Nat/Rob harmonic mean on novel classes across five datasets, including 52.52% PGD-100 robust accuracy on Caltech101.

Background & Motivation

CLIP's few-shot transfer ability depends on its pretrained shared image-text space: an image can be classified by its proximity to the corresponding text prototype, with relatively little labeled adaptation data. Strong clean accuracy, however, does not imply perturbation robustness. In the paper, frozen CLIP achieves 95.34% natural accuracy on Caltech101 but only 2.80% under PGD-100. Adversarial fine-tuning on base classes can improve robustness while shifting representations toward task-specific distributions and damaging general semantics needed for unseen classes.

Limited supervision makes this tension sharper. A natural image that is already misclassified or assigned low confidence may be an unreliable reference for aligning its attacked counterpart. Transfer difficulty also varies across tasks and classes: changing text prompts alone may not repair visual features that attacks have mixed together, while excessive base-class specialization can hurt classes that would otherwise transfer easily. Training therefore needs a reliable alignment direction, and inference needs control over how much specialized knowledge is used.

Core idea: use inverse-adversarial examples as high-confidence semantic anchors, learn robust specialized knowledge through visual prompts and a text adapter, and retain general semantics according to the text-space distance between evaluation and base classes.

Method

Overall Architecture

The inputs are a small labeled set of base-class images and class names. The backbone is CLIP ViT-B/16, and the output is an image classification result over novel classes, not a generated visual answer. During training, each image produces an adversarial version that increases classification loss and an inverse-adversarial version that decreases it. Inverse-adversarial semantic anchors guide robust adaptation, primarily updating visual prompts and a text adapter while keeping the pretrained backbone frozen.

Inference does not require the true image label to construct inverse-adversarial examples. Instead, the model estimates semantic transfer difficulty for each candidate class, fuses its frozen and adapted text prototypes, and compares them with image features extracted using visual prompts. The adaptation is class-dependent text-prototype fusion, not an image-wise attack-strength detector or an ensemble of prediction probabilities from two complete models.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Base images and labels"] --> Anchor["Inverse-Adversarial<br/>Semantic Anchors"]
    Anchor --> Adapt["Dual-Modal<br/>Expertise Adaptation"]
    Classes["Evaluation and base class text"] --> Fusion["Class-Difficulty-Adaptive<br/>Fusion"]
    Adapt -->|Adapted text prototypes| Fusion
    Test["Test image"] -->|Visual prompt encoding| Compare["Image-text similarity classification"]
    Adapt -->|Learned visual prompts| Compare
    Fusion --> Compare

Key Designs

1. Inverse-Adversarial Semantic Anchors: give adversarial representations a more reliable alignment direction

A conventional attack increases cross-entropy for the true label within a bounded neighborhood, making correct classification harder. Inverse-adversarial generation does the opposite: it decreases that loss within another bounded neighborhood. The image is updated in the input-gradient descent direction and projected back into the permitted perturbation set. "Inverse" refers to the opposite optimization direction, not an exact inversion of an earlier attack, nor a guarantee that the resulting image is more reliable for every model. Ground-truth labels guide the search toward a representation closer to the correct text semantics in the current model's embedding space.

Training only on these easier examples would not establish defensive capability, so the method retains cross-entropy on adversarial images. An additional constraint compares both image variants against all text prototypes, converts the similarities into class distributions using softmax, and aligns the adversarial distribution with the inverse-adversarial distribution. Unlike a hard target alone, distribution alignment also constrains relative relationships among other classes. Unlike directly following the natural image, it first improves the reference example's class confidence. This still assumes trustworthy labels and should not be presented as automatic label-noise correction.

The following objective is reconstructed from the prose in Section 3.1 and the readable parts of Eq. (9). Operators are missing in the cached extraction, so this is not a verbatim transcription of an intact equation; refer to the original paper for implementation details:

\[ \mathcal{L}=\mathcal{L}_{\mathrm{CE}}(\hat{x},y)+\lambda\,\mathrm{KL}\!\left(p^{\mathrm{inv}}\|p^{\mathrm{adv}}\right). \]

Here, \(\hat{x}\) is the adversarial image, the two distributions come from inverse-adversarial and adversarial image-text similarities, and \(\lambda\) controls the semantic constraint. The cache does not clearly specify whether gradients through the inverse-adversarial distribution are stopped, so describing it as a frozen teacher would be unwarranted. The inverse perturbation budget, generation step count, and numerical value of this loss weight are also not reliably disclosed.

2. Dual-Modal Expertise Adaptation: improve visual separability and then adjust text prototypes

Visual prompt tuning, or VPT, inserts learnable tokens into intermediate visual Transformer layers, where self-attention lets them interact with image patches. A small set of prompt parameters can therefore alter feature extraction without updating the entire visual backbone. The paper uses prompt length 2 across 10 visual layers. Since attacks can mix features from different classes, VPT directly addresses the visual representation problem. Text prompt tuning, or TPT, changes class text representations instead and cannot directly restructure those entangled visual features.

Better visual separability does not necessarily place class prototypes appropriately. The authors therefore add a lightweight linear text adapter, TA, after the frozen text encoder. A learnable matrix and bias adjust class vectors, adapting the semantic decision prototypes while preserving useful general visual representations. This is neither a language model nor an additional text-prompt-learning branch: VPT determines how images are represented, while TA determines where class prototypes lie. Both are optimized using the robust training objective above.

This division also explains the asymmetry in the adaptation ablation: arbitrary combinations of a prompt on one side and an adapter on the other are not equally effective. On Caltech101, TPT plus a visual adapter, VA, achieves 46.29% robust accuracy, versus 52.52% for VPT+TA. TA alone reaches 48.92%, already above VPT alone at 45.24%. These results support coordinated adjustment of visual features and text prototypes in this setting, but feature visualizations do not establish a universal theorem across architectures.

3. Class-Difficulty-Adaptive Fusion: use semantic distance to control specialization

The paper first characterizes dataset-level robust transfer difficulty by dividing random-classifier accuracy by the adversarial accuracy of a simple 5-shot classifier head trained on frozen CLIP features. This 5-shot baseline belongs to the difficulty definition, not the subsequent 16-shot main adaptation protocol. In this definition, \(C\) is the number of classes in the classification task.

\[ \mathrm{RTD}^{\mathrm{adv}}=\frac{1/C}{\mathrm{Prec}_{g}^{\mathrm{adv}}}. \]

A larger ratio indicates that simple feature transfer struggles more to establish robust classification. Classes within a dataset can nevertheless differ, so final fusion does not directly use this dataset-level ratio. Instead, it uses text-space distances: the average cosine distance from an evaluation class to all base-class text prototypes, and its distance to the nearest base class. To distinguish this class count from the one above, the note uses \(C_b\) for the number of base classes; this is a notation clarification, not an additional mechanism:

\[ d_{\mathrm{avg}}=1-\frac{1}{C_b}\sum_{j=1}^{C_b}\mathrm{sim}(t_{\mathrm{eval}},t_j),\qquad d_{\mathrm{nn}}=1-\max_{1\leq j\leq C_b}\mathrm{sim}(t_{\mathrm{eval}},t_j). \]

Average distance summarizes the relationship to the learned semantic set, while nearest-neighbor distance helps identify whether a very similar base class already exists. A distance-dependent exponential term, scaling parameter \(\beta\), and threshold \(\xi\) control the fusion coefficient. The prose explains that the threshold avoids unnecessary adaptation for highly similar classes. However, the indicator-function subscript in cached Eq. (16) is truncated, preventing reliable verification of the full gate. This note therefore does not reconstruct that equation or invent a numerical value for \(\xi\).

The meaning of fusion itself is clear: larger \(\alpha\) uses more adapted expertise, while smaller \(\alpha\) preserves more frozen text semantics. The following expression is likewise reconstructed from Eq. (15) and its surrounding explanation because the cache loses some connecting operators; refer to the original paper for implementation details:

\[ t_{\mathrm{eval}}^{\mathrm{fused}}=\alpha\,\mathrm{TxtAdapt}(\tilde{t}_{\mathrm{eval}})+(1-\alpha)\tilde{t}_{\mathrm{eval}}. \]

Only the text branch is fused. Even when frozen text semantics are fully retained, the visual model does not revert to original CLIP because images still pass through learned VPT. A further boundary is that semantic distance is a proxy for robust difficulty, not a direct measurement of whether an individual image has been attacked.

Loss & Training

The model uses CLIP ViT-B/16, adapts on base classes in a 16-shot setting, and evaluates novel classes excluded from adaptation. Training lasts 15 epochs with Adadelta and cosine annealing, using a single NVIDIA A6000 and FP16 mixed precision. Reported results are averages over 20 random seeds.

Adversarial examples are generated online using 2-step PGD with \(\epsilon=2/255\). Evaluation uses 100-step PGD and AutoAttack, with test budgets of \(2/255\) and \(4/255\). Visual prompt length is 2, prompt depth is 10, and fusion scaling is \(\beta=4.0\). These disclosed settings do not fill in the missing inverse-adversarial generation hyperparameters, numerical learning rate, or gating threshold, so the cache does not support a fully specified reproduction recipe.

Key Experimental Results

Main Results

The following selection comes from Table 1 and evaluates novel classes. Rob uses PGD-100 at \(\epsilon=2/255\). Nat and Rob are percentages; HM retains the source table's numerical scale and should not be read as a relative percentage improvement. The comparator is the existing method with the highest HM for each dataset, and the final column is an absolute difference calculated for this note.

\[ \mathrm{HM}=\frac{2\,\mathrm{Nat}\,\mathrm{Rob}}{\mathrm{Nat}+\mathrm{Rob}}. \]

This expression follows the paper's harmonic-mean definition; cached Eq. (17) itself has extraction damage.

Dataset	Ours Nat	Ours Rob	Ours HM	Best Existing Method	Existing HM	HM Difference
Caltech101	76.54	52.52	62.27	AdvMaPLe	61.35	+0.92
OxfordPets	72.16	28.43	40.63	AdvMaPLe	40.25	+0.38
Flowers102	50.64	24.36	32.91	AdvMaPLe	29.65	+3.26
DTD	35.02	20.62	26.02	FAP	23.06	+2.96
EuroSAT	28.33	13.76	19.02	FAP	16.94	+2.08

The best HM does not mean clean accuracy is preserved without cost. Caltech101 Nat falls from original CLIP's 95.34% to 76.54%, a reduction of 18.80 percentage points. The contribution is an improved trade-off among robust adaptation methods, not superiority to frozen CLIP on every metric.

Ablation Study

The following results come from Table 4. IA denotes inverse-adversarial learning, EA denotes expertise adaptation with VPT+TA, and DA denotes difficulty-adaptive fusion. "None enabled" is the training baseline for this component ablation and must not be confused with frozen CLIP in Table 1. All values are accuracy percentages, with robust accuracy evaluated using PGD-100.

Configuration	Caltech101 Nat	Caltech101 Rob	EuroSAT Nat	EuroSAT Rob
None enabled	70.20	39.52	22.51	8.25
IA	72.43	42.78	26.32	9.36
EA	73.04	47.60	25.98	10.96
IA + EA	75.86	50.75	27.25	12.18
EA + DA	74.75	51.31	27.06	12.82
IA + EA + DA	76.54	52.52	28.33	13.76

Adding DA to IA+EA increases Rob by 1.77 and 1.58 percentage points on the two datasets, respectively. Adding IA to EA+DA increases it by 1.21 and 0.94 points. Relative to the baseline with none enabled, EA alone adds 8.08 and 2.71 points, exceeding the 3.26 and 1.11 points from IA alone. However, DA alone is not reported: this is not a complete factorial experiment, and these rows cannot independently identify every interaction effect.

Key Findings

• Advantages persist under stronger attacks, but the method does not win every setting. In Table 2 under AutoAttack at \(\epsilon=4/255\), it achieves 46.91% on Caltech101 versus AdvMaPLe's 46.17%; on OxfordPets, its 24.03% is below AdvMaPLe's 24.96%.
• FAP's DTD result is internally inconsistent: Table 1 reports PGD Rob of 17.07%, whereas Table 2 reports 18.07% for PGD at the same \(\epsilon=2/255\). The main table in this note follows Table 1 rather than silently merging or correcting the discrepancy.
• The prose accompanying Figure 6 says larger \(\beta\) generally helps low-difficulty datasets but can become unstable when too large on high-difficulty datasets. Deeper visual prompting generally helps high-difficulty tasks more. Individual curve values cannot be reliably recovered from the cache, so only the trends and final settings are recorded.

Highlights & Insights

• Robust training needs to examine the reliability of its alignment reference, not just search for the hardest examples. The inverse-adversarial branch improves the supervision direction, while adversarial cross-entropy keeps training focused on difficult inputs; neither role replaces the other.
• The two modalities do different jobs. VPT changes how perturbed images are represented, whereas TA adjusts class prototypes, providing a more interpretable division of work than indiscriminately increasing trainable capacity.
• Whether specialized knowledge should apply to an unseen class can be controlled explicitly. Semantic-distance-based prototype fusion could inform other few-shot transfer tasks, but its correlation with failure risk must be checked in each target setting.

Limitations & Future Work

• The authors acknowledge additional training overhead from inverse-adversarial generation and distribution alignment, and identify efficiency and broader backbone coverage as future directions. The cache provides no runtime, memory, or overhead ratio, so the claimed slight increase cannot be quantified.
• Evidence is limited to one CLIP ViT-B/16 backbone, five classification datasets, and base-to-novel evaluation. It does not establish effectiveness for generative VLMs, visual question answering, open-vocabulary detection, or text attacks.
• Natural accuracy remains substantially below frozen CLIP, and natural/robust accuracy on the difficult EuroSAT dataset is only 28.33%/13.76%. The best HM establishes a better relative trade-off, not readiness for high-stakes deployment.
• Semantic distance is not actual visual attack difficulty, and the analysis does not establish class-wise correspondence between them. Useful follow-up evidence would include class-level correlations, fusion-weight calibration, and threshold sensitivity rather than dataset trends alone.
• Although results average 20 seeds, the main table provides no standard deviations or confidence intervals. The OxfordPets HM advantage is only 0.38 and does not establish statistical significance. Damaged cached equations, incomplete hyperparameter disclosure, and the conflicting FAP entries also limit reproducibility checks.

Related Work & Insights

• Relative to inverse-adversarial training: the method builds on using loss-reducing neighborhood examples for robust learning and turns them into semantic references for CLIP's image-text distributions. Its contribution centers on combining this idea with few-shot dual-modal adaptation and class-dependent fusion, not inventing inverse-adversarial examples.
• Relative to C-AVP / TeCoA-VPT: these visual prompting methods also address visual robustness directly. IADA-RVLM additionally adjusts text prototypes and introduces inverse-adversarial constraints, with ablations supporting coordinated adaptation across modalities.
• Relative to AdvPT / APT / AdvMaPLe / FAP: these approaches cover textual or multimodal robust prompting. IADA-RVLM emphasizes selectively applying specialized knowledge to new classes; its advantage is primarily HM in this protocol, not universal superiority under every attack.
• Relative to RTD and CLIP-Adapter: the former motivates a reference measure for transfer difficulty, and the latter provides the frozen-backbone, lightweight-adapter approach. This paper uses text geometry to control adaptation strength at inference, but the proxy's scope still needs independent validation.

Rating

• Novelty: 4/5. The combination of inverse-adversarial learning, parameter-efficient adaptation, and difficulty-aware fusion is well targeted, although its constituent ideas have precedents.
• Experimental Thoroughness: 3/5. Five datasets, two attack families, two budgets, and component ablations provide useful coverage, but multiple backbones, error bars, and overhead measurements are missing.
• Writing Quality: 3/5. Training and inference roles are clear, but conflicting table entries and claims of universal superiority need qualification; damaged cached equations do not establish a typesetting problem in the original PDF.
• Value: 4/5. The approach offers practical ideas for few-shot robust CLIP adaptation, while substantial gaps remain before high-stakes deployment.