Skip to content

Realistic Test-Time Adaptation of Vision-Language Models

Conference: CVPR 2025
arXiv: 2501.03729
Code: https://github.com/MaxZanella/StatA
Area: Multimodal VLMs
Keywords: Test-Time Adaptation, Vision-Language Models, Transductive Learning, Statistical Anchor, Zero-Shot Classification

TL;DR

This paper reveals that existing test-time adaptation (TTA) / transductive methods for VLMs can severely damage the zero-shot robustness of CLIP in realistic scenarios (variable number of active classes, non-i.i.d. data streams). It proposes StatA, which introduces a KL-divergence regularization based on text encoder knowledge (statistical anchors) on the parameters of a Gaussian mixture model, maintaining stable improvements across all deployment scenarios.

Background & Motivation

The zero-shot capability of VLMs (such as CLIP) enables classification without labeled data, and recent TTA methods (transductive inference, online adaptation) have further improved performance. However, existing methods are built upon unrealistic assumptions: (1) the batch contains all classes and is uniformly distributed; (2) the data stream is i.i.d. In real-world deployments, satellite image patches may contain only a few classes, and samples in video frames are highly correlated. Key Challenge: The performance gains of existing methods under favorable assumptions come at the cost of sacrificing zero-shot robustness in other scenarios. TransCLIP performance drops by 26.3% and ZLaP drops by 37.8% in scenarios with a "Very Low" number of active classes. Key Insight: Constrain both assignment variables and statistical parameters simultaneously (instead of only constraining assignments), utilizing text encoder knowledge for "anchoring". Core Idea: Use text embeddings as statistical anchors to keep model parameters close to the text prior when data is scarce.

Method

Overall Architecture

StatA belongs to the family of soft probabilistic clustering methods. Given CLIP vision features \(\mathbf{f}_i\) and text embeddings \(\mathbf{t}_k\), the method alternately optimizes two sets of variables: (1) assignment vectors \(\mathbf{z}_i\) (the probability of a sample belonging to each class); (2) multivariate Gaussian model parameters \((\boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k)\) for each class. The key lies in adding a StatA regularization term to the standard MLE objective, using KL divergence to constrain model parameters from deviating from the prior anchors provided by the text encoder.

Key Designs

  1. Statistical Anchor (StatA) Regularization:

    • Function: Prevents model parameters from deviating from text priors in cases of few classes or low data volume.
    • Mechanism: Constructs an anchor distribution \(\mathcal{N}'_k = \mathcal{N}(\boldsymbol{\mu}'_k, \boldsymbol{\Sigma}')\) for each class \(k\), where the mean anchor is \(\boldsymbol{\mu}'_k = \mathbf{t}_k\) (text embedding) and the covariance anchor \(\boldsymbol{\Sigma}'\) is calculated from the variance of vision features weighted by zero-shot predictions. It then penalizes model parameter deviation from the anchors via \(\text{KL}(\mathcal{N}'_k || \mathcal{N}_k)\).
    • Design Motivation: Existing methods (PADDLE, Dirichlet, TransCLIP) only regularize the assignment variables \(\mathbf{z}\) and do not constrain model parameters \(\mathbf{M}\). However, the text encoder of VLMs naturally provides prior knowledge of prototypes for each class, which is wasted if not utilized.

  2. Adaptive Convex Combination Update:

    • Function: Closed-form updates for \(\boldsymbol{\mu}_k\) and \(\boldsymbol{\Sigma}_k\), automatically balancing between MLE estimation and text anchors.
    • Mechanism: \(\boldsymbol{\mu}_k = \beta_k \mathbf{v}_k + (1-\beta_k) \boldsymbol{\mu}'_k\), where \(\beta_k = \frac{n_k}{n_k + \alpha}\), and \(n_k\) is the number of predicted samples for class \(k\). The more samples there are, the more trustworthy the MLE is; the fewer samples, the more trustworthy the text anchor is.
    • Design Motivation: When a class has very few or even zero samples, the MLE estimation is unreliable; in this case, it should fall back to the text prior. \(\alpha=1\) works across all experiments without tuning.

  3. Hard-Assignment-based \(\beta_k\) Calculation:

    • Function: More robust estimation of the predicted sample count for each class.
    • Mechanism: Replaces the soft assignments \(z_{i,k}\) in \(\beta_k\) with hard assignments \(\mathbb{1}[k = \arg\max_r z_{i,r}]\), avoiding noise introduced by the residual components in soft probabilities.
    • Design Motivation: Experiments show that the hard assignment version is more stable across scenarios, especially in those with a high number of active classes.

Loss & Training

Total objective function: \(\mathcal{L}_\mathcal{A}(\mathbf{z}; \boldsymbol{\mu}, \boldsymbol{\Sigma}) = \mathcal{L}_{\text{MLE}}(\mathbf{z}; \boldsymbol{\mu}, \boldsymbol{\Sigma}) + \alpha \sum_{k=1}^K \text{KL}(\mathcal{N}'_k || \mathcal{N}_k)\)

Where \(\mathcal{L}_{\text{MLE}}\) is the standard log-likelihood objective of the Gaussian mixture model. Optimization utilizes block coordinate descent: first fixing parameters to update \(\mathbf{z}\), then fixing \(\mathbf{z}\) to update parameters in closed-form. Initialization uses zero-shot softmax predictions. StatA is training-free, only requiring a few iteration steps during inference.

Key Experimental Results

Main Results (Batch Size=64, Average of 11 Datasets)

Method Very Low (1-4 classes) Low (2-10 classes) Medium (5-25 classes) All
CLIP (zero-shot) 65.2 65.2 65.2 65.2
Dirichlet 68.5 (+3.3) 70.3 (+5.1) 67.5 (+2.2) 59.2 (-6.0)
ZLaP 27.5 (-37.8) 35.2 (-30.0) 44.7 (-20.6) 65.5 (+0.3)
TransCLIP 38.9 (-26.3) 40.4 (-24.8) 42.7 (-22.5) 66.1 (+0.9)
StatA 70.4 (+5.1) 69.3 (+4.1) 67.4 (+2.2) 66.5 (+1.3)

StatA is the only method that achieves stable positive gains across all scenarios.

Online Adaptation Experiments

Method Low Correlation High Correlation Class Separation
CLIP 65.2 65.2 65.2
MTA +1.3 +1.3 +1.3
TDA +1.7 -0.3 -1.3
DMN-ZS +2.3 +0.2 -2.9
StatA +3.7 +2.9 +2.6

Key Findings

  • Existing transductive methods (such as ZLaP and TransCLIP) suffer from catastrophic performance drops (-20% to -38%) in low active class number scenarios, which is fundamentally due to the class balance bias of MLE.
  • Dirichlet performs strongly in scenarios with few classes but drops by 6% in the "All" class scenario, because its MDL regularization biases towards a small number of classes.
  • StatA with \(\alpha=1\) requires no hyperparameter tuning and is the only method that achieves positive gains in all scenarios ranging from "Very Low" to "All".
  • StatA takes only a few seconds to process thousands of samples, demonstrating high computational efficiency.

Highlights & Insights

  • Addresses the core pain point of real-world deployment: Existing TTA methods perform remarkably well under ideal distribution assumptions but collapse when deployed in different scenarios. This is an important overlooked issue.
  • Elegant mathematical framework: The convex combination update of StatA has a clear intuitive explanation—"rely on data when abundant, rely on text priors when scarce."
  • Black-box compatibility: Only requires access to the feature space and does not need internal model parameters, making it deployable via APIs.

Limitations & Future Work

  • Experiments are limited to image classification tasks and have not been extended to other vision tasks like detection or segmentation.
  • The anchor distribution uses a shared diagonal covariance matrix, which may limit representation capacity.
  • Improvements are moderate in the "All" class scenario (+1.3%), indicating that the constraint effect of the anchor weakens when data is abundant.
  • Scenarios with open worlds where the label space changes over time are not considered.
  • vs TransCLIP: TransCLIP constrains assignment variables with KL divergence, whereas StatA constrains model parameters with KL—shifting from "regularizing predictions" to "regularizing statistics".
  • vs EM-Dirichlet: Dirichlet's MDL regularization biases towards a small number of classes, leading to collapse in the "All" scenario. StatA avoids this bias through the adaptive \(\beta_k\).
  • vs TDA/DMN-ZS: These online methods construct a memory bank and rely on a uniform data stream, leading to a sharp performance drop under non-i.i.d. conditions.

Rating

  • Novelty: ⭐⭐⭐⭐ The perspective shift from "regularizing assignments" to "regularizing model parameters" is novel, and the design of StatA is elegant.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extremely comprehensive evaluation covering 6 active class number settings \(\times\) batch/online/stream configurations \(\times\) 11 datasets.
  • Writing Quality: ⭐⭐⭐⭐ Mathematical derivations are clear, though the dense LaTeX notation requires careful reading.
  • Value: ⭐⭐⭐⭐ Exposes the "false prosperity" of existing TTA methods and provides a reliable baseline for realistic deployment.